feast.infra.online_stores package
Subpackages
- feast.infra.online_stores.contrib package
- Subpackages
- Submodules
- feast.infra.online_stores.contrib.cassandra_repo_configuration module
- feast.infra.online_stores.contrib.hbase_repo_configuration module
- feast.infra.online_stores.contrib.mysql_repo_configuration module
- feast.infra.online_stores.contrib.postgres module
- feast.infra.online_stores.contrib.postgres_repo_configuration module
- feast.infra.online_stores.contrib.rockset_repo_configuration module
- Module contents
Submodules
feast.infra.online_stores.bigtable module
- class feast.infra.online_stores.bigtable.BigtableOnlineStore[source]
Bases:
OnlineStore
- online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str] | None = None) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Reads features values for the given entity keys.
- Parameters:
config – The config for the current feature store.
table – The feature view whose feature values should be read.
entity_keys – The list of entity keys for which feature values should be read.
requested_features – The list of features that should be read.
- Returns:
A list of the same length as entity_keys. Each item in the list is a tuple where the first item is the event timestamp for the row, and the second item is a dict mapping feature names to values, which are returned in proto format.
- online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Writes a batch of feature rows to the online store.
If a tz-naive timestamp is passed to this method, it is assumed to be UTC.
- Parameters:
config – The config for the current feature store.
table – Feature view to which these feature rows correspond.
data – A list of quadruplets containing feature data. Each quadruplet contains an entity key, a dict containing feature values, an event timestamp for the row, and the created timestamp for the row if it exists.
progress – Function to be called once a batch of rows is written to the online store, used to show progress.
- teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
Tears down all cloud resources for the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Creates the appropriate tables and column family in Bigtable.
We use a dedicated table for each entity combination. For example, if a FeatureView uses the entities shop and customer, the resulting table would be called customer-shop (entities are sorted lexicographically first).
- class feast.infra.online_stores.bigtable.BigtableOnlineStoreConfig(*, type: Literal['bigtable'] = 'bigtable', project_id: StrictStr | None = None, instance: StrictStr, max_versions: int = 2)[source]
Bases:
FeastConfigBaseModel
Online store config for GCP Bigtable
- instance: StrictStr
The Bigtable instance’s ID
feast.infra.online_stores.datastore module
- class feast.infra.online_stores.datastore.DatastoreOnlineStore[source]
Bases:
OnlineStore
Google Cloud Datastore implementation of the online store interface.
See https://github.com/feast-dev/feast/blob/master/docs/specs/online_store_format.md#google-datastore-online-store-format for more details about the data model for this implementation.
- _client
Datastore connection.
- Type:
google.cloud.datastore.client.Client | None
- online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str] | None = None) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Reads features values for the given entity keys.
- Parameters:
config – The config for the current feature store.
table – The feature view whose feature values should be read.
entity_keys – The list of entity keys for which feature values should be read.
requested_features – The list of features that should be read.
- Returns:
A list of the same length as entity_keys. Each item in the list is a tuple where the first item is the event timestamp for the row, and the second item is a dict mapping feature names to values, which are returned in proto format.
- online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Writes a batch of feature rows to the online store.
If a tz-naive timestamp is passed to this method, it is assumed to be UTC.
- Parameters:
config – The config for the current feature store.
table – Feature view to which these feature rows correspond.
data – A list of quadruplets containing feature data. Each quadruplet contains an entity key, a dict containing feature values, an event timestamp for the row, and the created timestamp for the row if it exists.
progress – Function to be called once a batch of rows is written to the online store, used to show progress.
- teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
Tears down all cloud resources for the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Reconciles cloud resources with the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables_to_delete – Feature views whose corresponding infrastructure should be deleted.
tables_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
partial – If true, tables_to_delete and tables_to_keep are not exhaustive lists, so infrastructure corresponding to other feature views should be not be touched.
- class feast.infra.online_stores.datastore.DatastoreOnlineStoreConfig(*, type: Literal['datastore'] = 'datastore', project_id: StrictStr | None = None, namespace: StrictStr | None = None, write_concurrency: PositiveInt | None = 40, write_batch_size: PositiveInt | None = 50)[source]
Bases:
FeastConfigBaseModel
Online store config for GCP Datastore
- class feast.infra.online_stores.datastore.DatastoreTable(project: str, name: str, project_id: str | None = None, namespace: str | None = None)[source]
Bases:
InfraObject
A Datastore table managed by Feast.
- name
The name of the table.
- project_id
The GCP project id.
- Type:
optional
- namespace
Datastore namespace.
- Type:
optional
- static from_infra_object_proto(infra_object_proto: InfraObject) Any [source]
Returns an InfraObject created from a protobuf representation.
- Parameters:
infra_object_proto – A protobuf representation of an InfraObject.
- Raises:
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
- static from_proto(datastore_table_proto: DatastoreTable) Any [source]
Converts a protobuf representation of a subclass to an object of that subclass.
- Parameters:
infra_object_proto – A protobuf representation of an InfraObject.
- Raises:
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
feast.infra.online_stores.dynamodb module
- class feast.infra.online_stores.dynamodb.DynamoDBOnlineStore[source]
Bases:
OnlineStore
AWS DynamoDB implementation of the online store interface.
- _dynamodb_client
Boto3 DynamoDB client.
- _dynamodb_resource
Boto3 DynamoDB resource.
- online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str] | None = None) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Retrieve feature values from the online DynamoDB store.
- Parameters:
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView.
entity_keys – a list of entity keys that should be read from the FeatureStore.
- online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Write a batch of feature rows to online DynamoDB store.
Note: This method applies a
batch_writer
to automatically handle any unprocessed items and resend them as needed, this is useful if you’re loading a lot of data at a time.- Parameters:
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView.
data – a list of quadruplets containing Feature data. Each quadruplet contains an Entity Key,
values (a dict containing feature) –
row (an event timestamp for the) –
and –
exists. (the created timestamp for the row if it) –
progress – Optional function to be called once every mini-batch of rows is written to
progress. (the online store. Can be used to display) –
- teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
Delete tables from the DynamoDB Online Store.
- Parameters:
config – The RepoConfig for the current FeatureStore.
tables – Tables to delete from the feature repo.
- update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Update tables from the DynamoDB Online Store.
- Parameters:
config – The RepoConfig for the current FeatureStore.
tables_to_delete – Tables to delete from the DynamoDB Online Store.
tables_to_keep – Tables to keep in the DynamoDB Online Store.
- class feast.infra.online_stores.dynamodb.DynamoDBOnlineStoreConfig(*, type: Literal['dynamodb'] = 'dynamodb', batch_size: int = 40, endpoint_url: str | None = None, region: StrictStr, table_name_template: StrictStr = '{project}.{table_name}', consistent_reads: StrictBool = False)[source]
Bases:
FeastConfigBaseModel
Online store config for DynamoDB store
- consistent_reads: StrictBool
Whether to read from Dynamodb by forcing consistent reads
- endpoint_url: str | None
8000
- Type:
DynamoDB local development endpoint Url, i.e. http
- Type:
//localhost
- region: StrictStr
AWS Region Name
- table_name_template: StrictStr
DynamoDB table name template
- class feast.infra.online_stores.dynamodb.DynamoDBTable(name: str, region: str, endpoint_url: str | None = None)[source]
Bases:
InfraObject
A DynamoDB table managed by Feast.
- name
The name of the table.
- endpoint_url
Local DynamoDB Endpoint Url.
- _dynamodb_client
Boto3 DynamoDB client.
- _dynamodb_resource
Boto3 DynamoDB resource.
- endpoint_url = None
- static from_infra_object_proto(infra_object_proto: InfraObject) Any [source]
Returns an InfraObject created from a protobuf representation.
- Parameters:
infra_object_proto – A protobuf representation of an InfraObject.
- Raises:
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
- static from_proto(dynamodb_table_proto: DynamoDBTable) Any [source]
Converts a protobuf representation of a subclass to an object of that subclass.
- Parameters:
infra_object_proto – A protobuf representation of an InfraObject.
- Raises:
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
feast.infra.online_stores.helpers module
- feast.infra.online_stores.helpers.compute_entity_id(entity_key: EntityKey, entity_key_serialization_version=1) str [source]
Compute Entity id given Feast Entity Key for online stores. Remember that Entity here refers to EntityKeyProto which is used in some online stores to encode the keys. It has nothing to do with the Entity concept we have in Feast.
- feast.infra.online_stores.helpers.get_online_store_from_config(online_store_config: Any) OnlineStore [source]
Creates an online store corresponding to the given online store config.
feast.infra.online_stores.online_store module
- class feast.infra.online_stores.online_store.OnlineStore[source]
Bases:
ABC
The interface that Feast uses to interact with the storage system that handles online features.
- abstract online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str] | None = None) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Reads features values for the given entity keys.
- Parameters:
config – The config for the current feature store.
table – The feature view whose feature values should be read.
entity_keys – The list of entity keys for which feature values should be read.
requested_features – The list of features that should be read.
- Returns:
A list of the same length as entity_keys. Each item in the list is a tuple where the first item is the event timestamp for the row, and the second item is a dict mapping feature names to values, which are returned in proto format.
- abstract online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Writes a batch of feature rows to the online store.
If a tz-naive timestamp is passed to this method, it is assumed to be UTC.
- Parameters:
config – The config for the current feature store.
table – Feature view to which these feature rows correspond.
data – A list of quadruplets containing feature data. Each quadruplet contains an entity key, a dict containing feature values, an event timestamp for the row, and the created timestamp for the row if it exists.
progress – Function to be called once a batch of rows is written to the online store, used to show progress.
- plan(config: RepoConfig, desired_registry_proto: Registry) List[InfraObject] [source]
Returns the set of InfraObjects required to support the desired registry.
- Parameters:
config – The config for the current feature store.
desired_registry_proto – The desired registry, in proto form.
- abstract teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
Tears down all cloud resources for the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- abstract update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Reconciles cloud resources with the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables_to_delete – Feature views whose corresponding infrastructure should be deleted.
tables_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
partial – If true, tables_to_delete and tables_to_keep are not exhaustive lists, so infrastructure corresponding to other feature views should be not be touched.
feast.infra.online_stores.redis module
- class feast.infra.online_stores.redis.RedisOnlineStore[source]
Bases:
OnlineStore
Redis implementation of the online store interface.
See https://github.com/feast-dev/feast/blob/master/docs/specs/online_store_format.md#redis-online-store-format for more details about the data model for this implementation.
- _client
Redis connection.
- Type:
redis.client.Redis | redis.cluster.RedisCluster | None
- delete_entity_values(config: RepoConfig, join_keys: List[str])[source]
- online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str] | None = None) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Reads features values for the given entity keys.
- Parameters:
config – The config for the current feature store.
table – The feature view whose feature values should be read.
entity_keys – The list of entity keys for which feature values should be read.
requested_features – The list of features that should be read.
- Returns:
A list of the same length as entity_keys. Each item in the list is a tuple where the first item is the event timestamp for the row, and the second item is a dict mapping feature names to values, which are returned in proto format.
- online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Writes a batch of feature rows to the online store.
If a tz-naive timestamp is passed to this method, it is assumed to be UTC.
- Parameters:
config – The config for the current feature store.
table – Feature view to which these feature rows correspond.
data – A list of quadruplets containing feature data. Each quadruplet contains an entity key, a dict containing feature values, an event timestamp for the row, and the created timestamp for the row if it exists.
progress – Function to be called once a batch of rows is written to the online store, used to show progress.
- teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
We delete the keys in redis for tables/views being removed.
- update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Look for join_keys (list of entities) that are not in use anymore (usually this happens when the last feature view that was using specific compound key is deleted) and remove all features attached to this “join_keys”.
- class feast.infra.online_stores.redis.RedisOnlineStoreConfig(*, type: Literal['redis'] = 'redis', redis_type: RedisType = RedisType.redis, connection_string: StrictStr = 'localhost:6379', key_ttl_seconds: int | None = None)[source]
Bases:
FeastConfigBaseModel
Online store config for Redis store
- connection_string: StrictStr
Connection string containing the host, port, and configuration parameters for Redis format: host:port,parameter1,parameter2 eg. redis:6379,db=0
feast.infra.online_stores.snowflake module
- class feast.infra.online_stores.snowflake.SnowflakeOnlineStore[source]
Bases:
OnlineStore
- online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str]) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Reads features values for the given entity keys.
- Parameters:
config – The config for the current feature store.
table – The feature view whose feature values should be read.
entity_keys – The list of entity keys for which feature values should be read.
requested_features – The list of features that should be read.
- Returns:
A list of the same length as entity_keys. Each item in the list is a tuple where the first item is the event timestamp for the row, and the second item is a dict mapping feature names to values, which are returned in proto format.
- online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Writes a batch of feature rows to the online store.
If a tz-naive timestamp is passed to this method, it is assumed to be UTC.
- Parameters:
config – The config for the current feature store.
table – Feature view to which these feature rows correspond.
data – A list of quadruplets containing feature data. Each quadruplet contains an entity key, a dict containing feature values, an event timestamp for the row, and the created timestamp for the row if it exists.
progress – Function to be called once a batch of rows is written to the online store, used to show progress.
- teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
Tears down all cloud resources for the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Reconciles cloud resources with the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables_to_delete – Feature views whose corresponding infrastructure should be deleted.
tables_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
partial – If true, tables_to_delete and tables_to_keep are not exhaustive lists, so infrastructure corresponding to other feature views should be not be touched.
- class feast.infra.online_stores.snowflake.SnowflakeOnlineStoreConfig(*, type: Literal['snowflake.online'] = 'snowflake.online', config_path: str | None = '/home/docs/.snowsql/config', account: str | None = None, user: str | None = None, password: str | None = None, role: str | None = None, warehouse: str | None = None, authenticator: str | None = None, database: StrictStr, schema: str | None = 'PUBLIC')[source]
Bases:
FeastConfigBaseModel
Online store config for Snowflake
- database: StrictStr
Snowflake database name
feast.infra.online_stores.sqlite module
- class feast.infra.online_stores.sqlite.SqliteOnlineStore[source]
Bases:
OnlineStore
SQLite implementation of the online store interface. Not recommended for production usage.
- _conn
SQLite connection.
- Type:
sqlite3.Connection | None
- online_read(config: RepoConfig, table: FeatureView, entity_keys: List[EntityKey], requested_features: List[str] | None = None) List[Tuple[datetime | None, Dict[str, Value] | None]] [source]
Reads features values for the given entity keys.
- Parameters:
config – The config for the current feature store.
table – The feature view whose feature values should be read.
entity_keys – The list of entity keys for which feature values should be read.
requested_features – The list of features that should be read.
- Returns:
A list of the same length as entity_keys. Each item in the list is a tuple where the first item is the event timestamp for the row, and the second item is a dict mapping feature names to values, which are returned in proto format.
- online_write_batch(config: RepoConfig, table: FeatureView, data: List[Tuple[EntityKey, Dict[str, Value], datetime, datetime | None]], progress: Callable[[int], Any] | None) None [source]
Writes a batch of feature rows to the online store.
If a tz-naive timestamp is passed to this method, it is assumed to be UTC.
- Parameters:
config – The config for the current feature store.
table – Feature view to which these feature rows correspond.
data – A list of quadruplets containing feature data. Each quadruplet contains an entity key, a dict containing feature values, an event timestamp for the row, and the created timestamp for the row if it exists.
progress – Function to be called once a batch of rows is written to the online store, used to show progress.
- plan(config: RepoConfig, desired_registry_proto: Registry) List[InfraObject] [source]
Returns the set of InfraObjects required to support the desired registry.
- Parameters:
config – The config for the current feature store.
desired_registry_proto – The desired registry, in proto form.
- teardown(config: RepoConfig, tables: Sequence[FeatureView], entities: Sequence[Entity])[source]
Tears down all cloud resources for the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- update(config: RepoConfig, tables_to_delete: Sequence[FeatureView], tables_to_keep: Sequence[FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool)[source]
Reconciles cloud resources with the specified set of Feast objects.
- Parameters:
config – The config for the current feature store.
tables_to_delete – Feature views whose corresponding infrastructure should be deleted.
tables_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
partial – If true, tables_to_delete and tables_to_keep are not exhaustive lists, so infrastructure corresponding to other feature views should be not be touched.
- class feast.infra.online_stores.sqlite.SqliteOnlineStoreConfig(*, type: Literal['sqlite', 'feast.infra.online_stores.sqlite.SqliteOnlineStore'] = 'sqlite', path: StrictStr = 'data/online.db')[source]
Bases:
FeastConfigBaseModel
Online store config for local (SQLite-based) store
- path: StrictStr
(optional) Path to sqlite db
- class feast.infra.online_stores.sqlite.SqliteTable(path: str, name: str)[source]
Bases:
InfraObject
A Sqlite table managed by Feast.
- name
The name of the table.
- conn
SQLite connection.
- Type:
- conn: Connection
- static from_infra_object_proto(infra_object_proto: InfraObject) Any [source]
Returns an InfraObject created from a protobuf representation.
- Parameters:
infra_object_proto – A protobuf representation of an InfraObject.
- Raises:
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
- static from_proto(sqlite_table_proto: SqliteTable) Any [source]
Converts a protobuf representation of a subclass to an object of that subclass.
- Parameters:
infra_object_proto – A protobuf representation of an InfraObject.
- Raises:
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.