feast.infra.online_stores package
Subpackages
Submodules
feast.infra.online_stores.datastore module
- class feast.infra.online_stores.datastore.DatastoreOnlineStore[source]
Bases:
feast.infra.online_stores.online_store.OnlineStore
OnlineStore is an object used for all interaction between Feast and the service used for offline storage of features.
- online_read(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, entity_keys: List[feast.types.EntityKey_pb2.EntityKey], requested_features: Optional[List[str]] = None) List[Tuple[Optional[datetime.datetime], Optional[Dict[str, feast.types.Value_pb2.Value]]]] [source]
Read feature values given an Entity Key. This is a low level interface, not expected to be used by the users directly.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
entity_keys – a list of entity keys that should be read from the FeatureStore.
requested_features – (Optional) A subset of the features that should be read from the FeatureStore.
- Returns
Data is returned as a list, one item per entity key in the original order as the entity_keys argument. Each item in the list is a tuple of event_ts for the row, and the feature data as a dict from feature names to values. Values are returned as Value proto message.
- online_write_batch(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, data: List[Tuple[feast.types.EntityKey_pb2.EntityKey, Dict[str, feast.types.Value_pb2.Value], datetime.datetime, Optional[datetime.datetime]]], progress: Optional[Callable[[int], Any]]) None [source]
Write a batch of feature rows to the online store. This is a low level interface, not expected to be used by the users directly.
If a tz-naive timestamp is passed to this method, it should be assumed to be UTC by implementors.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
data – a list of quadruplets containing Feature data. Each quadruplet contains an Entity Key,
values (a dict containing feature) –
row (an event timestamp for the) –
and –
exists. (the created timestamp for the row if it) –
progress – Optional function to be called once every mini-batch of rows is written to
progress. (the online store. Can be used to display) –
- teardown(config: feast.repo_config.RepoConfig, tables: Sequence[feast.feature_view.FeatureView], entities: Sequence[feast.entity.Entity])[source]
- update(config: feast.repo_config.RepoConfig, tables_to_delete: Sequence[feast.feature_view.FeatureView], tables_to_keep: Sequence[feast.feature_view.FeatureView], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity], partial: bool)[source]
- class feast.infra.online_stores.datastore.DatastoreOnlineStoreConfig(*, type: Literal['datastore'] = 'datastore', project_id: pydantic.types.StrictStr = None, namespace: pydantic.types.StrictStr = None, write_concurrency: pydantic.types.PositiveInt = 40, write_batch_size: pydantic.types.PositiveInt = 50)[source]
Bases:
feast.repo_config.FeastConfigBaseModel
Online store config for GCP Datastore
- namespace: Optional[pydantic.types.StrictStr]
(optional) Datastore namespace
- project_id: Optional[pydantic.types.StrictStr]
(optional) GCP Project Id
- type: Literal['datastore']
Online store type selector
- write_batch_size: Optional[pydantic.types.PositiveInt]
(optional) Amount of feature rows per batch being written into Datastore
- write_concurrency: Optional[pydantic.types.PositiveInt]
(optional) Amount of threads to use when writing batches of feature rows into Datastore
- class feast.infra.online_stores.datastore.DatastoreTable(project: str, name: str, project_id: Optional[str] = None, namespace: Optional[str] = None)[source]
Bases:
feast.infra.infra_object.InfraObject
A Datastore table managed by Feast.
- name
The name of the table.
- project_id
The GCP project id.
- Type
optional
- namespace
Datastore namespace.
- Type
optional
- static from_infra_object_proto(infra_object_proto: feast.core.InfraObject_pb2.InfraObject) Any [source]
Returns an InfraObject created from a protobuf representation.
- Parameters
infra_object_proto – A protobuf representation of an InfraObject.
- Raises
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
- static from_proto(datastore_table_proto: feast.core.DatastoreTable_pb2.DatastoreTable) Any [source]
Converts a protobuf representation of a subclass to an object of that subclass.
- Parameters
infra_object_proto – A protobuf representation of an InfraObject.
- Raises
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
feast.infra.online_stores.dynamodb module
- class feast.infra.online_stores.dynamodb.DynamoDBOnlineStore[source]
Bases:
feast.infra.online_stores.online_store.OnlineStore
Online feature store for AWS DynamoDB.
- _dynamodb_client
Boto3 DynamoDB client.
- _dynamodb_resource
Boto3 DynamoDB resource.
- online_read(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, entity_keys: List[feast.types.EntityKey_pb2.EntityKey], requested_features: Optional[List[str]] = None) List[Tuple[Optional[datetime.datetime], Optional[Dict[str, feast.types.Value_pb2.Value]]]] [source]
Retrieve feature values from the online DynamoDB store.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView.
entity_keys – a list of entity keys that should be read from the FeatureStore.
- online_write_batch(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, data: List[Tuple[feast.types.EntityKey_pb2.EntityKey, Dict[str, feast.types.Value_pb2.Value], datetime.datetime, Optional[datetime.datetime]]], progress: Optional[Callable[[int], Any]]) None [source]
Write a batch of feature rows to online DynamoDB store.
Note: This method applies a
batch_writer
to automatically handle any unprocessed items and resend them as needed, this is useful if you’re loading a lot of data at a time.- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView.
data – a list of quadruplets containing Feature data. Each quadruplet contains an Entity Key,
values (a dict containing feature) –
row (an event timestamp for the) –
and –
exists. (the created timestamp for the row if it) –
progress – Optional function to be called once every mini-batch of rows is written to
progress. (the online store. Can be used to display) –
- teardown(config: feast.repo_config.RepoConfig, tables: Sequence[feast.feature_view.FeatureView], entities: Sequence[feast.entity.Entity])[source]
Delete tables from the DynamoDB Online Store.
- Parameters
config – The RepoConfig for the current FeatureStore.
tables – Tables to delete from the feature repo.
- update(config: feast.repo_config.RepoConfig, tables_to_delete: Sequence[feast.feature_view.FeatureView], tables_to_keep: Sequence[feast.feature_view.FeatureView], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity], partial: bool)[source]
Update tables from the DynamoDB Online Store.
- Parameters
config – The RepoConfig for the current FeatureStore.
tables_to_delete – Tables to delete from the DynamoDB Online Store.
tables_to_keep – Tables to keep in the DynamoDB Online Store.
- class feast.infra.online_stores.dynamodb.DynamoDBOnlineStoreConfig(*, type: Literal['dynamodb'] = 'dynamodb', batch_size: int = 40, endpoint_url: str = None, region: pydantic.types.StrictStr, table_name_template: pydantic.types.StrictStr = '{project}.{table_name}')[source]
Bases:
feast.repo_config.FeastConfigBaseModel
Online store config for DynamoDB store
- endpoint_url: Optional[str]
8000
- Type
DynamoDB local development endpoint Url, i.e. http
- Type
//localhost
- region: pydantic.types.StrictStr
AWS Region Name
- table_name_template: pydantic.types.StrictStr
DynamoDB table name template
- type: Literal['dynamodb']
Online store type selector
- class feast.infra.online_stores.dynamodb.DynamoDBTable(name: str, region: str, endpoint_url: Optional[str] = None)[source]
Bases:
feast.infra.infra_object.InfraObject
A DynamoDB table managed by Feast.
- name
The name of the table.
- endpoint_url
Local DynamoDB Endpoint Url.
- _dynamodb_client
Boto3 DynamoDB client.
- _dynamodb_resource
Boto3 DynamoDB resource.
- endpoint_url = None
- static from_infra_object_proto(infra_object_proto: feast.core.InfraObject_pb2.InfraObject) Any [source]
Returns an InfraObject created from a protobuf representation.
- Parameters
infra_object_proto – A protobuf representation of an InfraObject.
- Raises
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
- static from_proto(dynamodb_table_proto: feast.core.DynamoDBTable_pb2.DynamoDBTable) Any [source]
Converts a protobuf representation of a subclass to an object of that subclass.
- Parameters
infra_object_proto – A protobuf representation of an InfraObject.
- Raises
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
feast.infra.online_stores.helpers module
- feast.infra.online_stores.helpers.compute_entity_id(entity_key: feast.types.EntityKey_pb2.EntityKey) str [source]
Compute Entity id given Feast Entity Key for online stores. Remember that Entity here refers to EntityKeyProto which is used in some online stores to encode the keys. It has nothing to do with the Entity concept we have in Feast.
- feast.infra.online_stores.helpers.get_online_store_from_config(online_store_config: Any) feast.infra.online_stores.online_store.OnlineStore [source]
Creates an online store corresponding to the given online store config.
feast.infra.online_stores.online_store module
- class feast.infra.online_stores.online_store.OnlineStore[source]
Bases:
abc.ABC
OnlineStore is an object used for all interaction between Feast and the service used for online storage of features.
- abstract online_read(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, entity_keys: List[feast.types.EntityKey_pb2.EntityKey], requested_features: Optional[List[str]] = None) List[Tuple[Optional[datetime.datetime], Optional[Dict[str, feast.types.Value_pb2.Value]]]] [source]
Read feature values given an Entity Key. This is a low level interface, not expected to be used by the users directly.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
entity_keys – a list of entity keys that should be read from the FeatureStore.
requested_features – (Optional) A subset of the features that should be read from the FeatureStore.
- Returns
Data is returned as a list, one item per entity key in the original order as the entity_keys argument. Each item in the list is a tuple of event_ts for the row, and the feature data as a dict from feature names to values. Values are returned as Value proto message.
- abstract online_write_batch(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, data: List[Tuple[feast.types.EntityKey_pb2.EntityKey, Dict[str, feast.types.Value_pb2.Value], datetime.datetime, Optional[datetime.datetime]]], progress: Optional[Callable[[int], Any]]) None [source]
Write a batch of feature rows to the online store. This is a low level interface, not expected to be used by the users directly.
If a tz-naive timestamp is passed to this method, it should be assumed to be UTC by implementors.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
data – a list of quadruplets containing Feature data. Each quadruplet contains an Entity Key,
values (a dict containing feature) –
row (an event timestamp for the) –
and –
exists. (the created timestamp for the row if it) –
progress – Optional function to be called once every mini-batch of rows is written to
progress. (the online store. Can be used to display) –
- plan(config: feast.repo_config.RepoConfig, desired_registry_proto: feast.core.Registry_pb2.Registry) List[feast.infra.infra_object.InfraObject] [source]
Returns the set of InfraObjects required to support the desired registry.
- Parameters
config – The RepoConfig for the current FeatureStore.
desired_registry_proto – The desired registry, in proto form.
- abstract teardown(config: feast.repo_config.RepoConfig, tables: Sequence[feast.feature_view.FeatureView], entities: Sequence[feast.entity.Entity])[source]
- abstract update(config: feast.repo_config.RepoConfig, tables_to_delete: Sequence[feast.feature_view.FeatureView], tables_to_keep: Sequence[feast.feature_view.FeatureView], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity], partial: bool)[source]
feast.infra.online_stores.redis module
- class feast.infra.online_stores.redis.RedisOnlineStore[source]
Bases:
feast.infra.online_stores.online_store.OnlineStore
- delete_entity_values(config: feast.repo_config.RepoConfig, join_keys: List[str])[source]
- online_read(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, entity_keys: List[feast.types.EntityKey_pb2.EntityKey], requested_features: Optional[List[str]] = None) List[Tuple[Optional[datetime.datetime], Optional[Dict[str, feast.types.Value_pb2.Value]]]] [source]
Read feature values given an Entity Key. This is a low level interface, not expected to be used by the users directly.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
entity_keys – a list of entity keys that should be read from the FeatureStore.
requested_features – (Optional) A subset of the features that should be read from the FeatureStore.
- Returns
Data is returned as a list, one item per entity key in the original order as the entity_keys argument. Each item in the list is a tuple of event_ts for the row, and the feature data as a dict from feature names to values. Values are returned as Value proto message.
- online_write_batch(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, data: List[Tuple[feast.types.EntityKey_pb2.EntityKey, Dict[str, feast.types.Value_pb2.Value], datetime.datetime, Optional[datetime.datetime]]], progress: Optional[Callable[[int], Any]]) None [source]
Write a batch of feature rows to the online store. This is a low level interface, not expected to be used by the users directly.
If a tz-naive timestamp is passed to this method, it should be assumed to be UTC by implementors.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
data – a list of quadruplets containing Feature data. Each quadruplet contains an Entity Key,
values (a dict containing feature) –
row (an event timestamp for the) –
and –
exists. (the created timestamp for the row if it) –
progress – Optional function to be called once every mini-batch of rows is written to
progress. (the online store. Can be used to display) –
- teardown(config: feast.repo_config.RepoConfig, tables: Sequence[feast.feature_view.FeatureView], entities: Sequence[feast.entity.Entity])[source]
We delete the keys in redis for tables/views being removed.
- update(config: feast.repo_config.RepoConfig, tables_to_delete: Sequence[feast.feature_view.FeatureView], tables_to_keep: Sequence[feast.feature_view.FeatureView], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity], partial: bool)[source]
Look for join_keys (list of entities) that are not in use anymore (usually this happens when the last feature view that was using specific compound key is deleted) and remove all features attached to this “join_keys”.
- class feast.infra.online_stores.redis.RedisOnlineStoreConfig(*, type: Literal['redis'] = 'redis', redis_type: feast.infra.online_stores.redis.RedisType = RedisType.redis, connection_string: pydantic.types.StrictStr = 'localhost:6379', key_ttl_seconds: int = None)[source]
Bases:
feast.repo_config.FeastConfigBaseModel
Online store config for Redis store
- connection_string: pydantic.types.StrictStr
Connection string containing the host, port, and configuration parameters for Redis format: host:port,parameter1,parameter2 eg. redis:6379,db=0
- redis_type: feast.infra.online_stores.redis.RedisType
redis or redis_cluster
- Type
Redis type
- type: Literal['redis']
Online store type selector
feast.infra.online_stores.sqlite module
- class feast.infra.online_stores.sqlite.SqliteOnlineStore[source]
Bases:
feast.infra.online_stores.online_store.OnlineStore
OnlineStore is an object used for all interaction between Feast and the service used for offline storage of features.
- _conn
SQLite connection.
- Type
Optional[sqlite3.Connection]
- online_read(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, entity_keys: List[feast.types.EntityKey_pb2.EntityKey], requested_features: Optional[List[str]] = None) List[Tuple[Optional[datetime.datetime], Optional[Dict[str, feast.types.Value_pb2.Value]]]] [source]
Read feature values given an Entity Key. This is a low level interface, not expected to be used by the users directly.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
entity_keys – a list of entity keys that should be read from the FeatureStore.
requested_features – (Optional) A subset of the features that should be read from the FeatureStore.
- Returns
Data is returned as a list, one item per entity key in the original order as the entity_keys argument. Each item in the list is a tuple of event_ts for the row, and the feature data as a dict from feature names to values. Values are returned as Value proto message.
- online_write_batch(config: feast.repo_config.RepoConfig, table: feast.feature_view.FeatureView, data: List[Tuple[feast.types.EntityKey_pb2.EntityKey, Dict[str, feast.types.Value_pb2.Value], datetime.datetime, Optional[datetime.datetime]]], progress: Optional[Callable[[int], Any]]) None [source]
Write a batch of feature rows to the online store. This is a low level interface, not expected to be used by the users directly.
If a tz-naive timestamp is passed to this method, it should be assumed to be UTC by implementors.
- Parameters
config – The RepoConfig for the current FeatureStore.
table – Feast FeatureView
data – a list of quadruplets containing Feature data. Each quadruplet contains an Entity Key,
values (a dict containing feature) –
row (an event timestamp for the) –
and –
exists. (the created timestamp for the row if it) –
progress – Optional function to be called once every mini-batch of rows is written to
progress. (the online store. Can be used to display) –
- plan(config: feast.repo_config.RepoConfig, desired_registry_proto: feast.core.Registry_pb2.Registry) List[feast.infra.infra_object.InfraObject] [source]
Returns the set of InfraObjects required to support the desired registry.
- Parameters
config – The RepoConfig for the current FeatureStore.
desired_registry_proto – The desired registry, in proto form.
- teardown(config: feast.repo_config.RepoConfig, tables: Sequence[feast.feature_view.FeatureView], entities: Sequence[feast.entity.Entity])[source]
- update(config: feast.repo_config.RepoConfig, tables_to_delete: Sequence[feast.feature_view.FeatureView], tables_to_keep: Sequence[feast.feature_view.FeatureView], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity], partial: bool)[source]
- class feast.infra.online_stores.sqlite.SqliteOnlineStoreConfig(*, type: Literal['sqlite', 'feast.infra.online_stores.sqlite.SqliteOnlineStore'] = 'sqlite', path: pydantic.types.StrictStr = 'data/online.db')[source]
Bases:
feast.repo_config.FeastConfigBaseModel
Online store config for local (SQLite-based) store
- path: pydantic.types.StrictStr
(optional) Path to sqlite db
- type: Literal['sqlite', 'feast.infra.online_stores.sqlite.SqliteOnlineStore']
Online store type selector
- class feast.infra.online_stores.sqlite.SqliteTable(path: str, name: str)[source]
Bases:
feast.infra.infra_object.InfraObject
A Sqlite table managed by Feast.
- name
The name of the table.
- conn
SQLite connection.
- Type
- conn: sqlite3.Connection
- static from_infra_object_proto(infra_object_proto: feast.core.InfraObject_pb2.InfraObject) Any [source]
Returns an InfraObject created from a protobuf representation.
- Parameters
infra_object_proto – A protobuf representation of an InfraObject.
- Raises
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.
- static from_proto(sqlite_table_proto: feast.core.SqliteTable_pb2.SqliteTable) Any [source]
Converts a protobuf representation of a subclass to an object of that subclass.
- Parameters
infra_object_proto – A protobuf representation of an InfraObject.
- Raises
FeastInvalidInfraObjectType – The type of InfraObject could not be identified.