InMemoryDatastore¶

class lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore(config, registry=None)¶

Bases: lsst.daf.butler.Datastore

Basic Datastore for writing to an in memory cache.

This datastore is ephemeral in that the contents of the datastore disappear when the Python process completes. This also means that other processes can not access this datastore.

Parameters:	config : `DatastoreConfig` or `str` Configuration.
Attributes:	config : `DatastoreConfig` Configuration used to create Datastore. storageClassFactory : `StorageClassFactory` Factory for creating storage class instances from name. name : `str` Label associated with this Datastore.

Attributes Summary

`containerKey`
`defaultConfigFile`	Path to configuration defaults.
`isEphemeral`	A new datastore is created every time and datasets disappear when the process shuts down.

Methods Summary

`addStoredItemInfo`(ref, info)	Record internal storage information associated with this `DatasetRef`.
`exists`(ref)	Check if the dataset exists in the datastore.
`fromConfig`(config, registry)	Create datastore from type specified in config file.
`get`(ref[, parameters])	Load an InMemoryDataset from the store.
`getStoredItemInfo`(ref)	Retrieve information associated with object stored in this `Datastore`.
`getUri`(ref[, predict])	URI to the Dataset.
`ingest`(path, ref[, formatter, transfer])	Add an on-disk file with the given `DatasetRef` to the store, possibly transferring it.
`put`(inMemoryDataset, ref)	Write a InMemoryDataset with a given `DatasetRef` to the store.
`remove`(ref)	Indicate to the Datastore that a Dataset can be removed.
`removeStoredItemInfo`(ref)	Remove information about the object associated with this dataset.
`setConfigRoot`(root, config, full)	Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
`transaction`()	Context manager supporting `Datastore` transactions.
`transfer`(inputDatastore, ref)	Retrieve a Dataset from an input `Datastore`, and store the result in this `Datastore`.

Attributes Documentation

containerKey = None¶

defaultConfigFile = 'datastores/inMemoryDatastore.yaml'¶: Path to configuration defaults. Relative to $DAF_BUTLER_DIR/config or absolute path. Can be None if no defaults specified.

isEphemeral = True¶: A new datastore is created every time and datasets disappear when the process shuts down.

Methods Documentation

addStoredItemInfo(ref, info)¶

Record internal storage information associated with this DatasetRef.

Parameters:	ref : `DatasetRef` The Dataset that has been stored. info : `StoredItemInfo` Metadata associated with the stored Dataset.
Raises:	KeyError An entry with this DatasetRef already exists.

exists(ref)¶

Check if the dataset exists in the datastore.

Parameters:	ref : `DatasetRef` Reference to the required dataset.
Returns:	exists : `bool` `True` if the entity exists in the `Datastore`.

static fromConfig(config, registry)¶

Create datastore from type specified in config file.

Parameters:	config : `Config` Configuration instance.

get(ref, parameters=None)¶

Load an InMemoryDataset from the store.

Parameters:	ref : `DatasetRef` Reference to the required Dataset. parameters : `dict` `StorageClass`-specific parameters that specify, for example, a slice of the Dataset to be loaded.
Returns:	inMemoryDataset : `object` Requested Dataset or slice thereof as an InMemoryDataset.
Raises:	FileNotFoundError Requested dataset can not be retrieved. TypeError Return value from formatter has unexpected type. ValueError Formatter failed to process the dataset.

getStoredItemInfo(ref)¶

Retrieve information associated with object stored in this Datastore.

Parameters:	ref : `DatasetRef` The Dataset that is to be queried.
Returns:	info : `StoredItemInfo` Stored information about the internal location of this file and its formatter.
Raises:	KeyError Dataset with that id can not be found.

getUri(ref, predict=False)¶

URI to the Dataset.

Always uses “mem://” URI prefix.

Parameters:	ref : `DatasetRef` Reference to the required Dataset. predict : `bool` If `True`, allow URIs to be returned of datasets that have not been written.
Returns:	uri : `str` URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, and if `predict` is `True`, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.
Raises:	FileNotFoundError A URI has been requested for a dataset that does not exist and guessing is not allowed.

ingest(path, ref, formatter=None, transfer=None)¶

Add an on-disk file with the given DatasetRef to the store, possibly transferring it.

The caller is responsible for ensuring that the given (or predicted) Formatter is consistent with how the file was written; ingest will in general silently ignore incorrect formatters (as it cannot efficiently verify their correctness), deferring errors until get is first called on the ingested dataset.

Datastores are not required to implement this method, but must do so in order to support direct raw data ingest.

Parameters:

path : str: File path, relative to the repository root.
ref : DatasetRef: Reference to the associated Dataset.
formatter : Formatter (optional): Formatter that should be used to retreive the Dataset. If not provided, the formatter will be constructed according to Datastore configuration.
transfer : str (optional): If not None, must be one of ‘move’, ‘copy’, ‘hardlink’, or ‘symlink’ indicating how to transfer the file. Datastores need not support all options, but must raise NotImplementedError if the passed option is not supported. That includes None, which indicates that the file should be ingested at its current location with no transfer. If a Datastore does support ingest-without-transfer in general, but the given path is not appropriate, an exception other than NotImplementedError that better describes the problem should be raised.

Raises:

NotImplementedError: Raised if the given transfer mode is not supported.

put(inMemoryDataset, ref)¶

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:	inMemoryDataset : `object` The Dataset to store. ref : `DatasetRef` Reference to the associated Dataset.
Raises:	TypeError Supplied object and storage class are inconsistent.

remove(ref)¶

Indicate to the Datastore that a Dataset can be removed.

Parameters:	ref : `DatasetRef` Reference to the required Dataset.
Raises:	FileNotFoundError Attempt to remove a dataset that does not exist.

removeStoredItemInfo(ref)¶

Remove information about the object associated with this dataset.

Parameters:	ref : `DatasetRef` The Dataset that has been removed.

classmethod setConfigRoot(root, config, full)¶

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Does nothing in this implementation.

Parameters:

root : str: Filesystem path to the root of the data repository.
config : Config: A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.
full : Config: A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to Config.

transaction()¶

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

transfer(inputDatastore, ref)¶

Retrieve a Dataset from an input Datastore, and store the result in this Datastore.

Parameters:	inputDatastore : `Datastore` The external `Datastore` from which to retreive the Dataset. ref : `DatasetRef` Reference to the required Dataset in the input data store.

Navigation

InMemoryDatastore¶