InMemoryDatastore

class lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore(config: Union[Config, str], bridgeManager: DatastoreRegistryBridgeManager, butlerRoot: Optional[str] = None)

Bases: lsst.daf.butler.datastores.genericDatastore.GenericBaseDatastore

Basic Datastore for writing to an in memory cache.

This datastore is ephemeral in that the contents of the datastore disappear when the Python process completes. This also means that other processes can not access this datastore.

Parameters:
config : DatastoreConfig or str

Configuration.

bridgeManager : DatastoreRegistryBridgeManager

Object that manages the interface between Registry and datastores.

butlerRoot : str, optional

Unused parameter.

Notes

InMemoryDatastore does not support any file-based ingest.

Attributes Summary

bridge Object that manages the interface between this Datastore and the Registry (DatastoreRegistryBridge).
containerKey
defaultConfigFile Path to configuration defaults.
isEphemeral A new datastore is created every time and datasets disappear when the process shuts down.
names Names associated with this datastore returned as a list.

Methods Summary

addStoredItemInfo(refs, infos) Record internal storage information associated with one or more datasets.
emptyTrash(ignore_errors) Remove all datasets from the trash.
exists(ref) Check if the dataset exists in the datastore.
export(refs, *, directory, transfer) Export datasets for transfer to another data repository.
forget(refs) Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
fromConfig(config, bridgeManager, …) Create datastore from type specified in config file.
get(ref, parameters, Any]] = None) Load an InMemoryDataset from the store.
getLookupKeys() Return all the lookup keys relevant to this datastore.
getStoredItemInfo(ref)
getStoredItemsInfo(ref) Retrieve information associated with files stored in this Datastore associated with this dataset ref.
getURI(ref, predict) URI to the Dataset.
getURIs(ref, predict) Return URIs associated with dataset.
ingest(*datasets, transfer) Ingest one or more files into the datastore.
knows(ref) Check if the dataset is known to the datastore.
needs_expanded_data_ids(transfer, entity, …) Test whether this datastore needs expanded data IDs to ingest.
put(inMemoryDataset, ref) Write a InMemoryDataset with a given DatasetRef to the store.
remove(ref) Indicate to the Datastore that a dataset can be removed.
removeStoredItemInfo(ref) Remove information about the file associated with this dataset.
retrieveArtifacts(refs, destination, …) Retrieve the file artifacts associated with the supplied refs.
setConfigRoot(root, config, full, overwrite) Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
transaction() Context manager supporting Datastore transactions.
transfer(inputDatastore, ref) Retrieve a dataset from an input Datastore, and store the result in this Datastore.
transfer_from(source_datastore, refs, …) Transfer dataset artifacts from another datastore to this one.
trash(ref, …) Indicate to the Datastore that a dataset can be removed.
validateConfiguration(entities, DatasetType, …) Validate some of the configuration for this datastore.
validateKey(lookupKey, entity, DatasetType, …) Validate a specific look up key with supplied entity.

Attributes Documentation

bridge

Object that manages the interface between this Datastore and the Registry (DatastoreRegistryBridge).

containerKey = None
defaultConfigFile = 'datastores/inMemoryDatastore.yaml'

Path to configuration defaults. Accessed within the configs resource or relative to a search path. Can be None if no defaults specified.

isEphemeral = True

A new datastore is created every time and datasets disappear when the process shuts down.

names

Names associated with this datastore returned as a list.

Can be different to name for a chaining datastore.

Methods Documentation

addStoredItemInfo(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef], infos: Iterable[lsst.daf.butler.datastores.inMemoryDatastore.StoredMemoryItemInfo]) → None

Record internal storage information associated with one or more datasets.

Parameters:
refs : sequence of DatasetRef

The datasets that have been stored.

infos : sequence of StoredDatastoreItemInfo

Metadata associated with the stored datasets.

emptyTrash(ignore_errors: bool = False) → None

Remove all datasets from the trash.

Parameters:
ignore_errors : bool, optional

Ignore errors.

Notes

The internal tracking of datasets is affected by this method and transaction handling is not supported if there is a problem before the datasets themselves are deleted.

Concurrency should not normally be an issue for the in memory datastore since all internal changes are isolated to solely this process and the registry only changes rows associated with this process.

exists(ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → bool

Check if the dataset exists in the datastore.

Parameters:
ref : DatasetRef

Reference to the required dataset.

Returns:
exists : bool

True if the entity exists in the Datastore.

export(refs: Iterable[DatasetRef], *, directory: Optional[str] = None, transfer: Optional[str] = None) → Iterable[FileDataset]

Export datasets for transfer to another data repository.

Parameters:
refs : iterable of DatasetRef

Dataset references to be exported.

directory : str, optional

Path to a directory that should contain files corresponding to output datasets. Ignored if transfer is None.

transfer : str, optional

Mode that should be used to move datasets out of the repository. Valid options are the same as those of the transfer argument to ingest, and datastores may similarly signal that a transfer mode is not supported by raising NotImplementedError.

Returns:
dataset : iterable of DatasetTransfer

Structs containing information about the exported datasets, in the same order as refs.

Raises:
NotImplementedError

Raised if the given transfer mode is not supported.

forget(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]) → None

Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.

Parameters:
refs : Iterable [ DatasetRef ]

References to the datasets being forgotten.

Notes

Asking a datastore to forget a DatasetRef it does not hold should be a silent no-op, not an error.

static fromConfig(config: Config, bridgeManager: DatastoreRegistryBridgeManager, butlerRoot: Optional[Union[str, ButlerURI]] = None) → 'Datastore'

Create datastore from type specified in config file.

Parameters:
config : Config

Configuration instance.

bridgeManager : DatastoreRegistryBridgeManager

Object that manages the interface between Registry and datastores.

butlerRoot : str, optional

Butler root directory.

get(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, parameters: Optional[Mapping[str, Any]] = None) → Any

Load an InMemoryDataset from the store.

Parameters:
ref : DatasetRef

Reference to the required Dataset.

parameters : dict

StorageClass-specific parameters that specify, for example, a slice of the dataset to be loaded.

Returns:
inMemoryDataset : object

Requested dataset or slice thereof as an InMemoryDataset.

Raises:
FileNotFoundError

Requested dataset can not be retrieved.

TypeError

Return value from formatter has unexpected type.

ValueError

Formatter failed to process the dataset.

getLookupKeys() → Set[LookupKey]

Return all the lookup keys relevant to this datastore.

Returns:
keys : set of LookupKey

The keys stored internally for looking up information based on DatasetType name or StorageClass.

getStoredItemInfo(ref: DatasetIdRef) → StoredMemoryItemInfo
getStoredItemsInfo(ref: DatasetIdRef) → List[StoredMemoryItemInfo]

Retrieve information associated with files stored in this Datastore associated with this dataset ref.

Parameters:
ref : DatasetRef

The dataset that is to be queried.

Returns:
items : list [StoredDatastoreItemInfo]

Stored information about the files and associated formatters associated with this dataset. Only one file will be returned if the dataset has not been disassembled. Can return an empty list if no matching datasets can be found.

getURI(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, predict: bool = False) → lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI

URI to the Dataset.

Always uses “mem://” URI prefix.

Parameters:
ref : DatasetRef

Reference to the required Dataset.

predict : bool

If True, allow URIs to be returned of datasets that have not been written.

Returns:
uri : str

URI pointing to the dataset within the datastore. If the dataset does not exist in the datastore, and if predict is True, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

Raises:
FileNotFoundError

A URI has been requested for a dataset that does not exist and guessing is not allowed.

AssertionError

Raised if an internal error occurs.

getURIs(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, predict: bool = False) → Tuple[Optional[lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI], Dict[str, lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI]]

Return URIs associated with dataset.

Parameters:
ref : DatasetRef

Reference to the required dataset.

predict : bool, optional

If the datastore does not know about the dataset, should it return a predicted URI or not?

Returns:
primary : ButlerURI

The URI to the primary artifact associated with this dataset. If the dataset was disassembled within the datastore this may be None.

components : dict

URIs to any components associated with the dataset artifact. Can be empty if there are no components.

Notes

The URIs returned for in-memory datastores are not usable but provide an indication of the associated dataset.

ingest(*datasets, transfer: Optional[str] = None) → None

Ingest one or more files into the datastore.

Parameters:
datasets : FileDataset

Each positional argument is a struct containing information about a file to be ingested, including its path (either absolute or relative to the datastore root, if applicable), a complete DatasetRef (with dataset_id not None), and optionally a formatter class or its fully-qualified string name. If a formatter is not provided, the one the datastore would use for put on that dataset is assumed.

transfer : str, optional

How (and whether) the dataset should be added to the datastore. If None (default), the file must already be in a location appropriate for the datastore (e.g. within its root directory), and will not be modified. Other choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” is a special option that will let the data store choose the most natural option for itself.

Raises:
NotImplementedError

Raised if the datastore does not support the given transfer mode (including the case where ingest is not supported at all).

DatasetTypeNotSupportedError

Raised if one or more files to be ingested have a dataset type that is not supported by the datastore.

FileNotFoundError

Raised if one of the given files does not exist.

FileExistsError

Raised if transfer is not None but the (internal) location the file would be moved to is already occupied.

Notes

Subclasses should implement _prepIngest and _finishIngest instead of implementing ingest directly. Datastores that hold and delegate to child datastores may want to call those methods as well.

Subclasses are encouraged to document their supported transfer modes in their class documentation.

knows(ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → bool

Check if the dataset is known to the datastore.

This datastore does not distinguish dataset existence from knowledge of a dataset.

Parameters:
ref : DatasetRef

Reference to the required dataset.

Returns:
exists : bool

True if the dataset is known to the datastore.

needs_expanded_data_ids(transfer: Optional[str], entity: Optional[Union[DatasetRef, DatasetType, StorageClass]] = None) → bool

Test whether this datastore needs expanded data IDs to ingest.

Parameters:
transfer : str or None

Transfer mode for ingest.

entity, optional

Object representing what will be ingested. If not provided (or not specific enough), True may be returned even if expanded data IDs aren’t necessary.

Returns:
needed : bool

If True, expanded data IDs may be needed. False only if expansion definitely isn’t necessary.

put(inMemoryDataset: Any, ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → None

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:
inMemoryDataset : object

The dataset to store.

ref : DatasetRef

Reference to the associated Dataset.

Raises:
TypeError

Supplied object and storage class are inconsistent.

DatasetTypeNotSupportedError

The associated DatasetType is not handled by this datastore.

Notes

If the datastore is configured to reject certain dataset types it is possible that the put will fail and raise a DatasetTypeNotSupportedError. The main use case for this is to allow ChainedDatastore to put to multiple datastores without requiring that every datastore accepts the dataset.

remove(ref: DatasetRef) → None

Indicate to the Datastore that a dataset can be removed.

Warning

This method deletes the artifact associated with this dataset and can not be reversed.

Parameters:
ref : DatasetRef

Reference to the required Dataset.

Raises:
FileNotFoundError

Attempt to remove a dataset that does not exist.

Notes

This method is used for immediate removal of a dataset and is generally reserved for internal testing of datastore APIs. It is implemented by calling trash() and then immediately calling emptyTrash(). This call is meant to be immediate so errors encountered during removal are not ignored.

removeStoredItemInfo(ref: DatasetIdRef) → None

Remove information about the file associated with this dataset.

Parameters:
ref : DatasetRef

The dataset that has been removed.

retrieveArtifacts(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef], destination: lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI, transfer: str = 'auto', preserve_path: bool = True, overwrite: Optional[bool] = False) → List[lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI]

Retrieve the file artifacts associated with the supplied refs.

Notes

Not implemented by this datastore.

classmethod setConfigRoot(root: str, config: Config, full: Config, overwrite: bool = True) → None

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Does nothing in this implementation.

Parameters:
root : str

Filesystem path to the root of the data repository.

config : Config

A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.

full : Config

A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.

overwrite : bool, optional

If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transaction() → Iterator[lsst.daf.butler.core.datastore.DatastoreTransaction]

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

transfer(inputDatastore: Datastore, ref: DatasetRef) → None

Retrieve a dataset from an input Datastore, and store the result in this Datastore.

Parameters:
inputDatastore : Datastore

The external Datastore from which to retreive the Dataset.

ref : DatasetRef

Reference to the required dataset in the input data store.

transfer_from(source_datastore: Datastore, refs: Iterable[DatasetRef], local_refs: Optional[Iterable[DatasetRef]] = None, transfer: str = 'auto') → None

Transfer dataset artifacts from another datastore to this one.

Parameters:
source_datastore : Datastore

The datastore from which to transfer artifacts. That datastore must be compatible with this datastore receiving the artifacts.

refs : iterable of DatasetRef

The datasets to transfer from the source datastore.

local_refs : iterable of DatasetRef, optional

The dataset refs associated with the registry associated with this datastore. Can be None if the source and target datastore are using UUIDs.

transfer : str, optional

How (and whether) the dataset should be added to the datastore. Choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” (the default) is a special option that will let the data store choose the most natural option for itself. If the source location and transfer location are identical the transfer mode will be ignored.

Raises:
TypeError

Raised if the two datastores are not compatible.

trash(ref: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]], ignore_errors: bool = False) → None

Indicate to the Datastore that a dataset can be removed.

Parameters:
ref : DatasetRef or iterable thereof

Reference to the required Dataset(s).

ignore_errors: `bool`, optional

Indicate that errors should be ignored.

Raises:
FileNotFoundError

Attempt to remove a dataset that does not exist. Only relevant if a single dataset ref is given.

Notes

Concurrency should not normally be an issue for the in memory datastore since all internal changes are isolated to solely this process and the registry only changes rows associated with this process.

validateConfiguration(entities: Iterable[Union[DatasetRef, DatasetType, StorageClass]], logFailures: bool = False) → None

Validate some of the configuration for this datastore.

Parameters:
entities : iterable of DatasetRef, DatasetType, or StorageClass

Entities to test against this configuration. Can be differing types.

logFailures : bool, optional

If True, output a log message for every validation error detected.

Raises:
DatastoreValidationError

Raised if there is a validation problem with a configuration. All the problems are reported in a single exception.

Notes

This method is a no-op.

validateKey(lookupKey: LookupKey, entity: Union[DatasetRef, DatasetType, StorageClass]) → None

Validate a specific look up key with supplied entity.

Parameters:
lookupKey : LookupKey

Key to use to retrieve information from the datastore configuration.

entity : DatasetRef, DatasetType, or StorageClass

Entity to compare with configuration retrieved using the specified lookup key.

Raises:
DatastoreValidationError

Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.