InMemoryDatastore

class lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore(config: DatastoreConfig, bridgeManager: DatastoreRegistryBridgeManager)

Bases: GenericBaseDatastore[StoredMemoryItemInfo]

Basic Datastore for writing to an in memory cache.

This datastore is ephemeral in that the contents of the datastore disappear when the Python process completes. This also means that other processes can not access this datastore.

Parameters:

configDatastoreConfig or str

Configuration.

bridgeManagerDatastoreRegistryBridgeManager

Object that manages the interface between Registry and datastores.

Notes

InMemoryDatastore does not support any file-based ingest.

Attributes Summary

defaultConfigFile	Path to configuration defaults.
isEphemeral	A new datastore is created every time and datasets disappear when the process shuts down.

Methods Summary

clone(bridgeManager)	Make an independent copy of this Datastore with a different DatastoreRegistryBridgeManager instance.
emptyTrash([ignore_errors])	Remove all datasets from the trash.
exists(ref)	Check if the dataset exists in the datastore.
export_records(refs)	Export datastore records and locations to an in-memory data structure.
forget(refs)	Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
get(ref[, parameters, storageClass])	Load an InMemoryDataset from the store.
getLookupKeys()	Return all the lookup keys relevant to this datastore.
getURI(ref[, predict])	URI to the Dataset.
getURIs(ref[, predict])	Return URIs associated with dataset.
get_opaque_table_definitions()	Make definitions of the opaque tables used by this Datastore.
import_records(data)	Import datastore location and record data from an in-memory data structure.
ingest_zip(zip_path, transfer)	Ingest an indexed Zip file and contents.
knows(ref)	Check if the dataset is known to the datastore.
needs_expanded_data_ids(transfer[, entity])	Test whether this datastore needs expanded data IDs to ingest.
put(inMemoryDataset, ref)	Write a InMemoryDataset with a given DatasetRef to the store.
put_new(in_memory_dataset, ref)	Write a InMemoryDataset with a given DatasetRef to the store.
removeStoredItemInfo(ref)	Remove information about the file associated with this dataset.
retrieveArtifacts(refs, destination[, ...])	Retrieve the file artifacts associated with the supplied refs.
setConfigRoot(root, config, full[, overwrite])	Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
trash(ref[, ignore_errors])	Indicate to the Datastore that a dataset can be removed.
validateConfiguration(entities[, logFailures])	Validate some of the configuration for this datastore.
validateKey(lookupKey, entity)	Validate a specific look up key with supplied entity.

Attributes Documentation

defaultConfigFile: ClassVar[str | None] = 'datastores/inMemoryDatastore.yaml'

Path to configuration defaults. Accessed within the configs resource or relative to a search path. Can be None if no defaults specified.

isEphemeral: bool = True

A new datastore is created every time and datasets disappear when the process shuts down.

Methods Documentation

clone(bridgeManager: DatastoreRegistryBridgeManager) → InMemoryDatastore

Make an independent copy of this Datastore with a different DatastoreRegistryBridgeManager instance.

Parameters:

bridgeManagerDatastoreRegistryBridgeManager

New DatastoreRegistryBridgeManager object to use when instantiating managers.

Returns:

datastoreDatastore

New Datastore instance with the same configuration as the existing instance.

emptyTrash(ignore_errors: bool = False) → None

Remove all datasets from the trash.

Parameters:

ignore_errorsbool, optional

Ignore errors.

Notes

The internal tracking of datasets is affected by this method and transaction handling is not supported if there is a problem before the datasets themselves are deleted.

Concurrency should not normally be an issue for the in memory datastore since all internal changes are isolated to solely this process and the registry only changes rows associated with this process.

exists(ref: DatasetRef) → bool

Check if the dataset exists in the datastore.

Parameters:

refDatasetRef

Reference to the required dataset.

Returns:

existsbool

True if the entity exists in the Datastore.

export_records(refs: Iterable[DatasetIdRef]) → Mapping[str, DatastoreRecordData]

Export datastore records and locations to an in-memory data structure.

Parameters:

refsIterable [ DatasetIdRef ]

Datasets to save. This may include datasets not known to this datastore, which should be ignored. May not include component datasets.

Returns:

dataMapping [ str, DatastoreRecordData ]

Exported datastore records indexed by datastore name.

forget(refs: Iterable[DatasetRef]) → None

Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.

Parameters:

refsIterable [ DatasetRef ]

References to the datasets being forgotten.

Notes

Asking a datastore to forget a DatasetRef it does not hold should be a silent no-op, not an error.

get(ref: DatasetRef, parameters: Mapping[str, Any] | None = None, storageClass: StorageClass | str | None = None) → Any

Load an InMemoryDataset from the store.

Parameters:

refDatasetRef

Reference to the required Dataset.

parametersdict

StorageClass-specific parameters that specify, for example, a slice of the dataset to be loaded.

storageClassStorageClass or str, optional

The storage class to be used to override the Python type returned by this method. By default the returned type matches the dataset type definition for this dataset. Specifying a read StorageClass can force a different type to be returned. This type must be compatible with the original type.

Returns:

inMemoryDatasetobject

Requested dataset or slice thereof as an InMemoryDataset.

Raises:

FileNotFoundError

Requested dataset can not be retrieved.

TypeError

Return value from formatter has unexpected type.

ValueError

Formatter failed to process the dataset.

getLookupKeys() → set[LookupKey]

Return all the lookup keys relevant to this datastore.

Returns:

keysset of LookupKey

The keys stored internally for looking up information based on DatasetType name or StorageClass.

getURI(ref: DatasetRef, predict: bool = False) → ResourcePath

URI to the Dataset.

Always uses “mem://” URI prefix.

Parameters:

refDatasetRef

Reference to the required Dataset.

predictbool

If True, allow URIs to be returned of datasets that have not been written.

Returns:

uristr

URI pointing to the dataset within the datastore. If the dataset does not exist in the datastore, and if predict is True, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

Raises:

FileNotFoundError

A URI has been requested for a dataset that does not exist and guessing is not allowed.

AssertionError

Raised if an internal error occurs.

getURIs(ref: DatasetRef, predict: bool = False) → DatasetRefURIs

Return URIs associated with dataset.

Parameters:

refDatasetRef

Reference to the required dataset.

predictbool, optional

If the datastore does not know about the dataset, controls whether it should return a predicted URI or not.

Returns:

urisDatasetRefURIs

The URI to the primary artifact associated with this dataset (if the dataset was disassembled within the datastore this may be None), and the URIs to any components associated with the dataset artifact. (can be empty if there are no components).

Notes

The URIs returned for in-memory datastores are not usable but provide an indication of the associated dataset.

get_opaque_table_definitions() → Mapping[str, DatastoreOpaqueTable]

Make definitions of the opaque tables used by this Datastore.

Returns:

tablesMapping [ str, ddl.TableSpec ]

Mapping of opaque table names to their definitions. This can be an empty mapping if Datastore does not use opaque tables to keep datastore records.

import_records(data: Mapping[str, DatastoreRecordData]) → None

Import datastore location and record data from an in-memory data structure.

Parameters:

dataMapping [ str, DatastoreRecordData ]

Datastore records indexed by datastore name. May contain data for other Datastore instances (generally because they are chained to this one), which should be ignored.

Notes

Implementations should generally not check that any external resources (e.g. files) referred to by these records actually exist, for performance reasons; we expect higher-level code to guarantee that they do.

Implementations are responsible for calling DatastoreRegistryBridge.insert on all datasets in data.locations where the key is in names, as well as loading any opaque table data.

Implementations may assume that datasets are either fully present or not at all (single-component exports are not permitted).

ingest_zip(zip_path: ResourcePath, transfer: str | None) → None

Ingest an indexed Zip file and contents.

The Zip file must have an index file as created by retrieveArtifacts.

Parameters:

zip_pathlsst.resources.ResourcePath

Path to the Zip file.

transferstr

Method to use for transferring the Zip file into the datastore.

knows(ref: DatasetRef) → bool

Check if the dataset is known to the datastore.

This datastore does not distinguish dataset existence from knowledge of a dataset.

Parameters:

refDatasetRef

Reference to the required dataset.

Returns:

existsbool

True if the dataset is known to the datastore.

needs_expanded_data_ids(transfer: str | None, entity: DatasetRef | DatasetType | StorageClass | None = None) → bool

Test whether this datastore needs expanded data IDs to ingest.

Parameters:

transferstr or None

Transfer mode for ingest.

entityDatasetRef or DatasetType or StorageClass or None, optional

Object representing what will be ingested. If not provided (or not specific enough), True may be returned even if expanded data IDs aren’t necessary.

Returns:

neededbool

If True, expanded data IDs may be needed. False only if expansion definitely isn’t necessary.

put(inMemoryDataset: Any, ref: DatasetRef) → None

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:

inMemoryDatasetobject

The dataset to store.

refDatasetRef

Reference to the associated Dataset.

Raises:

TypeError

Supplied object and storage class are inconsistent.

DatasetTypeNotSupportedError

The associated DatasetType is not handled by this datastore.

Notes

If the datastore is configured to reject certain dataset types it is possible that the put will fail and raise a DatasetTypeNotSupportedError. The main use case for this is to allow ChainedDatastore to put to multiple datastores without requiring that every datastore accepts the dataset.

put_new(in_memory_dataset: Any, ref: DatasetRef) → Mapping[str, DatasetRef]

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:

in_memory_datasetobject

The Dataset to store.

refDatasetRef

Reference to the associated Dataset.

Returns:

datastore_refsMapping [str, DatasetRef]

Mapping of a datastore name to dataset reference stored in that datastore, reference will include datastore records. Only non-ephemeral datastores will appear in this mapping.

removeStoredItemInfo(ref: DatasetIdRef) → None

Remove information about the file associated with this dataset.

Parameters:

refDatasetRef

The dataset that has been removed.

Notes

This method is actually not used by this implementation, but there are some tests that check that this method works, so we keep it for now.

retrieveArtifacts(refs: Iterable[DatasetRef], destination: ResourcePath, transfer: str = 'auto', preserve_path: bool = True, overwrite: bool | None = False, write_index: bool = True, add_prefix: bool = False) → dict[ResourcePath, ArtifactIndexInfo]

Retrieve the file artifacts associated with the supplied refs.

Parameters:

refsiterable of DatasetRef

The datasets for which artifacts are to be retrieved. A single ref can result in multiple artifacts. The refs must be resolved.

destinationlsst.resources.ResourcePath

Location to write the artifacts.

transferstr, optional

Method to use to transfer the artifacts. Must be one of the options supported by lsst.resources.ResourcePath.transfer_from(). “move” is not allowed.

preserve_pathbool, optional

If True the full path of the artifact within the datastore is preserved. If False the final file component of the path is used.

overwritebool, optional

If True allow transfers to overwrite existing files at the destination.

write_indexbool, optional

If True write a file at the top level containing a serialization of a ZipIndex for the downloaded datasets.

add_prefixbool, optional

If True and if preserve_path is False, apply a prefix to the filenames corresponding to some part of the dataset ref ID. This can be used to guarantee uniqueness.

Notes

Not implemented by this datastore.

classmethod setConfigRoot(root: str, config: Config, full: Config, overwrite: bool = True) → None

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Does nothing in this implementation.

Parameters:

rootstr

Filesystem path to the root of the data repository.

configConfig

A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.

fullConfig

A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.

overwritebool, optional

If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

trash(ref: DatasetRef | Iterable[DatasetRef], ignore_errors: bool = False) → None

Indicate to the Datastore that a dataset can be removed.

Parameters:

refDatasetRef or iterable thereof

Reference to the required Dataset(s).

ignore_errorsbool, optional

Indicate that errors should be ignored.

Raises:

FileNotFoundError

Attempt to remove a dataset that does not exist. Only relevant if a single dataset ref is given.

Notes

Concurrency should not normally be an issue for the in memory datastore since all internal changes are isolated to solely this process and the registry only changes rows associated with this process.

validateConfiguration(entities: Iterable[DatasetRef | DatasetType | StorageClass], logFailures: bool = False) → None

Validate some of the configuration for this datastore.

Parameters:

entitiesiterable of DatasetRef, DatasetType, or StorageClass

Entities to test against this configuration. Can be differing types.

logFailuresbool, optional

If True, output a log message for every validation error detected.

Raises:

DatastoreValidationError

Raised if there is a validation problem with a configuration. All the problems are reported in a single exception.

Notes

This method is a no-op.

validateKey(lookupKey: LookupKey, entity: DatasetRef | DatasetType | StorageClass) → None

Validate a specific look up key with supplied entity.

Parameters:

lookupKeyLookupKey

Key to use to retrieve information from the datastore configuration.

entityDatasetRef, DatasetType, or StorageClass

Entity to compare with configuration retrieved using the specified lookup key.

Raises:

DatastoreValidationError

Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.