InMemoryDatastore¶
- class lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore(config: DatastoreConfig, bridgeManager: DatastoreRegistryBridgeManager)¶
Bases:
GenericBaseDatastore
[StoredMemoryItemInfo
]Basic Datastore for writing to an in memory cache.
This datastore is ephemeral in that the contents of the datastore disappear when the Python process completes. This also means that other processes can not access this datastore.
- Parameters:
- config
DatastoreConfig
orstr
Configuration.
- bridgeManager
DatastoreRegistryBridgeManager
Object that manages the interface between
Registry
and datastores.
- config
Notes
InMemoryDatastore does not support any file-based ingest.
Attributes Summary
Path to configuration defaults.
A new datastore is created every time and datasets disappear when the process shuts down.
Methods Summary
clone
(bridgeManager)Make an independent copy of this Datastore with a different
DatastoreRegistryBridgeManager
instance.emptyTrash
([ignore_errors])Remove all datasets from the trash.
exists
(ref)Check if the dataset exists in the datastore.
export_records
(refs)Export datastore records and locations to an in-memory data structure.
forget
(refs)Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
get
(ref[, parameters, storageClass])Load an InMemoryDataset from the store.
Return all the lookup keys relevant to this datastore.
getURI
(ref[, predict])URI to the Dataset.
getURIs
(ref[, predict])Return URIs associated with dataset.
Make definitions of the opaque tables used by this Datastore.
import_records
(data)Import datastore location and record data from an in-memory data structure.
knows
(ref)Check if the dataset is known to the datastore.
needs_expanded_data_ids
(transfer[, entity])Test whether this datastore needs expanded data IDs to ingest.
put
(inMemoryDataset, ref)Write a InMemoryDataset with a given
DatasetRef
to the store.put_new
(in_memory_dataset, ref)Write a
InMemoryDataset
with a givenDatasetRef
to the store.removeStoredItemInfo
(ref)Remove information about the file associated with this dataset.
retrieveArtifacts
(refs, destination[, ...])Retrieve the file artifacts associated with the supplied refs.
setConfigRoot
(root, config, full[, overwrite])Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
trash
(ref[, ignore_errors])Indicate to the Datastore that a dataset can be removed.
validateConfiguration
(entities[, logFailures])Validate some of the configuration for this datastore.
validateKey
(lookupKey, entity)Validate a specific look up key with supplied entity.
Attributes Documentation
- defaultConfigFile: ClassVar[str | None] = 'datastores/inMemoryDatastore.yaml'¶
Path to configuration defaults. Accessed within the
configs
resource or relative to a search path. Can be None if no defaults specified.
- isEphemeral: bool = True¶
A new datastore is created every time and datasets disappear when the process shuts down.
Methods Documentation
- clone(bridgeManager: DatastoreRegistryBridgeManager) InMemoryDatastore ¶
Make an independent copy of this Datastore with a different
DatastoreRegistryBridgeManager
instance.- Parameters:
- bridgeManager
DatastoreRegistryBridgeManager
New
DatastoreRegistryBridgeManager
object to use when instantiating managers.
- bridgeManager
- Returns:
- datastore
Datastore
New
Datastore
instance with the same configuration as the existing instance.
- datastore
- emptyTrash(ignore_errors: bool = False) None ¶
Remove all datasets from the trash.
- Parameters:
- ignore_errors
bool
, optional Ignore errors.
- ignore_errors
Notes
The internal tracking of datasets is affected by this method and transaction handling is not supported if there is a problem before the datasets themselves are deleted.
Concurrency should not normally be an issue for the in memory datastore since all internal changes are isolated to solely this process and the registry only changes rows associated with this process.
- exists(ref: DatasetRef) bool ¶
Check if the dataset exists in the datastore.
- export_records(refs: Iterable[DatasetIdRef]) Mapping[str, DatastoreRecordData] ¶
Export datastore records and locations to an in-memory data structure.
- forget(refs: Iterable[DatasetRef]) None ¶
Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
- Parameters:
- refs
Iterable
[DatasetRef
] References to the datasets being forgotten.
- refs
Notes
Asking a datastore to forget a
DatasetRef
it does not hold should be a silent no-op, not an error.
- get(ref: DatasetRef, parameters: Mapping[str, Any] | None = None, storageClass: StorageClass | str | None = None) Any ¶
Load an InMemoryDataset from the store.
- Parameters:
- ref
DatasetRef
Reference to the required Dataset.
- parameters
dict
StorageClass
-specific parameters that specify, for example, a slice of the dataset to be loaded.- storageClass
StorageClass
orstr
, optional The storage class to be used to override the Python type returned by this method. By default the returned type matches the dataset type definition for this dataset. Specifying a read
StorageClass
can force a different type to be returned. This type must be compatible with the original type.
- ref
- Returns:
- inMemoryDataset
object
Requested dataset or slice thereof as an InMemoryDataset.
- inMemoryDataset
- Raises:
- FileNotFoundError
Requested dataset can not be retrieved.
- TypeError
Return value from formatter has unexpected type.
- ValueError
Formatter failed to process the dataset.
- getLookupKeys() set[LookupKey] ¶
Return all the lookup keys relevant to this datastore.
- Returns:
- keys
set
ofLookupKey
The keys stored internally for looking up information based on
DatasetType
name orStorageClass
.
- keys
- getURI(ref: DatasetRef, predict: bool = False) ResourcePath ¶
URI to the Dataset.
Always uses “mem://” URI prefix.
- Parameters:
- Returns:
- uri
str
URI pointing to the dataset within the datastore. If the dataset does not exist in the datastore, and if
predict
isTrue
, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.
- uri
- Raises:
- FileNotFoundError
A URI has been requested for a dataset that does not exist and guessing is not allowed.
- AssertionError
Raised if an internal error occurs.
- getURIs(ref: DatasetRef, predict: bool = False) DatasetRefURIs ¶
Return URIs associated with dataset.
- Parameters:
- ref
DatasetRef
Reference to the required dataset.
- predict
bool
, optional If the datastore does not know about the dataset, controls whether it should return a predicted URI or not.
- ref
- Returns:
- uris
DatasetRefURIs
The URI to the primary artifact associated with this dataset (if the dataset was disassembled within the datastore this may be
None
), and the URIs to any components associated with the dataset artifact. (can be empty if there are no components).
- uris
Notes
The URIs returned for in-memory datastores are not usable but provide an indication of the associated dataset.
- get_opaque_table_definitions() Mapping[str, DatastoreOpaqueTable] ¶
Make definitions of the opaque tables used by this Datastore.
- Returns:
- tables
Mapping
[str
,ddl.TableSpec
] Mapping of opaque table names to their definitions. This can be an empty mapping if Datastore does not use opaque tables to keep datastore records.
- tables
- import_records(data: Mapping[str, DatastoreRecordData]) None ¶
Import datastore location and record data from an in-memory data structure.
- Parameters:
Notes
Implementations should generally not check that any external resources (e.g. files) referred to by these records actually exist, for performance reasons; we expect higher-level code to guarantee that they do.
Implementations are responsible for calling
DatastoreRegistryBridge.insert
on all datasets indata.locations
where the key is innames
, as well as loading any opaque table data.Implementations may assume that datasets are either fully present or not at all (single-component exports are not permitted).
- knows(ref: DatasetRef) bool ¶
Check if the dataset is known to the datastore.
This datastore does not distinguish dataset existence from knowledge of a dataset.
- needs_expanded_data_ids(transfer: str | None, entity: DatasetRef | DatasetType | StorageClass | None = None) bool ¶
Test whether this datastore needs expanded data IDs to ingest.
- Parameters:
- Returns:
- put(inMemoryDataset: Any, ref: DatasetRef) None ¶
Write a InMemoryDataset with a given
DatasetRef
to the store.- Parameters:
- inMemoryDataset
object
The dataset to store.
- ref
DatasetRef
Reference to the associated Dataset.
- inMemoryDataset
- Raises:
- TypeError
Supplied object and storage class are inconsistent.
- DatasetTypeNotSupportedError
The associated
DatasetType
is not handled by this datastore.
Notes
If the datastore is configured to reject certain dataset types it is possible that the put will fail and raise a
DatasetTypeNotSupportedError
. The main use case for this is to allowChainedDatastore
to put to multiple datastores without requiring that every datastore accepts the dataset.
- put_new(in_memory_dataset: Any, ref: DatasetRef) Mapping[str, DatasetRef] ¶
Write a
InMemoryDataset
with a givenDatasetRef
to the store.- Parameters:
- in_memory_dataset
object
The Dataset to store.
- ref
DatasetRef
Reference to the associated Dataset.
- in_memory_dataset
- Returns:
- removeStoredItemInfo(ref: DatasetIdRef) None ¶
Remove information about the file associated with this dataset.
- Parameters:
- ref
DatasetRef
The dataset that has been removed.
- ref
Notes
This method is actually not used by this implementation, but there are some tests that check that this method works, so we keep it for now.
- retrieveArtifacts(refs: Iterable[DatasetRef], destination: ResourcePath, transfer: str = 'auto', preserve_path: bool = True, overwrite: bool | None = False) list[lsst.resources._resourcePath.ResourcePath] ¶
Retrieve the file artifacts associated with the supplied refs.
- Parameters:
- refsiterable of
DatasetRef
The datasets for which artifacts are to be retrieved. A single ref can result in multiple artifacts. The refs must be resolved.
- destination
lsst.resources.ResourcePath
Location to write the artifacts.
- transfer
str
, optional Method to use to transfer the artifacts. Must be one of the options supported by
lsst.resources.ResourcePath.transfer_from()
. “move” is not allowed.- preserve_path
bool
, optional If
True
the full path of the artifact within the datastore is preserved. IfFalse
the final file component of the path is used.- overwrite
bool
, optional If
True
allow transfers to overwrite existing files at the destination.
- refsiterable of
Notes
Not implemented by this datastore.
- classmethod setConfigRoot(root: str, config: Config, full: Config, overwrite: bool = True) None ¶
Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
Does nothing in this implementation.
- Parameters:
- root
str
Filesystem path to the root of the data repository.
- config
Config
A
Config
to update. Only the subset understood by this component will be updated. Will not expand defaults.- full
Config
A complete config with all defaults expanded that can be converted to a
DatastoreConfig
. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied fromfull
toconfig
.- overwrite
bool
, optional If
False
, do not modify a value inconfig
if the value already exists. Default is always to overwrite with the providedroot
.
- root
Notes
If a keyword is explicitly defined in the supplied
config
it will not be overridden by this method ifoverwrite
isFalse
. This allows explicit values set in external configs to be retained.
- trash(ref: DatasetRef | Iterable[DatasetRef], ignore_errors: bool = False) None ¶
Indicate to the Datastore that a dataset can be removed.
- Parameters:
- ref
DatasetRef
or iterable thereof Reference to the required Dataset(s).
- ignore_errors
bool
, optional Indicate that errors should be ignored.
- ref
- Raises:
- FileNotFoundError
Attempt to remove a dataset that does not exist. Only relevant if a single dataset ref is given.
Notes
Concurrency should not normally be an issue for the in memory datastore since all internal changes are isolated to solely this process and the registry only changes rows associated with this process.
- validateConfiguration(entities: Iterable[DatasetRef | DatasetType | StorageClass], logFailures: bool = False) None ¶
Validate some of the configuration for this datastore.
- Parameters:
- Raises:
- DatastoreValidationError
Raised if there is a validation problem with a configuration. All the problems are reported in a single exception.
Notes
This method is a no-op.
- validateKey(lookupKey: LookupKey, entity: DatasetRef | DatasetType | StorageClass) None ¶
Validate a specific look up key with supplied entity.
- Parameters:
- lookupKey
LookupKey
Key to use to retrieve information from the datastore configuration.
- entity
DatasetRef
,DatasetType
, orStorageClass
Entity to compare with configuration retrieved using the specified lookup key.
- lookupKey
- Raises:
- DatastoreValidationError
Raised if there is a problem with the combination of entity and lookup key.
Notes
Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.