ChainedDatastore¶

class lsst.daf.butler.datastores.chainedDatastore.ChainedDatastore(config: Union[Config, str], bridgeManager: DatastoreRegistryBridgeManager, butlerRoot: str | None = None)¶

Bases: lsst.daf.butler.Datastore

Chained Datastores to allow read and writes from multiple datastores.

A ChainedDatastore is configured with multiple datastore configurations. A put() is always sent to each datastore. A get() operation is sent to each datastore in turn and the first datastore to return a valid dataset is used.

Parameters:

config : DatastoreConfig or str: Configuration. This configuration must include a datastores field as a sequence of datastore configurations. The order in this sequence indicates the order to use for read operations.
bridgeManager : DatastoreRegistryBridgeManager: Object that manages the interface between Registry and datastores.
butlerRoot : str, optional: New datastore root to use to override the configuration value. This root is sent to each child datastore.

Notes

ChainedDatastore never supports None or "move" as an ingest transfer mode. It supports "copy", "symlink", "relsymlink" and "hardlink" if and only if all its child datastores do.

Attributes Summary

`containerKey`	Key to specify where child datastores are configured.
`defaultConfigFile`	Path to configuration defaults.
`isEphemeral`
`names`	Names associated with this datastore returned as a list.

Methods Summary

`emptyTrash`(ignore_errors)	Remove all datasets from the trash.
`exists`(ref)	Check if the dataset exists in one of the datastores.
`export`(refs, *, directory, transfer)	Export datasets for transfer to another data repository.
`export_records`(refs)	Export datastore records and locations to an in-memory data structure.
`forget`(refs)	Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
`fromConfig`(config, bridgeManager, butlerRoot)	Create datastore from type specified in config file.
`get`(ref, parameters, Any]] = None, …)	Load an InMemoryDataset from the store.
`getLookupKeys`()	Return all the lookup keys relevant to this datastore.
`getManyURIs`(refs, predict, allow_missing)	Return URIs associated with many datasets.
`getURI`(ref, predict)	URI to the Dataset.
`getURIs`(ref, predict)	Return URIs associated with dataset.
`import_records`(data, …)	Import datastore location and record data from an in-memory data structure.
`ingest`(*datasets, transfer, None] = None, …)	Ingest one or more files into the datastore.
`knows`(ref)	Check if the dataset is known to any of the datastores.
`knows_these`(refs)	Check which of the given datasets are known to this datastore.
`mexists`(refs, artifact_existence, bool], …)	Check the existence of multiple datasets at once.
`needs_expanded_data_ids`(transfer, entity, …)	Test whether this datastore needs expanded data IDs to ingest.
`put`(inMemoryDataset, ref)	Write a InMemoryDataset with a given `DatasetRef` to each datastore.
`remove`(ref)	Indicate to the datastore that a dataset can be removed.
`retrieveArtifacts`(refs, destination, …)	Retrieve the file artifacts associated with the supplied refs.
`setConfigRoot`(root, config, full, overwrite)	Set any filesystem-dependent config options for child Datastores to be appropriate for a new empty repository with the given root.
`transaction`()	Context manager supporting `Datastore` transactions.
`transfer`(inputDatastore, ref)	Retrieve a dataset from an input `Datastore`, and store the result in this `Datastore`.
`transfer_from`(source_datastore, refs, …)	Transfer dataset artifacts from another datastore to this one.
`trash`(ref, …)	Indicate to the Datastore that a Dataset can be moved to the trash.
`validateConfiguration`(entities, DatasetType, …)	Validate some of the configuration for this datastore.
`validateKey`(lookupKey, entity, DatasetType, …)	Validate a specific look up key with supplied entity.

Attributes Documentation

containerKey = 'datastores'¶: Key to specify where child datastores are configured.

defaultConfigFile = 'datastores/chainedDatastore.yaml'¶: Path to configuration defaults. Accessed within the configs resource or relative to a search path. Can be None if no defaults specified.

isEphemeral = False¶

names¶

Names associated with this datastore returned as a list.

Can be different to name for a chaining datastore.

Methods Documentation

emptyTrash(ignore_errors: bool = True) → None¶

Remove all datasets from the trash.

Parameters:	ignore_errors : `bool`, optional Determine whether errors should be ignored.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

exists(ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → bool¶

Check if the dataset exists in one of the datastores.

Parameters:	ref : `DatasetRef` Reference to the required dataset.
Returns:	exists : `bool` `True` if the entity exists in one of the child datastores.

export(refs: Iterable[DatasetRef], *, directory: Optional[ResourcePathExpression] = None, transfer: Optional[str] = 'auto') → Iterable[FileDataset]¶

Export datasets for transfer to another data repository.

Parameters:	refs : iterable of `DatasetRef` Dataset references to be exported. directory : `str`, optional Path to a directory that should contain files corresponding to output datasets. Ignored if `transfer` is explicitly `None`. transfer : `str`, optional Mode that should be used to move datasets out of the repository. Valid options are the same as those of the `transfer` argument to `ingest`, and datastores may similarly signal that a transfer mode is not supported by raising `NotImplementedError`. If “auto” is given and no `directory` is specified, `None` will be implied.
Returns:	dataset : iterable of `DatasetTransfer` Structs containing information about the exported datasets, in the same order as `refs`.
Raises:	NotImplementedError Raised if the given transfer mode is not supported.

export_records(refs: Iterable[DatasetIdRef]) → Mapping[str, DatastoreRecordData]¶

Export datastore records and locations to an in-memory data structure.

Parameters:	refs : `Iterable` [ `DatasetIdRef` ] Datasets to save. This may include datasets not known to this datastore, which should be ignored.
Returns:	data : `Mapping` [ `str`, `DatastoreRecordData` ] Exported datastore records indexed by datastore name.

forget(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]) → None¶

Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.

Parameters:	refs : `Iterable` [ `DatasetRef` ] References to the datasets being forgotten.

Notes

Asking a datastore to forget a DatasetRef it does not hold should be a silent no-op, not an error.

static fromConfig(config: Config, bridgeManager: DatastoreRegistryBridgeManager, butlerRoot: Optional[ResourcePathExpression] = None) → 'Datastore'¶

Create datastore from type specified in config file.

Parameters:	config : `Config` Configuration instance. bridgeManager : `DatastoreRegistryBridgeManager` Object that manages the interface between `Registry` and datastores. butlerRoot : `str`, optional Butler root directory.

get(ref: DatasetRef, parameters: Optional[Mapping[str, Any]] = None, storageClass: Optional[Union[StorageClass, str]] = None) → Any¶

Load an InMemoryDataset from the store.

The dataset is returned from the first datastore that has the dataset.

Parameters:	ref : `DatasetRef` Reference to the required Dataset. parameters : `dict` `StorageClass`-specific parameters that specify, for example, a slice of the dataset to be loaded. storageClass : `StorageClass` or `str`, optional The storage class to be used to override the Python type returned by this method. By default the returned type matches the dataset type definition for this dataset. Specifying a read `StorageClass` can force a different type to be returned. This type must be compatible with the original type.
Returns:	inMemoryDataset : `object` Requested dataset or slice thereof as an InMemoryDataset.
Raises:	FileNotFoundError Requested dataset can not be retrieved. TypeError Return value from formatter has unexpected type. ValueError Formatter failed to process the dataset.

getLookupKeys() → Set[LookupKey]¶

Return all the lookup keys relevant to this datastore.

Returns:	keys : `set` of `LookupKey` The keys stored internally for looking up information based on `DatasetType` name or `StorageClass`.

getManyURIs(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef], predict: bool = False, allow_missing: bool = False) → Dict[lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datastore.DatasetRefURIs]¶

Return URIs associated with many datasets.

Parameters:	refs : iterable of `DatasetIdRef` References to the required datasets. predict : `bool`, optional If the datastore does not know about a dataset, should it return a predicted URI or not? allow_missing : `bool` If `False`, and `predict` is `False`, will raise if a `DatasetRef` does not exist.
Returns:	URIs : `dict` of [`DatasetRef`, `DatasetRefUris`] A dict of primary and component URIs, indexed by the passed-in refs.
Raises:	FileNotFoundError A URI has been requested for a dataset that does not exist and guessing is not allowed.

Notes

In file-based datastores, getManuURIs does not check that the file is really there, it’s assuming it is if datastore is aware of the file then it actually exists.

getURI(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, predict: bool = False) → lsst.resources._resourcePath.ResourcePath¶

URI to the Dataset.

The returned URI is from the first datastore in the list that has the dataset with preference given to the first dataset coming from a permanent datastore. If no datastores have the dataset and prediction is allowed, the predicted URI for the first datastore in the list will be returned.

Parameters:	ref : `DatasetRef` Reference to the required Dataset. predict : `bool` If `True`, allow URIs to be returned of datasets that have not been written.
Returns:	uri : `lsst.resources.ResourcePath` URI pointing to the dataset within the datastore. If the dataset does not exist in the datastore, and if `predict` is `True`, the URI will be a prediction and will include a URI fragment “#predicted”.
Raises:	FileNotFoundError A URI has been requested for a dataset that does not exist and guessing is not allowed. RuntimeError Raised if a request is made for a single URI but multiple URIs are associated with this dataset.

Notes

If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

getURIs(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, predict: bool = False) → lsst.daf.butler.core.datastore.DatasetRefURIs¶

Return URIs associated with dataset.

Parameters:	ref : `DatasetRef` Reference to the required dataset. predict : `bool`, optional If the datastore does not know about the dataset, should it return a predicted URI or not?
Returns:	uris : `DatasetRefURIs` The URI to the primary artifact associated with this dataset (if the dataset was disassembled within the datastore this may be `None`), and the URIs to any components associated with the dataset artifact. (can be empty if there are no components).

Notes

The returned URI is from the first datastore in the list that has the dataset with preference given to the first dataset coming from a permanent datastore. If no datastores have the dataset and prediction is allowed, the predicted URI for the first datastore in the list will be returned.

import_records(data: Mapping[str, lsst.daf.butler.core.datastoreRecordData.DatastoreRecordData]) → None¶

Import datastore location and record data from an in-memory data structure.

Parameters:	data : `Mapping` [ `str`, `DatastoreRecordData` ] Datastore records indexed by datastore name. May contain data for other `Datastore` instances (generally because they are chained to this one), which should be ignored.

Notes

Implementations should generally not check that any external resources (e.g. files) referred to by these records actually exist, for performance reasons; we expect higher-level code to guarantee that they do.

Implementations are responsible for calling DatastoreRegistryBridge.insert on all datasets in data.locations where the key is in names, as well as loading any opaque table data.

ingest(*datasets, transfer: Optional[str, None] = None, record_validation_info: bool = True) → None¶

Ingest one or more files into the datastore.

Parameters:

datasets : FileDataset: Each positional argument is a struct containing information about a file to be ingested, including its path (either absolute or relative to the datastore root, if applicable), a complete DatasetRef (with dataset_id not None), and optionally a formatter class or its fully-qualified string name. If a formatter is not provided, the one the datastore would use for put on that dataset is assumed.
transfer : str, optional: How (and whether) the dataset should be added to the datastore. If None (default), the file must already be in a location appropriate for the datastore (e.g. within its root directory), and will not be modified. Other choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” is a special option that will let the data store choose the most natural option for itself.
record_validation_info : bool, optional: If True, the default, the datastore can record validation information associated with the file. If False the datastore will not attempt to track any information such as checksums or file sizes. This can be useful if such information is tracked in an external system or if the file is to be compressed in place. It is up to the datastore whether this parameter is relevant.

Raises:

NotImplementedError: Raised if the datastore does not support the given transfer mode (including the case where ingest is not supported at all).
DatasetTypeNotSupportedError: Raised if one or more files to be ingested have a dataset type that is not supported by the datastore.
FileNotFoundError: Raised if one of the given files does not exist.
FileExistsError: Raised if transfer is not None but the (internal) location the file would be moved to is already occupied.

Notes

Subclasses should implement _prepIngest and _finishIngest instead of implementing ingest directly. Datastores that hold and delegate to child datastores may want to call those methods as well.

Subclasses are encouraged to document their supported transfer modes in their class documentation.

knows(ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → bool¶

Check if the dataset is known to any of the datastores.

Does not check for existence of any artifact.

Parameters:	ref : `DatasetRef` Reference to the required dataset.
Returns:	exists : `bool` `True` if the dataset is known to the datastore.

knows_these(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]) → dict¶

Check which of the given datasets are known to this datastore.

This is like mexist() but does not check that the file exists.

Parameters:	refs : iterable `DatasetRef` The datasets to check.
Returns:	exists : dict`[`DatasetRef, `bool`] Mapping of dataset to boolean indicating whether the dataset is known to the datastore.

mexists(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef], artifact_existence: Optional[Dict[lsst.resources._resourcePath.ResourcePath, bool], None] = None) → Dict[lsst.daf.butler.core.datasets.ref.DatasetRef, bool]¶

Check the existence of multiple datasets at once.

Parameters:	refs : iterable of `DatasetRef` The datasets to be checked. artifact_existence : `dict` [`lsst.resources.ResourcePath`, `bool`] Optional mapping of datastore artifact to existence. Updated by this method with details of all artifacts tested. Can be `None` if the caller is not interested.
Returns:	existence : `dict` of [`DatasetRef`, `bool`] Mapping from dataset to boolean indicating existence in any of the child datastores.

needs_expanded_data_ids(transfer: Optional[str], entity: Optional[Union[DatasetRef, DatasetType, StorageClass]] = None) → bool¶

Test whether this datastore needs expanded data IDs to ingest.

Parameters:	transfer : `str` or `None` Transfer mode for ingest. entity, optional Object representing what will be ingested. If not provided (or not specific enough), `True` may be returned even if expanded data IDs aren’t necessary.
Returns:	needed : `bool` If `True`, expanded data IDs may be needed. `False` only if expansion definitely isn’t necessary.

put(inMemoryDataset: Any, ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → None¶

Write a InMemoryDataset with a given DatasetRef to each datastore.

The put() to child datastores can fail with DatasetTypeNotSupportedError. The put() for this datastore will be deemed to have succeeded so long as at least one child datastore accepted the inMemoryDataset.

Parameters:	inMemoryDataset : `object` The dataset to store. ref : `DatasetRef` Reference to the associated Dataset.
Raises:	TypeError Supplied object and storage class are inconsistent. DatasetTypeNotSupportedError All datastores reported `DatasetTypeNotSupportedError`.

remove(ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → None¶

Indicate to the datastore that a dataset can be removed.

The dataset will be removed from each datastore. The dataset is not required to exist in every child datastore.

Parameters:	ref : `DatasetRef` Reference to the required dataset.
Raises:	FileNotFoundError Attempt to remove a dataset that does not exist. Raised if none of the child datastores removed the dataset.

retrieveArtifacts(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef], destination: lsst.resources._resourcePath.ResourcePath, transfer: str = 'auto', preserve_path: bool = True, overwrite: bool = False) → List[lsst.resources._resourcePath.ResourcePath]¶

Retrieve the file artifacts associated with the supplied refs.

Parameters:

refs : iterable of DatasetRef: The datasets for which file artifacts are to be retrieved. A single ref can result in multiple files. The refs must be resolved.
destination : lsst.resources.ResourcePath: Location to write the file artifacts.
transfer : str, optional: Method to use to transfer the artifacts. Must be one of the options supported by lsst.resources.ResourcePath.transfer_from(). “move” is not allowed.
preserve_path : bool, optional: If True the full path of the file artifact within the datastore is preserved. If False the final file component of the path is used.
overwrite : bool, optional: If True allow transfers to overwrite existing files at the destination.

Returns:

targets : list of lsst.resources.ResourcePath: URIs of file artifacts in destination location. Order is not preserved.

classmethod setConfigRoot(root: str, config: Config, full: Config, overwrite: bool = True) → None¶

Set any filesystem-dependent config options for child Datastores to be appropriate for a new empty repository with the given root.

Parameters:

root : str: Filesystem path to the root of the data repository.
config : Config: A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.
full : Config: A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.
overwrite : bool, optional: If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transaction() → Iterator[lsst.daf.butler.core.datastore.DatastoreTransaction]¶

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

transfer(inputDatastore: lsst.daf.butler.core.datastore.Datastore, ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → None¶

Retrieve a dataset from an input Datastore, and store the result in this Datastore.

Parameters:	inputDatastore : `Datastore` The external `Datastore` from which to retreive the Dataset. ref : `DatasetRef` Reference to the required dataset in the input data store.
Returns:	results : `list` List containing the return value from the `put()` to each child datastore.

transfer_from(source_datastore: Datastore, refs: Iterable[DatasetRef], local_refs: Optional[Iterable[DatasetRef]] = None, transfer: str = 'auto', artifact_existence: Optional[Dict[ResourcePath, bool]] = None) → None¶

Transfer dataset artifacts from another datastore to this one.

Parameters:

source_datastore : Datastore: The datastore from which to transfer artifacts. That datastore must be compatible with this datastore receiving the artifacts.
refs : iterable of DatasetRef: The datasets to transfer from the source datastore.
local_refs : iterable of DatasetRef, optional: The dataset refs associated with the registry associated with this datastore. Can be None if the source and target datastore are using UUIDs.
transfer : str, optional: How (and whether) the dataset should be added to the datastore. Choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” (the default) is a special option that will let the data store choose the most natural option for itself. If the source location and transfer location are identical the transfer mode will be ignored.
artifact_existence : dict [lsst.resources.ResourcePath, bool]: Optional mapping of datastore artifact to existence. Updated by this method with details of all artifacts tested. Can be None if the caller is not interested.

Raises:

TypeError: Raised if the two datastores are not compatible.

trash(ref: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]], ignore_errors: bool = True) → None¶

Indicate to the Datastore that a Dataset can be moved to the trash.

Parameters:	ref : `DatasetRef` or iterable thereof Reference(s) to the required Dataset. ignore_errors : `bool`, optional Determine whether errors should be ignored. When multiple refs are being trashed there will be no per-ref check.
Raises:	FileNotFoundError When Dataset does not exist and errors are not ignored. Only checked if a single ref is supplied (and not in a list).

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

validateConfiguration(entities: Iterable[Union[DatasetRef, DatasetType, StorageClass]], logFailures: bool = False) → None¶

Validate some of the configuration for this datastore.

Parameters:	entities : iterable of `DatasetRef`, `DatasetType`, or `StorageClass` Entities to test against this configuration. Can be differing types. logFailures : `bool`, optional If `True`, output a log message for every validation error detected.
Raises:	DatastoreValidationError Raised if there is a validation problem with a configuration. All the problems are reported in a single exception.

Notes

This method checks each datastore in turn.

validateKey(lookupKey: LookupKey, entity: Union[DatasetRef, DatasetType, StorageClass]) → None¶

Validate a specific look up key with supplied entity.

Parameters:	lookupKey : `LookupKey` Key to use to retrieve information from the datastore configuration. entity : `DatasetRef`, `DatasetType`, or `StorageClass` Entity to compare with configuration retrieved using the specified lookup key.
Raises:	DatastoreValidationError Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.

Navigation

ChainedDatastore¶