Datastore¶

class lsst.daf.butler.Datastore(config: Union[Config, str], bridgeManager: DatastoreRegistryBridgeManager, butlerRoot: str = None)¶

Bases: object

Datastore interface.

Parameters

configDatastoreConfig or str: Load configuration either from an existing config instance or by referring to a configuration file.
bridgeManagerDatastoreRegistryBridgeManager: Object that manages the interface between Registry and datastores.
butlerRootstr, optional: New datastore root to use to override the configuration value.

Attributes Summary

`containerKey`	Name of the key containing a list of subconfigurations that also need to be merged with defaults and will likely use different Python datastore classes (but all using DatastoreConfig).
`defaultConfigFile`	Path to configuration defaults.
`isEphemeral`	Indicate whether this Datastore is ephemeral or not.
`names`	Names associated with this datastore returned as a list.

Methods Summary

`emptyTrash`([ignore_errors])	Remove all datasets from the trash.
`exists`(datasetRef)	Check if the dataset exists in the datastore.
`export`(refs, *[, directory, transfer])	Export datasets for transfer to another data repository.
`forget`(refs)	Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
`fromConfig`(config, bridgeManager[, butlerRoot])	Create datastore from type specified in config file.
`get`(datasetRef[, parameters])	Load an `InMemoryDataset` from the store.
`getLookupKeys`()	Return all the lookup keys relevant to this datastore.
`getURI`(datasetRef[, predict])	URI to the Dataset.
`getURIs`(datasetRef[, predict])	Return URIs associated with dataset.
`ingest`(*datasets[, transfer])	Ingest one or more files into the datastore.
`needs_expanded_data_ids`(transfer[, entity])	Test whether this datastore needs expanded data IDs to ingest.
`put`(inMemoryDataset, datasetRef)	Write a `InMemoryDataset` with a given `DatasetRef` to the store.
`remove`(datasetRef)	Indicate to the Datastore that a Dataset can be removed.
`setConfigRoot`(root, config, full[, overwrite])	Set filesystem-dependent config options for this datastore.
`transaction`()	Context manager supporting `Datastore` transactions.
`transfer`(inputDatastore, datasetRef)	Transfer a dataset from another datastore to this datastore.
`trash`(datasetRef[, ignore_errors])	Indicate to the Datastore that a Dataset can be moved to the trash.
`validateConfiguration`(entities[, logFailures])	Validate some of the configuration for this datastore.
`validateKey`(lookupKey, entity)	Validate a specific look up key with supplied entity.

Attributes Documentation

containerKey: ClassVar[Optional[str]] = None¶: Name of the key containing a list of subconfigurations that also need to be merged with defaults and will likely use different Python datastore classes (but all using DatastoreConfig). Assumed to be a list of configurations that can be represented in a DatastoreConfig and containing a “cls” definition. None indicates that no containers are expected in this Datastore.

defaultConfigFile: ClassVar[Optional[str]] = None¶: Path to configuration defaults. Accessed within the config resource or relative to a search path. Can be None if no defaults specified.

isEphemeral: bool = False¶: Indicate whether this Datastore is ephemeral or not. An ephemeral datastore is one where the contents of the datastore will not exist across process restarts. This value can change per-instance.

names¶

Names associated with this datastore returned as a list.

Can be different to name for a chaining datastore.

Methods Documentation

abstract emptyTrash(ignore_errors: bool = True) → None ¶

Remove all datasets from the trash.

Parameters

ignore_errorsbool, optional: Determine whether errors should be ignored.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

abstract exists(datasetRef: DatasetRef) → bool ¶

Check if the dataset exists in the datastore.

Parameters

datasetRefDatasetRef: Reference to the required dataset.

Returns

existsbool: True if the entity exists in the Datastore.

export(refs: Iterable[DatasetRef], *, directory: Optional[str] = None, transfer: Optional[str] = None) → Iterable[FileDataset]¶

Export datasets for transfer to another data repository.

Parameters

refsiterable of DatasetRef: Dataset references to be exported.
directorystr, optional: Path to a directory that should contain files corresponding to output datasets. Ignored if transfer is None.
transferstr, optional: Mode that should be used to move datasets out of the repository. Valid options are the same as those of the transfer argument to ingest, and datastores may similarly signal that a transfer mode is not supported by raising NotImplementedError.

Returns

datasetiterable of DatasetTransfer: Structs containing information about the exported datasets, in the same order as refs.

Raises

NotImplementedError: Raised if the given transfer mode is not supported.

abstract forget(refs: Iterable[DatasetRef]) → None ¶

Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.

Parameters

refsIterable [ DatasetRef ]: References to the datasets being forgotten.

Notes

Asking a datastore to forget a DatasetRef it does not hold should be a silent no-op, not an error.

static fromConfig(config: Config, bridgeManager: DatastoreRegistryBridgeManager, butlerRoot: Optional[Union[str, ButlerURI]] = None) → Datastore ¶

Create datastore from type specified in config file.

Parameters

configConfig: Configuration instance.
bridgeManagerDatastoreRegistryBridgeManager: Object that manages the interface between Registry and datastores.
butlerRootstr, optional: Butler root directory.

abstract get(datasetRef: DatasetRef, parameters: Mapping[str, Any] = None) → Any¶

Load an InMemoryDataset from the store.

Parameters

datasetRefDatasetRef: Reference to the required Dataset.
parametersdict: StorageClass-specific parameters that specify a slice of the Dataset to be loaded.

Returns

inMemoryDatasetobject: Requested Dataset or slice thereof as an InMemoryDataset.

abstract getLookupKeys() → Set[LookupKey]¶

Return all the lookup keys relevant to this datastore.

Returns

keysset of LookupKey: The keys stored internally for looking up information based on DatasetType name or StorageClass.

abstract getURI(datasetRef: DatasetRef, predict: bool = False) → ButlerURI ¶

URI to the Dataset.

Parameters

datasetRefDatasetRef: Reference to the required Dataset.
predictbool: If True attempt to predict the URI for a dataset if it does not exist in datastore.

Returns

uristr: URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, the URI may be a guess. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

Raises

FileNotFoundError: A URI has been requested for a dataset that does not exist and guessing is not allowed.

abstract getURIs(datasetRef: DatasetRef, predict: bool = False) → Tuple[Optional[ButlerURI], Dict[str, ButlerURI]]¶

Return URIs associated with dataset.

Parameters

refDatasetRef: Reference to the required dataset.
predictbool, optional: If the datastore does not know about the dataset, should it return a predicted URI or not?

Returns

primaryButlerURI: The URI to the primary artifact associated with this dataset. If the dataset was disassembled within the datastore this may be None.
componentsdict: URIs to any components associated with the dataset artifact. Can be empty if there are no components.

ingest(*datasets: lsst.daf.butler.FileDataset, transfer: Optional[str] = None) → None ¶

Ingest one or more files into the datastore.

Parameters

datasetsFileDataset: Each positional argument is a struct containing information about a file to be ingested, including its path (either absolute or relative to the datastore root, if applicable), a complete DatasetRef (with dataset_id not None), and optionally a formatter class or its fully-qualified string name. If a formatter is not provided, the one the datastore would use for put on that dataset is assumed.
transferstr, optional: How (and whether) the dataset should be added to the datastore. If None (default), the file must already be in a location appropriate for the datastore (e.g. within its root directory), and will not be modified. Other choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” is a special option that will let the data store choose the most natural option for itself.

Raises

NotImplementedError: Raised if the datastore does not support the given transfer mode (including the case where ingest is not supported at all).
DatasetTypeNotSupportedError: Raised if one or more files to be ingested have a dataset type that is not supported by the datastore.
FileNotFoundError: Raised if one of the given files does not exist.
FileExistsError: Raised if transfer is not None but the (internal) location the file would be moved to is already occupied.

Notes

Subclasses should implement _prepIngest and _finishIngest instead of implementing ingest directly. Datastores that hold and delegate to child datastores may want to call those methods as well.

Subclasses are encouraged to document their supported transfer modes in their class documentation.

needs_expanded_data_ids(transfer: Optional[str], entity: Optional[Union[DatasetRef, DatasetType, StorageClass]] = None) → bool ¶

Test whether this datastore needs expanded data IDs to ingest.

Parameters

transferstr or None: Transfer mode for ingest.
entity, optional: Object representing what will be ingested. If not provided (or not specific enough), True may be returned even if expanded data IDs aren’t necessary.

Returns

neededbool: If True, expanded data IDs may be needed. False only if expansion definitely isn’t necessary.

abstract put(inMemoryDataset: Any, datasetRef: DatasetRef) → None ¶

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters

inMemoryDatasetobject: The Dataset to store.
datasetRefDatasetRef: Reference to the associated Dataset.

abstract remove(datasetRef: DatasetRef) → None ¶

Indicate to the Datastore that a Dataset can be removed.

Parameters

datasetRefDatasetRef: Reference to the required Dataset.

Raises

FileNotFoundError: When Dataset does not exist.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

abstract classmethod setConfigRoot(root: str, config: lsst.daf.butler.Config, full: lsst.daf.butler.Config, overwrite: bool = True) → None ¶

Set filesystem-dependent config options for this datastore.

The options will be appropriate for a new empty repository with the given root.

Parameters

rootstr: Filesystem path to the root of the data repository.
configConfig: A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.
fullConfig: A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.
overwritebool, optional: If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transaction() → Iterator[lsst.daf.butler.core.datastore.DatastoreTransaction]¶

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

abstract transfer(inputDatastore: Datastore, datasetRef: DatasetRef) → None ¶

Transfer a dataset from another datastore to this datastore.

Parameters

inputDatastoreDatastore: The external Datastore from which to retrieve the Dataset.
datasetRefDatasetRef: Reference to the required Dataset.

abstract trash(datasetRef: DatasetRef, ignore_errors: bool = True) → None ¶

Indicate to the Datastore that a Dataset can be moved to the trash.

Parameters

datasetRefDatasetRef: Reference to the required Dataset.
ignore_errorsbool, optional: Determine whether errors should be ignored.

Raises

FileNotFoundError: When Dataset does not exist.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

abstract validateConfiguration(entities: Iterable[Union[DatasetRef, DatasetType, StorageClass]], logFailures: bool = False) → None ¶

Validate some of the configuration for this datastore.

Parameters

entitiesiterable of DatasetRef, DatasetType, or StorageClass: Entities to test against this configuration. Can be differing types.
logFailuresbool, optional: If True, output a log message for every validation error detected.

Raises

DatastoreValidationError: Raised if there is a validation problem with a configuration.

Notes

Which parts of the configuration are validated is at the discretion of each Datastore implementation.

abstract validateKey(lookupKey: LookupKey, entity: Union[DatasetRef, DatasetType, StorageClass]) → None ¶

Validate a specific look up key with supplied entity.

Parameters

lookupKeyLookupKey: Key to use to retrieve information from the datastore configuration.
entityDatasetRef, DatasetType, or StorageClass: Entity to compare with configuration retrieved using the specified lookup key.

Raises

DatastoreValidationError: Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.

Navigation

Datastore¶