Datastore¶

class lsst.daf.butler.Datastore(config, registry, butlerRoot=None)¶

Bases: object

Datastore interface.

Parameters:	config : `DatastoreConfig` or `str` Load configuration either from an existing config instance or by referring to a configuration file. registry : `Registry` Registry to use for storing internal information about the datasets. butlerRoot : `str`, optional New datastore root to use to override the configuration value.

Attributes Summary

`containerKey`	Name of the key containing a list of subconfigurations that also need to be merged with defaults and will likely use different Python datastore classes (but all using DatastoreConfig).
`defaultConfigFile`	Path to configuration defaults.
`isEphemeral`	Indicate whether this Datastore is ephemeral or not.

Methods Summary

`exists`(datasetRef)	Check if the dataset exists in the datastore.
`fromConfig`(config, registry, butlerRoot)	Create datastore from type specified in config file.
`get`(datasetRef[, parameters])	Load an `InMemoryDataset` from the store.
`getLookupKeys`()	Return all the lookup keys relevant to this datastore.
`getUri`(datasetRef)	URI to the Dataset.
`ingest`(path, ref[, formatter, transfer])	Add an on-disk file with the given `DatasetRef` to the store, possibly transferring it.
`put`(inMemoryDataset, datasetRef)	Write a `InMemoryDataset` with a given `DatasetRef` to the store.
`remove`(datasetRef)	Indicate to the Datastore that a Dataset can be removed.
`setConfigRoot`(root, config, full, overwrite)	Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
`transaction`()	Context manager supporting `Datastore` transactions.
`transfer`(inputDatastore, datasetRef)	Retrieve a Dataset from an input `Datastore`, and store the result in this `Datastore`.
`validateConfiguration`(entities[, logFailures])	Validate some of the configuration for this datastore.
`validateKey`(lookupKey, entity[, logFailures])	Validate a specific look up key with supplied entity.

Attributes Documentation

containerKey = None¶: Name of the key containing a list of subconfigurations that also need to be merged with defaults and will likely use different Python datastore classes (but all using DatastoreConfig). Assumed to be a list of configurations that can be represented in a DatastoreConfig and containing a “cls” definition. None indicates that no containers are expected in this Datastore.

defaultConfigFile = None¶: Path to configuration defaults. Relative to $DAF_BUTLER_DIR/config or absolute path. Can be None if no defaults specified.

isEphemeral = False¶: Indicate whether this Datastore is ephemeral or not. An ephemeral datastore is one where the contents of the datastore will not exist across process restarts.

Methods Documentation

exists(datasetRef)¶

Check if the dataset exists in the datastore.

Parameters:	datasetRef : `DatasetRef` Reference to the required dataset.
Returns:	exists : `bool` `True` if the entity exists in the `Datastore`.

static fromConfig(config: lsst.daf.butler.core.config.Config, registry: lsst.daf.butler.core.registry.Registry, butlerRoot: Optional[str] = None) → lsst.daf.butler.core.datastore.Datastore¶

Create datastore from type specified in config file.

Parameters:	config : `Config` Configuration instance. registry : `Registry` Registry to be used by the Datastore for internal data. butlerRoot : `str`, optional Butler root directory.

get(datasetRef, parameters=None)¶

Load an InMemoryDataset from the store.

Parameters:	datasetRef : `DatasetRef` Reference to the required Dataset. parameters : `dict` `StorageClass`-specific parameters that specify a slice of the Dataset to be loaded.
Returns:	inMemoryDataset : `object` Requested Dataset or slice thereof as an InMemoryDataset.

getLookupKeys()¶

Return all the lookup keys relevant to this datastore.

Returns:	keys : `set` of `LookupKey` The keys stored internally for looking up information based on `DatasetType` name or `StorageClass`.

getUri(datasetRef)¶

URI to the Dataset.

Parameters:	datasetRef : `DatasetRef` Reference to the required Dataset.
Returns:	uri : `str` URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, the URI may be a guess. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

ingest(path, ref, formatter=None, transfer=None)¶

Add an on-disk file with the given DatasetRef to the store, possibly transferring it.

The caller is responsible for ensuring that the given (or predicted) Formatter is consistent with how the file was written; ingest will in general silently ignore incorrect formatters (as it cannot efficiently verify their correctness), deferring errors until get is first called on the ingested dataset.

Datastores are not required to implement this method, but must do so in order to support direct raw data ingest.

Parameters:

path : str: File path, relative to the repository root.
ref : DatasetRef: Reference to the associated Dataset.
formatter : Formatter (optional): Formatter that should be used to retreive the Dataset. If not provided, the formatter will be constructed according to Datastore configuration.
transfer : str (optional): If not None, must be one of ‘move’, ‘copy’, ‘hardlink’, or ‘symlink’ indicating how to transfer the file. Datastores need not support all options, but must raise NotImplementedError if the passed option is not supported. That includes None, which indicates that the file should be ingested at its current location with no transfer. If a Datastore does support ingest-without-transfer in general, but the given path is not appropriate, an exception other than NotImplementedError that better describes the problem should be raised.

Raises:

NotImplementedError: Raised if the given transfer mode is not supported.
DatasetTypeNotSupportedError: The associated DatasetType is not handled by this datastore.

put(inMemoryDataset, datasetRef)¶

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:	inMemoryDataset : `InMemoryDataset` The Dataset to store. datasetRef : `DatasetRef` Reference to the associated Dataset.

remove(datasetRef)¶

Indicate to the Datastore that a Dataset can be removed.

Parameters:	datasetRef : `DatasetRef` Reference to the required Dataset.
Raises:	FileNotFoundError When Dataset does not exist.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

classmethod setConfigRoot(root: str, config: lsst.daf.butler.core.config.Config, full: lsst.daf.butler.core.config.Config, overwrite: bool = True)¶

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Parameters:

root : str: Filesystem path to the root of the data repository.
config : Config: A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.
full : Config: A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.
overwrite : bool, optional: If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transaction()¶

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

transfer(inputDatastore, datasetRef)¶

Retrieve a Dataset from an input Datastore, and store the result in this Datastore.

Parameters:	inputDatastore : `Datastore` The external `Datastore` from which to retreive the Dataset. datasetRef : `DatasetRef` Reference to the required Dataset.

validateConfiguration(entities, logFailures=False)¶

Validate some of the configuration for this datastore.

Parameters:	entities : `DatasetRef`, `DatasetType`, or `StorageClass` Entities to test against this configuration. Can be differing types. logFailures : `bool`, optional If `True`, output a log message for every validation error detected.
Raises:	DatastoreValidationError Raised if there is a validation problem with a configuration.

Notes

Which parts of the configuration are validated is at the discretion of each Datastore implementation.

validateKey(lookupKey, entity, logFailures=False)¶

Validate a specific look up key with supplied entity.

Parameters:	lookupKey : `LookupKey` Key to use to retrieve information from the datastore configuration. entity : `DatasetRef`, `DatasetType`, or `StorageClass` Entity to compare with configuration retrieved using the specified lookup key.
Raises:	DatastoreValidationError Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.

Navigation

Datastore¶