Datastore

class lsst.daf.butler.Datastore(config, registry, butlerRoot=None)

Bases: object

Datastore interface.

Parameters:
config : DatastoreConfig or str

Load configuration either from an existing config instance or by referring to a configuration file.

registry : Registry

Registry to use for storing internal information about the datasets.

butlerRoot : str, optional

New datastore root to use to override the configuration value.

Attributes Summary

containerKey Name of the key containing a list of subconfigurations that also need to be merged with defaults and will likely use different Python datastore classes (but all using DatastoreConfig).
defaultConfigFile Path to configuration defaults.
isEphemeral Indicate whether this Datastore is ephemeral or not.
names List of names associated with this Datastore.

Methods Summary

exists(datasetRef) Check if the dataset exists in the datastore.
export(refs, *, directory, transfer) Export datasets for transfer to another data repository.
fromConfig(config, registry, butlerRoot) Create datastore from type specified in config file.
get(datasetRef[, parameters]) Load an InMemoryDataset from the store.
getLookupKeys() Return all the lookup keys relevant to this datastore.
getUri(datasetRef) URI to the Dataset.
ingest(*datasets, transfer) Ingest one or more files into the datastore.
put(inMemoryDataset, datasetRef) Write a InMemoryDataset with a given DatasetRef to the store.
remove(datasetRef) Indicate to the Datastore that a Dataset can be removed.
setConfigRoot(root, config, full, overwrite) Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
transaction() Context manager supporting Datastore transactions.
transfer(inputDatastore, datasetRef) Retrieve a Dataset from an input Datastore, and store the result in this Datastore.
validateConfiguration(entities[, logFailures]) Validate some of the configuration for this datastore.
validateKey(lookupKey, entity[, logFailures]) Validate a specific look up key with supplied entity.

Attributes Documentation

containerKey = None

Name of the key containing a list of subconfigurations that also need to be merged with defaults and will likely use different Python datastore classes (but all using DatastoreConfig). Assumed to be a list of configurations that can be represented in a DatastoreConfig and containing a “cls” definition. None indicates that no containers are expected in this Datastore.

defaultConfigFile = None

Path to configuration defaults. Relative to $DAF_BUTLER_DIR/config or absolute path. Can be None if no defaults specified.

isEphemeral = False

Indicate whether this Datastore is ephemeral or not. An ephemeral datastore is one where the contents of the datastore will not exist across process restarts.

names

List of names associated with this Datastore. Can be different to name for a chaining datastore.

Methods Documentation

exists(datasetRef)

Check if the dataset exists in the datastore.

Parameters:
datasetRef : DatasetRef

Reference to the required dataset.

Returns:
exists : bool

True if the entity exists in the Datastore.

export(refs: Iterable[DatasetRef], *, directory: Optional[str] = None, transfer: Optional[str] = None) → Iterable[FileDataset]

Export datasets for transfer to another data repository.

Parameters:
refs : iterable of DatasetRef

Dataset references to be exported.

directory : str, optional

Path to a directory that should contain files corresponding to output datasets. Ignored if transfer is None.

transfer : str, optional

Mode that should be used to move datasets out of the repository. Valid options are the same as those of the transfer argument to ingest, and datastores may similarly signal that a transfer mode is not supported by raising NotImplementedError.

Returns:
dataset : iterable of DatasetTransfer

Structs containing information about the exported datasets, in the same order as refs.

Raises:
NotImplementedError

Raised if the given transfer mode is not supported.

static fromConfig(config: Config, registry: Registry, butlerRoot: Optional[str] = None) → 'Datastore'

Create datastore from type specified in config file.

Parameters:
config : Config

Configuration instance.

registry : Registry

Registry to be used by the Datastore for internal data.

butlerRoot : str, optional

Butler root directory.

get(datasetRef, parameters=None)

Load an InMemoryDataset from the store.

Parameters:
datasetRef : DatasetRef

Reference to the required Dataset.

parameters : dict

StorageClass-specific parameters that specify a slice of the Dataset to be loaded.

Returns:
inMemoryDataset : object

Requested Dataset or slice thereof as an InMemoryDataset.

getLookupKeys()

Return all the lookup keys relevant to this datastore.

Returns:
keys : set of LookupKey

The keys stored internally for looking up information based on DatasetType name or StorageClass.

getUri(datasetRef)

URI to the Dataset.

Parameters:
datasetRef : DatasetRef

Reference to the required Dataset.

Returns:
uri : str

URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, the URI may be a guess. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

ingest(*datasets, transfer: Optional[str] = None)

Ingest one or more files into the datastore.

Parameters:
datasets : FileDataset

Each positional argument is a struct containing information about a file to be ingested, including its path (either absolute or relative to the datastore root, if applicable), a complete DatasetRef (with dataset_id not None), and optionally a formatter class or its fully-qualified string name. If a formatter is not provided, the one the datastore would use for put on that dataset is assumed.

transfer : str, optional

How (and whether) the dataset should be added to the datastore. If None (default), the file must already be in a location appropriate for the datastore (e.g. within its root directory), and will not be modified. Other choices include “move”, “copy”, “symlink”, and “hardlink”. Most datastores do not support all transfer modes.

Raises:
NotImplementedError

Raised if the datastore does not support the given transfer mode (including the case where ingest is not supported at all).

DatasetTypeNotSupportedError

Raised if one or more files to be ingested have a dataset type that is not supported by the datastore.

FileNotFoundError

Raised if one of the given files does not exist.

FileExistsError

Raised if transfer is not None but the (internal) location the file would be moved to is already occupied.

Notes

Subclasses should implement _prepIngest and _finishIngest instead of implementing ingest directly. Datastores that hold and delegate to child datastores may want to call those methods as well.

Subclasses are encouraged to document their supported transfer modes in their class documentation.

put(inMemoryDataset, datasetRef)

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:
inMemoryDataset : InMemoryDataset

The Dataset to store.

datasetRef : DatasetRef

Reference to the associated Dataset.

remove(datasetRef)

Indicate to the Datastore that a Dataset can be removed.

Parameters:
datasetRef : DatasetRef

Reference to the required Dataset.

Raises:
FileNotFoundError

When Dataset does not exist.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

classmethod setConfigRoot(root: str, config: lsst.daf.butler.core.config.Config, full: lsst.daf.butler.core.config.Config, overwrite: bool = True)

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Parameters:
root : str

Filesystem path to the root of the data repository.

config : Config

A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.

full : Config

A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.

overwrite : bool, optional

If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transaction()

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

transfer(inputDatastore, datasetRef)

Retrieve a Dataset from an input Datastore, and store the result in this Datastore.

Parameters:
inputDatastore : Datastore

The external Datastore from which to retreive the Dataset.

datasetRef : DatasetRef

Reference to the required Dataset.

validateConfiguration(entities, logFailures=False)

Validate some of the configuration for this datastore.

Parameters:
entities : DatasetRef, DatasetType, or StorageClass

Entities to test against this configuration. Can be differing types.

logFailures : bool, optional

If True, output a log message for every validation error detected.

Raises:
DatastoreValidationError

Raised if there is a validation problem with a configuration.

Notes

Which parts of the configuration are validated is at the discretion of each Datastore implementation.

validateKey(lookupKey, entity, logFailures=False)

Validate a specific look up key with supplied entity.

Parameters:
lookupKey : LookupKey

Key to use to retrieve information from the datastore configuration.

entity : DatasetRef, DatasetType, or StorageClass

Entity to compare with configuration retrieved using the specified lookup key.

Raises:
DatastoreValidationError

Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.