PosixDatastore

class lsst.daf.butler.datastores.posixDatastore.PosixDatastore(config, registry, butlerRoot=None)

Bases: lsst.daf.butler.datastores.fileLikeDatastore.FileLikeDatastore

Basic POSIX filesystem backed Datastore.

Parameters:

config : DatastoreConfig or str

Configuration. A string should refer to the name of the config file.

registry : Registry

Registry to use for storing internal information about the datasets.

butlerRoot : str, optional

New datastore root to use to override the configuration value.

Raises:

ValueError

If root location does not exist and create is False in the configuration.

Attributes Summary

containerKey
defaultConfigFile	Path to configuration defaults.
isEphemeral

Methods Summary

addStoredItemInfo(ref, info)	Record internal storage information associated with this DatasetRef
computeChecksum(filename[, algorithm, …])	Compute the checksum of the supplied file.
exists(ref)	Check if the dataset exists in the datastore.
export(refs, *, directory, transfer)	Export datasets for transfer to another data repository.
fromConfig(config, registry, butlerRoot)	Create datastore from type specified in config file.
get(ref[, parameters])	Load an InMemoryDataset from the store.
getLookupKeys()	Return all the lookup keys relevant to this datastore.
getStoredItemInfo(ref)	Retrieve information associated with file stored in this Datastore.
getUri(ref[, predict])	URI to the Dataset.
ingest(path, ref[, formatter, transfer])	Add an on-disk file with the given DatasetRef to the store, possibly transferring it.
put(inMemoryDataset, ref)	Write a InMemoryDataset with a given DatasetRef to the store.
remove(ref)	Indicate to the Datastore that a Dataset can be removed.
removeStoredItemInfo(ref)	Remove information about the file associated with this dataset.
setConfigRoot(root, config, full[, overwrite])	Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
transaction()	Context manager supporting Datastore transactions.
transfer(inputDatastore, ref)	Retrieve a Dataset from an input Datastore, and store the result in this Datastore.
validateConfiguration(entities[, logFailures])	Validate some of the configuration for this datastore.
validateKey(lookupKey, entity)	Validate a specific look up key with supplied entity.

Attributes Documentation

containerKey = None

defaultConfigFile = 'datastores/posixDatastore.yaml'

Path to configuration defaults. Relative to $DAF_BUTLER_DIR/config or absolute path. Can be None if no defaults specified.

isEphemeral = False

Methods Documentation

addStoredItemInfo(ref, info)

Record internal storage information associated with this DatasetRef

Parameters:

ref : DatasetRef

The Dataset that has been stored.

info : StoredDatastoreItemInfo

Metadata associated with the stored Dataset.

static computeChecksum(filename, algorithm='blake2b', block_size=8192)

Compute the checksum of the supplied file.

Parameters:

filename : str

Name of file to calculate checksum from.

algorithm : str, optional

Name of algorithm to use. Must be one of the algorithms supported by :py:class`hashlib`.

block_size : int

Number of bytes to read from file at one time.

Returns:

hexdigest : str

Hex digest of the file.

exists(ref)

Check if the dataset exists in the datastore.

Parameters:

ref : DatasetRef

Reference to the required dataset.

Returns:

exists : bool

True if the entity exists in the Datastore.

export(refs: Iterable[DatasetRef], *, directory: Optional[str] = None, transfer: Optional[str] = None) → Iterable[DatasetExport]

Export datasets for transfer to another data repository.

Parameters:

refs : iterable of DatasetRef

Dataset references to be exported.

directory : str, optional

Path to a directory that should contain files corresponding to output datasets. Ignored if transfer is None.

transfer : str, optional

Mode that should be used to move datasets out of the repository. Valid options are the same as those of the transfer argument to ingest, and datastores may similarly signal that a transfer mode is not supported by raising NotImplementedError.

Returns:

dataset : iterable of DatasetTransfer

Structs containing information about the exported datasets, in the same order as refs.

Raises:

NotImplementedError

Raised if the given transfer mode is not supported.

static fromConfig(config: lsst.daf.butler.core.config.Config, registry: lsst.daf.butler.core.registry.Registry, butlerRoot: Optional[str] = None) → Datastore

Create datastore from type specified in config file.

Parameters:

config : Config

Configuration instance.

registry : Registry

Registry to be used by the Datastore for internal data.

butlerRoot : str, optional

Butler root directory.

get(ref, parameters=None)

Load an InMemoryDataset from the store.

Parameters:

ref : DatasetRef

Reference to the required Dataset.

parameters : dict

StorageClass-specific parameters that specify, for example, a slice of the Dataset to be loaded.

Returns:

inMemoryDataset : object

Requested Dataset or slice thereof as an InMemoryDataset.

Raises:

FileNotFoundError

Requested dataset can not be retrieved.

TypeError

Return value from formatter has unexpected type.

ValueError

Formatter failed to process the dataset.

getLookupKeys()

Return all the lookup keys relevant to this datastore.

Returns:

keys : set of LookupKey

The keys stored internally for looking up information based on DatasetType name or StorageClass.

getStoredItemInfo(ref)

Retrieve information associated with file stored in this Datastore.

Parameters:

ref : DatasetRef

The Dataset that is to be queried.

Returns:

info : StoredFilenfo

Stored information about this file and its formatter.

Raises:

KeyError

Dataset with that id can not be found.

getUri(ref, predict=False)

URI to the Dataset.

Parameters:

ref : DatasetRef

Reference to the required Dataset.

predict : bool

If True, allow URIs to be returned of datasets that have not been written.

Returns:

uri : str

URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, and if predict is True, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

Raises:

FileNotFoundError

A URI has been requested for a dataset that does not exist and guessing is not allowed.

ingest(path, ref, formatter=None, transfer=None)

Add an on-disk file with the given DatasetRef to the store, possibly transferring it.

The caller is responsible for ensuring that the given (or predicted) Formatter is consistent with how the file was written; ingest will in general silently ignore incorrect formatters (as it cannot efficiently verify their correctness), deferring errors until get is first called on the ingested dataset.

Parameters:

path : str

File path. Treated as relative to the repository root if not absolute.

ref : DatasetRef

Reference to the associated Dataset.

formatter : Formatter, optional

Formatter that should be used to retreive the Dataset. If not provided, the formatter will be constructed according to Datastore configuration. Can be a the Formatter class or an instance.

transfer : str (optional)

If not None, must be one of ‘move’, ‘copy’, ‘hardlink’, or ‘symlink’ indicating how to transfer the file. The new filename and location will be determined via template substitution, as with put. If the file is outside the datastore root, it must be transferred somehow.

Raises:

RuntimeError

Raised if transfer is None and path is outside the repository root.

FileNotFoundError

Raised if the file at path does not exist.

FileExistsError

Raised if transfer is not None but a file already exists at the location computed from the template.

DatasetTypeNotSupportedError

The associated DatasetType is not handled by this datastore.

put(inMemoryDataset, ref)

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:

inMemoryDataset : object

The Dataset to store.

ref : DatasetRef

Reference to the associated Dataset.

Raises:

TypeError

Supplied object and storage class are inconsistent.

DatasetTypeNotSupportedError

The associated DatasetType is not handled by this datastore.

Notes

If the datastore is configured to reject certain dataset types it is possible that the put will fail and raise a DatasetTypeNotSupportedError. The main use case for this is to allow ChainedDatastore to put to multiple datastores without requiring that every datastore accepts the dataset.

remove(ref)

Indicate to the Datastore that a Dataset can be removed.

Warning

This method does not support transactions; removals are immediate, cannot be undone, and are not guaranteed to be atomic if deleting either the file or the internal database records fails.

Parameters:

ref : DatasetRef

Reference to the required Dataset.

Raises:

FileNotFoundError

Attempt to remove a dataset that does not exist.

removeStoredItemInfo(ref)

Remove information about the file associated with this dataset.

Parameters:

ref : DatasetRef

The Dataset that has been removed.

classmethod setConfigRoot(root, config, full, overwrite=True)

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Parameters:

root : str

URI to the root of the data repository.

config : Config

A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.

full : Config

A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.

overwrite : bool, optional

If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transaction()

Context manager supporting Datastore transactions.

Transactions can be nested, and are to be used in combination with Registry.transaction.

transfer(inputDatastore, ref)

Retrieve a Dataset from an input Datastore, and store the result in this Datastore.

Parameters:

inputDatastore : Datastore

The external Datastore from which to retreive the Dataset.

ref : DatasetRef

Reference to the required Dataset in the input data store.

validateConfiguration(entities, logFailures=False)

Validate some of the configuration for this datastore.

Parameters:

entities : iterable of DatasetRef, DatasetType, or StorageClass

Entities to test against this configuration. Can be differing types.

logFailures : bool, optional

If True, output a log message for every validation error detected.

Raises:

DatastoreValidationError

Raised if there is a validation problem with a configuration. All the problems are reported in a single exception.

Notes

This method checks that all the supplied entities have valid file templates and also have formatters defined.

validateKey(lookupKey, entity)

Validate a specific look up key with supplied entity.

Parameters:

lookupKey : LookupKey

Key to use to retrieve information from the datastore configuration.

entity : DatasetRef, DatasetType, or StorageClass

Entity to compare with configuration retrieved using the specified lookup key.

Raises:

DatastoreValidationError

Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.