PosixDatastore¶

class lsst.daf.butler.datastores.posixDatastore.PosixDatastore(config, registry)¶

Bases: lsst.daf.butler.Datastore

Basic POSIX filesystem backed Datastore.

Parameters:	config : `DatastoreConfig` or `str` Configuration.
Raises:	ValueError If root location does not exist and `create` is `False` in the configuration.
Attributes:	config : `DatastoreConfig` Configuration used to create Datastore. registry : `Registry` `Registry` to use when recording the writing of Datasets. root : `str` Root directory of this `Datastore`. locationFactory : `LocationFactory` Factory for creating locations relative to this root. formatterFactory : `FormatterFactory` Factory for creating instances of formatters. storageClassFactory : `StorageClassFactory` Factory for creating storage class instances from name. templates : `FileTemplates` File templates that can be used by this `Datastore`. name : `str` Label associated with this Datastore.

Attributes Summary

defaultConfigFile Path to configuration defaults.

Methods Summary

`addStoredFileInfo`(ref, info)	Record internal storage information associated with this `DatasetRef`
`computeChecksum`(filename[, algorithm, …])	Compute the checksum of the supplied file.
`exists`(ref)	Check if the dataset exists in the datastore.
`get`(ref[, parameters])	Load an InMemoryDataset from the store.
`getStoredFileInfo`(ref)	Retrieve information associated with file stored in this `Datastore`.
`getUri`(ref[, predict])	URI to the Dataset.
`ingest`(path, ref[, formatter, transfer])	Add an on-disk file with the given `DatasetRef` to the store, possibly transferring it.
`put`(inMemoryDataset, ref)	Write a InMemoryDataset with a given `DatasetRef` to the store.
`remove`(ref)	Indicate to the Datastore that a Dataset can be removed.
`removeStoredFileInfo`(ref)	Remove information about the file associated with this dataset.
`setConfigRoot`(root, config, full)	Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.
`transfer`(inputDatastore, ref)	Retrieve a Dataset from an input `Datastore`, and store the result in this `Datastore`.

Attributes Documentation

defaultConfigFile = 'datastores/posixDatastore.yaml'¶: Path to configuration defaults. Relative to $DAF_BUTLER_DIR/config or absolute path. Can be None if no defaults specified.

Methods Documentation

addStoredFileInfo(ref, info)¶

Record internal storage information associated with this DatasetRef

Parameters:	ref : `DatasetRef` The Dataset that has been stored. info : `StoredFileInfo` Metadata associated with the stored Dataset.

static computeChecksum(filename, algorithm='blake2b', block_size=8192)¶

Compute the checksum of the supplied file.

Parameters:	filename : `str` Name of file to calculate checksum from. algorithm : `str`, optional Name of algorithm to use. Must be one of the algorithms supported by :py:class`hashlib`. block_size : `int` Number of bytes to read from file at one time.
Returns:	hexdigest : `str` Hex digest of the file.

exists(ref)¶

Check if the dataset exists in the datastore.

Parameters:	ref : `DatasetRef` Reference to the required dataset.
Returns:	exists : `bool` `True` if the entity exists in the `Datastore`.

get(ref, parameters=None)¶

Load an InMemoryDataset from the store.

Parameters:	ref : `DatasetRef` Reference to the required Dataset. parameters : `dict` `StorageClass`-specific parameters that specify, for example, a slice of the Dataset to be loaded.
Returns:	inMemoryDataset : `object` Requested Dataset or slice thereof as an InMemoryDataset.
Raises:	FileNotFoundError Requested dataset can not be retrieved. TypeError Return value from formatter has unexpected type. ValueError Formatter failed to process the dataset.

getStoredFileInfo(ref)¶

Retrieve information associated with file stored in this Datastore.

Parameters:	ref : `DatasetRef` The Dataset that is to be queried.
Returns:	info : `StoredFileInfo` Stored information about this file and its formatter.
Raises:	KeyError Dataset with that id can not be found.

getUri(ref, predict=False)¶

URI to the Dataset.

Parameters:	ref : `DatasetRef` Reference to the required Dataset. predict : `bool` If `True`, allow URIs to be returned of datasets that have not been written.
Returns:	uri : `str` URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, and if `predict` is `True`, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.
Raises:	FileNotFoundError A URI has been requested for a dataset that does not exist and guessing is not allowed.

ingest(path, ref, formatter=None, transfer=None)¶

Add an on-disk file with the given DatasetRef to the store, possibly transferring it.

The caller is responsible for ensuring that the given (or predicted) Formatter is consistent with how the file was written; ingest will in general silently ignore incorrect formatters (as it cannot efficiently verify their correctness), deferring errors until get is first called on the ingested dataset.

Parameters:

path : str: File path. Treated as relative to the repository root if not absolute.
ref : DatasetRef: Reference to the associated Dataset.
formatter : Formatter (optional): Formatter that should be used to retreive the Dataset. If not provided, the formatter will be constructed according to Datastore configuration.
transfer : str (optional): If not None, must be one of ‘move’, ‘copy’, ‘hardlink’, or ‘symlink’ indicating how to transfer the file. The new filename and location will be determined via template substitution, as with put. If the file is outside the datastore root, it must be transferred somehow.

Raises:

RuntimeError: Raised if transfer is None and path is outside the repository root.
FileNotFoundError: Raised if the file at path does not exist.
FileExistsError: Raised if transfer is not None but a file already exists at the location computed from the template.

put(inMemoryDataset, ref)¶

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:	inMemoryDataset : `object` The Dataset to store. ref : `DatasetRef` Reference to the associated Dataset.
Raises:	TypeError Supplied object and storage class are inconsistent. DatasetTypeNotSupportedError The associated `DatasetType` is not handled by this datastore.

remove(ref)¶

Indicate to the Datastore that a Dataset can be removed.

Warning

This method does not support transactions; removals are immediate, cannot be undone, and are not guaranteed to be atomic if deleting either the file or the internal database records fails.

Parameters:	ref : `DatasetRef` Reference to the required Dataset.
Raises:	FileNotFoundError Attempt to remove a dataset that does not exist.

removeStoredFileInfo(ref)¶

Remove information about the file associated with this dataset.

Parameters:	ref : `DatasetRef` The Dataset that has been removed.

classmethod setConfigRoot(root, config, full)¶

Set any filesystem-dependent config options for this Datastore to be appropriate for a new empty repository with the given root.

Parameters:

root : str: Filesystem path to the root of the data repository.
config : Config: A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.
full : Config: A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to Config.

transfer(inputDatastore, ref)¶

Retrieve a Dataset from an input Datastore, and store the result in this Datastore.

Parameters:	inputDatastore : `Datastore` The external `Datastore` from which to retreive the Dataset. ref : `DatasetRef` Reference to the required Dataset in the input data store.

Navigation

PosixDatastore¶