Butler¶

class lsst.daf.butler.Butler(config=None, collection=None, run=None)¶

Bases: object

Main entry point for the data access system.

Parameters:	config : `Config` Configuration. collection : `str`, optional Collection to use for all input lookups, overriding config[“collection”] if provided. run : `str`, `Run`, optional Collection associated with the `Run` to use for outputs, overriding config[“run”]. If a `Run` associated with the given Collection does not exist, it will be created. If “collection” is None, this collection will be used for input lookups as well; if not, it must have the same value as “run”.
Raises:	ValueError Raised if neither “collection” nor “run” are provided by argument or config, or if both are provided and are inconsistent.
Attributes:	config : `str`, `ButlerConfig` or `Config`, optional (filename to) configuration. If this is not a `ButlerConfig`, defaults will be read. If a `str`, may be the path to a directory containing a “butler.yaml” file. datastore : `Datastore` Datastore to use for storage. registry : `Registry` Registry to use for lookups.

Attributes Summary

GENERATION This is a Generation 3 Butler.

Methods Summary

`close`()	This method should be called to properly close any resources the butler may have open.
`datasetExists`(datasetRefOrType[, dataId])	Return True if the Dataset is actually present in the Datastore.
`get`(datasetRefOrType[, dataId, parameters])	Retrieve a stored dataset.
`getDirect`(ref[, parameters])	Retrieve a stored dataset.
`getUri`(datasetRefOrType[, dataId, predict])	Return the URI to the Dataset.
`makeRepo`(root[, config, standalone, …])	Create an empty data repository by adding a butler.yaml config to a repository root directory.
`put`(obj, datasetRefOrType[, dataId, producer])	Store and register a dataset.
`remove`(datasetRefOrType[, dataId, delete, …])	Remove a dataset from the collection and possibly the repository.
`transaction`()	Context manager supporting `Butler` transactions.
`validateConfiguration`([logFailures, …])	Validate butler configuration.

Attributes Documentation

GENERATION = 3¶

This is a Generation 3 Butler.

This attribute may be removed in the future, once the Generation 2 Butler interface has been fully retired; it should only be used in transitional code.

Methods Documentation

close()¶: This method should be called to properly close any resources the butler may have open. The instance on which this method is closed should be considered unusable and no further methods should be called on it.

datasetExists(datasetRefOrType, dataId=None, **kwds)¶

Return True if the Dataset is actually present in the Datastore.

Parameters:

datasetRefOrType : DatasetRef, DatasetType, or str: When DatasetRef the dataId should be None. Otherwise the DatasetType or name thereof.
dataId : dict or DataId: A dict of Dimension link name, value pairs that label the DatasetRef within a Collection. When None, a DatasetRef should be provided as the first argument.
kwds: Additional keyword arguments used to augment or construct a DataId. See DataId parameters.

Raises:

LookupError: Raised if the Dataset is not even present in the Registry.

get(datasetRefOrType, dataId=None, parameters=None, **kwds)¶

Retrieve a stored dataset.

Parameters:

datasetRefOrType : DatasetRef, DatasetType, or str: When DatasetRef the dataId should be None. Otherwise the DatasetType or name thereof.
dataId : dict or DataId: A dict of Dimension link name, value pairs that label the DatasetRef within a Collection. When None, a DatasetRef should be provided as the first argument.
parameters : dict: Additional StorageClass-defined options to control reading, typically used to efficiently read only a subset of the dataset.
kwds: Additional keyword arguments used to augment or construct a DataId. See DataId parameters.

Returns:

obj : object: The dataset.

getDirect(ref, parameters=None)¶

Retrieve a stored dataset.

Unlike Butler.get, this method allows datasets outside the Butler’s collection to be read as long as the DatasetRef that identifies them can be obtained separately.

Parameters:	ref : `DatasetRef` Reference to an already stored dataset. parameters : `dict` Additional StorageClass-defined options to control reading, typically used to efficiently read only a subset of the dataset.
Returns:	obj : `object` The dataset.

getUri(datasetRefOrType, dataId=None, predict=False, **kwds)¶

Return the URI to the Dataset.

Parameters:	datasetRefOrType : `DatasetRef`, `DatasetType`, or `str` When `DatasetRef` the `dataId` should be `None`. Otherwise the `DatasetType` or name thereof. dataId : `dict` or `DataId` A `dict` of `Dimension` link name, value pairs that label the `DatasetRef` within a Collection. When `None`, a `DatasetRef` should be provided as the first argument. predict : `bool` If `True`, allow URIs to be returned of datasets that have not been written. kwds Additional keyword arguments used to augment or construct a `DataId`. See `DataId` parameters.
Returns:	uri : `str` URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, and if `predict` is `True`, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.
Raises:	FileNotFoundError A URI has been requested for a dataset that does not exist and guessing is not allowed.

static makeRepo(root, config=None, standalone=False, createRegistry=True)¶

Create an empty data repository by adding a butler.yaml config to a repository root directory.

Parameters:	root : `str` Filesystem path to the root of the new repository. Will be created if it does not exist. config : `Config`, optional Configuration to write to the repository, after setting any root-dependent Registry or Datastore config options. If `None`, default configuration will be used. standalone : `bool` If True, write all expanded defaults, not just customized or repository-specific settings. This (mostly) decouples the repository from the default configuration, insulating it from changes to the defaults (which may be good or bad, depending on the nature of the changes). Future additions to the defaults will still be picked up when initializing `Butlers` to repos created with `standalone=True`. createRegistry : `bool` If `True` create a new Registry. Note that when ``standalone=False`` (the default), the configuration search path (see `ConfigSubset.defaultSearchPaths`) that was used to construct the repository should also be used to construct any Butlers to it to avoid configuration inconsistencies.
Returns:	config : `Config` The updated `Config` instance written to the repo.
Raises:	ValueError Raised if a ButlerConfig or ConfigSubset is passed instead of a regular Config (as these subclasses would make it impossible to support `standalone=False`). os.error Raised if the directory does not exist, exists but is not a directory, or cannot be created.

put(obj, datasetRefOrType, dataId=None, producer=None, **kwds)¶

Store and register a dataset.

Parameters:	obj : `object` The dataset. datasetRefOrType : `DatasetRef`, `DatasetType`, or `str` When `DatasetRef` is provided, `dataId` should be `None`. Otherwise the `DatasetType` or name thereof. dataId : `dict` or `DataId` A `dict` of `Dimension` link name, value pairs that label the `DatasetRef` within a Collection. When `None`, a `DatasetRef` should be provided as the second argument. producer : `Quantum`, optional The producer. kwds Additional keyword arguments used to augment or construct a `DataId`. See `DataId` parameters.
Returns:	ref : `DatasetRef` A reference to the stored dataset, updated with the correct id if given.
Raises:	TypeError Raised if the butler was not constructed with a Run, and is hence read-only.

remove(datasetRefOrType, dataId=None, *, delete=True, remember=True, **kwds)¶

Remove a dataset from the collection and possibly the repository.

The identified dataset is always at least removed from the Butler’s collection. By default it is also deleted from the Datastore (e.g. files are actually deleted), but the dataset is “remembered” by retaining its row in the dataset and provenance tables in the registry.

If the dataset is a composite, all components will also be removed.

Parameters:

datasetRefOrType : DatasetRef, DatasetType, or str: When DatasetRef the dataId should be None. Otherwise the DatasetType or name thereof.
dataId : dict or DataId: A dict of Dimension link name, value pairs that label the DatasetRef within a Collection. When None, a DatasetRef should be provided as the first argument.
delete : bool: If True (default) actually delete the dataset from the Datastore (i.e. actually remove files).
remember : bool: If True (default), retain dataset and provenance records in the Registry for this dataset.
kwds: Additional keyword arguments used to augment or construct a DataId. See DataId parameters.

Raises:

ValueError: Raised if delete and remember are both False; a dataset cannot remain in a Datastore if all of its Registry entries are removed.
OrphanedRecordError: Raised if remember is False but the dataset is still present in a Datastore not recognized by this Butler client.

transaction()¶

Context manager supporting Butler transactions.

Transactions can be nested.

validateConfiguration(logFailures=False, datasetTypeNames=None, ignore=None)¶

Validate butler configuration.

Checks that each DatasetType can be stored in the Datastore.

Parameters:

logFailures : bool, optional: If True, output a log message for every validation error detected.
datasetTypeNames : iterable of str, optional: The DatasetType names that should be checked. This allows only a subset to be selected.
ignore : iterable of str, optional: Names of DatasetTypes to skip over. This can be used to skip known problems. If a named DatasetType corresponds to a composite, all component of that DatasetType will also be ignored.

Raises:

ButlerValidationError: Raised if there is some inconsistency with how this Butler is configured.

Navigation

Butler¶