Butler¶

class lsst.daf.butler.Butler(config: Union[lsst.daf.butler.core.config.Config, str, None] = None, *, butler: Optional[lsst.daf.butler._butler.Butler] = None, collection: Optional[str] = None, run: Optional[str] = None, searchPaths: Optional[List[str]] = None, writeable: Optional[bool] = None)¶

Bases: object

Main entry point for the data access system.

Parameters:	config : `ButlerConfig`, `Config` or `str`, optional. Configuration. Anything acceptable to the `ButlerConfig` constructor. If a directory path is given the configuration will be read from a `butler.yaml` file in that location. If `None` is given default values will be used. butler : `Butler`, optional. If provided, construct a new Butler that uses the same registry and datastore as the given one, but with the given collection and run. Incompatible with the `config`, `searchPaths`, and `writeable` arguments. collection : `str`, optional Collection to use for all input lookups. May be `None` to either use the value passed to `run`, or to defer passing a collection until the methods that require one are called. run : `str`, optional Name of the run datasets should be output to; also used as a tagged collection name these dataset will be associated with. If the run does not exist, it will be created. If `collection` is `None`, this collection will be used for input lookups as well; if not, it must have the same value as `run`. searchPaths : `list` of `str`, optional Directory paths to search when calculating the full Butler configuration. Not used if the supplied config is already a `ButlerConfig`. writeable : `bool`, optional Explicitly sets whether the butler supports write operations. If not provided, a read-only butler is created unless `run` is passed.
Raises:	ValueError Raised if neither “collection” nor “run” are provided by argument or config, or if both are provided and are inconsistent.
Attributes:	config : `str`, `ButlerConfig` or `Config`, optional (filename to) configuration. If this is not a `ButlerConfig`, defaults will be read. If a `str`, may be the path to a directory containing a “butler.yaml” file. datastore : `Datastore` Datastore to use for storage. registry : `Registry` Registry to use for lookups.

Attributes Summary

GENERATION This is a Generation 3 Butler.

Methods Summary

`datasetExists`(datasetRefOrType, …)	Return True if the Dataset is actually present in the Datastore.
`export`(*, directory, filename, format, transfer)	Export datasets from the repository represented by this `Butler`.
`get`(datasetRefOrType, …)	Retrieve a stored dataset.
`getDeferred`(datasetRefOrType, …)	Create a `DeferredDatasetHandle` which can later retrieve a dataset
`getDirect`(ref, *, parameters, Any]] = None)	Retrieve a stored dataset.
`getUri`(datasetRefOrType, …)	Return the URI to the Dataset.
`import_`(*, directory, filename, format, transfer)	Import datasets exported from a different butler repository.
`ingest`(*datasets, transfer, run)	Store and register one or more datasets that already exist on disk.
`isWriteable`()	Return `True` if this `Butler` supports write operations.
`makeRepo`(root, config, str, None] = None, …)	Create an empty data repository by adding a butler.yaml config to a repository root directory.
`put`(obj, datasetRefOrType, DatasetType, …)	Store and register a dataset.
`remove`(datasetRefOrType, …)	Remove a dataset from the collection and possibly the repository.
`transaction`()	Context manager supporting `Butler` transactions.
`validateConfiguration`(logFailures, …)	Validate butler configuration.

Attributes Documentation

GENERATION = 3¶

This is a Generation 3 Butler.

This attribute may be removed in the future, once the Generation 2 Butler interface has been fully retired; it should only be used in transitional code.

Methods Documentation

datasetExists(datasetRefOrType: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datasets.type.DatasetType, str], dataId: Union[lsst.daf.butler.core.dimensions.coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, collection: Optional[str] = None, **kwds) → bool¶

Return True if the Dataset is actually present in the Datastore.

Parameters:

datasetRefOrType : DatasetRef, DatasetType, or str: When DatasetRef the dataId should be None. Otherwise the DatasetType or name thereof.
dataId : dict or DataCoordinate: A dict of Dimension link name, value pairs that label the DatasetRef within a Collection. When None, a DatasetRef should be provided as the first argument.
collection : str, optional: Collection to search, overriding self.collection.
kwds: Additional keyword arguments used to augment or construct a DataCoordinate. See DataCoordinate.standardize parameters.

Raises:

LookupError: Raised if the dataset is not even present in the Registry.
ValueError: Raised if a resolved DatasetRef was passed as an input, but it differs from the one found in the registry in this collection.
TypeError: Raised if collection and self.collection are both None.

export(*, directory: Optional[str] = None, filename: Optional[str] = None, format: Optional[str] = None, transfer: Optional[str] = None) → ContextManager[RepoExport]¶

Export datasets from the repository represented by this Butler.

This method is a context manager that returns a helper object (RepoExport) that is used to indicate what information from the repository should be exported.

Parameters:

directory : str, optional: Directory dataset files should be written to if transfer is not None.
filename : str, optional: Name for the file that will include database information associated with the exported datasets. If this is not an absolute path and directory is not None, it will be written to directory instead of the current working directory. Defaults to “export.{format}”.
format : str, optional: File format for the database information file. If None, the extension of filename will be used.
transfer : str, optional: Transfer mode passed to Datastore.export.

Raises:

TypeError: Raised if the set of arguments passed is inconsistent.

Examples

Typically the Registry.queryDimensions and Registry.queryDatasets methods are used to provide the iterables over data IDs and/or datasets to be exported:

with butler.export("exports.yaml") as export:
    # Export all flats, and the calibration_label dimensions
    # associated with them.
    export.saveDatasets(butler.registry.queryDatasets("flat"),
                        elements=[butler.registry.dimensions["calibration_label"]])
    # Export all datasets that start with "deepCoadd_" and all of
    # their associated data ID information.
    export.saveDatasets(butler.registry.queryDatasets("deepCoadd_*"))

get(datasetRefOrType: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datasets.type.DatasetType, str], dataId: Union[lsst.daf.butler.core.dimensions.coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, parameters: Optional[Dict[str, Any]] = None, collection: Optional[str] = None, **kwds) → Any¶

Retrieve a stored dataset.

Parameters:	datasetRefOrType : `DatasetRef`, `DatasetType`, or `str` When `DatasetRef` the `dataId` should be `None`. Otherwise the `DatasetType` or name thereof. dataId : `dict` or `DataCoordinate` A `dict` of `Dimension` link name, value pairs that label the `DatasetRef` within a Collection. When `None`, a `DatasetRef` should be provided as the first argument. parameters : `dict` Additional StorageClass-defined options to control reading, typically used to efficiently read only a subset of the dataset. collection : `str`, optional Collection to search, overriding `self.collection`. kwds Additional keyword arguments used to augment or construct a `DataCoordinate`. See `DataCoordinate.standardize` parameters.
Returns:	obj : `object` The dataset.
Raises:	ValueError Raised if a resolved `DatasetRef` was passed as an input, but it differs from the one found in the registry in this collection. LookupError Raised if no matching dataset exists in the `Registry`. TypeError Raised if `collection` and `self.collection` are both `None`.

getDeferred(datasetRefOrType: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datasets.type.DatasetType, str], dataId: Union[lsst.daf.butler.core.dimensions.coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, parameters: Optional[dict] = None, collection: Optional[str] = None, **kwds) → lsst.daf.butler._deferredDatasetHandle.DeferredDatasetHandle¶

Create a DeferredDatasetHandle which can later retrieve a dataset

Parameters:	datasetRefOrType : `DatasetRef`, `DatasetType`, or `str` When `DatasetRef` the `dataId` should be `None`. Otherwise the `DatasetType` or name thereof. dataId : `dict` or `DataCoordinate`, optional A `dict` of `Dimension` link name, value pairs that label the `DatasetRef` within a Collection. When `None`, a `DatasetRef` should be provided as the first argument. collection : `str`, optional Name of the collection to search, overriding `self.collection`. parameters : `dict` Additional StorageClass-defined options to control reading, typically used to efficiently read only a subset of the dataset. collection : `str`, optional Collection to search, overriding `self.collection`. kwds Additional keyword arguments used to augment or construct a `DataId`. See `DataId` parameters.
Returns:	obj : `DeferredDatasetHandle` A handle which can be used to retrieve a dataset at a later time.
Raises:	LookupError Raised if no matching dataset exists in the `Registry` (and `allowUnresolved is False`). ValueError Raised if a resolved `DatasetRef` was passed as an input, but it differs from the one found in the registry in this collection. TypeError Raised if `collection` and `self.collection` are both `None`.

getDirect(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, *, parameters: Optional[Dict[str, Any]] = None)¶

Retrieve a stored dataset.

Unlike Butler.get, this method allows datasets outside the Butler’s collection to be read as long as the DatasetRef that identifies them can be obtained separately.

Parameters:	ref : `DatasetRef` Reference to an already stored dataset. parameters : `dict` Additional StorageClass-defined options to control reading, typically used to efficiently read only a subset of the dataset.
Returns:	obj : `object` The dataset.

getUri(datasetRefOrType: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datasets.type.DatasetType, str], dataId: Union[lsst.daf.butler.core.dimensions.coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, predict: bool = False, collection: Optional[str] = None, run: Optional[str] = None, **kwds) → str¶

Return the URI to the Dataset.

Parameters:	datasetRefOrType : `DatasetRef`, `DatasetType`, or `str` When `DatasetRef` the `dataId` should be `None`. Otherwise the `DatasetType` or name thereof. dataId : `dict` or `DataCoordinate` A `dict` of `Dimension` link name, value pairs that label the `DatasetRef` within a Collection. When `None`, a `DatasetRef` should be provided as the first argument. predict : `bool` If `True`, allow URIs to be returned of datasets that have not been written. collection : `str`, optional Collection to search, overriding `self.collection`. run : `str`, optional Run to use for predictions, overriding `self.run`. kwds Additional keyword arguments used to augment or construct a `DataCoordinate`. See `DataCoordinate.standardize` parameters.
Returns:	uri : `str` URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, and if `predict` is `True`, the URI will be a prediction and will include a URI fragment “#predicted”. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.
Raises:	LookupError A URI has been requested for a dataset that does not exist and guessing is not allowed. ValueError Raised if a resolved `DatasetRef` was passed as an input, but it differs from the one found in the registry in this collection. TypeError Raised if `collection` and `self.collection` are both `None`.

import_(*, directory: Optional[str] = None, filename: Optional[str] = None, format: Optional[str] = None, transfer: Optional[str] = None)¶

Import datasets exported from a different butler repository.

Parameters:

directory : str, optional: Directory containing dataset files. If None, all file paths must be absolute.
filename : str, optional: Name for the file that containing database information associated with the exported datasets. If this is not an absolute path, does not exist in the current working directory, and directory is not None, it is assumed to be in directory. Defaults to “export.{format}”.
format : str, optional: File format for the database information file. If None, the extension of filename will be used.
transfer : str, optional: Transfer mode passed to Datastore.export.

Raises:

TypeError: Raised if the set of arguments passed is inconsistent, or if the butler is read-only.

ingest(*datasets, transfer: Optional[str] = None, run: Optional[str] = None)¶

Store and register one or more datasets that already exist on disk.

Parameters:

datasets : FileDataset: Each positional argument is a struct containing information about a file to be ingested, including its path (either absolute or relative to the datastore root, if applicable), a DatasetRef, and optionally a formatter class or its fully-qualified string name. If a formatter is not provided, the formatter that would be used for put is assumed. On successful return, all FileDataset.ref attributes will have their DatasetRef.id attribute populated and all FileDataset.formatter attributes will be set to the formatter class used. FileDataset.path attributes may be modified to put paths in whatever the datastore considers a standardized form.
transfer : str, optional: If not None, must be one of ‘move’, ‘copy’, ‘hardlink’, or ‘symlink’, indicating how to transfer the file.
run : str, optional: The name of the run ingested datasets should be added to, overriding self.run.

Raises:

TypeError: Raised if the butler is read-only or if no run was provided.
NotImplementedError: Raised if the Datastore does not support the given transfer mode.
DatasetTypeNotSupportedError: Raised if one or more files to be ingested have a dataset type that is not supported by the Datastore..
FileNotFoundError: Raised if one of the given files does not exist.
FileExistsError: Raised if transfer is not None but the (internal) location the file would be moved to is already occupied.

Notes

This operation is not fully exception safe: if a database operation fails, the given FileDataset instances may be only partially updated.

It is atomic in terms of database operations (they will either all succeed or all fail) providing the database engine implements transactions correctly. It will attempt to be atomic in terms of filesystem operations as well, but this cannot be implemented rigorously for most datastores.

isWriteable() → bool¶: Return True if this Butler supports write operations.

static makeRepo(root: str, config: Union[lsst.daf.butler.core.config.Config, str, None] = None, standalone: bool = False, createRegistry: bool = True, searchPaths: Optional[List[str]] = None, forceConfigRoot: bool = True, outfile: Optional[str] = None) → lsst.daf.butler.core.config.Config¶

Create an empty data repository by adding a butler.yaml config to a repository root directory.

Parameters:	root : `str` Filesystem path to the root of the new repository. Will be created if it does not exist. config : `Config` or `str`, optional Configuration to write to the repository, after setting any root-dependent Registry or Datastore config options. Can not be a `ButlerConfig` or a `ConfigSubset`. If `None`, default configuration will be used. Root-dependent config options specified in this config are overwritten if `forceConfigRoot` is `True`. standalone : `bool` If True, write all expanded defaults, not just customized or repository-specific settings. This (mostly) decouples the repository from the default configuration, insulating it from changes to the defaults (which may be good or bad, depending on the nature of the changes). Future additions to the defaults will still be picked up when initializing `Butlers` to repos created with `standalone=True`. createRegistry : `bool`, optional If `True` create a new Registry. searchPaths : `list` of `str`, optional Directory paths to search when calculating the full butler configuration. forceConfigRoot : `bool`, optional If `False`, any values present in the supplied `config` that would normally be reset are not overridden and will appear directly in the output config. This allows non-standard overrides of the root directory for a datastore or registry to be given. If this parameter is `True` the values for `root` will be forced into the resulting config if appropriate. outfile : `str`, optional If not-`None`, the output configuration will be written to this location rather than into the repository itself. Can be a URI string. Can refer to a directory that will be used to write `butler.yaml`.
Returns:	config : `Config` The updated `Config` instance written to the repo.
Raises:	ValueError Raised if a ButlerConfig or ConfigSubset is passed instead of a regular Config (as these subclasses would make it impossible to support `standalone=False`). os.error Raised if the directory does not exist, exists but is not a directory, or cannot be created.

Notes

Note that when standalone=False (the default), the configuration search path (see ConfigSubset.defaultSearchPaths) that was used to construct the repository should also be used to construct any Butlers to avoid configuration inconsistencies.

put(obj: Any, datasetRefOrType: Union[DatasetRef, DatasetType, str], dataId: Optional[DataId] = None, *, producer: Optional[Quantum] = None, run: Optional[str] = None, **kwds) → DatasetRef¶

Store and register a dataset.

Parameters:	obj : `object` The dataset. datasetRefOrType : `DatasetRef`, `DatasetType`, or `str` When `DatasetRef` is provided, `dataId` should be `None`. Otherwise the `DatasetType` or name thereof. dataId : `dict` or `DataCoordinate` A `dict` of `Dimension` link name, value pairs that label the `DatasetRef` within a Collection. When `None`, a `DatasetRef` should be provided as the second argument. producer : `Quantum`, optional The producer. run : `str`, optional The name of the run the dataset should be added to, overriding `self.run`. kwds Additional keyword arguments used to augment or construct a `DataCoordinate`. See `DataCoordinate.standardize` parameters.
Returns:	ref : `DatasetRef` A reference to the stored dataset, updated with the correct id if given.
Raises:	TypeError Raised if the butler is read-only or if no run has been provided.

remove(datasetRefOrType: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datasets.type.DatasetType, str], dataId: Union[lsst.daf.butler.core.dimensions.coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, delete: bool = True, remember: bool = True, collection: Optional[str] = None, **kwds)¶

Remove a dataset from the collection and possibly the repository.

The identified dataset is always at least removed from the Butler’s collection. By default it is also deleted from the Datastore (e.g. files are actually deleted), but the dataset is “remembered” by retaining its row in the dataset and provenance tables in the registry.

If the dataset is a composite, all components will also be removed.

Parameters:

datasetRefOrType : DatasetRef, DatasetType, or str: When DatasetRef the dataId should be None. Otherwise the DatasetType or name thereof.
dataId : dict or DataId: A dict of Dimension link name, value pairs that label the DatasetRef within a Collection. When None, a DatasetRef should be provided as the first argument.
delete : bool: If True (default) actually delete the dataset from the Datastore (i.e. actually remove files).
remember : bool: If True (default), retain dataset and provenance records in the Registry for this dataset.
collection : str, optional: Collection to search, overriding self.collection.
kwds: Additional keyword arguments used to augment or construct a DataId. See DataId parameters.

Raises:

TypeError: Raised if the butler is read-only, if no collection was provided, or if delete and remember are both False; a dataset cannot remain in a Datastore if its Registry entries is removed.
OrphanedRecordError: Raised if remember is False but the dataset is still present in a Datastore not recognized by this Butler client.
ValueError: Raised if a resolved DatasetRef was passed as an input, but it differs from the one found in the registry in this collection.

transaction()¶

Context manager supporting Butler transactions.

Transactions can be nested.

validateConfiguration(logFailures: bool = False, datasetTypeNames: Optional[Iterable[str]] = None, ignore: Optional[Iterable[str]] = None)¶

Validate butler configuration.

Checks that each DatasetType can be stored in the Datastore.

Parameters:

logFailures : bool, optional: If True, output a log message for every validation error detected.
datasetTypeNames : iterable of str, optional: The DatasetType names that should be checked. This allows only a subset to be selected.
ignore : iterable of str, optional: Names of DatasetTypes to skip over. This can be used to skip known problems. If a named DatasetType corresponds to a composite, all component of that DatasetType will also be ignored.

Raises:

ButlerValidationError: Raised if there is some inconsistency with how this Butler is configured.

Navigation

Butler¶