lsst.daf.butler

This module provides an abstracted data access interface, known as the Butler. It can be used to read and write data without having to know the details of file formats or locations.

Contributing

lsst.daf.butler is developed at https://github.com/lsst/daf_butler. You can find Jira issues for this module under the daf_butler component.

Using the Butler

Python API reference

lsst.daf.butler Package

Functions

conformSet(container, universe) Transform an iterable of DimensionElement or names thereof into an object with a set-like .names attribute.
disableWhenLimited(func) Decorator that indicates that a method should raise NotImplementedError on Registries whose limited attribute is True.
makeBoxWcsRegion(box, wcs, margin) Construct a spherical ConvexPolygon from a WCS and a bounding box.

Classes

AmbiguousDatasetError Exception raised when a DatasetRef has no ID and a Registry operation requires one.
Butler([config, butler, collection, run, …]) Main entry point for the data access system.
ButlerConfig([other, searchPaths]) Contains the configuration for a Butler
ButlerURI(uri[, root, forceAbsolute]) Convenience wrapper around URI parsers.
ButlerValidationError There is a problem with the Butler configuration.
CompositeAssembler(storageClass) Class for providing assembler and disassembler support for composites.
CompositesConfig([other, validate, …])
CompositesMap(config, *, universe) Determine whether a specific datasetType or StorageClass should be disassembled.
Config([other]) Implements a datatype that is used by Butler for configuration parameters.
ConfigSubset([other, validate, …]) Config representing a subset of a more general configuration.
ConflictingDefinitionError Exception raised when trying to insert a database record when a conflicting record already exists.
Constraints(config, *, universe) Determine whether a DatasetRef, DatasetType, or StorageClass is allowed to be handled.
ConstraintsConfig([other]) Configuration information for Constraints
ConstraintsValidationError Exception thrown when a constraints list has mutually exclusive definitions.
DataId([dataId, dimensions, dimension, …]) A dict-like identifier for data usable across multiple collections and DatasetTypes.
DataIdPacker An abstract base class for a bidirectional mappings between a DataId and a packed integer or bytes blob.
DataIdPackerDimensions(given, required) A helper class that holds the three categories of dimensions associated with DataIdPacker objects.
DatabaseDict(config, types, key, value[, …]) An abstract base class for dict-like objects with a specific key type and namedtuple values, backed by a database.
DatasetComponent(name, storageClass, component) Component of a dataset and associated information.
DatasetOriginInfo Interface for classes providing collection information for pre-flight.
DatasetOriginInfoDef(defaultInputs, …[, …]) Default implementation of the DatasetOriginInfo.
DatasetRef(datasetType, dataId[, id, run, …]) Reference to a Dataset in a Registry.
DatasetType(name, dimensions, storageClass, *) A named category of Datasets that defines how they are organized, related, and stored.
DatasetTypeNotSupportedError A DatasetType is not handled by this routine.
Datastore(config, registry[, butlerRoot]) Datastore interface.
DatastoreConfig([other, validate, …])
DatastoreValidationError There is a problem with the Datastore configuration.
Dimension(universe, name, config) A discrete dimension of data used to organize Datasets and associate them with metadata.
DimensionConfig([other, validate, …])
DimensionElement(universe, name, hasRegion, …) Base class for elements in the dimension schema.
DimensionGraph(universe, elements, …) An automatically-expanded collection of both Dimensions and DimensionJoins.
DimensionJoin(universe, name, config) A join that relates two or more Dimensions.
DimensionKeyDict([other, universe, keys, …]) An immutable mapping that uses DimensionElement instances as keys.
DimensionSet(universe, elements[, expand, …]) A custom set/dict hybrid class for collections of DimensionElements.
DimensionUniverse() Specialization for a DimensionGraph that defines a complete system of dimensions.
Execution([startTime, endTime, host, id]) Any step in a production.
FileDescriptor(location, storageClass[, …]) Describes a particular file.
FileTemplate(template) Format a path template into a fully expanded path.
FileTemplateValidationError Exception thrown when a file template is not consistent with the associated DatasetType.
FileTemplates(config[, default]) Collection of FileTemplate templates.
FileTemplatesConfig([other]) Configuration information for FileTemplates
Formatter(fileDescriptor) Interface for reading and writing Datasets with a particular StorageClass.
FormatterFactory() Factory for Formatter instances.
Location(datastoreRootUri, path) Identifies a location within the Datastore.
LocationFactory(datastoreRoot) Factory for Location instances.
LookupKey([name, dimensions, dataId, universe]) Representation of key that can be used to lookup information based on dataset type name, storage class name, dimensions.
MappingFactory(refType) Register the mapping of some key to a python type and retrieve instances.
OrphanedRecordError Exception raised when trying to remove or modify a database record that is still being used in some other table.
Quantum(*[, taskName, taskClass, dataId, …]) A discrete unit of work that may depend on one or more datasets and produces one or more datasets.
Registry(registryConfig[, schemaConfig, …]) Registry interface.
RegistryConfig([other, validate, …])
Run(collection[, environment, pipeline]) Represent a processing run.
Schema([config, limited]) The SQL schema for a Butler Registry.
SchemaBuilder(config[, limited]) Builds a Schema step-by-step.
SchemaConfig([other, validate, …])
StorageClass([name, pytype, components, …]) Class describing how a label maps to a particular Python type.
StorageClassConfig([other, validate, …])
StorageClassFactory([config]) Factory for StorageClass instances.
StoredFileInfo(formatter, path, storageClass) Information associated with a stored file in a Datastore.
ValidationError Some sort of validation error has occurred.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.registry.AmbiguousDatasetError, lsst.daf.butler.butler.Butler, lsst.daf.butler.core.butlerConfig.ButlerConfig, lsst.daf.butler.core.location.ButlerURI, lsst.daf.butler.butler.ButlerValidationError, lsst.daf.butler.core.assembler.CompositeAssembler, lsst.daf.butler.core.composites.CompositesConfig, lsst.daf.butler.core.composites.CompositesMap, lsst.daf.butler.core.config.Config, lsst.daf.butler.core.config.ConfigSubset, lsst.daf.butler.core.registry.ConflictingDefinitionError, lsst.daf.butler.core.constraints.Constraints, lsst.daf.butler.core.constraints.ConstraintsConfig, lsst.daf.butler.core.constraints.ConstraintsValidationError, lsst.daf.butler.core.dimensions.dataId.DataId, lsst.daf.butler.core.dataIdPacker.DataIdPacker, lsst.daf.butler.core.dataIdPacker.DataIdPackerDimensions, lsst.daf.butler.core.databaseDict.DatabaseDict, lsst.daf.butler.core.assembler.DatasetComponent, lsst.daf.butler.core.preFlight.DatasetOriginInfo, lsst.daf.butler.core.preFlight.DatasetOriginInfoDef, lsst.daf.butler.core.datasets.DatasetRef, lsst.daf.butler.core.datasets.DatasetType, lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError, lsst.daf.butler.core.datastore.Datastore, lsst.daf.butler.core.datastore.DatastoreConfig, lsst.daf.butler.core.datastore.DatastoreValidationError, lsst.daf.butler.core.dimensions.elements.Dimension, lsst.daf.butler.core.dimensions.graph.DimensionConfig, lsst.daf.butler.core.dimensions.elements.DimensionElement, lsst.daf.butler.core.dimensions.graph.DimensionGraph, lsst.daf.butler.core.dimensions.elements.DimensionJoin, lsst.daf.butler.core.dimensions.dataId.DimensionKeyDict, lsst.daf.butler.core.dimensions.sets.DimensionSet, lsst.daf.butler.core.dimensions.graph.DimensionUniverse, lsst.daf.butler.core.execution.Execution, lsst.daf.butler.core.fileDescriptor.FileDescriptor, lsst.daf.butler.core.fileTemplates.FileTemplate, lsst.daf.butler.core.fileTemplates.FileTemplateValidationError, lsst.daf.butler.core.fileTemplates.FileTemplates, lsst.daf.butler.core.fileTemplates.FileTemplatesConfig, lsst.daf.butler.core.formatter.Formatter, lsst.daf.butler.core.formatter.FormatterFactory, lsst.daf.butler.core.location.Location, lsst.daf.butler.core.location.LocationFactory, lsst.daf.butler.core.configSupport.LookupKey, lsst.daf.butler.core.mappingFactory.MappingFactory, lsst.daf.butler.core.registry.OrphanedRecordError, lsst.daf.butler.core.quantum.Quantum, lsst.daf.butler.core.registry.Registry, lsst.daf.butler.core.registry.RegistryConfig, lsst.daf.butler.core.run.Run, lsst.daf.butler.core.schema.Schema, lsst.daf.butler.core.schema.SchemaBuilder, lsst.daf.butler.core.schema.SchemaConfig, lsst.daf.butler.core.storageClass.StorageClass, lsst.daf.butler.core.storageClass.StorageClassConfig, lsst.daf.butler.core.storageClass.StorageClassFactory, lsst.daf.butler.core.storedFileInfo.StoredFileInfo, lsst.daf.butler.core.exceptions.ValidationError

Example datastores

lsst.daf.butler.datastores.posixDatastore Module

Classes
PosixDatastore(config, registry[, butlerRoot]) Basic POSIX filesystem backed Datastore.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.posixDatastore.PosixDatastore

lsst.daf.butler.datastores.inMemoryDatastore Module

Classes
StoredItemInfo(timestamp, storageClass[, …]) Internal Metadata associated with a DatasetRef.
InMemoryDatastore(config[, registry, butlerRoot]) Basic Datastore for writing to an in memory cache.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.inMemoryDatastore.StoredItemInfo, lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore

lsst.daf.butler.datastores.chainedDatastore Module

Classes
ChainedDatastore(config[, registry, butlerRoot]) Chained Datastores to allow read and writes from multiple datastores.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.chainedDatastore.ChainedDatastore

Example registries

lsst.daf.butler.registries.sqlRegistry Module

Classes
SqlRegistryConfig([other, validate, …])
SqlRegistry(registryConfig, schemaConfig, …) Registry backed by a SQL database.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.registries.sqlRegistry.SqlRegistryConfig, lsst.daf.butler.registries.sqlRegistry.SqlRegistry

Example formatters

lsst.daf.butler.formatters.fileFormatter Module

Classes
FileFormatter(fileDescriptor) Interface for reading and writing files on a POSIX file system.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.fileFormatter.FileFormatter

lsst.daf.butler.formatters.jsonFormatter Module

Classes
JsonFormatter(fileDescriptor) Interface for reading and writing Python objects to and from JSON files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.jsonFormatter.JsonFormatter

lsst.daf.butler.formatters.yamlFormatter Module

Classes
YamlFormatter(fileDescriptor) Interface for reading and writing Python objects to and from YAML files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.yamlFormatter.YamlFormatter

lsst.daf.butler.formatters.pickleFormatter Module

Classes
PickleFormatter(fileDescriptor) Interface for reading and writing Python objects to and from pickle files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.pickleFormatter.PickleFormatter

Support API

lsst.daf.butler.core.safeFileIo Module

Functions
safeMakeDir(directory) Make a directory in a manner avoiding race conditions
setFileMode(filename) Set a file mode according to the user’s umask
FileForWriteOnceCompareSame(name) Context manager to get a file that can be written only once and all other writes will succeed only if they match the initial write.
SafeFile(name) Context manager to create a file in a manner avoiding race conditions
SafeFilename(name) Context manager for creating a file in a manner avoiding race conditions.
SafeLockedFileForRead(name) Context manager for reading a file that may be locked with an exclusive lock via SafeLockedFileForWrite.
Classes
DoNotWrite
FileForWriteOnceCompareSameFailure
SafeLockedFileForWrite(name) File-like object that is used to create a file if needed, lock it with an exclusive lock, and contain file descriptors to readable and writable versions of the file.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.safeFileIo.DoNotWrite, lsst.daf.butler.core.safeFileIo.FileForWriteOnceCompareSameFailure, lsst.daf.butler.core.safeFileIo.SafeLockedFileForWrite

lsst.daf.butler.core.utils Module

Functions
iterable(a) Make input iterable.
allSlots(self) Return combined __slots__ for all classes in objects mro.
slotValuesAreEqual(self, other) Test for equality by the contents of all slots, including those of its parents.
slotValuesToHash(self) Generate a hash from slot values.
getFullTypeName(cls) Return full type name of the supplied entity.
getInstanceOf(typeOrName, *args, **kwargs) Given the type name or a type, instantiate an object of that type.
transactional(func) Decorator that wraps a method and makes it transactional.
getObjectSize(obj[, seen]) Recursively finds size of objects.
stripIfNotNone(s) Strip leading and trailing whitespace if the given object is not None.
Classes
Singleton Metaclass to convert a class to a Singleton.
PrivateConstructorMeta A metaclass that disables regular construction syntax.
NamedKeyDict(*args) A dictionary wrapper that require keys to have a .name attribute, and permits lookups using either key objects or their names.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.utils.Singleton, lsst.daf.butler.core.utils.PrivateConstructorMeta, lsst.daf.butler.core.utils.NamedKeyDict

lsst.daf.butler.core.repoRelocation Module

Functions
replaceRoot(configRoot, butlerRoot) Update a configuration root with the butler root location.
Variables
BUTLER_ROOT_TAG The special string to be used in configuration files to indicate that the butler root location should be used.