This module provides an abstracted data access interface, known as the Butler. It can be used to read and write data without having to know the details of file formats or locations.


lsst.daf.butler is developed at You can find Jira issues for this module under the daf_butler component.

Python API reference

lsst.daf.butler Package


conformSet(container) Transform an iterable of DimensionElement or names thereof into an object with a set-like .names attribute.
disableWhenLimited(func) Decorator that indicates that a method should raise NotImplementedError on Registries whose limited attribute is True.
makeBoxWcsRegion(box, wcs, margin) Construct a spherical ConvexPolygon from a WCS and a bounding box.


AmbiguousDatasetError Exception raised when a DatasetRef has no ID and a Registry operation requires one.
Butler([config, collection, run]) Main entry point for the data access system.
ButlerConfig([other]) Contains the configuration for a Butler
CompositeAssembler(storageClass) Class for providing assembler and disassembler support for composites.
CompositesConfig([other, validate, …])
CompositesMap(config) Determine whether a specific datasetType or StorageClass should be disassembled.
Config([other]) Implements a datatype that is used by Butler for configuration parameters.
ConfigSubset([other, validate, …]) Config representing a subset of a more general configuration.
ConflictingDefinitionError Exception raised when trying to insert a database record when a conflicting record already exists.
DataId([dataId, dimensions, dimension, …]) A dict-like identifier for data usable across multiple collections and DatasetTypes.
DataIdPacker An abstract base class for a bidirectional mappings between a DataId and a packed integer or bytes blob.
DataIdPackerDimensions(given, required) A helper class that holds the three categories of dimensions associated with DataIdPacker objects.
DatabaseDict(config, types, key, value) An abstract base class for dict-like objects with a specific key type and namedtuple values, backed by a database.
DatasetComponent(name, storageClass, component) Component of a dataset and associated information.
DatasetOriginInfo Interface for classes providing collection information for pre-flight.
DatasetOriginInfoDef(defaultInputs, …[, …]) Default implementation of the DatasetOriginInfo.
DatasetRef(datasetType, dataId[, id, run, …]) Reference to a Dataset in a Registry.
DatasetType(name, dimensions, storageClass) A named category of Datasets that defines how they are organized, related, and stored.
DatasetTypeNotSupportedError A DatasetType is not handled by this routine.
Datastore(config, registry) Datastore interface.
DatastoreConfig([other, validate, …])
Dimension(universe, name, config) A discrete dimension of data used to organize Datasets and associate them with metadata.
DimensionConfig([other, validate, …])
DimensionElement(universe, name, hasRegion, …) Base class for elements in the dimension schema.
DimensionGraph(universe, elements, …) An automatically-expanded collection of both Dimensions and DimensionJoins.
DimensionJoin(universe, name, config) A join that relates two or more Dimensions.
DimensionKeyDict([other, universe, keys, …]) An immutable mapping that uses DimensionElement instances as keys.
DimensionNameSet(names) An incomplete, name-only stand-in for DimensionSet or DimensionGraph.
DimensionSet(universe, elements[, expand, …]) A custom set/dict hybrid class for collections of DimensionElements.
DimensionUniverse() Specialization for a DimensionGraph that defines a complete system of dimensions.
Execution([startTime, endTime, host, id]) Any step in a production.
FileDescriptor(location, storageClass[, …]) Describes a particular file.
FileTemplate(template) Format a path template into a fully expanded path.
FileTemplates(config[, default]) Collection of FileTemplate templates.
FileTemplatesConfig([other]) Configuration information for FileTemplates
Formatter Interface for reading and writing Datasets with a particular StorageClass.
FormatterFactory() Factory for Formatter instances.
Location(datastoreRoot, uri) Identifies a location in the Datastore.
LocationFactory(datastoreRoot) Factory for Location instances.
MappingFactory(refType) Register the mapping of some key to a python type and retrieve instances.
OrphanedRecordError Exception raised when trying to remove or modify a database record that is still being used in some other table.
PreFlightDimensionsRow(dataId, datasetRefs) Simple data class holding combination of Dimension values for one row returned by pre-flight solver.
Quantum(task, run, *args, **kwargs) A discrete unit of work that may depend on one or more datasets and produces one or more datasets.
Registry(registryConfig[, schemaConfig, …]) Registry interface.
RegistryConfig([other, validate, …])
Run(collection[, environment, pipeline]) Represent a processing run.
Schema([config, limited]) The SQL schema for a Butler Registry.
SchemaBuilder(config[, limited]) Builds a Schema step-by-step.
SchemaConfig([other, validate, …])
StorageClass([name, pytype, components, …]) Class describing how a label maps to a particular Python type.
StorageClassConfig([other, validate, …])
StorageClassFactory([config]) Factory for StorageClass instances.
StoredFileInfo(formatter, path, storageClass) Information associated with a stored file in a Datastore.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.registry.AmbiguousDatasetError, lsst.daf.butler.butler.Butler, lsst.daf.butler.core.butlerConfig.ButlerConfig, lsst.daf.butler.core.assembler.CompositeAssembler, lsst.daf.butler.core.composites.CompositesConfig, lsst.daf.butler.core.composites.CompositesMap, lsst.daf.butler.core.config.Config, lsst.daf.butler.core.config.ConfigSubset, lsst.daf.butler.core.registry.ConflictingDefinitionError, lsst.daf.butler.core.dimensions.dataId.DataId, lsst.daf.butler.core.dataIdPacker.DataIdPacker, lsst.daf.butler.core.dataIdPacker.DataIdPackerDimensions, lsst.daf.butler.core.databaseDict.DatabaseDict, lsst.daf.butler.core.assembler.DatasetComponent, lsst.daf.butler.core.preFlight.DatasetOriginInfo, lsst.daf.butler.core.preFlight.DatasetOriginInfoDef, lsst.daf.butler.core.datasets.DatasetRef, lsst.daf.butler.core.datasets.DatasetType, lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError, lsst.daf.butler.core.datastore.Datastore, lsst.daf.butler.core.datastore.DatastoreConfig, lsst.daf.butler.core.dimensions.elements.Dimension, lsst.daf.butler.core.dimensions.graph.DimensionConfig, lsst.daf.butler.core.dimensions.elements.DimensionElement, lsst.daf.butler.core.dimensions.graph.DimensionGraph, lsst.daf.butler.core.dimensions.elements.DimensionJoin, lsst.daf.butler.core.dimensions.dataId.DimensionKeyDict, lsst.daf.butler.core.dimensions.sets.DimensionNameSet, lsst.daf.butler.core.dimensions.sets.DimensionSet, lsst.daf.butler.core.dimensions.graph.DimensionUniverse, lsst.daf.butler.core.execution.Execution, lsst.daf.butler.core.fileDescriptor.FileDescriptor, lsst.daf.butler.core.fileTemplates.FileTemplate, lsst.daf.butler.core.fileTemplates.FileTemplates, lsst.daf.butler.core.fileTemplates.FileTemplatesConfig, lsst.daf.butler.core.formatter.Formatter, lsst.daf.butler.core.formatter.FormatterFactory, lsst.daf.butler.core.location.Location, lsst.daf.butler.core.location.LocationFactory, lsst.daf.butler.core.mappingFactory.MappingFactory, lsst.daf.butler.core.registry.OrphanedRecordError, lsst.daf.butler.core.preFlight.PreFlightDimensionsRow, lsst.daf.butler.core.quantum.Quantum, lsst.daf.butler.core.registry.Registry, lsst.daf.butler.core.registry.RegistryConfig,, lsst.daf.butler.core.schema.Schema, lsst.daf.butler.core.schema.SchemaBuilder, lsst.daf.butler.core.schema.SchemaConfig, lsst.daf.butler.core.storageClass.StorageClass, lsst.daf.butler.core.storageClass.StorageClassConfig, lsst.daf.butler.core.storageClass.StorageClassFactory, lsst.daf.butler.core.storedFileInfo.StoredFileInfo

Example datastores

lsst.daf.butler.datastores.posixDatastore Module

PosixDatastore(config, registry) Basic POSIX filesystem backed Datastore.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.posixDatastore.PosixDatastore

lsst.daf.butler.datastores.inMemoryDatastore Module

StoredItemInfo(timestamp, storageClass[, …]) Internal Metadata associated with a DatasetRef.
InMemoryDatastore(config[, registry]) Basic Datastore for writing to an in memory cache.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.inMemoryDatastore.StoredItemInfo, lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore

lsst.daf.butler.datastores.chainedDatastore Module

ChainedDatastore(config[, registry]) Chained Datastores to allow read and writes from multiple datastores.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.chainedDatastore.ChainedDatastore

Example registries

lsst.daf.butler.registries.sqlRegistry Module

SqlRegistryConfig([other, validate, …])
SqlRegistry(registryConfig, schemaConfig, …) Registry backed by a SQL database.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.registries.sqlRegistry.SqlRegistryConfig, lsst.daf.butler.registries.sqlRegistry.SqlRegistry

Example formatters

lsst.daf.butler.formatters.fileFormatter Module

FileFormatter Interface for reading and writing files on a POSIX file system.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.fileFormatter.FileFormatter

lsst.daf.butler.formatters.jsonFormatter Module

JsonFormatter Interface for reading and writing Python objects to and from JSON files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.jsonFormatter.JsonFormatter

lsst.daf.butler.formatters.yamlFormatter Module

YamlFormatter Interface for reading and writing Python objects to and from YAML files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.yamlFormatter.YamlFormatter

lsst.daf.butler.formatters.pickleFormatter Module

PickleFormatter Interface for reading and writing Python objects to and from pickle files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.pickleFormatter.PickleFormatter

Support API

lsst.daf.butler.core.safeFileIo Module

safeMakeDir(directory) Make a directory in a manner avoiding race conditions
setFileMode(filename) Set a file mode according to the user’s umask
FileForWriteOnceCompareSame(name) Context manager to get a file that can be written only once and all other writes will succeed only if they match the initial write.
SafeFile(name) Context manager to create a file in a manner avoiding race conditions
SafeFilename(name) Context manager for creating a file in a manner avoiding race conditions.
SafeLockedFileForRead(name) Context manager for reading a file that may be locked with an exclusive lock via SafeLockedFileForWrite.
SafeLockedFileForWrite(name) File-like object that is used to create a file if needed, lock it with an exclusive lock, and contain file descriptors to readable and writable versions of the file.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.safeFileIo.DoNotWrite, lsst.daf.butler.core.safeFileIo.FileForWriteOnceCompareSameFailure, lsst.daf.butler.core.safeFileIo.SafeLockedFileForWrite

lsst.daf.butler.core.utils Module

iterable(a) Make input iterable.
allSlots(self) Return combined __slots__ for all classes in objects mro.
slotValuesAreEqual(self, other) Test for equality by the contents of all slots, including those of its parents.
slotValuesToHash(self) Generate a hash from slot values.
getFullTypeName(cls) Return full type name of the supplied entity.
getInstanceOf(typeOrName) Given the type name or a type, instantiate an object of that type.
transactional(func) Decorator that wraps a method and makes it transactional.
getObjectSize(obj[, seen]) Recursively finds size of objects.
stripIfNotNone(s) Strip leading and trailing whitespace if the given object is not None.
Singleton Metaclass to convert a class to a Singleton.
PrivateConstructorMeta A metaclass that disables regular construction syntax.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.utils.Singleton, lsst.daf.butler.core.utils.PrivateConstructorMeta