lsst.daf.butler

This module provides an abstracted data access interface, known as the Butler. It can be used to read and write data without having to know the details of file formats or locations.

Contributing

lsst.daf.butler is developed at https://github.com/lsst/daf_butler. You can find Jira issues for this module under the daf_butler component.

Concrete Storage Classes

Python API reference

lsst.daf.butler Package

Functions

makeBoxWcsRegion(box, wcs, margin) Construct a spherical ConvexPolygon from a WCS and a bounding box.

Classes

AmbiguousDatasetError Exception raised when a DatasetRef has no ID and a Registry operation requires one.
Butler([config, butler, collection, run, …]) Main entry point for the data access system.
ButlerConfig([other, searchPaths]) Contains the configuration for a Butler
ButlerURI(uri[, root, forceAbsolute]) Convenience wrapper around URI parsers.
ButlerValidationError There is a problem with the Butler configuration.
CompositeAssembler(storageClass) Class for providing assembler and disassembler support for composites.
CompositesConfig([other, validate, …])
CompositesMap(config, *, universe) Determine whether a specific datasetType or StorageClass should be disassembled.
Config([other]) Implements a datatype that is used by Butler for configuration parameters.
ConfigSubset([other, validate, …]) Config representing a subset of a more general configuration.
ConflictingDefinitionError Exception raised when trying to insert a database record when a conflicting record already exists.
Constraints(config, *, universe) Determine whether a DatasetRef, DatasetType, or StorageClass is allowed to be handled.
ConstraintsConfig([other]) Configuration information for Constraints
ConstraintsValidationError Exception thrown when a constraints list has mutually exclusive definitions.
DataCoordinate An immutable data ID dictionary that guarantees that its key-value pairs identify all required dimensions in a DimensionGraph.
DatasetComponent(name, storageClass, component) Component of a dataset and associated information.
DatasetRef Reference to a Dataset in a Registry.
DatasetType(name, dimensions, storageClass, *) A named category of Datasets that defines how they are organized, related, and stored.
DatasetTypeNotSupportedError A DatasetType is not handled by this routine.
Datastore(config, registry[, butlerRoot]) Datastore interface.
DatastoreConfig([other, validate, …])
DatastoreValidationError There is a problem with the Datastore configuration.
DbAuth([path, envVar, authList]) Retrieves authentication information for database connections.
DbAuthError A problem has occurred retrieving database authentication information.
DbAuthPermissionsError Credentials file has incorrect permissions.
DeferredDatasetHandle(butler, …) This is a class to support deferred loading of a dataset from a butler.
Dimension(name, *, uniqueKeys, **kwds) A named data-organization concept that can be used as a key in a data ID.
DimensionConfig([other, validate, …]) Configuration that defines a DimensionUniverse.
DimensionElement(name, *, …) A named data-organization concept that defines a label and/or metadata in the dimensions system.
DimensionGraph An immutable, dependency-complete collection of dimensions.
DimensionPacker(fixed, dimensions) An abstract base class for bidirectional mappings between a DataCoordinate and a packed integer ID.
DimensionRecord(*args) Base class for the Python representation of database records for a DimensionElement.
DimensionUniverse A special DimensionGraph that constructs and manages a complete set of compatible dimensions.
ExpandedDataCoordinate A data ID that has been expanded to include all relevant metadata.
FieldSpec(name, dtype, length, nbytes, …) A struct-like class used to define a column in a logical Registry table.
FileDataset(path, ref, *, formatter, str, …) A struct that represents a dataset exported to a file.
FileDescriptor(location, storageClass[, …]) Describes a particular file.
FileTemplate(template) Format a path template into a fully expanded path.
FileTemplateValidationError Exception thrown when a file template is not consistent with the associated DatasetType.
FileTemplates(config[, default]) Collection of FileTemplate templates.
FileTemplatesConfig([other]) Configuration information for FileTemplates
ForeignKeySpec(table, source, …], target, …) A struct-like class used to define a foreign key constraint in a logical Registry table.
Formatter(fileDescriptor) Interface for reading and writing Datasets with a particular StorageClass.
FormatterFactory() Factory for Formatter instances.
Location(datastoreRootUri, path) Identifies a location within the Datastore.
LocationFactory(datastoreRoot) Factory for Location instances.
LookupKey([name, dimensions, dataId, universe]) Representation of key that can be used to lookup information based on dataset type name, storage class name, dimensions.
MappingFactory(refType) Register the mapping of some key to a python type and retrieve instances.
OrphanedRecordError Exception raised when trying to remove or modify a database record that is still being used in some other table.
Quantum(*[, taskName, taskClass, dataId, …]) A discrete unit of work that may depend on one or more datasets and produces one or more datasets.
Registry(registryConfig[, schemaConfig, …]) Registry interface.
RepoExport(registry, datastore, backend, *, …) Public interface for exporting a subset of a data repository.
RepoExportBackend An abstract interface for data repository export implementations.
RepoImportBackend An abstract interface for data repository import implementations.
RepoTransferFormatConfig([other, validate, …]) The section of butler configuration that associates repo import/export backends with file formats.
Schema([spec]) The SQL schema for a Butler Registry.
SchemaConfig([other, validate, …])
SkyPixDimension(name, pixelization) A special Dimension subclass for hierarchical pixelizations of the sky.
StorageClass([name, pytype, components, …]) Class describing how a label maps to a particular Python type.
StorageClassConfig([other, validate, …])
StorageClassFactory([config]) Factory for StorageClass instances.
StoredDatastoreItemInfo Internal information associated with a stored dataset in a Datastore.
StoredFileInfo(formatter, path, …) Datastore-private metadata associated with a file stored in a Datastore.
TableSpec(fields, unique, …]], …) A struct-like class used to define a logical Registry table (which may also be implemented in the database as a view).
ValidationError Some sort of validation error has occurred.
YamlRepoExportBackend(stream) A repository export implementation that saves to a YAML file.
YamlRepoImportBackend(stream) A repository import implementation that reads from a YAML file.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.registry.AmbiguousDatasetError, lsst.daf.butler.butler.Butler, lsst.daf.butler.core.butlerConfig.ButlerConfig, lsst.daf.butler.core.location.ButlerURI, lsst.daf.butler.butler.ButlerValidationError, lsst.daf.butler.core.assembler.CompositeAssembler, lsst.daf.butler.core.composites.CompositesConfig, lsst.daf.butler.core.composites.CompositesMap, lsst.daf.butler.core.config.Config, lsst.daf.butler.core.config.ConfigSubset, lsst.daf.butler.core.registry.ConflictingDefinitionError, lsst.daf.butler.core.constraints.Constraints, lsst.daf.butler.core.constraints.ConstraintsConfig, lsst.daf.butler.core.constraints.ConstraintsValidationError, lsst.daf.butler.core.dimensions.coordinate.DataCoordinate, lsst.daf.butler.core.assembler.DatasetComponent, lsst.daf.butler.core.datasets.ref.DatasetRef, lsst.daf.butler.core.datasets.type.DatasetType, lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError, lsst.daf.butler.core.datastore.Datastore, lsst.daf.butler.core.datastore.DatastoreConfig, lsst.daf.butler.core.datastore.DatastoreValidationError, lsst.daf.butler.core.dbAuth.DbAuth, lsst.daf.butler.core.dbAuth.DbAuthError, lsst.daf.butler.core.dbAuth.DbAuthPermissionsError, lsst.daf.butler.core.deferredDatasetHandle.DeferredDatasetHandle, lsst.daf.butler.core.dimensions.elements.Dimension, lsst.daf.butler.core.dimensions.config.DimensionConfig, lsst.daf.butler.core.dimensions.elements.DimensionElement, lsst.daf.butler.core.dimensions.graph.DimensionGraph, lsst.daf.butler.core.dimensions.packer.DimensionPacker, lsst.daf.butler.core.dimensions.records.DimensionRecord, lsst.daf.butler.core.dimensions.universe.DimensionUniverse, lsst.daf.butler.core.dimensions.coordinate.ExpandedDataCoordinate, lsst.daf.butler.core.schema.FieldSpec, lsst.daf.butler.core.repoTransfers.FileDataset, lsst.daf.butler.core.fileDescriptor.FileDescriptor, lsst.daf.butler.core.fileTemplates.FileTemplate, lsst.daf.butler.core.fileTemplates.FileTemplateValidationError, lsst.daf.butler.core.fileTemplates.FileTemplates, lsst.daf.butler.core.fileTemplates.FileTemplatesConfig, lsst.daf.butler.core.schema.ForeignKeySpec, lsst.daf.butler.core.formatter.Formatter, lsst.daf.butler.core.formatter.FormatterFactory, lsst.daf.butler.core.location.Location, lsst.daf.butler.core.location.LocationFactory, lsst.daf.butler.core.configSupport.LookupKey, lsst.daf.butler.core.mappingFactory.MappingFactory, lsst.daf.butler.core.registry.OrphanedRecordError, lsst.daf.butler.core.quantum.Quantum, lsst.daf.butler.core.registry.Registry, lsst.daf.butler.core.repoTransfers.RepoExport, lsst.daf.butler.core.repoTransfers.RepoExportBackend, lsst.daf.butler.core.repoTransfers.RepoImportBackend, lsst.daf.butler.core.repoTransfers.RepoTransferFormatConfig, lsst.daf.butler.core.schema.Schema, lsst.daf.butler.core.schema.SchemaConfig, lsst.daf.butler.core.dimensions.elements.SkyPixDimension, lsst.daf.butler.core.storageClass.StorageClass, lsst.daf.butler.core.storageClass.StorageClassConfig, lsst.daf.butler.core.storageClass.StorageClassFactory, lsst.daf.butler.core.storedFileInfo.StoredDatastoreItemInfo, lsst.daf.butler.core.storedFileInfo.StoredFileInfo, lsst.daf.butler.core.schema.TableSpec, lsst.daf.butler.core.exceptions.ValidationError, lsst.daf.butler.core.repoTransfers.YamlRepoExportBackend, lsst.daf.butler.core.repoTransfers.YamlRepoImportBackend

Example datastores

lsst.daf.butler.datastores.posixDatastore Module

Classes
PosixDatastore(config, registry[, butlerRoot]) Basic POSIX filesystem backed Datastore.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.posixDatastore.PosixDatastore

lsst.daf.butler.datastores.inMemoryDatastore Module

Classes
StoredMemoryItemInfo(timestamp, …) Internal InMemoryDatastore Metadata associated with a stored DatasetRef.
InMemoryDatastore(config[, registry, butlerRoot]) Basic Datastore for writing to an in memory cache.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.inMemoryDatastore.StoredMemoryItemInfo, lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore

lsst.daf.butler.datastores.chainedDatastore Module

Classes
ChainedDatastore(config[, registry, butlerRoot]) Chained Datastores to allow read and writes from multiple datastores.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.chainedDatastore.ChainedDatastore

Example registries

lsst.daf.butler.registries.sqlRegistry Module

Classes
SqlRegistryConfig([other, validate, …])
SqlRegistry(registryConfig, schemaConfig, …) Registry backed by a SQL database.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.registries.sqlRegistry.SqlRegistryConfig, lsst.daf.butler.registries.sqlRegistry.SqlRegistry

Example formatters

lsst.daf.butler.formatters.fileFormatter Module

Classes
FileFormatter(fileDescriptor) Interface for reading and writing files on a POSIX file system.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.fileFormatter.FileFormatter

lsst.daf.butler.formatters.jsonFormatter Module

Classes
JsonFormatter(fileDescriptor) Interface for reading and writing Python objects to and from JSON files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.jsonFormatter.JsonFormatter

lsst.daf.butler.formatters.yamlFormatter Module

Classes
YamlFormatter(fileDescriptor) Interface for reading and writing Python objects to and from YAML files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.yamlFormatter.YamlFormatter

lsst.daf.butler.formatters.pickleFormatter Module

Classes
PickleFormatter(fileDescriptor) Interface for reading and writing Python objects to and from pickle files.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.pickleFormatter.PickleFormatter

Support API

lsst.daf.butler.core.safeFileIo Module

Functions
safeMakeDir(directory) Make a directory in a manner avoiding race conditions
setFileMode(filename) Set a file mode according to the user’s umask
FileForWriteOnceCompareSame(name) Context manager to get a file that can be written only once and all other writes will succeed only if they match the initial write.
SafeFile(name) Context manager to create a file in a manner avoiding race conditions
SafeFilename(name) Context manager for creating a file in a manner avoiding race conditions.
SafeLockedFileForRead(name) Context manager for reading a file that may be locked with an exclusive lock via SafeLockedFileForWrite.
Classes
DoNotWrite
FileForWriteOnceCompareSameFailure
SafeLockedFileForWrite(name) File-like object that is used to create a file if needed, lock it with an exclusive lock, and contain file descriptors to readable and writable versions of the file.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.safeFileIo.DoNotWrite, lsst.daf.butler.core.safeFileIo.FileForWriteOnceCompareSameFailure, lsst.daf.butler.core.safeFileIo.SafeLockedFileForWrite

lsst.daf.butler.core.utils Module

Functions
allSlots(self) Return combined __slots__ for all classes in objects mro.
getClassOf(typeOrName) Given the type name or a type, return the python type.
getFullTypeName(cls) Return full type name of the supplied entity.
getInstanceOf(typeOrName, *args, **kwargs) Given the type name or a type, instantiate an object of that type.
getObjectSize(obj[, seen]) Recursively finds size of objects.
immutable(cls) A class decorator that simulates a simple form of immutability for the decorated class.
iterable(a) Make input iterable.
slotValuesAreEqual(self, other) Test for equality by the contents of all slots, including those of its parents.
slotValuesToHash(self) Generate a hash from slot values.
stripIfNotNone(s) Strip leading and trailing whitespace if the given object is not None.
transactional(func) Decorator that wraps a method and makes it transactional.
Classes
IndexedTupleDict An immutable mapping that combines a tuple of values with a (possibly shared) mapping from key to tuple index.
NamedKeyDict(*args) A dictionary wrapper that require keys to have a .name attribute, and permits lookups using either key objects or their names.
NamedValueSet(elements) A custom mutable set class that requires elements to have a .name attribute, which can then be used as keys in dict-like lookup.
PrivateConstructorMeta A metaclass that disables regular construction syntax.
Singleton Metaclass to convert a class to a Singleton.
Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.utils.IndexedTupleDict, lsst.daf.butler.core.utils.NamedKeyDict, lsst.daf.butler.core.utils.NamedValueSet, lsst.daf.butler.core.utils.PrivateConstructorMeta, lsst.daf.butler.core.utils.Singleton

lsst.daf.butler.core.repoRelocation Module

Functions
replaceRoot(configRoot, butlerRoot) Update a configuration root with the butler root location.
Variables
BUTLER_ROOT_TAG The special string to be used in configuration files to indicate that the butler root location should be used.