lsst.daf.butler

This module provides an abstracted data access interface.

Python API reference

lsst.daf.butler Package

Data Access Butler

Functions

conformSet(container) Transform an iterable of DimensionElement or names thereof into an object with a set-like .names attribute.
disableWhenLimited(func) Decorator that indicates that a method should raise NotImplementedError on Registries whose limited attribute is True.
makeBoxWcsRegion(box, wcs, margin) Construct a spherical ConvexPolygon from a WCS and a bounding box.

Classes

AmbiguousDatasetError Exception raised when a DatasetRef has no ID and a Registry operation requires one.
Butler([config, collection, run]) Main entry point for the data access system.
ButlerConfig([other]) Contains the configuration for a Butler
CompositeAssembler(storageClass) Class for providing assembler and disassembler support for composites.
CompositesConfig([other, validate, …])
CompositesMap(config) Determine whether a specific datasetType or StorageClass should be disassembled.
Config([other]) Implements a datatype that is used by Butler for configuration parameters.
ConfigSubset([other, validate, …]) Config representing a subset of a more general configuration.
ConflictingDefinitionError Exception raised when trying to insert a database record when a conflicting record already exists.
DataId([dataId, dimensions, dimension, …]) A dict-like identifier for data usable across multiple collections and DatasetTypes.
DataIdPacker An abstract base class for a bidirectional mappings between a DataId and a packed integer or bytes blob.
DataIdPackerDimensions(given, required) A helper class that holds the three categories of dimensions associated with DataIdPacker objects.
DatabaseDict(config, types, key, value) An abstract base class for dict-like objects with a specific key type and namedtuple values, backed by a database.
DatasetComponent(name, storageClass, component) Component of a dataset and associated information.
DatasetOriginInfo Interface for classes providing collection information for pre-flight.
DatasetOriginInfoDef(defaultInputs, …[, …]) Default implementation of the DatasetOriginInfo.
DatasetRef(datasetType, dataId[, id, run, …]) Reference to a Dataset in a Registry.
DatasetType(name, dimensions, storageClass) A named category of Datasets that defines how they are organized, related, and stored.
DatasetTypeNotSupportedError A DatasetType is not handled by this routine.
Datastore(config, registry) Datastore interface.
DatastoreConfig([other, validate, …])
Dimension(universe, name, config) A discrete dimension of data used to organize Datasets and associate them with metadata.
DimensionConfig([other, validate, …])
DimensionElement(universe, name, hasRegion, …) Base class for elements in the dimension schema.
DimensionGraph(universe, elements, …) An automatically-expanded collection of both Dimensions and DimensionJoins.
DimensionJoin(universe, name, config) A join that relates two or more Dimensions.
DimensionKeyDict([other, universe, keys, …]) An immutable mapping that uses DimensionElement instances as keys.
DimensionNameSet(names) An incomplete, name-only stand-in for DimensionSet or DimensionGraph.
DimensionSet(universe, elements[, expand, …]) A custom set/dict hybrid class for collections of DimensionElements.
DimensionUniverse() Specialization for a DimensionGraph that defines a complete system of dimensions.
Execution([startTime, endTime, host, id]) Any step in a production.
FileDescriptor(location, storageClass[, …]) Describes a particular file.
FileTemplate(template) Format a path template into a fully expanded path.
FileTemplates(config[, default]) Collection of FileTemplate templates.
FileTemplatesConfig([other]) Configuration information for FileTemplates
Formatter Interface for reading and writing Datasets with a particular StorageClass.
FormatterFactory() Factory for Formatter instances.
Location(datastoreRoot, uri) Identifies a location in the Datastore.
LocationFactory(datastoreRoot) Factory for Location instances.
MappingFactory(refType) Register the mapping of some key to a python type and retrieve instances.
PreFlightDimensionsRow(dataId, datasetRefs) Simple data class holding combination of Dimension values for one row returned by pre-flight solver.
Quantum(task, run, *args, **kwargs) A discrete unit of work that may depend on one or more datasets and produces one or more datasets.
Registry(registryConfig[, schemaConfig, …]) Registry interface.
RegistryConfig([other, validate, …])
Run(collection[, environment, pipeline]) Represent a processing run.
Schema([config, limited]) The SQL schema for a Butler Registry.
SchemaBuilder(config[, limited]) Builds a Schema step-by-step.
SchemaConfig([other, validate, …])
StorageClass([name, pytype, components, …]) Class describing how a label maps to a particular Python type.
StorageClassConfig([other, validate, …])
StorageClassFactory([config]) Factory for StorageClass instances.
StoredFileInfo(formatter, path, storageClass) Information associated with a stored file in a Datastore.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.registry.AmbiguousDatasetError, lsst.daf.butler.butler.Butler, lsst.daf.butler.core.butlerConfig.ButlerConfig, lsst.daf.butler.core.assembler.CompositeAssembler, lsst.daf.butler.core.composites.CompositesConfig, lsst.daf.butler.core.composites.CompositesMap, lsst.daf.butler.core.config.Config, lsst.daf.butler.core.config.ConfigSubset, lsst.daf.butler.core.registry.ConflictingDefinitionError, lsst.daf.butler.core.dimensions.dataId.DataId, lsst.daf.butler.core.dataIdPacker.DataIdPacker, lsst.daf.butler.core.dataIdPacker.DataIdPackerDimensions, lsst.daf.butler.core.databaseDict.DatabaseDict, lsst.daf.butler.core.assembler.DatasetComponent, lsst.daf.butler.core.preFlight.DatasetOriginInfo, lsst.daf.butler.core.preFlight.DatasetOriginInfoDef, lsst.daf.butler.core.datasets.DatasetRef, lsst.daf.butler.core.datasets.DatasetType, lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError, lsst.daf.butler.core.datastore.Datastore, lsst.daf.butler.core.datastore.DatastoreConfig, lsst.daf.butler.core.dimensions.elements.Dimension, lsst.daf.butler.core.dimensions.graph.DimensionConfig, lsst.daf.butler.core.dimensions.elements.DimensionElement, lsst.daf.butler.core.dimensions.graph.DimensionGraph, lsst.daf.butler.core.dimensions.elements.DimensionJoin, lsst.daf.butler.core.dimensions.dataId.DimensionKeyDict, lsst.daf.butler.core.dimensions.sets.DimensionNameSet, lsst.daf.butler.core.dimensions.sets.DimensionSet, lsst.daf.butler.core.dimensions.graph.DimensionUniverse, lsst.daf.butler.core.execution.Execution, lsst.daf.butler.core.fileDescriptor.FileDescriptor, lsst.daf.butler.core.fileTemplates.FileTemplate, lsst.daf.butler.core.fileTemplates.FileTemplates, lsst.daf.butler.core.fileTemplates.FileTemplatesConfig, lsst.daf.butler.core.formatter.Formatter, lsst.daf.butler.core.formatter.FormatterFactory, lsst.daf.butler.core.location.Location, lsst.daf.butler.core.location.LocationFactory, lsst.daf.butler.core.mappingFactory.MappingFactory, lsst.daf.butler.core.preFlight.PreFlightDimensionsRow, lsst.daf.butler.core.quantum.Quantum, lsst.daf.butler.core.registry.Registry, lsst.daf.butler.core.registry.RegistryConfig, lsst.daf.butler.core.run.Run, lsst.daf.butler.core.schema.Schema, lsst.daf.butler.core.schema.SchemaBuilder, lsst.daf.butler.core.schema.SchemaConfig, lsst.daf.butler.core.storageClass.StorageClass, lsst.daf.butler.core.storageClass.StorageClassConfig, lsst.daf.butler.core.storageClass.StorageClassFactory, lsst.daf.butler.core.storedFileInfo.StoredFileInfo

Example Datastores

lsst.daf.butler.datastores.posixDatastore Module

POSIX datastore.

Classes

PosixDatastore(config, registry) Basic POSIX filesystem backed Datastore.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.posixDatastore.PosixDatastore

lsst.daf.butler.datastores.inMemoryDatastore Module

In-memory datastore.

Classes

StoredItemInfo(timestamp, storageClass[, …]) Internal Metadata associated with a DatasetRef.
InMemoryDatastore(config[, registry]) Basic Datastore for writing to an in memory cache.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.inMemoryDatastore.StoredItemInfo, lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore

lsst.daf.butler.datastores.chainedDatastore Module

Chained datastore.

Classes

ChainedDatastore(config[, registry]) Chained Datastores to allow read and writes from multiple datastores.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.chainedDatastore.ChainedDatastore

Example Registries

lsst.daf.butler.registries.sqlRegistry Module

Classes

SqlRegistryConfig([other, validate, …])
SqlRegistry(registryConfig, schemaConfig, …) Registry backed by a SQL database.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.registries.sqlRegistry.SqlRegistryConfig, lsst.daf.butler.registries.sqlRegistry.SqlRegistry

Example Formatters

lsst.daf.butler.formatters.fileFormatter Module

Support for reading and writing files to a POSIX file system.

Classes

FileFormatter Interface for reading and writing files on a POSIX file system.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.fileFormatter.FileFormatter

lsst.daf.butler.formatters.jsonFormatter Module

Classes

JsonFormatter Interface for reading and writing Python objects to and from JSON files.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.jsonFormatter.JsonFormatter

lsst.daf.butler.formatters.yamlFormatter Module

Classes

YamlFormatter Interface for reading and writing Python objects to and from YAML files.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.yamlFormatter.YamlFormatter

lsst.daf.butler.formatters.pickleFormatter Module

Formatter associated with Python pickled objects.

Classes

PickleFormatter Interface for reading and writing Python objects to and from pickle files.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.pickleFormatter.PickleFormatter

Support API

lsst.daf.butler.core.safeFileIo Module

Utilities for safe file IO

Functions

safeMakeDir(directory) Make a directory in a manner avoiding race conditions
setFileMode(filename) Set a file mode according to the user’s umask
FileForWriteOnceCompareSame(name) Context manager to get a file that can be written only once and all other writes will succeed only if they match the initial write.
SafeFile(name) Context manager to create a file in a manner avoiding race conditions
SafeFilename(name) Context manager for creating a file in a manner avoiding race conditions.
SafeLockedFileForRead(name) Context manager for reading a file that may be locked with an exclusive lock via SafeLockedFileForWrite.

Classes

DoNotWrite
FileForWriteOnceCompareSameFailure
SafeLockedFileForWrite(name) File-like object that is used to create a file if needed, lock it with an exclusive lock, and contain file descriptors to readable and writable versions of the file.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.safeFileIo.DoNotWrite, lsst.daf.butler.core.safeFileIo.FileForWriteOnceCompareSameFailure, lsst.daf.butler.core.safeFileIo.SafeLockedFileForWrite

lsst.daf.butler.core.utils Module

Functions

iterable(a) Make input iterable.
allSlots(self) Return combined __slots__ for all classes in objects mro.
slotValuesAreEqual(self, other) Test for equality by the contents of all slots, including those of its parents.
slotValuesToHash(self) Generate a hash from slot values.
getFullTypeName(cls) Return full type name of the supplied entity.
getInstanceOf(typeOrName) Given the type name or a type, instantiate an object of that type.
transactional(func) Decorator that wraps a method and makes it transactional.
getObjectSize(obj[, seen]) Recursively finds size of objects.
stripIfNotNone(s) Strip leading and trailing whitespace if the given object is not None.

Classes

Singleton Metaclass to convert a class to a Singleton.
PrivateConstructorMeta A metaclass that disables regular construction syntax.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.utils.Singleton, lsst.daf.butler.core.utils.PrivateConstructorMeta