lsst.daf.butler

This module provides an abstracted data access interface.

Python API reference

lsst.daf.butler Package

Data Access Butler

Functions

disableWhenLimited(func) Decorator that indicates that a method should raise NotImplementedError on Registries whose limited attribute is True.
makeBoxWcsRegion(box, wcs, margin) Construct a spherical ConvexPolygon from a WCS and a bounding box.
toNameSet(container) Transform an iterable of DimensionElement or names thereof into a DimensionNameSet.

Classes

Butler([config, collection, run]) Main entry point for the data access system.
ButlerConfig([other]) Contains the configuration for a Butler
CompositeAssembler(storageClass) Class for providing assembler and disassembler support for composites.
CompositesConfig([other, validate, …])
CompositesMap(config) Determine whether a specific datasetType or StorageClass should be disassembled.
Config([other]) Implements a datatype that is used by Butler for configuration parameters.
ConfigSubset([other, validate, …]) Config representing a subset of a more general configuration.
DataId([dataId, dimensions, dimension, …]) A dict-like identifier for data usable across multiple collections and DatasetTypes.
DatabaseDict(config, types, key, value) An abstract base class for dict-like objects with a specific key type and namedtuple values, backed by a database.
DatasetComponent(name, storageClass, component) Component of a dataset and associated information.
DatasetOriginInfo Interface for classes providing collection information for pre-flight.
DatasetOriginInfoDef(defaultInputs, …[, …]) Default implementation of the DatasetOriginInfo.
DatasetRef(datasetType, dataId[, id, run]) Reference to a Dataset in a Registry.
DatasetType(name, dimensions, storageClass) A named category of Datasets that defines how they are organized, related, and stored.
DatasetTypeNotSupportedError A DatasetType is not handled by this routine.
Datastore(config, registry) Datastore interface.
DatastoreConfig([other, validate, …])
Dimension(universe, name, config) A discrete dimension of data used to organize Datasets and associate them with metadata.
DimensionConfig([other, validate, …])
DimensionElement(universe, name, hasRegion, …) Base class for elements in the dimension schema.
DimensionGraph(universe[, dimensions, …]) An automatically-expanded collection of both Dimensions and DimensionJoins.
DimensionJoin(universe, name, config) A join that relates two or more Dimensions.
DimensionNameSet(names) An incomplete, name-only stand-in for DimensionSet or DimensionGraph.
DimensionSet(universe, elements[, expand, …]) A custom set/dict hybrid class for collections of DimensionElements.
FileDescriptor(location, storageClass[, …]) Describes a particular file.
FileTemplate(template) Format a path template into a fully expanded path.
FileTemplates(config[, default]) Collection of FileTemplate templates.
FileTemplatesConfig([other]) Configuration information for FileTemplates
Formatter Interface for reading and writing Datasets with a particular StorageClass.
FormatterFactory() Factory for Formatter instances.
Location(datastoreRoot, uri) Identifies a location in the Datastore.
LocationFactory(datastoreRoot) Factory for Location instances.
MappingFactory(refType) Register the mapping of some key to a python type and retrieve instances.
PreFlightDimensionsRow(dataId, datasetRefs) Simple data class holding combination of Dimension values for one row returned by pre-flight solver.
Quantum(task, run, *args, **kwargs) A discrete unit of work that may depend on one or more datasets and produces one or more datasets.
Registry(registryConfig[, schemaConfig, …]) Registry interface.
RegistryConfig([other, validate, …])
Run(collection[, environment, pipeline]) Represent a processing run.
Schema([config, limited]) The SQL schema for a Butler Registry.
SchemaBuilder(config[, limited]) Builds a Schema step-by-step.
SchemaConfig([other, validate, …])
StorageClass([name, pytype, components, …]) Class describing how a label maps to a particular Python type.
StorageClassConfig([other, validate, …])
StorageClassFactory([config]) Factory for StorageClass instances.
StoredFileInfo(formatter, path, storageClass) Information associated with a stored file in a Datastore.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.butler.Butler, lsst.daf.butler.core.butlerConfig.ButlerConfig, lsst.daf.butler.core.assembler.CompositeAssembler, lsst.daf.butler.core.composites.CompositesConfig, lsst.daf.butler.core.composites.CompositesMap, lsst.daf.butler.core.config.Config, lsst.daf.butler.core.config.ConfigSubset, lsst.daf.butler.core.dimensions.dataId.DataId, lsst.daf.butler.core.databaseDict.DatabaseDict, lsst.daf.butler.core.assembler.DatasetComponent, lsst.daf.butler.core.preFlight.DatasetOriginInfo, lsst.daf.butler.core.preFlight.DatasetOriginInfoDef, lsst.daf.butler.core.datasets.DatasetRef, lsst.daf.butler.core.datasets.DatasetType, lsst.daf.butler.core.exceptions.DatasetTypeNotSupportedError, lsst.daf.butler.core.datastore.Datastore, lsst.daf.butler.core.datastore.DatastoreConfig, lsst.daf.butler.core.dimensions.elements.Dimension, lsst.daf.butler.core.dimensions.graph.DimensionConfig, lsst.daf.butler.core.dimensions.elements.DimensionElement, lsst.daf.butler.core.dimensions.graph.DimensionGraph, lsst.daf.butler.core.dimensions.elements.DimensionJoin, lsst.daf.butler.core.dimensions.sets.DimensionNameSet, lsst.daf.butler.core.dimensions.sets.DimensionSet, lsst.daf.butler.core.fileDescriptor.FileDescriptor, lsst.daf.butler.core.fileTemplates.FileTemplate, lsst.daf.butler.core.fileTemplates.FileTemplates, lsst.daf.butler.core.fileTemplates.FileTemplatesConfig, lsst.daf.butler.core.formatter.Formatter, lsst.daf.butler.core.formatter.FormatterFactory, lsst.daf.butler.core.location.Location, lsst.daf.butler.core.location.LocationFactory, lsst.daf.butler.core.mappingFactory.MappingFactory, lsst.daf.butler.core.preFlight.PreFlightDimensionsRow, lsst.daf.butler.core.quantum.Quantum, lsst.daf.butler.core.registry.Registry, lsst.daf.butler.core.registry.RegistryConfig, lsst.daf.butler.core.run.Run, lsst.daf.butler.core.schema.Schema, lsst.daf.butler.core.schema.SchemaBuilder, lsst.daf.butler.core.schema.SchemaConfig, lsst.daf.butler.core.storageClass.StorageClass, lsst.daf.butler.core.storageClass.StorageClassConfig, lsst.daf.butler.core.storageClass.StorageClassFactory, lsst.daf.butler.core.storedFileInfo.StoredFileInfo

Example Datastores

lsst.daf.butler.datastores.posixDatastore Module

POSIX datastore.

Classes

PosixDatastore(config, registry) Basic POSIX filesystem backed Datastore.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.posixDatastore.PosixDatastore

lsst.daf.butler.datastores.inMemoryDatastore Module

In-memory datastore.

Classes

StoredItemInfo(timestamp, storageClass[, …]) Internal Metadata associated with a DatasetRef.
InMemoryDatastore(config[, registry]) Basic Datastore for writing to an in memory cache.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.inMemoryDatastore.StoredItemInfo, lsst.daf.butler.datastores.inMemoryDatastore.InMemoryDatastore

lsst.daf.butler.datastores.chainedDatastore Module

Chained datastore.

Classes

ChainedDatastore(config[, registry]) Chained Datastores to allow read and writes from multiple datastores.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.datastores.chainedDatastore.ChainedDatastore

Example Registries

lsst.daf.butler.registries.sqlRegistry Module

Classes

SqlRegistryConfig([other, validate, …])
SqlRegistry(registryConfig, schemaConfig, …) Registry backed by a SQL database.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.registries.sqlRegistry.SqlRegistryConfig, lsst.daf.butler.registries.sqlRegistry.SqlRegistry

Example Formatters

lsst.daf.butler.formatters.fileFormatter Module

Support for reading and writing files to a POSIX file system.

Classes

FileFormatter Interface for reading and writing files on a POSIX file system.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.fileFormatter.FileFormatter

lsst.daf.butler.formatters.jsonFormatter Module

Classes

JsonFormatter Interface for reading and writing Python objects to and from JSON files.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.jsonFormatter.JsonFormatter

lsst.daf.butler.formatters.yamlFormatter Module

Classes

YamlFormatter Interface for reading and writing Python objects to and from YAML files.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.yamlFormatter.YamlFormatter

lsst.daf.butler.formatters.pickleFormatter Module

Formatter associated with Python pickled objects.

Classes

PickleFormatter Interface for reading and writing Python objects to and from pickle files.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.formatters.pickleFormatter.PickleFormatter

Support API

lsst.daf.butler.core.safeFileIo Module

Utilities for safe file IO

Functions

safeMakeDir(directory) Make a directory in a manner avoiding race conditions
setFileMode(filename) Set a file mode according to the user’s umask
FileForWriteOnceCompareSame(name) Context manager to get a file that can be written only once and all other writes will succeed only if they match the initial write.
SafeFile(name) Context manager to create a file in a manner avoiding race conditions
SafeFilename(name) Context manager for creating a file in a manner avoiding race conditions.
SafeLockedFileForRead(name) Context manager for reading a file that may be locked with an exclusive lock via SafeLockedFileForWrite.

Classes

DoNotWrite
FileForWriteOnceCompareSameFailure
SafeLockedFileForWrite(name) File-like object that is used to create a file if needed, lock it with an exclusive lock, and contain file descriptors to readable and writable versions of the file.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.safeFileIo.DoNotWrite, lsst.daf.butler.core.safeFileIo.FileForWriteOnceCompareSameFailure, lsst.daf.butler.core.safeFileIo.SafeLockedFileForWrite

lsst.daf.butler.core.utils Module

Functions

iterable(a) Make input iterable.
allSlots(self) Return combined __slots__ for all classes in objects mro.
slotValuesAreEqual(self, other) Test for equality by the contents of all slots, including those of its parents.
slotValuesToHash(self) Generate a hash from slot values.
getFullTypeName(cls) Return full type name of the supplied entity.
getInstanceOf(typeOrName) Given the type name or a type, instantiate an object of that type.
transactional(func) Decorator that wraps a method and makes it transactional.
getObjectSize(obj[, seen]) Recursively finds size of objects.
stripIfNotNone(s) Strip leading and trailing whitespace if the given object is not None.

Classes

Singleton Metaclass to convert a class to a Singleton.
PrivateConstructorMeta A metaclass that disables regular construction syntax.

Class Inheritance Diagram

Inheritance diagram of lsst.daf.butler.core.utils.Singleton, lsst.daf.butler.core.utils.PrivateConstructorMeta