lsst.daf.butler¶
Changes¶
Using the Butler¶
This module provides an abstracted data access interface, known as the Butler. It can be used to read and write data without having to know the details of file formats or locations.
The Dimensions System¶
Concrete Storage Classes¶
Butler Command-Line Reference¶
Design and Development¶
lsst.daf.butler
is developed at https://github.com/lsst/daf_butler.
You can find Jira issues for this module under the daf_butler component.
Butler Command Line Interface Development¶
Python API reference¶
lsst.daf.butler Package¶
Functions¶
|
Add a field and possibly a foreign key to a table specification. |
Classes¶
|
An abstract base class for managing caching in a Datastore. |
Raised when a |
|
|
Main entry point for the data access system. |
|
Contains the configuration for a |
|
Class representing a collection of |
Index of all known butler repositories. |
|
There is a problem with the Butler configuration. |
|
|
An ordered search path of collections. |
|
Enumeration used to label different types of collections. |
|
Configuration specifics for Composites. |
|
Determine whether something should be disassembled. |
|
Implements a datatype that is used by |
|
Config representing a subset of a more general configuration. |
|
Determine whether an entity is allowed to be handled. |
|
Configuration information for |
Thrown when a constraints list has mutually exclusive definitions. |
|
Data ID dictionary. |
|
An abstract base class for homogeneous iterables of data IDs. |
|
|
Iterable supporting the full Sequence interface. |
|
Iterable iteration that is set-like. |
|
A |
|
A combination class that maps directly to a database table or query. |
|
An intermediate base class for |
|
Database topological family implementation. |
|
Class representing the membership of a dataset in a single collection. |
|
Component of a dataset and associated information. |
|
This enum is used to specify dataset ID generation options for |
|
Reference to a Dataset in a |
|
Represents the primary and component ResourcePath(s) associated with a DatasetRef. |
|
A named category of Datasets. |
A |
|
|
Datastore interface. |
|
A class for managing caching in a Datastore using local files. |
|
Configuration information for |
|
Configuration for Datastores. |
|
A variant of the datastore cache where no cache is enabled. |
|
A struct that represents a tabular data export from a single datastore. |
There is a problem with the Datastore configuration. |
|
|
Proxy class that provides deferred loading of datasets from a butler. |
A dimension. |
|
Element with extra information. |
|
|
Configuration that defines a |
A label and/or metadata in the dimensions system. |
|
|
An immutable, dependency-complete collection of dimensions. |
|
Class for going from |
|
Base class for the Python representation of database records. |
Class used to accumulate dimension records for serialization. |
|
|
Self-consistent set of dimensions. |
|
A struct that represents a dataset exported to a file. |
|
Describes a particular file. |
|
Format a path template into a fully expanded path. |
Exception for file template inconsistent with associated DatasetType. |
|
|
Collection of |
|
Configuration information for |
|
Interface for reading and writing Datasets. |
Factory for |
|
|
Governor dimension. |
|
Identifies a location within the |
|
Factory for |
|
Representation of key that can be used to lookup information. |
|
Register the mapping of some key to a python type and retrieve instances. |
|
A lightweight implementation of |
|
Dictionary wrapper for named keys. |
Custom mapping class. |
|
Custom sets with named elements. |
|
Mutable variant of |
|
|
Custom mutable set class. |
|
Public interface for reporting incremental progress in the butler and related tools. |
Base class for errors relating to Butler.pruneCollections input arguments. |
|
Raised when purge is True but is not supported for the given collection. |
|
Raised when purge and unstore are both required to be True, and purge is True but unstore is False. |
|
|
Class representing a discrete unit of work. |
|
An implementation of |
|
A serializable struct for per-quantum provenance information and datastore records. |
|
Abstract Registry interface. |
|
|
Raised when pruning a RUN collection but purge is False. |
|
|
Simplified model for serializing a |
|
Simplified model of a |
|
Simplified model of a |
|
Representation of a |
|
Simplified model of a |
|
Simplified model for serializing a |
|
Simplified model of a |
A struct that combines SQLAlchemy objects. |
|
|
Special dimension for sky pixelizations. |
|
Class for hierarchical pixelization of the sky. |
|
Class reflecting how spatial regions are represented inside the DB. |
|
Class describing how a label maps to a particular Python type. |
|
Configuration class for defining Storage Classes. |
|
Delegate class for StorageClass components and parameters. |
Factory for |
|
Internal information associated with a stored dataset in a |
|
|
Datastore-private metadata associated with a Datastore file. |
|
A half-open time interval with nanosecond precision. |
Representation of a time span within a database engine. |
|
Mapping of in-memory representation of a region to DB representation. |
|
|
A grouping of |
Representation of a logical table that can participate in overlap joins. |
|
|
Enumeration of continuous-variable relationships for dimensions. |
Some sort of validation error has occurred. |
|
|
A repository export implementation that saves to a YAML file. |
|
A repository import implementation that reads from a YAML file. |
Class Inheritance Diagram¶
- class lsst.daf.butler.ButlerURI(uri)¶
ButlerURI
implementation. Exists for backwards compatibility. New code should uselsst.resources.ResourcePath
.
lsst.daf.butler.registry Package¶
Classes¶
Exception raised when method arguments are invalid or inconsistent. |
|
Exception raised for collection-related errors. |
|
Exception raised for an incorrect collection expression. |
|
|
An ordered search path of collections. |
|
Enumeration used to label different types of collections. |
Exception raised when type of a collection is incorrect. |
|
Exception raised when trying to insert a database record when a conflicting record already exists. |
|
Exception raised for incorrect data IDs, this is a base class for other specific error types. |
|
Exception raised when a value specified in a data ID does not exist. |
|
|
This enum is used to specify dataset ID generation options for |
Exception raised for problems with dataset types. |
|
Exception raised for an incorrect dataset type expression. |
|
|
Retrieves authentication information for database connections. |
Exception raised when a problem has occurred retrieving database authentication information. |
|
Credentials file has incorrect permissions. |
|
Exception raised when a dimension specified in a data ID does not exist or required dimension is not provided. |
|
Exception raised when a data ID contains contradictory key-value pairs, according to dimension relationships. |
|
Exception raised when an operation attempts to use a collection that does not exist. |
|
Exception raised when a collection is needed, but collection argument is not provided and default collection is not defined in registry. |
|
Exception raised when trying to remove or modify a database record that is still being used in some other table. |
|
|
Abstract Registry interface. |
|
|
|
A struct used to provide the default collections searched or written to by a |
Base class for many exception classes produced by Registry methods. |
|
Exception raised when an unsupported |
|
Exception raised for problems with user expression. |
|
Exception raised when a user query expression cannot be parsed. |
Class Inheritance Diagram¶
lsst.daf.butler.registry.interfaces Package¶
Classes¶
Exception raised when trying to update existing attribute without specifying |
|
An interface for managing butler attributes in a |
|
|
A subclass of |
An interface for managing the collections (including runs) in a |
|
|
A struct used to represent a collection in internal |
|
An abstract interface that represents a particular database engine's representation of a single schema/namespace/database. |
Exception raised when database content (row values or schema entities) are inconsistent with what this client expects. |
|
A base class for objects that manage overlaps between a pair of database-backed dimensions. |
|
Intermediate interface for |
|
|
This enum is used to specify dataset ID generation options for |
|
An interface that manages the records associated with a particular |
An interface that manages the tables that describe datasets. |
|
|
An abstract base class that defines the interface that a |
|
An abstract base class that defines the interface between |
An abstract base class that represents a way of storing the records associated with a single |
|
|
An interface for managing the dimension records in a |
|
A fake |
Intermediate interface for |
|
|
An interface that manages the records associated with a particular opaque table in a |
An interface that manages the opaque tables in a |
|
Exception raised when a write operation is called on a read-only |
|
|
A subclass of |
Exception raised when trying to initialize database schema when some tables already exist. |
|
Intermediate interface for |
|
|
Helper class used to declare the static schema for a registry layer in a database. |
|
Class representing a version number. |
Interface for extension classes with versions. |
Class Inheritance Diagram¶
lsst.daf.butler.registry.queries Package¶
Classes¶
|
A |
|
An enhanced implementation of |
|
Implementation of DimensionRecordQueryResults using database query. |
An interface for objects that represent the results of queries for datasets. |
|
An interface for objects that represent the results of queries for dimension records. |
|
|
An object that represents results from a query for datasets with a single parent |
|
An abstract base class for queries that return some combination of |
|
A builder for potentially complex queries that join tables based on dimension relationships. |
|
A struct that holds and categorizes the dimensions involved in a query. |
|
Struct used to pass around the manager objects that back a |
Class Inheritance Diagram¶
lsst.daf.butler.registry.summaries Module¶
Classes¶
|
A summary of the datasets that can be found in a collection. |
|
A custom mapping that represents a restriction on the values one or more governor dimensions may take in some context. |
Class Inheritance Diagram¶
lsst.daf.butler.registry.wildcards Module¶
Classes¶
|
The results of preprocessing a wildcard expression to separate match patterns from strings. |
|
An unordered query for collections and dataset type restrictions. |
|
An ordered search path of collections. |
Class Inheritance Diagram¶
Example datastores¶
lsst.daf.butler.datastores.chainedDatastore Module¶
Classes¶
|
Chained Datastores to allow read and writes from multiple datastores. |
Class Inheritance Diagram¶
lsst.daf.butler.datastores.inMemoryDatastore Module¶
Classes¶
|
Internal InMemoryDatastore Metadata associated with a stored DatasetRef. |
|
Basic Datastore for writing to an in memory cache. |
Class Inheritance Diagram¶
lsst.daf.butler.datastores.fileDatastore Module¶
Classes¶
|
Generic Datastore for file-based implementations. |
Class Inheritance Diagram¶
Example formatters¶
lsst.daf.butler.formatters.file Module¶
Classes¶
|
Interface for reading and writing files on a POSIX file system. |
Class Inheritance Diagram¶
lsst.daf.butler.formatters.json Module¶
Classes¶
|
Formatter implementation for JSON files. |
Class Inheritance Diagram¶
lsst.daf.butler.formatters.matplotlib Module¶
Classes¶
|
Interface for writing matplotlib figures. |
Class Inheritance Diagram¶
lsst.daf.butler.formatters.parquet Module¶
Classes¶
|
Interface for reading and writing Pandas DataFrames to and from Parquet files. |
Class Inheritance Diagram¶
lsst.daf.butler.formatters.pickle Module¶
Classes¶
|
Interface for reading and writing Python objects to and from pickle files. |
Class Inheritance Diagram¶
lsst.daf.butler.formatters.yaml Module¶
Classes¶
|
Formatter implementation for YAML files. |
Class Inheritance Diagram¶
Database backends¶
lsst.daf.butler.registry.databases.sqlite Module¶
Classes¶
|
An implementation of the |
Class Inheritance Diagram¶
lsst.daf.butler.registry.databases.postgresql Module¶
Classes¶
|
An implementation of the |
Class Inheritance Diagram¶
Support API¶
lsst.daf.butler.core.utils Module¶
Functions¶
Strip leading and trailing whitespace if the given object is not None. |
|
|
Decorate a method and makes it transactional. |
lsst.daf.butler.core.repoRelocation Module¶
Functions¶
|
Update a configuration root with the butler root location. |
Variables¶
The special string to be used in configuration files to indicate that the butler root location should be used. |
Test utilities¶
lsst.daf.butler.tests Package¶
Functions¶
|
Add the records that back a new data ID to a repository. |
|
Add a new dataset type to a repository. |
|
Return a complete data ID matching some criterion. |
|
Create a read/write Butler to a fresh collection. |
|
Create an empty test repository. |
|
Modify a repository to support reading and writing |
Classes¶
|
A formatter that always fails without writing anything. |
|
A formatter that never works but does leave a file behind. |
A test case base that is used to verify click command functions import and call their respective script functions correctly. |
|
Tests log initialization, reset, and setting log levels. |
|
Helper methods for Datasets |
|
Mocks a butler datastore. |
|
Helper methods for Datastore tests |
|
Dummy Registry, for Datastore test purposes. |
|
|
Parameter handler for list parameters |
|
Parameter handler for parameters using Metrics |
|
Smorgasboard of information that might be the result of some processing. |
|
A variant of |
|
Command Line Inteface API¶
Warning
The command line interface API (everything in lsst.daf.butler.cli
) is for only for developer use to write command line interfaces, and is not intended for general use.
lsst.daf.butler.cli.butler Module¶
Functions¶
|
Classes¶
|
Extends |
|
Class Inheritance Diagram¶
lsst.daf.butler.cli.cmd Package¶
Variables¶
Add existing datasets to a tagged collection; searches for datasets with the options and adds them to the named COLLECTION. |
|
Import data into a butler repository. |
|
Certify calibrations in a repository. |
|
Create an empty Gen3 Butler repository. |
|
Define a collection chain. |
|
Dump either a subset or full Butler configuration to standard output. |
|
Validate the configuration files for a Gen3 Butler repository. |
|
Export calibrations from the butler for import elsewhere. |
|
Ingest files from table file. |
|
Remove a collection and possibly prune datasets within it. |
|
Query for and remove one or more datasets from a collection and/or storage. |
|
Get the collections whose names match an expression. |
|
List the data IDs in a repository. |
|
Get the dataset types in a repository. |
|
List the datasets in a repository. |
|
Query for dimension information. |
|
Register a new dataset type with this butler repository. |
|
Retrieve file artifacts associated with datasets in a repository. |
|
Remove one or more non-RUN collections. |
|
Remove one or more RUN collections. |
|
Remove a dataset type definition from a repository. |
|
Transfer datasets from a source butler to a destination butler. |
lsst.daf.butler.cli.opt Package¶
Classes¶
|
Variables¶
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
Class Inheritance Diagram¶
lsst.daf.butler.cli.utils Module¶
Functions¶
|
Render astropy tables to string as they are displayed in the CLI. |
|
Print astropy tables to be displayed in the CLI. |
|
Get the text type string for CLI help documentation. |
|
Get a standard assert message from a click result |
|
A context manager that creates (and then cleans up) an environment that provides a CLI plugin command with the given name. |
|
Add a Click argument's help message to a function's documentation. |
|
Process a tuple of values, where each value may contain comma-separated values, and return a single list of all the passed-in values. |
|
Process a tuple of values that are key-value pairs separated by a given separator. |
|
Convert a value to upper case. |
|
Remove newlines and leading whitespace from a multi-line string with a consistent indentation level. |
|
Click callback that reads additional values from the supplied YAML file. |
|
Sort an astropy table, with prioritization given to columns in this order: 1. |
|
Decorator which catches all exceptions, prints an exception traceback and signals click to exit. |
Classes¶
|
A test runner to use when the logging system will be initialized by code under test, calls CliLog.resetLog(), which undoes any logging setup that was done with the CliLog interface. |
|
Decorator to add a section label between options in the help text of a command. |
|
Overrides click.Path to implement file-does-not-exist checking. |
|
Overrides click.Option with desired behaviors. |
|
Overrides click.Argument with desired behaviors. |
|
Implements an Option that prints a section label in the help text and does not pass any value to the command function. |
|
Wraps the click.option decorator to enable shared options to be declared and allows inspection of the shared option. |
|
Wraps the click.argument decorator to enable shared arguments to be declared. |
|
Command subclass that stores a copy of the args list for use by the command. |
|
Command subclass with butler-command specific overrides. |
Base class for an option group decorator. |
|
|
Helper object for managing the |
Class Inheritance Diagram¶
lsst.daf.butler.cli.cliLog Module¶
Classes¶
|
A log formatter that issues accurate timezone-aware timestamps. |
|
Interface for managing python logging and |
Class Inheritance Diagram¶
lsst.daf.butler.cli.progress Module¶
Classes¶
|
A |