lsst.pipe.base¶
The lsst.pipe.base
module provides base classes for the task framework.
Tasks package the algorithmic units of the LSST Science Pipelines.
You can create, configure, and run tasks with their Python APIs.
Some tasks, called pipeline tasks, can be packaged into data processing pipelines that you can run from the command line.
Changes¶
Using lsst.pipe.base¶
Overview¶
Developing tasks and pipeline tasks¶
Developing Pipelines¶
Contributing¶
lsst.pipe.base
is developed at https://github.com/lsst/pipe_base.
You can find Jira issues for this module under the pipe_base component.
Python API reference¶
lsst.pipe.base Package¶
Functions¶
|
Create an execution butler. |
|
Create an iterator over the selected connections type which yields all the defined connections of that type. |
Classes¶
|
Helper class for calling |
Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate a repeatable algorithmic failure that will not be addressed by retries. |
|
Exception that runQuantum raises when the (partial) outputs it has written contain information about their own incompleteness or degraded quality. |
|
|
A wrapper class for |
|
A description of the resources available to a running quantum. |
|
Protocol for objects that can extract a possibly-nested mapping of primitives. |
|
Protocol for objects that can assign and extract a possibly-nested mapping of primitives. |
|
An in-memory version of a |
Exception class to indicate that a lookup by NodeId is impossible due to incompatibilities. |
|
|
Input variant of a |
|
Base class for instrument-specific logic for the Gen3 Butler. |
Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate logic bug or configuration problem. |
|
|
A structure to specify a subset of labels to load. |
An exception raised when a Quantum should not exist because there is no work for it to do. |
|
|
Deprecated, this class is used with QuantumGraph save formats of 1 and 2 when unpicking objects and must be retained until those formats are considered unloadable. |
|
A |
|
Config associated with a |
|
Output variant of a |
|
A |
|
Base class for all pipeline tasks. |
|
Configuration class for |
|
PipelineTaskConnections is a class used to declare desired IO when a PipelineTask is run by an activator. |
|
Report for the QuantumGraph creation or reading. |
|
Quanta information summarized for single PipelineTask. |
|
A Namespace to map defined variable names of connections to the associated |
|
A Butler-like class specialized for a single quantum along with context information that can influence how the task is executed. |
|
QuantumGraph is a directed acyclic graph of |
|
Class representing a node in the quantum graph. |
|
Flags that add caveats to a "successful" quantum. |
Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate that a repeatable problem that will not be addressed by retries. |
|
Exception raised when dataset type is configured as scalar but there are multiple data IDs in a Quantum for that dataset. |
|
|
Protocol for objects that can be assigned a possibly-nested |
|
A container to which you can add fields as attributes. |
|
Base class for data processing tasks. |
|
TaskDef is a collection of information about task needed by Pipeline. |
Use to report errors for which a traceback is not useful. |
|
Abstract base class for task factory. |
|
|
Dict-like object for storing task metadata. |
A specialization of |
|
A specialization of |
Class Inheritance Diagram¶
lsst.pipe.base.pipeline_graph Package¶
Functions¶
|
Compare two versions of Packages. |
|
Log messages about configuration mismatch. |
Classes¶
Exception raised when the tasks in a pipeline graph use different (and incompatible) connection types for the same dataset type. |
|
|
A node in a pipeline graph that represents a resolved dataset type. |
Exception raised when multiple tasks in one pipeline produce the same output dataset type. |
|
|
Base class for edges in a pipeline graph. |
Exception raised when the edges in one version of a pipeline graph are not consistent with those in another, but they were expected to be. |
|
Exception raised when the tasks in a pipeline graph define dataset types with the same name in incompatible ways, or when these are incompatible with the data repository definition. |
|
Exception raised when the step definitions are invalid. |
|
|
A special key type for nodes in networkx graphs. |
|
Enumeration of the types of nodes in a PipelineGraph. |
Exception raised when a pipeline graph contains a cycle. |
|
|
A graph representation of fully-configured pipeline. |
Base exception raised when there is a problem constructing or resolving a pipeline graph. |
|
Exception raised when a PipelineGraph method could not provide strong exception safety, and the graph may have been left in an inconsistent state. |
|
Exception raised when a serialized PipelineGraph cannot be read. |
|
|
Representation of an input connection (including init-inputs and prerequisites) in a pipeline graph. |
|
A collection of the 'steps' defined in a pipeline graph. |
|
Enumeration of the ways to handle importing tasks when reading a serialized PipelineGraph. |
|
A node in a pipeline graph that represents the construction of a |
|
A node in a pipeline graph that represents a labeled configuration of a |
Exception raised when accessing an attribute of a graph or graph node that is not available unless the task class has been imported and configured. |
|
|
A specialized set that represents a labeled subset of the tasks in a pipeline graph. |
Exception raised when an operation requires dimensions or dataset types to have been resolved, but they have not been. |
|
|
Representation of an output connection (including init-outputs) in a pipeline graph. |
Class Inheritance Diagram¶
lsst.pipe.base.testUtils Module¶
Functions¶
|
Test that a constructed task conforms to its own init-connections. |
|
Test that the output of a call to |
|
Return the initInputs object that would have been passed to a |
|
Inspect a connections class for common errors. |
|
Create a Quantum for a particular data ID(s). |
|
Run a PipelineTask on a Quantum. |
lsst.pipe.base.connectionTypes Module¶
Module defining connection types to be used within a
PipelineTaskConnections
class.
Classes¶
|
Base class used for declaring |
|
Connection for initInput dataset. |
|
Connection for initOutput dataset. |
|
Class used for declaring PipelineTask input connections. |
|
Connection for output dataset. |
|
Class used for declaring PipelineTask prerequisite connections. |
Class Inheritance Diagram¶
lsst.pipe.base.pipelineIR Module¶
Classes¶
|
Intermediate representation of configurations read from a pipeline yaml file. |
An exception that is raised when a pipeline contract is not satisfied. |
|
|
Intermediate representation of configuration contracts read from a pipeline yaml file. |
|
An intermediate representation of imported pipelines. |
|
Intermediate representation of named subset of task labels read from a pipeline yaml file. |
|
Intermediate representation of parameters that are global to a pipeline. |
|
Intermediate representation of a pipeline definition. |
|
Intermediate representation of tasks read from a pipeline yaml file. |
Class Inheritance Diagram¶
lsst.pipe.base.tests.mocks Package¶
A system for replacing the tasks in a pipeline with mocks that just read and write trivial datasets.
See Testing pipelines with mocks for details.
Functions¶
|
Return the name of the mock storage class, dataset type, or task label for the given original name. |
|
Return the name of the original storage class, dataset type, or task label that corresponds to the given mock name. |
|
Return whether the given name is that of a mock storage class, dataset type, or task label. |
|
Create mocks for a full pipeline graph. |
Classes¶
|
A marker class that represents a conversion from a regular in-memory dataset to a mock storage class. |
|
Class that can match DataId against the user-defined string expression. |
|
A config class that defines a completely dynamic connection. |
|
A test-utility implementation of |
|
Configuration for DynamicTestPipelineTask. |
|
Information about an exception that should be raised by one or more quanta. |
A subclass of |
|
|
The in-memory dataset type used by |
|
Description of the quantum that produced a mock dataset. |
|
A test-utility implementation of |
|
Configuration class for |
|
A reimplementation of |
|
Implementation of the StorageClassDelegate interface for mock datasets. |
Class Inheritance Diagram¶
lsst.pipe.base.execution_reports Module¶
Functions¶
|
Look up a dataId from a quantum graph and a list of quantum graph nodeIDs. |
Classes¶
|
A report on the number of produced datasets as well as the status of missing datasets based on metadata. |
|
A report on the execution of a quantum graph. |
|
A report on the status and content of a task in an executed quantum graph. |
Class Inheritance Diagram¶
QuantumGraph generation API reference¶
lsst.pipe.base.quantum_graph_builder Module¶
The base class for the QuantumGraph-generation algorithm and various helper classes.
Classes¶
|
Struct that holds the results of empty-dimensions dataset queries for |
Exception generated when output datasets already exist. |
|
Exception generated when a prerequisite dataset does not exist. |
|
|
An abstract base class for building |
Base class for exceptions generated by QuantumGraphBuilder. |
Class Inheritance Diagram¶
lsst.pipe.base.quantum_graph_skeleton Module¶
An under-construction version of QuantumGraph and various helper classes.
Classes¶
|
Identifier type for dataset keys in a |
Identifier type for prerequisite dataset keys in a |
|
|
An under-construction quantum graph. |
|
Identifier type for quantum keys in a |
|
Identifier type for task init keys in a |
Class Inheritance Diagram¶
lsst.pipe.base.prerequisite_helpers Module¶
Helper classes for finding prerequisite input datasets during QuantumGraph generation.
Classes¶
|
A QuantumGraph-generation helper class that manages the spatial and temporal bounds of a tasks' quanta, for the purpose of finding prerequisite inputs. |
|
A QuantumGraph-generation helper class that manages the searches for a prerequisite input connection. |
|
A QuantumGraph-generation helper class that manages the searches for all prerequisite input connections for a task. |
A base class for objects that accumulate the appropriate spatial bounds for a quantum. |
|
A base class for objects that accumulate the appropriate timespan for a quantum. |
Class Inheritance Diagram¶
lsst.pipe.base.all_dimensions_quantum_graph_builder Module¶
The standard, general-purpose implementation of the QuantumGraph-generation algorithm.
Classes¶
An implementation of |
Class Inheritance Diagram¶
lsst.pipe.base._datasetQueryConstraints Module¶
Symbols defined in this package should be imported from
all_dimensions_quantum_graph_builder
instead; it only appears in the docs
due to limitations in Sphinx.
Classes¶
Base for all the valid variants for controlling constraining graph building queries based on dataset type existence. |