lsst.pipe.base¶
The lsst.pipe.base
module provides base classes for the task framework.
Tasks package the algorithmic units of the LSST Science Pipelines.
You can create, configure, and run tasks with their Python APIs.
Some tasks, called pipeline tasks, can be packaged into data processing pipelines that you can run from the command line.
Changes¶
Using lsst.pipe.base¶
Overview¶
Developing tasks and pipeline tasks¶
Developing Pipelines¶
Contributing¶
lsst.pipe.base
is developed at https://github.com/lsst/pipe_base.
You can find Jira issues for this module under the pipe_base component.
Python API reference¶
lsst.pipe.base Package¶
Functions¶
|
buildExecutionButler is a function that is responsible for exporting input |
|
Creates an iterator over the selected connections type which yields all the defined connections of that type. |
|
Log timer information to |
|
Decorator to measure duration of a method. |
Classes¶
|
Helper class for calling |
|
A Butler-like class specialized for a single quantum |
|
A wrapper class for |
|
GraphBuilder class is responsible for building task execution graph from a Pipeline. |
|
An in-memory version of a |
Exception class to indicate that a lookup by NodeId is impossible due to incompatibilities |
|
|
|
|
Base class for instrument-specific logic for the Gen3 Butler. |
Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate logic bug or configuration problem. |
|
|
A structure to specify a subset of labels to load |
An exception raised when a Quantum should not exist because there is no work for it to do. |
|
|
Deprecated, this class is used with QuantumGraph save formats of 1 and 2 when unpicking objects and must be retained until those formats are considered unloadable. |
|
|
|
A |
|
An immutable struct that classifies the dataset types used in a |
|
Base class for all pipeline tasks. |
|
Configuration class for |
|
PipelineTaskConnections is a class used to declare desired IO when a PipelineTask is run by an activator |
|
QuantumGraph is a directed acyclic graph of |
|
This class represents a node in the quantum graph. |
Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate that a repeatable problem that will not be addressed by retries. |
|
|
Configuration for resource requirements. |
Exception raised when dataset type is configured as scalar but there are multiple data IDs in a Quantum for that dataset. |
|
|
A container to which you can add fields as attributes. |
|
Base class for data processing tasks. |
|
An immutable struct that extracts and classifies the dataset types used by a |
|
TaskDef is a collection of information about task needed by Pipeline. |
Use to report errors for which a traceback is not useful. |
|
Abstract base class for task factory. |
|
|
Dict-like object for storing task metadata. |
Class Inheritance Diagram¶
digraph inheritancef5788eaa78 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AdjustQuantumHelper" [URL="../../../py-api/lsst.pipe.base.AdjustQuantumHelper.html#lsst.pipe.base.AdjustQuantumHelper",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Helper class for calling `PipelineTaskConnections.adjustQuantum`."]; "BaseModel" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "Representation" -> "BaseModel" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ButlerQuantumContext" [URL="../../../py-api/lsst.pipe.base.ButlerQuantumContext.html#lsst.pipe.base.ButlerQuantumContext",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A Butler-like class specialized for a single quantum"]; "Config" [URL="../../../py-api/lsst.pex.config.Config.html#lsst.pex.config.Config",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for configuration (*config*) objects."]; "DeferredDatasetRef" [URL="../../../py-api/lsst.pipe.base.DeferredDatasetRef.html#lsst.pipe.base.DeferredDatasetRef",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A wrapper class for `DatasetRef` that indicates that a `PipelineTask`"]; "GraphBuilder" [URL="../../../py-api/lsst.pipe.base.GraphBuilder.html#lsst.pipe.base.GraphBuilder",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="GraphBuilder class is responsible for building task execution graph from"]; "InMemoryDatasetHandle" [URL="../../../py-api/lsst.pipe.base.InMemoryDatasetHandle.html#lsst.pipe.base.InMemoryDatasetHandle",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An in-memory version of a `~lsst.daf.butler.DeferredDatasetHandle`."]; "IncompatibleGraphError" [URL="../../../py-api/lsst.pipe.base.IncompatibleGraphError.html#lsst.pipe.base.IncompatibleGraphError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exception class to indicate that a lookup by NodeId is impossible due"]; "InputQuantizedConnection" [URL="../../../py-api/lsst.pipe.base.InputQuantizedConnection.html#lsst.pipe.base.InputQuantizedConnection",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "QuantizedConnection" -> "InputQuantizedConnection" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Instrument" [URL="../../../py-api/lsst.pipe.base.Instrument.html#lsst.pipe.base.Instrument",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for instrument-specific logic for the Gen3 Butler."]; "InvalidQuantumError" [URL="../../../py-api/lsst.pipe.base.InvalidQuantumError.html#lsst.pipe.base.InvalidQuantumError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exception that may be raised by PipelineTasks (and code they delegate"]; "LabelSpecifier" [URL="../../../py-api/lsst.pipe.base.LabelSpecifier.html#lsst.pipe.base.LabelSpecifier",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A structure to specify a subset of labels to load"]; "NoWorkFound" [URL="../../../py-api/lsst.pipe.base.NoWorkFound.html#lsst.pipe.base.NoWorkFound",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An exception raised when a Quantum should not exist because there is no"]; "NodeId" [URL="../../../py-api/lsst.pipe.base.NodeId.html#lsst.pipe.base.NodeId",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Deprecated, this class is used with QuantumGraph save formats of"]; "OutputQuantizedConnection" [URL="../../../py-api/lsst.pipe.base.OutputQuantizedConnection.html#lsst.pipe.base.OutputQuantizedConnection",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "QuantizedConnection" -> "OutputQuantizedConnection" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Pipeline" [URL="../../../py-api/lsst.pipe.base.Pipeline.html#lsst.pipe.base.Pipeline",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A `Pipeline` is a representation of a series of tasks to run, and the"]; "PipelineDatasetTypes" [URL="../../../py-api/lsst.pipe.base.PipelineDatasetTypes.html#lsst.pipe.base.PipelineDatasetTypes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An immutable struct that classifies the dataset types used in a"]; "PipelineTask" [URL="../../../py-api/lsst.pipe.base.PipelineTask.html#lsst.pipe.base.PipelineTask",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for all pipeline tasks."]; "Task" -> "PipelineTask" [arrowsize=0.5,style="setlinewidth(0.5)"]; "PipelineTaskConfig" [URL="../../../py-api/lsst.pipe.base.PipelineTaskConfig.html#lsst.pipe.base.PipelineTaskConfig",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration class for `PipelineTask`"]; "Config" -> "PipelineTaskConfig" [arrowsize=0.5,style="setlinewidth(0.5)"]; "PipelineTaskConnections" [URL="../../../py-api/lsst.pipe.base.PipelineTaskConnections.html#lsst.pipe.base.PipelineTaskConnections",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="PipelineTaskConnections is a class used to declare desired IO when a"]; "QuantizedConnection" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A Namespace to map defined variable names of connections to the"]; "SimpleNamespace" -> "QuantizedConnection" [arrowsize=0.5,style="setlinewidth(0.5)"]; "QuantumGraph" [URL="../../../py-api/lsst.pipe.base.QuantumGraph.html#lsst.pipe.base.QuantumGraph",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="QuantumGraph is a directed acyclic graph of `QuantumNode` objects"]; "QuantumNode" [URL="../../../py-api/lsst.pipe.base.QuantumNode.html#lsst.pipe.base.QuantumNode",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="This class represents a node in the quantum graph."]; "RepeatableQuantumError" [URL="../../../py-api/lsst.pipe.base.RepeatableQuantumError.html#lsst.pipe.base.RepeatableQuantumError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exception that may be raised by PipelineTasks (and code they delegate"]; "Representation" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Mixin to provide __str__, __repr__, and __pretty__ methods. See #884 for more details."]; "ResourceConfig" [URL="../../../py-api/lsst.pipe.base.ResourceConfig.html#lsst.pipe.base.ResourceConfig",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration for resource requirements."]; "Config" -> "ResourceConfig" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ScalarError" [URL="../../../py-api/lsst.pipe.base.ScalarError.html#lsst.pipe.base.ScalarError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exception raised when dataset type is configured as scalar"]; "SimpleNamespace" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A simple attribute-based namespace."]; "Struct" [URL="../../../py-api/lsst.pipe.base.Struct.html#lsst.pipe.base.Struct",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A container to which you can add fields as attributes."]; "Task" [URL="../../../py-api/lsst.pipe.base.Task.html#lsst.pipe.base.Task",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for data processing tasks."]; "TaskDatasetTypes" [URL="../../../py-api/lsst.pipe.base.TaskDatasetTypes.html#lsst.pipe.base.TaskDatasetTypes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An immutable struct that extracts and classifies the dataset types used"]; "TaskDef" [URL="../../../py-api/lsst.pipe.base.TaskDef.html#lsst.pipe.base.TaskDef",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="TaskDef is a collection of information about task needed by Pipeline."]; "TaskError" [URL="../../../py-api/lsst.pipe.base.TaskError.html#lsst.pipe.base.TaskError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Use to report errors for which a traceback is not useful."]; "TaskFactory" [URL="../../../py-api/lsst.pipe.base.TaskFactory.html#lsst.pipe.base.TaskFactory",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Abstract base class for task factory."]; "TaskMetadata" [URL="../../../py-api/lsst.pipe.base.TaskMetadata.html#lsst.pipe.base.TaskMetadata",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Dict-like object for storing task metadata."]; "BaseModel" -> "TaskMetadata" [arrowsize=0.5,style="setlinewidth(0.5)"]; }lsst.pipe.base.testUtils Module¶
Functions¶
|
Test that a constructed task conforms to its own init-connections. |
|
Test that the output of a call to |
|
Return the initInputs object that would have been passed to a |
|
Inspect a connections class for common errors. |
|
Create a Quantum for a particular data ID(s). |
|
Run a PipelineTask on a Quantum. |
lsst.pipe.base.connectionTypes Module¶
Classes¶
|
|
|
|
|
Class used for declaring PipelineTask input connections |
|
Class used for declaring PipelineTask prerequisite connections |
|
|
|
Base class used for declaring PipelineTask connections |
Class Inheritance Diagram¶
digraph inheritance465200a9d7 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BaseConnection" [URL="../../../py-api/lsst.pipe.base.connectionTypes.BaseConnection.html#lsst.pipe.base.connectionTypes.BaseConnection",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class used for declaring PipelineTask connections"]; "BaseInput" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Class used for declaring PipelineTask input connections"]; "DimensionedConnection" -> "BaseInput" [arrowsize=0.5,style="setlinewidth(0.5)"]; "DimensionedConnection" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Class used for declaring PipelineTask connections that includes"]; "BaseConnection" -> "DimensionedConnection" [arrowsize=0.5,style="setlinewidth(0.5)"]; "InitInput" [URL="../../../py-api/lsst.pipe.base.connectionTypes.InitInput.html#lsst.pipe.base.connectionTypes.InitInput",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="InitInput(name: str, storageClass: str, doc: str = '', multiple: bool = False)"]; "BaseConnection" -> "InitInput" [arrowsize=0.5,style="setlinewidth(0.5)"]; "InitOutput" [URL="../../../py-api/lsst.pipe.base.connectionTypes.InitOutput.html#lsst.pipe.base.connectionTypes.InitOutput",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="InitOutput(name: str, storageClass: str, doc: str = '', multiple: bool = False)"]; "BaseConnection" -> "InitOutput" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Input" [URL="../../../py-api/lsst.pipe.base.connectionTypes.Input.html#lsst.pipe.base.connectionTypes.Input",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class used for declaring PipelineTask input connections"]; "BaseInput" -> "Input" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Output" [URL="../../../py-api/lsst.pipe.base.connectionTypes.Output.html#lsst.pipe.base.connectionTypes.Output",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Output(name: str, storageClass: str, doc: str = '', multiple: bool = False, dimensions: Iterable[str] = (), isCalibration: bool = False)"]; "DimensionedConnection" -> "Output" [arrowsize=0.5,style="setlinewidth(0.5)"]; "PrerequisiteInput" [URL="../../../py-api/lsst.pipe.base.connectionTypes.PrerequisiteInput.html#lsst.pipe.base.connectionTypes.PrerequisiteInput",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class used for declaring PipelineTask prerequisite connections"]; "BaseInput" -> "PrerequisiteInput" [arrowsize=0.5,style="setlinewidth(0.5)"]; }lsst.pipe.base.pipelineIR Module¶
Classes¶
|
Intermediate representation of configurations read from a pipeline yaml file. |
An exception that is raised when a pipeline contract is not satisfied |
|
|
Intermediate representation of configuration contracts read from a pipeline yaml file. |
|
An intermediate representation of imported pipelines |
|
Intermediate representation of a pipeline definition |
|
Intermediate representation of tasks read from a pipeline yaml file. |
|
Intermediate representation of named subset of task labels read from a pipeline yaml file. |