lsst.pipe.base

The lsst.pipe.base module provides base classes for the task framework. Tasks package the algorithmic units of the LSST Science Pipelines. You can create, configure, and run tasks with their Python APIs. Some tasks, called pipeline tasks, can be packaged into data processing pipelines that you can run from the command line.

Changes

• lsst-pipe-base v25.0.0 (2023-02-28)
• lsst-pipe-base v24.0.0 (2022-08-26)
• lsst-pipe-base v23.0.1 (2022-02-02)
• lsst-pipe-base v23.0.0 (2021-12-10)
• lsst-pipe-base v22.0 (2021-04-01)

Using lsst.pipe.base

Overview

• Overview of the task framework

Developing tasks and pipeline tasks

• Creating a task
• Testing a pipeline task
• Creating a PipelineTask
• Retargeting subtasks of tasks

Developing Pipelines

• Creating a Pipeline
• Testing pipelines with mocks
• Working with Pipeline Graphs

Contributing

lsst.pipe.base is developed at https://github.com/lsst/pipe_base. You can find Jira issues for this module under the pipe_base component.

Python API reference

lsst.pipe.base Package

Functions

buildExecutionButler(butler, graph, ...[, ...])	Create an execution butler.
iterConnections(connections, connectionType)	Create an iterator over the selected connections type which yields all the defined connections of that type.

Classes

AdjustQuantumHelper(inputs, outputs[, ...])	Helper class for calling PipelineTaskConnections.adjustQuantum.
ButlerQuantumContext(*args, **kwargs)	Deprecated version of QuantumContext.
DeferredDatasetRef(datasetRef)	A wrapper class for DatasetRef that indicates that a PipelineTask should receive a DeferredDatasetHandle instead of an in-memory dataset.
ExecutionResources(*[, num_cores, max_mem, ...])	A description of the resources available to a running quantum.
GraphBuilder(registry[, skipExistingIn, ...])	GraphBuilder class is responsible for building task execution graph from a Pipeline.
InMemoryDatasetHandle(inMemoryDataset, *[, ...])	An in-memory version of a DeferredDatasetHandle.
IncompatibleGraphError	Exception class to indicate that a lookup by NodeId is impossible due to incompatibilities
InputQuantizedConnection(**kwargs)	Input variant of a QuantizedConnection.
Instrument([collection_prefix])	Base class for instrument-specific logic for the Gen3 Butler.
InvalidQuantumError	Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate logic bug or configuration problem.
LabelSpecifier([labels, begin, end])	A structure to specify a subset of labels to load
NoWorkFound	An exception raised when a Quantum should not exist because there is no work for it to do.
NodeId(number, buildId)	Deprecated, this class is used with QuantumGraph save formats of 1 and 2 when unpicking objects and must be retained until those formats are considered unloadable.
ObservationDimensionPacker(data_id, config)	A DimensionPacker for visit+detector or exposure+detector.
ObservationDimensionPackerConfig(*args, **kw)	Config associated with a ObservationDimensionPacker.
OutputQuantizedConnection(**kwargs)	Output variant of a QuantizedConnection.
Pipeline(description)	A Pipeline is a representation of a series of tasks to run, and the configuration for those tasks.
PipelineDatasetTypes(initInputs, ...)	An immutable struct that classifies the dataset types used in a Pipeline.
PipelineTask(*[, config, log, initInputs])	Base class for all pipeline tasks.
PipelineTaskConfig(*args, **kw)	Configuration class for PipelineTask
PipelineTaskConnections(*[, config])	PipelineTaskConnections is a class used to declare desired IO when a PipelineTask is run by an activator
QuantumContext(butler, quantum, *[, resources])	A Butler-like class specialized for a single quantum along with context information that can influence how the task is executed.
QuantumGraph(quanta[, metadata, pruneRefs, ...])	QuantumGraph is a directed acyclic graph of QuantumNode objects
QuantumNode(quantum, taskDef, nodeId)	Class representing a node in the quantum graph.
RepeatableQuantumError	Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate that a repeatable problem that will not be addressed by retries.
ScalarError	Exception raised when dataset type is configured as scalar but there are multiple data IDs in a Quantum for that dataset.
Struct(**keyArgs)	A container to which you can add fields as attributes.
Task([config, name, parentTask, log])	Base class for data processing tasks.
TaskDatasetTypes(initInputs, initOutputs, ...)	An immutable struct that extracts and classifies the dataset types used by a PipelineTask
TaskDef([taskName, config, taskClass, ...])	TaskDef is a collection of information about task needed by Pipeline.
TaskError	Use to report errors for which a traceback is not useful.
TaskFactory()	Abstract base class for task factory.
TaskMetadata(*[, scalars, arrays, metadata])	Dict-like object for storing task metadata.

Class Inheritance Diagram

lsst.pipe.base.pipeline_graph Package

Classes

ConnectionTypeConsistencyError	Exception raised when the tasks in a pipeline graph use different (and incompatible) connection types for the same dataset type.
DatasetTypeNode(dataset_type, ...)	A node in a pipeline graph that represents a resolved dataset type.
DuplicateOutputError	Exception raised when multiple tasks in one pipeline produce the same output dataset type.
Edge(*, task_key, dataset_type_key, ...)	Base class for edges in a pipeline graph.
EdgesChangedError	Exception raised when the edges in one version of a pipeline graph are not consistent with those in another, but they were expected to be.
IncompatibleDatasetTypeError	Exception raised when the tasks in a pipeline graph define dataset types with the same name in incompatible ways, or when these are incompatible with the data repository definition.
NodeKey(node_type, name)	A special key type for nodes in networkx graphs.
NodeType(value[, names, module, qualname, ...])	Enumeration of the types of nodes in a PipelineGraph.
PipelineDataCycleError	Exception raised when a pipeline graph contains a cycle.
PipelineGraph(*[, description, universe, ...])	A graph representation of fully-configured pipeline.
PipelineGraphError	Base exception raised when there is a problem constructing or resolving a pipeline graph.
PipelineGraphExceptionSafetyError	Exception raised when a PipelineGraph method could not provide strong exception safety, and the graph may have been left in an inconsistent state.
PipelineGraphReadError	Exception raised when a serialized PipelineGraph cannot be read.
ReadEdge(dataset_type_key, task_key, *, ...)	Representation of an input connection (including init-inputs and prerequisites) in a pipeline graph.
TaskImportMode(value[, names, module, ...])	Enumeration of the ways to handle importing tasks when reading a serialized PipelineGraph.
TaskInitNode(key, *, inputs, outputs, ...[, ...])	A node in a pipeline graph that represents the construction of a PipelineTask.
TaskNode(key, init, *, prerequisite_inputs, ...)	A node in a pipeline graph that represents a labeled configuration of a PipelineTask.
TaskNotImportedError	Exception raised when accessing an attribute of a graph or graph node that is not available unless the task class has been imported and configured.
TaskSubset(parent_xgraph, label, members, ...)	A specialized set that represents a labeles subset of the tasks in a pipeline graph.
UnresolvedGraphError	Exception raised when an operation requires dimensions or dataset types to have been resolved, but they have not been.
WriteEdge(*, task_key, dataset_type_key, ...)	Representation of an output connection (including init-outputs) in a pipeline graph.

Class Inheritance Diagram

lsst.pipe.base.testUtils Module

Functions

assertValidInitOutput(task)	Test that a constructed task conforms to its own init-connections.
assertValidOutput(task, result)	Test that the output of a call to run conforms to its own connections.
getInitInputs(butler, config)	Return the initInputs object that would have been passed to a PipelineTask constructor.
lintConnections(connections, *[, ...])	Inspect a connections class for common errors.
makeQuantum(task, butler, dataId, ioDataIds)	Create a Quantum for a particular data ID(s).
runTestQuantum(task, butler, quantum[, mockRun])	Run a PipelineTask on a Quantum.

lsst.pipe.base.connectionTypes Module

Classes

InitInput(name, storageClass[, doc, ...])	Connection for initInput dataset.
InitOutput(name, storageClass[, doc, ...])	Connection for initOutput dataset.
Input(name, storageClass[, doc, multiple, ...])	Class used for declaring PipelineTask input connections
PrerequisiteInput(name, storageClass[, doc, ...])	Class used for declaring PipelineTask prerequisite connections.
Output(name, storageClass[, doc, multiple, ...])	Connection for output dataset.
BaseConnection(name, storageClass[, doc, ...])	Base class used for declaring PipelineTask connections.

Class Inheritance Diagram

lsst.pipe.base.pipelineIR Module

Classes

ConfigIR(python, dataId, file, rest)	Intermediate representation of configurations read from a pipeline yaml file.
ContractError	An exception that is raised when a pipeline contract is not satisfied
ContractIR(contract[, msg])	Intermediate representation of configuration contracts read from a pipeline yaml file.
ImportIR(location, include, exclude, ...)	An intermediate representation of imported pipelines
LabeledSubset(label, subset, description)	Intermediate representation of named subset of task labels read from a pipeline yaml file.
ParametersIR(mapping)	Intermediate representation of parameters that are global to a pipeline.
PipelineIR(loaded_yaml)	Intermediate representation of a pipeline definition
TaskIR(label, klass[, config])	Intermediate representation of tasks read from a pipeline yaml file.

Class Inheritance Diagram

lsst.pipe.base.tests.mocks Package

A system for replacing the tasks in a pipeline with mocks that just read and write trivial datasets.

See Testing pipelines with mocks for details.

Functions

get_mock_name(original)	Return the name of the mock storage class, dataset type, or task label for the given original name.
get_original_name(mock)	Return the name of the original storage class, dataset type, or task label that corresponds to the given mock name.
is_mock_name(name)	Return whether the given name is that of a mock storage class, dataset type, or task label.
mock_task_defs(originals[, ...])	Create mocks for an iterable of TaskDefs.

Classes

DataIdMatch(expression)	Class that can match DataId against the user-defined string expression.
DynamicConnectionConfig(*args, **kw)	A config class that defines a completely dynamic connection.
DynamicTestPipelineTask(*, config, **kwargs)	A test-utility implementation of PipelineTask with dimensions and connections determined wholly from configuration.
DynamicTestPipelineTaskConfig(*args, **kw)	Configuration for DynamicTestPipelineTask.
MockDataset(*, ref[, quantum, ...])	The in-memory dataset type used by MockStorageClass.
MockDatasetQuantum(*, task_label, data_id, ...)	Description of the quantum that produced a mock dataset.
MockPipelineTask(*, config, **kwargs)	A test-utility implementation of PipelineTask with connections generated by mocking those of a real task.
MockPipelineTaskConfig(*args, **kw)	Configuration class for MockPipelineTask.
MockStorageClass(original[, factory])	A reimplementation of lsst.daf.butler.StorageClass for mock datasets.
MockStorageClassDelegate(storageClass)	Implementation of the StorageClassDelegate interface for mock datasets.

Class Inheritance Diagram