lsst.pipe.base

The lsst.pipe.base module provides base classes for the task framework. Tasks package the algorithmic units of the LSST Science Pipelines. You can create, configure, and run tasks with their Python APIs. Some tasks, called command-line tasks, are also packaged into data processing pipelines that you can run from the command line.

Changes

Using lsst.pipe.base

Overview

Using command-line tasks

Developing tasks and command-line tasks

Developing Pipelines

Contributing

lsst.pipe.base is developed at https://github.com/lsst/pipe_base. You can find Jira issues for this module under the pipe_base component.

Python API reference

lsst.pipe.base Package

Functions

buildExecutionButler(butler, graph, ...[, ...])

buildExecutionButler is a function that is responsible for exporting input QuantumGraphs into a new minimal Butler which only contains datasets specified by the QuantumGraph.

getTaskLogger([name, logger])

Get a logger compatible with LSST usage.

iterConnections(connections, connectionType)

Creates an iterator over the selected connections type which yields all the defined connections of that type.

logInfo(obj, prefix[, logLevel, metadata, ...])

Log timer information to obj.metadata and obj.log.

timeMethod(*args, **kwargs)

Decorator to measure duration of a method.

Classes

AdjustQuantumHelper(inputs, outputs[, ...])

Helper class for calling PipelineTaskConnections.adjustQuantum.

ArgumentParser(name[, usage])

Argument parser for command-line tasks that is based on argparse.ArgumentParser.

ButlerInitializedTaskRunner(TaskClass, parsedCmd)

A TaskRunner for CmdLineTasks that require a butler keyword argument to be passed to their constructor.

ButlerQuantumContext(butler, quantum)

A Butler-like class specialized for a single quantum

CmdLineTask([config, name, parentTask, log])

Base class for command-line tasks: tasks that may be executed from the command-line.

ConfigDatasetType(name)

Dataset type specified by a config parameter.

ConfigFileAction(option_strings, dest[, ...])

argparse action to load config overrides from one or more files.

ConfigValueAction(option_strings, dest[, ...])

argparse action callback to override config parameters using name=value pairs from the command-line.

DataIdContainer([level])

Container for data IDs and associated data references.

DatasetArgument([name, help, default])

Dataset type specified by a command-line argument.

DeferredDatasetRef(datasetRef)

A wrapper class for DatasetRef that indicates that a PipelineTask should receive a DeferredDatasetHandle instead of an in-memory dataset.

GraphBuilder(registry[, skipExistingIn, ...])

GraphBuilder class is responsible for building task execution graph from a Pipeline.

IncompatibleGraphError

Exception class to indicate that a lookup by NodeId is impossible due to incompatibilities

InputOnlyArgumentParser(name[, usage])

ArgumentParser for command-line tasks that don't write any output.

InputQuantizedConnection(**kwargs)

Instrument([collection_prefix])

Base class for instrument-specific logic for the Gen3 Butler.

InvalidQuantumError

Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate logic bug or configuration problem.

LabelSpecifier([labels, begin, end])

A structure to specify a subset of labels to load

LegacyTaskRunner(TaskClass, parsedCmd[, ...])

A TaskRunner for CmdLineTasks which calls the Task's run method on a dataRef rather than the runDataRef method.

NoWorkFound

An exception raised when a Quantum should not exist because there is no work for it to do.

NodeId(number, buildId)

Deprecated, this class is used with QuantumGraph save formats of 1 and 2 when unpicking objects and must be retained until those formats are considered unloadable.

OutputQuantizedConnection(**kwargs)

Pipeline(description)

A Pipeline is a representation of a series of tasks to run, and the configuration for those tasks.

PipelineDatasetTypes(initInputs, ...)

An immutable struct that classifies the dataset types used in a Pipeline.

PipelineTask(*[, config, log, initInputs])

Base class for all pipeline tasks.

PipelineTaskConfig(*args, **kw)

Configuration class for PipelineTask

PipelineTaskConnections(*[, config])

PipelineTaskConnections is a class used to declare desired IO when a PipelineTask is run by an activator

QuantumGraph(quanta[, metadata, pruneRefs, ...])

QuantumGraph is a directed acyclic graph of QuantumNode objects

QuantumNode(quantum, taskDef, nodeId)

This class represents a node in the quantum graph.

RepeatableQuantumError

Exception that may be raised by PipelineTasks (and code they delegate to) in order to indicate that a repeatable problem that will not be addressed by retries.

ResourceConfig(*args, **kw)

Configuration for resource requirements.

ScalarError

Exception raised when dataset type is configured as scalar but there are multiple data IDs in a Quantum for that dataset.

Struct(**keyArgs)

A container to which you can add fields as attributes.

Task([config, name, parentTask, log])

Base class for data processing tasks.

TaskDatasetTypes(initInputs, initOutputs, ...)

An immutable struct that extracts and classifies the dataset types used by a PipelineTask

TaskDef([taskName, config, taskClass, label])

TaskDef is a collection of information about task needed by Pipeline.

TaskError

Use to report errors for which a traceback is not useful.

TaskFactory()

Abstract base class for task factory.

TaskMetadata(*[, scalars, arrays, metadata])

Dict-like object for storing task metadata.

TaskRunner(TaskClass, parsedCmd[, ...])

Run a command-line task, using multiprocessing if requested.

Class Inheritance Diagram

Inheritance diagram of lsst.pipe.base.connections.AdjustQuantumHelper, lsst.pipe.base.argumentParser.ArgumentParser, lsst.pipe.base.cmdLineTask.ButlerInitializedTaskRunner, lsst.pipe.base.butlerQuantumContext.ButlerQuantumContext, lsst.pipe.base.cmdLineTask.CmdLineTask, lsst.pipe.base.argumentParser.ConfigDatasetType, lsst.pipe.base.argumentParser.ConfigFileAction, lsst.pipe.base.argumentParser.ConfigValueAction, lsst.pipe.base.argumentParser.DataIdContainer, lsst.pipe.base.argumentParser.DatasetArgument, lsst.pipe.base.connections.DeferredDatasetRef, lsst.pipe.base.graphBuilder.GraphBuilder, lsst.pipe.base.graph.graph.IncompatibleGraphError, lsst.pipe.base.argumentParser.InputOnlyArgumentParser, lsst.pipe.base.connections.InputQuantizedConnection, lsst.pipe.base._instrument.Instrument, lsst.pipe.base._status.InvalidQuantumError, lsst.pipe.base.pipeline.LabelSpecifier, lsst.pipe.base.cmdLineTask.LegacyTaskRunner, lsst.pipe.base._status.NoWorkFound, lsst.pipe.base.graph.quantumNode.NodeId, lsst.pipe.base.connections.OutputQuantizedConnection, lsst.pipe.base.pipeline.Pipeline, lsst.pipe.base.pipeline.PipelineDatasetTypes, lsst.pipe.base.pipelineTask.PipelineTask, lsst.pipe.base.config.PipelineTaskConfig, lsst.pipe.base.connections.PipelineTaskConnections, lsst.pipe.base.graph.graph.QuantumGraph, lsst.pipe.base.graph.quantumNode.QuantumNode, lsst.pipe.base._status.RepeatableQuantumError, lsst.pipe.base.config.ResourceConfig, lsst.pipe.base.connections.ScalarError, lsst.pipe.base.struct.Struct, lsst.pipe.base.task.Task, lsst.pipe.base.pipeline.TaskDatasetTypes, lsst.pipe.base.pipeline.TaskDef, lsst.pipe.base.task.TaskError, lsst.pipe.base.taskFactory.TaskFactory, lsst.pipe.base._task_metadata.TaskMetadata, lsst.pipe.base.cmdLineTask.TaskRunner

lsst.pipe.base.testUtils Module

Functions

assertValidInitOutput(task)

Test that a constructed task conforms to its own init-connections.

assertValidOutput(task, result)

Test that the output of a call to run conforms to its own connections.

getInitInputs(butler, config)

Return the initInputs object that would have been passed to a PipelineTask constructor.

lintConnections(connections, *[, ...])

Inspect a connections class for common errors.

makeQuantum(task, butler, dataId, ioDataIds)

Create a Quantum for a particular data ID(s).

runTestQuantum(task, butler, quantum[, mockRun])

Run a PipelineTask on a Quantum.

lsst.pipe.base.connectionTypes Module

Classes

InitInput(name, storageClass[, doc, multiple])

InitOutput(name, storageClass[, doc, multiple])

Input(name, storageClass[, doc, multiple, ...])

Class used for declaring PipelineTask input connections

PrerequisiteInput(name, storageClass[, doc, ...])

Class used for declaring PipelineTask prerequisite connections

Output(name, storageClass[, doc, multiple, ...])

BaseConnection(name, storageClass[, doc, ...])

Base class used for declaring PipelineTask connections

Class Inheritance Diagram

Inheritance diagram of lsst.pipe.base.connectionTypes.InitInput, lsst.pipe.base.connectionTypes.InitOutput, lsst.pipe.base.connectionTypes.Input, lsst.pipe.base.connectionTypes.PrerequisiteInput, lsst.pipe.base.connectionTypes.Output, lsst.pipe.base.connectionTypes.BaseConnection

lsst.pipe.base.pipelineIR Module

Classes

ConfigIR(python, dataId, file, rest)

Intermediate representation of configurations read from a pipeline yaml file.

ContractError

An exception that is raised when a pipeline contract is not satisfied

ContractIR(contract[, msg])

Intermediate representation of configuration contracts read from a pipeline yaml file.

ImportIR(location, include, exclude, ...)

An intermediate representation of imported pipelines

PipelineIR(loaded_yaml)

Intermediate representation of a pipeline definition

TaskIR(label, klass[, config])

Intermediate representation of tasks read from a pipeline yaml file.

LabeledSubset(label, subset, description)

Intermediate representation of named subset of task labels read from a pipeline yaml file.

Class Inheritance Diagram

Inheritance diagram of lsst.pipe.base.pipelineIR.ConfigIR, lsst.pipe.base.pipelineIR.ContractError, lsst.pipe.base.pipelineIR.ContractIR, lsst.pipe.base.pipelineIR.ImportIR, lsst.pipe.base.pipelineIR.PipelineIR, lsst.pipe.base.pipelineIR.TaskIR, lsst.pipe.base.pipelineIR.LabeledSubset