lsst.pipe.base

The lsst.pipe.base module provides base classes for the task framework. Tasks package the algorithmic units of the LSST Science Pipelines. You can create, configure, and run tasks with their Python APIs. Some tasks, called command-line tasks, are also packaged into data processing pipelines that you can run from the command line.

Using lsst.pipe.base

Overview

• Overview of the task framework

Using command-line tasks

• Using Butler data repositories and reruns with command-line tasks
• Specifying data IDs with command-line tasks
• Configuring command-line tasks
• Retargeting subtasks of command-line tasks
• Working with provenance checks in command-line tasks
• Logging with command-line tasks
• Parallel processing with command-line tasks
• Command-line task argument reference

Developing tasks and command-line tasks

• Creating a task
• Creating a command-line task

Contributing

lsst.pipe.base is developed at https://github.com/lsst/pipe_base. You can find Jira issues for this module under the pipe_base component.

Python API reference

lsst.pipe.base Package

Functions

InitInputDatasetField(*, doc, storageClass)	Factory function to create Config class instances of InitInputDatasetConfig
InitOutputDatasetField(*, doc, storageClass)	Factory function to create Config class instances of InitOutputDatasetConfig
InputDatasetField(*, doc, dimensions, …[, …])	Factory function to create Config class instances of InputDatasetConfig
OutputDatasetField(*, doc, dimensions, …)	Factory function to create Config class instances of OutputDatasetConfig
logInfo(obj, prefix[, logLevel])	Log timer information to obj.metadata and obj.log.
timeMethod(func)	Decorator to measure duration of a task method.

Classes

ArgumentParser(name[, usage])	Argument parser for command-line tasks that is based on argparse.ArgumentParser.
ButlerInitializedTaskRunner(TaskClass, parsedCmd)	A TaskRunner for CmdLineTasks that require a butler keyword argument to be passed to their constructor.
CmdLineTask([config, name, parentTask, log])	Base class for command-line tasks: tasks that may be executed from the command-line.
ConfigDatasetType(name)	Dataset type specified by a config parameter.
ConfigFileAction(option_strings, dest[, …])	argparse action to load config overrides from one or more files.
ConfigValueAction(option_strings, dest[, …])	argparse action callback to override config parameters using name=value pairs from the command-line.
DataIdContainer([level])	Container for data IDs and associated data references.
DatasetArgument([name, help, default])	Dataset type specified by a command-line argument.
DatasetTypeDescriptor(name, dimensionNames, …)	Description of an unnormalized proto-DatasetType and its relationship to a PipelineTask.
GraphBuilder(taskFactory, registry[, …])	GraphBuilder class is responsible for building task execution graph from a Pipeline.
InitInputDatasetConfig
InitOutputDatasetConfig
InputDatasetConfig
InputOnlyArgumentParser(name[, usage])	ArgumentParser for command-line tasks that don’t write any output.
LegacyTaskRunner(TaskClass, parsedCmd[, …])	A TaskRunner for CmdLineTasks which calls the Task’s run method on a dataRef rather than the runDataRef method.
OutputDatasetConfig
Pipeline([iterable])	Pipeline is a sequence of TaskDef objects.
PipelineBuilder(taskFactory[, pipeline])	PipelineBuilder class is responsible for building task pipeline.
PipelineDatasetTypes(initInputs, …)	An immutable struct that classifies the dataset types used in a Pipeline.
PipelineTask(*[, config, log, initInputs])	Base class for all pipeline tasks.
PipelineTaskConfig	Base class for all PipelineTask configurations.
QuantumConfig	Configuration class which defines PipelineTask quanta dimensions.
QuantumGraph([iterable])	QuantumGraph is a sequence of QuantumGraphTaskNodes objects.
QuantumGraphTaskNodes(taskDef, quanta, …)	QuantumGraphTaskNodes represents a bunch of nodes in an quantum graph corresponding to a single task.
QuantumIterData(index, quantum, taskDef, …)	Helper class for iterating over quanta in a graph.
ResourceConfig	Configuration for resource requirements.
Struct(**keyArgs)	A container to which you can add fields as attributes.
Task([config, name, parentTask, log])	Base class for data processing tasks.
TaskDatasetTypes(initInputs, initOutputs, …)	An immutable struct that extracts and classifies the dataset types used by a PipelineTask
TaskDef(taskName, config[, taskClass, label])	TaskDef is a collection of information about task needed by Pipeline.
TaskError	Use to report errors for which a traceback is not useful.
TaskFactory	Abstract base class for task factory.
TaskRunner(TaskClass, parsedCmd[, …])	Run a command-line task, using multiprocessing if requested.

Class Inheritance Diagram