lsst.pipe.base¶
The lsst.pipe.base
module provides base classes for the task framework.
Tasks package the algorithmic units of the LSST Science Pipelines.
You can create, configure, and run tasks with their Python APIs.
Some tasks, called command-line tasks, are also packaged into data processing pipelines that you can run from the command line.
Using lsst.pipe.base¶
Overview¶
Using command-line tasks¶
- Using Butler data repositories and reruns with command-line tasks
- Specifying data IDs with command-line tasks
- Configuring command-line tasks
- Retargeting subtasks of command-line tasks
- Working with provenance checks in command-line tasks
- Logging with command-line tasks
- Parallel processing with command-line tasks
- Command-line task argument reference
Developing tasks and command-line tasks¶
Contributing¶
lsst.pipe.base
is developed at https://github.com/lsst/pipe_base.
You can find Jira issues for this module under the pipe_base component.
Python API reference¶
lsst.pipe.base Package¶
Functions¶
iterConnections (connections, connectionType) |
Creates an iterator over the selected connections type which yields all the defined connections of that type. |
logInfo (obj, prefix[, logLevel]) |
Log timer information to obj.metadata and obj.log . |
timeMethod (func) |
Decorator to measure duration of a task method. |
Classes¶
ArgumentParser (name[, usage]) |
Argument parser for command-line tasks that is based on argparse.ArgumentParser . |
ButlerInitializedTaskRunner (TaskClass, parsedCmd) |
A TaskRunner for CmdLineTask s that require a butler keyword argument to be passed to their constructor. |
ButlerQuantumContext (butler, quantum) |
Butler like class specialized for a single quantum |
CmdLineTask ([config, name, parentTask, log]) |
Base class for command-line tasks: tasks that may be executed from the command-line. |
ConfigDatasetType (name) |
Dataset type specified by a config parameter. |
ConfigFileAction (option_strings, dest[, …]) |
argparse action to load config overrides from one or more files. |
ConfigValueAction (option_strings, dest[, …]) |
argparse action callback to override config parameters using name=value pairs from the command-line. |
DataIdContainer ([level]) |
Container for data IDs and associated data references. |
DatasetArgument ([name, help, default]) |
Dataset type specified by a command-line argument. |
DeferredDatasetRef |
Class which denotes that a datasetRef should be treated as deferred when interacting with the butler |
GraphBuilder (registry[, skipExisting, …]) |
GraphBuilder class is responsible for building task execution graph from a Pipeline. |
InputOnlyArgumentParser (name[, usage]) |
ArgumentParser for command-line tasks that don’t write any output. |
InputQuantizedConnection (**kwargs) |
|
LegacyTaskRunner (TaskClass, parsedCmd[, …]) |
A TaskRunner for CmdLineTask s which calls the Task ’s run method on a dataRef rather than the runDataRef method. |
OutputQuantizedConnection (**kwargs) |
|
Pipeline (description) |
A Pipeline is a representation of a series of tasks to run, and the configuration for those tasks. |
PipelineDatasetTypes (initInputs, …) |
An immutable struct that classifies the dataset types used in a Pipeline . |
PipelineTask (*[, config, log, initInputs]) |
Base class for all pipeline tasks. |
PipelineTaskConfig |
Configuration class for `PipelineTask`s |
PipelineTaskConnections (*, config) |
PipelineTaskConnections is a class used to declare desired IO when a PipelineTask is run by an activator |
QuantumGraph ([iterable]) |
QuantumGraph is a sequence of QuantumGraphTaskNodes objects. |
QuantumGraphTaskNodes (taskDef, quanta, …) |
QuantumGraphTaskNodes represents a bunch of nodes in an quantum graph corresponding to a single task. |
QuantumIterData (index, quantum, taskDef, …) |
Helper class for iterating over quanta in a graph. |
ResourceConfig |
Configuration for resource requirements. |
Struct (**keyArgs) |
A container to which you can add fields as attributes. |
Task ([config, name, parentTask, log]) |
Base class for data processing tasks. |
TaskDatasetTypes (initInputs, initOutputs, …) |
An immutable struct that extracts and classifies the dataset types used by a PipelineTask |
TaskDef (taskName, config[, taskClass, label]) |
TaskDef is a collection of information about task needed by Pipeline. |
TaskError |
Use to report errors for which a traceback is not useful. |
TaskFactory |
Abstract base class for task factory. |
TaskRunner (TaskClass, parsedCmd[, …]) |
Run a command-line task, using multiprocessing if requested. |