SeparablePipelineExecutor

class lsst.ctrl.mpexec.SeparablePipelineExecutor(butler: Butler, clobber_output: bool = False, skip_existing_in: Iterable[str] | None = None, task_factory: TaskFactory | None = None, resources: ExecutionResources | None = None)

Bases: object

An executor that allows each step of pipeline execution to be run independently.

The executor can run any or all of the following steps:

• pre-execution initialization

• pipeline building

• quantum graph generation

• quantum graph execution

Any of these steps can also be handed off to external code without compromising the remaining ones.

Parameters:

butlerlsst.daf.butler.Butler

A Butler whose collections and run attributes contain the input and output collections to use for processing.

clobber_outputbool, optional

If set, the pipeline execution overwrites existing output files. Otherwise, any conflict between existing and new outputs is an error.

skip_existing_initerable [str], optional

If not empty, the pipeline execution searches the listed collections for existing outputs, and skips any quanta that have run to completion (or have no work to do). Otherwise, all tasks are attempted (subject to clobber_output).

task_factorylsst.pipe.base.TaskFactory, optional

A custom task factory for use in pre-execution and execution. By default, a new instance of lsst.ctrl.mpexec.TaskFactory is used.

resourcesExecutionResources

The resources available to each quantum being executed.

Methods Summary

make_pipeline(pipeline_uri)	Build a pipeline from pipeline and configuration information.
make_quantum_graph(pipeline[, where, builder])	Build a quantum graph from a pipeline and input datasets.
pre_execute_qgraph(graph[, ...])	Run pre-execution initialization.
run_pipeline(graph[, fail_fast, graph_executor])	Run a pipeline in the form of a prepared quantum graph.

Methods Documentation

make_pipeline(pipeline_uri: str | ResourcePath) → Pipeline

Build a pipeline from pipeline and configuration information.

Parameters:

pipeline_uristr or lsst.resources.ResourcePath

URI to a file containing a pipeline definition. A URI fragment may be used to specify a subset of the pipeline, as described in Command line options for running Pipelines.

Returns:

pipelinelsst.pipe.base.Pipeline

The fully-built pipeline.

make_quantum_graph(pipeline: Pipeline, where: str = '', builder: _GraphBuilderLike | None = None) → QuantumGraph

Build a quantum graph from a pipeline and input datasets.

Parameters:

pipelinelsst.pipe.base.Pipeline

The pipeline for which to generate a quantum graph.

wherestr, optional

A data ID query that constrains the quanta generated.

builderlsst.pipe.base.GraphBuilder-like, optional

A graph builder that implements a makeGraph method. By default, a new instance of lsst.pipe.base.GraphBuilder is used.

Returns:

graphlsst.pipe.base.QuantumGraph

The quantum graph for pipeline as run on the datasets identified by where.

Notes

This method does no special handling of empty quantum graphs. If needed, clients can use len to test if the returned graph is empty.

pre_execute_qgraph(graph: QuantumGraph, register_dataset_types: bool = False, save_init_outputs: bool = True, save_versions: bool = True) → None

Run pre-execution initialization.

This method will be deprecated after DM-38041, to be replaced with a method that takes either a Pipeline or a ResolvedPipelineGraph instead of a QuantumGraph.

Parameters:

graphlsst.pipe.base.QuantumGraph

The quantum graph defining the pipeline and datasets to be initialized.

register_dataset_typesbool, optional

If True, register all output dataset types from the pipeline represented by graph.

save_init_outputsbool, optional

If True, create init-output datasets in this object’s output run.

save_versionsbool, optional

If True, save a package versions dataset.

run_pipeline(graph: QuantumGraph, fail_fast: bool = False, graph_executor: QuantumGraphExecutor | None = None) → None

Run a pipeline in the form of a prepared quantum graph.

Pre-execution initialization must have already been run; see pre_execute_qgraph.

Parameters:

graphlsst.pipe.base.QuantumGraph

The pipeline and datasets to execute.

fail_fastbool, optional

If True, abort all (parallel) execution if any task fails (only used with the default graph executor).

graph_executorlsst.ctrl.mpexec.QuantumGraphExecutor, optional

A custom graph executor. By default, a new instance of lsst.ctrl.mpexec.MPGraphExecutor is used.