SimplePipelineExecutor¶
- class lsst.ctrl.mpexec.SimplePipelineExecutor(quantum_graph: QuantumGraph, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True)¶
Bases:
object
A simple, high-level executor for pipelines.
- Parameters:
- quantum_graph
QuantumGraph
Graph to be executed.
- butler
Butler
Object that manages all I/O. Must be initialized with
collections
andrun
properties that correspond to the input and output collections, which must be consistent with those used to createquantum_graph
.- resources
ExecutionResources
The resources available to each quantum being executed.
- raise_on_partial_outputs
bool
, optional If
True
raise exceptions chained bylsst.pipe.base.AnnotatedPartialOutputError
immediately, instead of considering the partial result a success and continuing to run downstream tasks.
- quantum_graph
Notes
Most callers should use one of the
classmethod
factory functions (from_pipeline_filename
,from_task_class
,from_pipeline
) instead of invoking the constructor directly; these guarantee that theButler
andQuantumGraph
are created consistently.This class is intended primarily to support unit testing and small-scale integration testing of
PipelineTask
classes. It deliberately lacks many features present in the command-line-onlypipetask
tool in order to keep the implementation simple. Python callers that need more sophistication should call lower-level tools likeQuantumGraphBuilder
,PreExecInit
, andSingleQuantumExecutor
directly.Methods Summary
as_generator
([register_dataset_types, ...])Yield quanta in the
QuantumGraph
in topological order.from_pipeline
(pipeline, *[, where, bind, ...])Create an executor by building a QuantumGraph from an in-memory pipeline.
from_pipeline_filename
(pipeline_filename, *)Create an executor by building a QuantumGraph from an on-disk pipeline YAML file.
from_pipeline_graph
(pipeline_graph, *[, ...])Create an executor by building a QuantumGraph from an in-memory pipeline graph.
from_task_class
(task_class[, config, label, ...])Create an executor by building a QuantumGraph from a pipeline containing a single task.
prep_butler
(root, inputs, output[, output_run])Return configured
Butler
.run
([register_dataset_types, save_versions])Run all the quanta in the
QuantumGraph
in topological order.Methods Documentation
- as_generator(register_dataset_types: bool = False, save_versions: bool = True) Iterator[Quantum] ¶
Yield quanta in the
QuantumGraph
in topological order.These quanta will be run as the returned generator is iterated over. Use this method to run the quanta one at a time. Use
run
to run all quanta in the graph.- Parameters:
- Returns:
Notes
Global initialization steps (see
PreExecInit
) are performed immediately when this method is called, but individual quanta are not actually executed until the returned iterator is iterated over.A topological ordering is not in general unique, but no other guarantees are made about the order in which quanta are processed.
- classmethod from_pipeline(pipeline: Pipeline, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True) SimplePipelineExecutor ¶
Create an executor by building a QuantumGraph from an in-memory pipeline.
- Parameters:
- pipeline
Pipeline
orIterable
[TaskDef
] A Python object describing the tasks to run, along with their labels and configuration.
- where
str
, optional Data ID query expression that constraints the quanta generated.
- bind
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- butler
Butler
Butler that manages all I/O.
prep_butler
can be used to create one.- resources
ExecutionResources
The resources available to each quantum being executed.
- raise_on_partial_outputs
bool
, optional If
True
raise exceptions chained bylsst.pipe.base.AnnotatedPartialOutputError
immediately, instead of considering the partial result a success and continuing to run downstream tasks.
- pipeline
- Returns:
- executor
SimplePipelineExecutor
An executor instance containing the constructed
QuantumGraph
andButler
, ready forrun
to be called.
- executor
- classmethod from_pipeline_filename(pipeline_filename: str, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True) SimplePipelineExecutor ¶
Create an executor by building a QuantumGraph from an on-disk pipeline YAML file.
- Parameters:
- pipeline_filename
str
Name of the YAML file to load the pipeline definition from.
- where
str
, optional Data ID query expression that constraints the quanta generated.
- bind
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- butler
Butler
Butler that manages all I/O.
prep_butler
can be used to create one.- resources
ExecutionResources
The resources available to each quantum being executed.
- raise_on_partial_outputs
bool
, optional If
True
raise exceptions chained bylsst.pipe.base.AnnotatedPartialOutputError
immediately, instead of considering the partial result a success and continuing to run downstream tasks.
- pipeline_filename
- Returns:
- executor
SimplePipelineExecutor
An executor instance containing the constructed
QuantumGraph
andButler
, ready forrun
to be called.
- executor
- classmethod from_pipeline_graph(pipeline_graph: PipelineGraph, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True) SimplePipelineExecutor ¶
Create an executor by building a QuantumGraph from an in-memory pipeline graph.
- Parameters:
- pipeline_graph
PipelineGraph
A Python object describing the tasks to run, along with their labels and configuration, in graph form. Will be resolved against the given
butler
, with any existing resolutions ignored.- where
str
, optional Data ID query expression that constraints the quanta generated.
- bind
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- butler
Butler
Butler that manages all I/O.
prep_butler
can be used to create one. Must have itsrun
anddefaults
not empty and notNone
.- resources
ExecutionResources
The resources available to each quantum being executed.
- raise_on_partial_outputs
bool
, optional If
True
raise exceptions chained bylsst.pipe.base.AnnotatedPartialOutputError
immediately, instead of considering the partial result a success and continuing to run downstream tasks.
- pipeline_graph
- Returns:
- executor
SimplePipelineExecutor
An executor instance containing the constructed
QuantumGraph
andButler
, ready forrun
to be called.
- executor
- classmethod from_task_class(task_class: type[lsst.pipe.base.pipelineTask.PipelineTask], config: Config | None = None, label: str | None = None, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True) SimplePipelineExecutor ¶
Create an executor by building a QuantumGraph from a pipeline containing a single task.
- Parameters:
- task_class
type
A concrete
PipelineTask
subclass.- config
Config
, optional Configuration for the task. If not provided, task-level defaults will be used (no per-instrument overrides).
- label
str
, optional Label for the task in its pipeline; defaults to
task_class._DefaultName
.- where
str
, optional Data ID query expression that constraints the quanta generated.
- bind
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- butler
Butler
Butler that manages all I/O.
prep_butler
can be used to create one.- resources
ExecutionResources
The resources available to each quantum being executed.
- raise_on_partial_outputs
bool
, optional If
True
raise exceptions chained bylsst.pipe.base.AnnotatedPartialOutputError
immediately, instead of considering the partial result a success and continuing to run downstream tasks.
- task_class
- Returns:
- executor
SimplePipelineExecutor
An executor instance containing the constructed
QuantumGraph
andButler
, ready forrun
to be called.
- executor
- classmethod prep_butler(root: str, inputs: Iterable[str], output: str, output_run: str | None = None) Butler ¶
Return configured
Butler
.Helper method for creating
Butler
instances with collections appropriate for processing.- Parameters:
- root
str
Root of the butler data repository; must already exist, with all necessary input data.
- inputs
Iterable
[str
] Collections to search for all input datasets, in search order.
- output
str
Name of a new output
CHAINED
collection to create that will combine both inputs and outputs.- output_run
str
, optional Name of the output
RUN
that will directly hold all output datasets. If not provided, a name will be created fromoutput
and a timestamp.
- root
- Returns:
- butler
Butler
Butler client instance compatible with all
classmethod
factories. Always writeable.
- butler
- run(register_dataset_types: bool = False, save_versions: bool = True) list[lsst.daf.butler._quantum.Quantum] ¶
Run all the quanta in the
QuantumGraph
in topological order.Use this method to run all quanta in the graph. Use
as_generator
to get a generator to run the quanta one at a time.- Parameters:
- Returns:
Notes
A topological ordering is not in general unique, but no other guarantees are made about the order in which quanta are processed.