SeparablePipelineExecutor¶
- class lsst.ctrl.mpexec.SeparablePipelineExecutor(butler: Butler, clobber_output: bool = False, skip_existing_in: Iterable[str] | None = None, task_factory: TaskFactory | None = None, resources: ExecutionResources | None = None)¶
Bases:
object
An executor that allows each step of pipeline execution to be run independently.
The executor can run any or all of the following steps:
pre-execution initialization
pipeline building
quantum graph generation
quantum graph execution
Any of these steps can also be handed off to external code without compromising the remaining ones.
- Parameters:
- butler
lsst.daf.butler.Butler
A Butler whose
collections
andrun
attributes contain the input and output collections to use for processing.- clobber_output
bool
, optional If set, the pipeline execution overwrites existing output files. Otherwise, any conflict between existing and new outputs is an error.
- skip_existing_initerable [
str
], optional If not empty, the pipeline execution searches the listed collections for existing outputs, and skips any quanta that have run to completion (or have no work to do). Otherwise, all tasks are attempted (subject to
clobber_output
).- task_factory
lsst.pipe.base.TaskFactory
, optional A custom task factory for use in pre-execution and execution. By default, a new instance of
lsst.ctrl.mpexec.TaskFactory
is used.- resources
ExecutionResources
The resources available to each quantum being executed.
- butler
Methods Summary
make_pipeline
(pipeline_uri)Build a pipeline from pipeline and configuration information.
make_quantum_graph
(pipeline[, where, builder])Build a quantum graph from a pipeline and input datasets.
pre_execute_qgraph
(graph[, ...])Run pre-execution initialization.
run_pipeline
(graph[, fail_fast, graph_executor])Run a pipeline in the form of a prepared quantum graph.
Methods Documentation
- make_pipeline(pipeline_uri: str | ResourcePath) Pipeline ¶
Build a pipeline from pipeline and configuration information.
- Parameters:
- pipeline_uri
str
orlsst.resources.ResourcePath
URI to a file containing a pipeline definition. A URI fragment may be used to specify a subset of the pipeline, as described in Command line options for running Pipelines.
- pipeline_uri
- Returns:
- pipeline
lsst.pipe.base.Pipeline
The fully-built pipeline.
- pipeline
- make_quantum_graph(pipeline: Pipeline, where: str = '', builder: _GraphBuilderLike | None = None) QuantumGraph ¶
Build a quantum graph from a pipeline and input datasets.
- Parameters:
- pipeline
lsst.pipe.base.Pipeline
The pipeline for which to generate a quantum graph.
- where
str
, optional A data ID query that constrains the quanta generated.
- builder
lsst.pipe.base.GraphBuilder
-like, optional A graph builder that implements a
makeGraph
method. By default, a new instance oflsst.pipe.base.GraphBuilder
is used.
- pipeline
- Returns:
- graph
lsst.pipe.base.QuantumGraph
The quantum graph for
pipeline
as run on the datasets identified bywhere
.
- graph
Notes
This method does no special handling of empty quantum graphs. If needed, clients can use
len
to test if the returned graph is empty.
- pre_execute_qgraph(graph: QuantumGraph, register_dataset_types: bool = False, save_init_outputs: bool = True, save_versions: bool = True) None ¶
Run pre-execution initialization.
This method will be deprecated after DM-38041, to be replaced with a method that takes either a
Pipeline
or aResolvedPipelineGraph
instead of aQuantumGraph
.- Parameters:
- graph
lsst.pipe.base.QuantumGraph
The quantum graph defining the pipeline and datasets to be initialized.
- register_dataset_types
bool
, optional If
True
, register all output dataset types from the pipeline represented bygraph
.- save_init_outputs
bool
, optional If
True
, create init-output datasets in this object’s output run.- save_versions
bool
, optional If
True
, save a package versions dataset.
- graph
- run_pipeline(graph: QuantumGraph, fail_fast: bool = False, graph_executor: QuantumGraphExecutor | None = None) None ¶
Run a pipeline in the form of a prepared quantum graph.
Pre-execution initialization must have already been run; see
pre_execute_qgraph
.- Parameters:
- graph
lsst.pipe.base.QuantumGraph
The pipeline and datasets to execute.
- fail_fast
bool
, optional If
True
, abort all (parallel) execution if any task fails (only used with the default graph executor).- graph_executor
lsst.ctrl.mpexec.QuantumGraphExecutor
, optional A custom graph executor. By default, a new instance of
lsst.ctrl.mpexec.MPGraphExecutor
is used.
- graph