SeparablePipelineExecutor¶
- class lsst.ctrl.mpexec.SeparablePipelineExecutor(*args, **kwargs)¶
Bases:
SeparablePipelineExecutor
Deprecated since version v30: The SeparablePipelineExecutor class has moved to lsst.pipe.base.separable_pipeline_executor. This forwarding shim will be removed after v30.
Methods Summary
make_pipeline
(pipeline_uri)Build a pipeline from pipeline and configuration information.
make_quantum_graph
(pipeline[, where, ...])Build a quantum graph from a pipeline and input datasets.
pre_execute_qgraph
(graph[, ...])Run pre-execution initialization.
run_pipeline
(graph[, fail_fast, ...])Run a pipeline in the form of a prepared quantum graph.
Methods Documentation
- make_pipeline(pipeline_uri: str | ResourcePath) Pipeline ¶
Build a pipeline from pipeline and configuration information.
- Parameters:
- pipeline_uri
str
orlsst.resources.ResourcePath
URI to a file containing a pipeline definition. A URI fragment may be used to specify a subset of the pipeline, as described in Command line options for running Pipelines.
- pipeline_uri
- Returns:
- pipeline
Pipeline
The fully-built pipeline.
- pipeline
- make_quantum_graph(pipeline: ~lsst.pipe.base.pipeline.Pipeline, where: str = '', *, builder_class: type[lsst.pipe.base.quantum_graph_builder.QuantumGraphBuilder] = <class 'lsst.pipe.base.all_dimensions_quantum_graph_builder.AllDimensionsQuantumGraphBuilder'>, attach_datastore_records: bool = False, **kwargs: ~typing.Any) QuantumGraph ¶
Build a quantum graph from a pipeline and input datasets.
- Parameters:
- pipeline
Pipeline
The pipeline for which to generate a quantum graph.
- where
str
, optional A data ID query that constrains the quanta generated. Must not be provided if a custom
builder_class
is given and that class does not acceptwhere
as a construction argument.- builder_class
type
[quantum_graph_builder.QuantumGraphBuilder
], optional Quantum graph builder implementation. Ignored if
builder
is provided.- attach_datastore_records
bool
, optional Whether to attach datastore records. These are currently used only by
lsst.daf.butler.QuantumBackedButler
, which is not used bySeparablePipelineExecutor
for execution.- **kwargs
Additional keyword arguments are forwarded to
builder_class
when a quantum graph builder instance is constructed. All arguments accepted by theQuantumGraphBuilder
base class are provided automatically (from explicit arguments to this method and executor attributes) and do not need to be included as keyword arguments.
- pipeline
- Returns:
- graph
QuantumGraph
The quantum graph for
.Pipeline
as run on the datasets identified bywhere
.
- graph
Notes
This method does no special handling of empty quantum graphs. If needed, clients can use
len
to test if the returned graph is empty.
- pre_execute_qgraph(graph: QuantumGraph, register_dataset_types: bool = False, save_init_outputs: bool = True, save_versions: bool = True) None ¶
Run pre-execution initialization.
This method will be deprecated after DM-38041, to be replaced with a method that takes either a
Pipeline
or a resolvedpipeline_graph.PipelineGraph
instead of aQuantumGraph
.- Parameters:
- graph
QuantumGraph
The quantum graph defining the pipeline and datasets to be initialized.
- register_dataset_types
bool
, optional If
True
, register all output dataset types from the pipeline represented bygraph
.- save_init_outputs
bool
, optional If
True
, create init-output datasets in this object’s output run.- save_versions
bool
, optional If
True
, save a package versions dataset.
- graph
- run_pipeline(graph: QuantumGraph, fail_fast: bool = False, graph_executor: QuantumGraphExecutor | None = None, num_proc: int = 1) None ¶
Run a pipeline in the form of a prepared quantum graph.
Pre-execution initialization must have already been run; see
pre_execute_qgraph
.- Parameters:
- graph
QuantumGraph
The pipeline and datasets to execute.
- fail_fast
bool
, optional If
True
, abort all execution if any task fails when running with multiple processes. Only used with the default graph executor).- graph_executor
quantum_graph_executor.QuantumGraphExecutor
, optional A custom graph executor. By default, a new instance of
mp_graph_executor.MPGraphExecutor
is used.- num_proc
int
, optional The number of processes that can be used to run the pipeline. The default value ensures that no subprocess is created. Only used with the default graph executor.
- graph