SeparablePipelineExecutor

class lsst.ctrl.mpexec.SeparablePipelineExecutor(*args, **kwargs)

Bases: SeparablePipelineExecutor

Deprecated since version v30: The SeparablePipelineExecutor class has moved to lsst.pipe.base.separable_pipeline_executor. This forwarding shim will be removed after v30.

Methods Summary

make_pipeline(pipeline_uri)	Build a pipeline from pipeline and configuration information.
make_quantum_graph(pipeline[, where, ...])	Build a quantum graph from a pipeline and input datasets.
pre_execute_qgraph(graph[, ...])	Run pre-execution initialization.
run_pipeline(graph[, fail_fast, ...])	Run a pipeline in the form of a prepared quantum graph.

Methods Documentation

make_pipeline(pipeline_uri: str | ResourcePath) → Pipeline

Build a pipeline from pipeline and configuration information.

Parameters:

pipeline_uristr or lsst.resources.ResourcePath

URI to a file containing a pipeline definition. A URI fragment may be used to specify a subset of the pipeline, as described in Command line options for running Pipelines.

Returns:

pipelinePipeline

The fully-built pipeline.

make_quantum_graph(pipeline: ~lsst.pipe.base.pipeline.Pipeline, where: str = '', *, builder_class: type[lsst.pipe.base.quantum_graph_builder.QuantumGraphBuilder] = <class 'lsst.pipe.base.all_dimensions_quantum_graph_builder.AllDimensionsQuantumGraphBuilder'>, attach_datastore_records: bool = False, **kwargs: ~typing.Any) → QuantumGraph

Build a quantum graph from a pipeline and input datasets.

Parameters:

pipelinePipeline

The pipeline for which to generate a quantum graph.

wherestr, optional

A data ID query that constrains the quanta generated. Must not be provided if a custom builder_class is given and that class does not accept where as a construction argument.

builder_classtype [ quantum_graph_builder.QuantumGraphBuilder ], optional

Quantum graph builder implementation. Ignored if builder is provided.

attach_datastore_recordsbool, optional

Whether to attach datastore records. These are currently used only by lsst.daf.butler.QuantumBackedButler, which is not used by SeparablePipelineExecutor for execution.

**kwargs

Additional keyword arguments are forwarded to builder_class when a quantum graph builder instance is constructed. All arguments accepted by the QuantumGraphBuilder base class are provided automatically (from explicit arguments to this method and executor attributes) and do not need to be included as keyword arguments.

Returns:

graphQuantumGraph

The quantum graph for .Pipeline as run on the datasets identified by where.

Notes

This method does no special handling of empty quantum graphs. If needed, clients can use len to test if the returned graph is empty.

pre_execute_qgraph(graph: QuantumGraph, register_dataset_types: bool = False, save_init_outputs: bool = True, save_versions: bool = True) → None

Run pre-execution initialization.

This method will be deprecated after DM-38041, to be replaced with a method that takes either a Pipeline or a resolved pipeline_graph.PipelineGraph instead of a QuantumGraph.

Parameters:

graphQuantumGraph

The quantum graph defining the pipeline and datasets to be initialized.

register_dataset_typesbool, optional

If True, register all output dataset types from the pipeline represented by graph.

save_init_outputsbool, optional

If True, create init-output datasets in this object’s output run.

save_versionsbool, optional

If True, save a package versions dataset.

run_pipeline(graph: QuantumGraph, fail_fast: bool = False, graph_executor: QuantumGraphExecutor | None = None, num_proc: int = 1) → None

Run a pipeline in the form of a prepared quantum graph.

Pre-execution initialization must have already been run; see pre_execute_qgraph.

Parameters:

graphQuantumGraph

The pipeline and datasets to execute.

fail_fastbool, optional

If True, abort all execution if any task fails when running with multiple processes. Only used with the default graph executor).

graph_executorquantum_graph_executor.QuantumGraphExecutor, optional

A custom graph executor. By default, a new instance of mp_graph_executor.MPGraphExecutor is used.

num_procint, optional

The number of processes that can be used to run the pipeline. The default value ensures that no subprocess is created. Only used with the default graph executor.