SimplePipelineExecutor

class lsst.ctrl.mpexec.SimplePipelineExecutor(*args, **kwargs)

Bases: SimplePipelineExecutor

Deprecated since version v30: The SimplePipelineExecutor class has moved to lsst.pipe.base.simple_pipeline_executor. This forwarding shim will be removed after v30.

Methods Summary

as_generator([register_dataset_types, ...])	Yield quanta in the QuantumGraph in topological order.
from_pipeline(pipeline, *[, where, bind, ...])	Create an executor by building a QuantumGraph from an in-memory pipeline.
from_pipeline_filename(pipeline_filename, *)	Create an executor by building a QuantumGraph from an on-disk pipeline YAML file.
from_pipeline_graph(pipeline_graph, *[, ...])	Create an executor by building a QuantumGraph from an in-memory pipeline graph.
from_task_class(task_class[, config, label, ...])	Create an executor by building a QuantumGraph from a pipeline containing a single task.
prep_butler(root, inputs, output[, output_run])	Return configured Butler.
run([register_dataset_types, save_versions])	Run all the quanta in the QuantumGraph in topological order.
use_local_butler(root[, ...])	Transfer all inputs to a local data repository.

Methods Documentation

as_generator(register_dataset_types: bool = False, save_versions: bool = True) → Iterator[Quantum]

Yield quanta in the QuantumGraph in topological order.

These quanta will be run as the returned generator is iterated over. Use this method to run the quanta one at a time. Use run to run all quanta in the graph.

Parameters:

register_dataset_typesbool, optional

If True, register all output dataset types before executing any quanta.

save_versionsbool, optional

If True (default), save a package versions dataset.

Returns:

quantaIterator [ Quantum ]

Executed quanta.

Notes

Global initialization steps (see QuantumGraph.init_output_run) are performed immediately when this method is called, but individual quanta are not actually executed until the returned iterator is iterated over.

A topological ordering is not in general unique, but no other guarantees are made about the order in which quanta are processed.

classmethod from_pipeline(pipeline: Pipeline, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True, attach_datastore_records: bool = False, output: str | None = None, output_run: str | None = None) → SimplePipelineExecutor

Create an executor by building a QuantumGraph from an in-memory pipeline.

Parameters:

pipelinePipeline or Iterable [ TaskDef ]

A Python object describing the tasks to run, along with their labels and configuration.

wherestr, optional

Data ID query expression that constraints the quanta generated.

bindMapping, optional

Mapping containing literal values that should be injected into the where expression, keyed by the identifiers they replace.

butlerButler

Butler that manages all I/O. prep_butler can be used to create one.

resourcesExecutionResources

The resources available to each quantum being executed.

raise_on_partial_outputsbool, optional

If True raise exceptions chained by AnnotatedPartialOutputsError immediately, instead of considering the partial result a success and continuing to run downstream tasks.

attach_datastore_recordsbool, optional

Whether to attach datastore records to the quantum graph. This is usually unnecessary, unless the executor is used to test behavior that depends on datastore records.

outputstr, optional

Name of a new output CHAINED collection to create that will combine both inputs and outputs.

output_runstr, optional

Name of the output RUN that will directly hold all output datasets. If not provided, a name will be created from output and a timestamp.

Returns:

executorSimplePipelineExecutor

An executor instance containing the constructed QuantumGraph and Butler, ready for run to be called.

classmethod from_pipeline_filename(pipeline_filename: str, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True, attach_datastore_records: bool = False, output: str | None = None, output_run: str | None = None) → SimplePipelineExecutor

Create an executor by building a QuantumGraph from an on-disk pipeline YAML file.

Parameters:

pipeline_filenamestr

Name of the YAML file to load the pipeline definition from.

wherestr, optional

Data ID query expression that constraints the quanta generated.

bindMapping, optional

Mapping containing literal values that should be injected into the where expression, keyed by the identifiers they replace.

butlerButler

Butler that manages all I/O. prep_butler can be used to create one.

resourcesExecutionResources

The resources available to each quantum being executed.

raise_on_partial_outputsbool, optional

If True raise exceptions chained by AnnotatedPartialOutputsError immediately, instead of considering the partial result a success and continuing to run downstream tasks.

attach_datastore_recordsbool, optional

Whether to attach datastore records to the quantum graph. This is usually unnecessary, unless the executor is used to test behavior that depends on datastore records.

outputstr, optional

Name of a new output CHAINED collection to create that will combine both inputs and outputs.

output_runstr, optional

Name of the output RUN that will directly hold all output datasets. If not provided, a name will be created from output and a timestamp.

Returns:

executorSimplePipelineExecutor

An executor instance containing the constructed QuantumGraph and Butler, ready for run to be called.

classmethod from_pipeline_graph(pipeline_graph: PipelineGraph, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True, attach_datastore_records: bool = False, output: str | None = None, output_run: str | None = None) → SimplePipelineExecutor

Create an executor by building a QuantumGraph from an in-memory pipeline graph.

Parameters:

pipeline_graphPipelineGraph

A Python object describing the tasks to run, along with their labels and configuration, in graph form. Will be resolved against the given butler, with any existing resolutions ignored.

wherestr, optional

Data ID query expression that constraints the quanta generated.

bindMapping, optional

Mapping containing literal values that should be injected into the where expression, keyed by the identifiers they replace.

butlerButler

Butler that manages all I/O. prep_butler can be used to create one. Must have its run and butler.collections.defaults not empty and not None.

resourcesExecutionResources

The resources available to each quantum being executed.

raise_on_partial_outputsbool, optional

If True raise exceptions chained by AnnotatedPartialOutputsError immediately, instead of considering the partial result a success and continuing to run downstream tasks.

attach_datastore_recordsbool, optional

Whether to attach datastore records to the quantum graph. This is usually unnecessary, unless the executor is used to test behavior that depends on datastore records.

outputstr, optional

Name of a new output CHAINED collection to create that will combine both inputs and outputs.

output_runstr, optional

Name of the output RUN that will directly hold all output datasets. If not provided, a name will be created from output and a timestamp.

Returns:

executorSimplePipelineExecutor

An executor instance containing the constructed QuantumGraph and Butler, ready for run to be called.

classmethod from_task_class(task_class: type[lsst.pipe.base.pipelineTask.PipelineTask], config: Config | None = None, label: str | None = None, *, where: str = '', bind: Mapping[str, Any] | None = None, butler: Butler, resources: ExecutionResources | None = None, raise_on_partial_outputs: bool = True, attach_datastore_records: bool = False, output: str | None = None, output_run: str | None = None) → SimplePipelineExecutor

Create an executor by building a QuantumGraph from a pipeline containing a single task.

Parameters:

task_classtype

A concrete PipelineTask subclass.

configConfig, optional

Configuration for the task. If not provided, task-level defaults will be used (no per-instrument overrides).

labelstr, optional

Label for the task in its pipeline; defaults to task_class._DefaultName.

wherestr, optional

Data ID query expression that constraints the quanta generated.

bindMapping, optional

Mapping containing literal values that should be injected into the where expression, keyed by the identifiers they replace.

butlerButler

Butler that manages all I/O. prep_butler can be used to create one.

resourcesExecutionResources

The resources available to each quantum being executed.

raise_on_partial_outputsbool, optional

If True raise exceptions chained by AnnotatedPartialOutputsError immediately, instead of considering the partial result a success and continuing to run downstream tasks.

attach_datastore_recordsbool, optional

Whether to attach datastore records to the quantum graph. This is usually unnecessary, unless the executor is used to test behavior that depends on datastore records.

outputstr, optional

Name of a new output CHAINED collection to create that will combine both inputs and outputs.

output_runstr, optional

Name of the output RUN that will directly hold all output datasets. If not provided, a name will be created from output and a timestamp.

Returns:

executorSimplePipelineExecutor

An executor instance containing the constructed QuantumGraph and Butler, ready for run to be called.

classmethod prep_butler(root: str, inputs: Iterable[str], output: str, output_run: str | None = None) → Butler

Return configured Butler.

Helper method for creating Butler instances with collections appropriate for processing.

Parameters:

rootstr

Root of the butler data repository; must already exist, with all necessary input data.

inputsIterable [ str ]

Collections to search for all input datasets, in search order.

outputstr

Name of a new output CHAINED collection to create that will combine both inputs and outputs.

output_runstr, optional

Name of the output RUN that will directly hold all output datasets. If not provided, a name will be created from output and a timestamp.

Returns:

butlerButler

Butler client instance compatible with all classmethod factories. Always writeable.

run(register_dataset_types: bool = False, save_versions: bool = True) → list[lsst.daf.butler._quantum.Quantum]

Run all the quanta in the QuantumGraph in topological order.

Use this method to run all quanta in the graph. Use as_generator to get a generator to run the quanta one at a time.

Parameters:

register_dataset_typesbool, optional

If True, register all output dataset types before executing any quanta.

save_versionsbool, optional

If True (default), save a package versions dataset.

Returns:

quantalist [ Quantum ]

Executed quanta.

Notes

A topological ordering is not in general unique, but no other guarantees are made about the order in which quanta are processed.

use_local_butler(root: str, register_dataset_types: bool = True, transfer_dimensions: bool = True) → Butler

Transfer all inputs to a local data repository. and set the executor to write outputs to it.

Parameters:

rootstr

Path to the local data repository; created if it does not exist.

register_dataset_typesbool, optional

Whether to register dataset types in the new repository. If False, the local data repository must already exist and already have all input dataset types registered.

transfer_dimensionsbool, optional

Whether to transfer dimension records to the new repository. If False, the local data repository must already exist and already have all needed dimension records.

Returns:

butlerlsst.daf.butler.Butler

Writeable butler for local data repository.

Notes

The input collection structure from the original data repository is not preserved by this method (it cannot be reconstructed from the quantum graph). Instead, a TAGGED collection is created to gather all inputs, and appended to the output CHAINED collection after the output RUN collection. Calibration inputs with the same data ID but multiple validity ranges are not included in that TAGGED; they are still transferred to the local data repository, but can only be found via the quantum graph or their original RUN collections.