PreExecInit

class lsst.ctrl.mpexec.PreExecInit(butler: Butler, taskFactory: TaskFactory, extendRun: bool = False, mock: bool = False)

Bases: lsst.ctrl.mpexec.preExecInit.PreExecInitBase

Initialization of registry for QuantumGraph execution.

This class encapsulates all necessary operations that have to be performed on butler and registry to prepare them for QuantumGraph execution.

Parameters:

butler : Butler

Data butler instance.

taskFactory : TaskFactory

Task factory.

extendRun : bool, optional

If True then do not try to overwrite any datasets that might exist in butler.run; instead compare them when appropriate/possible. If False, then any existing conflicting dataset will cause a butler exception to be raised.

mock : bool, optional

If True then also do initialization needed for pipeline mocking.

Methods Summary

find_init_input_refs(taskDef, graph)	Return the list of resolved dataset references for task init inputs.
find_init_output(taskDef, dataset_type_name, …)	Find task init output for given dataset type.
find_packages(graph)	Find packages information.
initialize(graph, saveInitOutputs, …)	Perform all initialization steps.
initializeDatasetTypes(graph, …)	Save or check DatasetTypes output by the tasks in a graph.
saveConfigs(graph)	Write configurations for pipeline tasks to butler or check that existing configurations are equal to the new ones.
saveInitOutputs(graph)	Write any datasets produced by initializing tasks in a graph.
savePackageVersions(graph)	Write versions of software packages to butler.
transaction()	Context manager for transaction.

Methods Documentation

find_init_input_refs(taskDef: TaskDef, graph: QuantumGraph) → Iterable[DatasetRef]

Return the list of resolved dataset references for task init inputs.

Parameters:

taskDef : TaskDef

Pipeline task definition.

graph : QuantumGraph

Quantum graph.

Returns:

refs : Iterable [DatasetRef]

Resolved dataset references.

find_init_output(taskDef: TaskDef, dataset_type_name: str, graph: QuantumGraph) → tuple[Any | None, DatasetRef]

Find task init output for given dataset type.

Parameters:

taskDef : TaskDef

Pipeline task definition.

dataset_type : str

Dataset type name.

graph : QuantumGraph

Quantum graph.

Returns:

data

Existing init output object retrieved from butler, None if butler has no existing object.

ref : DatasetRef

Resolved reference for init output to be stored in butler.

Raises:

MissingReferenceError

Raised if reference cannot be found or generated.

find_packages(graph: QuantumGraph) → tuple[Packages | None, DatasetRef]

Find packages information.

Parameters:

graph : QuantumGraph

Quantum graph.

Returns:

packages : lsst.utils.packages.Packages or None

Existing packages data retrieved from butler, or None.

ref : DatasetRef

Resolved reference for packages to be stored in butler.

Raises:

MissingReferenceError

Raised if reference cannot be found or generated.

initialize(graph: QuantumGraph, saveInitOutputs: bool = True, registerDatasetTypes: bool = False, saveVersions: bool = True) → None

Perform all initialization steps.

Convenience method to execute all initialization steps. Instead of calling this method and providing all options it is also possible to call methods individually.

Parameters:

graph : QuantumGraph

Execution graph.

saveInitOutputs : bool, optional

If True (default) then save “init outputs”, configurations, and package versions to butler.

registerDatasetTypes : bool, optional

If True then register dataset types in registry, otherwise they must be already registered.

saveVersions : bool, optional

If False then do not save package versions even if saveInitOutputs is set to True.

initializeDatasetTypes(graph: QuantumGraph, registerDatasetTypes: bool = False) → None

Save or check DatasetTypes output by the tasks in a graph.

Iterates over all DatasetTypes for all tasks in a graph and either tries to add them to registry or compares them to existing ones.

Parameters:

graph : QuantumGraph

Execution graph.

registerDatasetTypes : bool, optional

If True then register dataset types in registry, otherwise they must be already registered.

Raises:

ValueError

Raised if existing DatasetType is different from DatasetType in a graph.

KeyError

Raised if registerDatasetTypes is False and DatasetType does not exist in registry.

saveConfigs(graph: QuantumGraph) → None

Write configurations for pipeline tasks to butler or check that existing configurations are equal to the new ones.

Parameters:

graph : QuantumGraph

Execution graph.

Raises:

TypeError

Raised if existing object in butler is different from new data.

Exception

Raised if extendRun is False and datasets already exists. Content of a butler collection should not be changed if exception is raised.

saveInitOutputs(graph: QuantumGraph) → None

Write any datasets produced by initializing tasks in a graph.

Parameters:

graph : QuantumGraph

Execution graph.

Raises:

TypeError

Raised if the type of existing object in butler is different from new data.

savePackageVersions(graph: QuantumGraph) → None

Write versions of software packages to butler.

Parameters:

graph : QuantumGraph

Execution graph.

Raises:

TypeError

Raised if existing object in butler is incompatible with new data.

transaction() → collections.abc.Iterator[None]

Context manager for transaction.

Default implementation has no transaction support.