PreExecInit

class lsst.ctrl.mpexec.PreExecInit(butler: Butler, taskFactory: TaskFactory, extendRun: bool = False, mock: bool = False)

Bases: PreExecInitBase

Initialization of registry for QuantumGraph execution.

This class encapsulates all necessary operations that have to be performed on butler and registry to prepare them for QuantumGraph execution.

Parameters:

butlerButler

Data butler instance.

taskFactoryTaskFactory

Task factory.

extendRunbool, optional

If True then do not try to overwrite any datasets that might exist in butler.run; instead compare them when appropriate/possible. If False, then any existing conflicting dataset will cause a butler exception to be raised.

mockbool, optional

If True then also do initialization needed for pipeline mocking.

Methods Summary

find_init_input_refs(taskDef, graph)	Return the list of resolved dataset references for task init inputs.
find_init_output(taskDef, dataset_type_name, ...)	Find task init output for given dataset type.
find_packages(graph)	Find packages information.
initialize(graph[, saveInitOutputs, ...])	Perform all initialization steps.
initializeDatasetTypes(graph[, ...])	Save or check DatasetTypes output by the tasks in a graph.
saveConfigs(graph)	Write configurations for pipeline tasks to butler or check that existing configurations are equal to the new ones.
saveInitOutputs(graph)	Write any datasets produced by initializing tasks in a graph.
savePackageVersions(graph)	Write versions of software packages to butler.
transaction()	Context manager for transaction.

Methods Documentation

find_init_input_refs(taskDef: TaskDef, graph: QuantumGraph) → Iterable[DatasetRef]

Return the list of resolved dataset references for task init inputs.

Parameters:

taskDefTaskDef

Pipeline task definition.

graphQuantumGraph

Quantum graph.

Returns:

refsIterable [DatasetRef]

Resolved dataset references.

find_init_output(taskDef: TaskDef, dataset_type_name: str, graph: QuantumGraph) → tuple[Any | None, DatasetRef]

Find task init output for given dataset type.

Parameters:

taskDefTaskDef

Pipeline task definition.

dataset_typestr

Dataset type name.

graphQuantumGraph

Quantum graph.

Returns:

data

Existing init output object retrieved from butler, None if butler has no existing object.

refDatasetRef

Resolved reference for init output to be stored in butler.

Raises:

MissingReferenceError

Raised if reference cannot be found or generated.

find_packages(graph: QuantumGraph) → tuple[Packages | None, DatasetRef]

Find packages information.

Parameters:

graphQuantumGraph

Quantum graph.

Returns:

packageslsst.utils.packages.Packages or None

Existing packages data retrieved from butler, or None.

refDatasetRef

Resolved reference for packages to be stored in butler.

Raises:

MissingReferenceError

Raised if reference cannot be found or generated.

initialize(graph: QuantumGraph, saveInitOutputs: bool = True, registerDatasetTypes: bool = False, saveVersions: bool = True) → None

Perform all initialization steps.

Convenience method to execute all initialization steps. Instead of calling this method and providing all options it is also possible to call methods individually.

Parameters:

graphQuantumGraph

Execution graph.

saveInitOutputsbool, optional

If True (default) then save “init outputs”, configurations, and package versions to butler.

registerDatasetTypesbool, optional

If True then register dataset types in registry, otherwise they must be already registered.

saveVersionsbool, optional

If False then do not save package versions even if saveInitOutputs is set to True.

initializeDatasetTypes(graph: QuantumGraph, registerDatasetTypes: bool = False) → None

Save or check DatasetTypes output by the tasks in a graph.

Iterates over all DatasetTypes for all tasks in a graph and either tries to add them to registry or compares them to existing ones.

Parameters:

graphQuantumGraph

Execution graph.

registerDatasetTypesbool, optional

If True then register dataset types in registry, otherwise they must be already registered.

Raises:

ValueError

Raised if existing DatasetType is different from DatasetType in a graph.

KeyError

Raised if registerDatasetTypes is False and DatasetType does not exist in registry.

saveConfigs(graph: QuantumGraph) → None

Write configurations for pipeline tasks to butler or check that existing configurations are equal to the new ones.

Parameters:

graphQuantumGraph

Execution graph.

Raises:

TypeError

Raised if existing object in butler is different from new data.

Exception

Raised if extendRun is False and datasets already exists. Content of a butler collection should not be changed if exception is raised.

saveInitOutputs(graph: QuantumGraph) → None

Write any datasets produced by initializing tasks in a graph.

Parameters:

graphQuantumGraph

Execution graph.

Raises:

TypeError

Raised if the type of existing object in butler is different from new data.

savePackageVersions(graph: QuantumGraph) → None

Write versions of software packages to butler.

Parameters:

graphQuantumGraph

Execution graph.

Raises:

TypeError

Raised if existing object in butler is incompatible with new data.

transaction() → Iterator[None]

Context manager for transaction.

Default implementation has no transaction support.