PredictedQuantumGraph¶

class lsst.pipe.base.quantum_graph.PredictedQuantumGraph(components: PredictedQuantumGraphComponents)¶

Bases: BaseQuantumGraph

A directed acyclic graph that predicts a processing run and supports it during execution.

Parameters:

componentsPredictedQuantumGraphComponents: A struct of components used to construct the graph.

Notes

Iteration over a PredictedQuantumGraph yields loaded quantum IDs in deterministic topological order (but the tiebreaker is unspecified). The len of a PredictedQuantumGraph is the number of loaded non-init quanta, i.e. the same as the number of quanta iterated over.

Attributes Summary

`bipartite_xgraph`	A directed acyclic graph with quantum and dataset nodes.
`datasets_by_type`	A nested mapping of all datasets, keyed first by dataset type name and then by data ID.
`dimension_data`	All dimension records needed to expand the data IDS in the graph.
`quanta_by_task`	A nested mapping of all quanta, keyed first by task name and then by data ID.
`quantum_only_xgraph`	A directed acyclic graph with quanta as nodes and datasets elided.

Methods Summary

`build_execution_quanta`([quantum_ids, task_label])	Build `lsst.daf.butler.Quantum` objects suitable for executing tasks.
`from_old_quantum_graph`(old_quantum_graph)	Construct from an old `QuantumGraph` instance.
`get_init_inputs`(task_label)	Return the init-input datasets for the given task.
`get_init_outputs`(task_label)	Return the init-output datasets for the given task.
`init_output_run`(butler[, existing])	Initialize a new output RUN collection by writing init-output datasets (including configs and packages).
`make_init_qbb`(butler_config, *[, ...])	Construct an quantum-backed butler suitable for reading and writing init input and init output datasets, respectively.
`open`(uri[, page_size, import_mode])	Open a quantum graph and return a reader to load from it.
`read_execution_quanta`(uri[, quantum_ids, ...])	Read one or more executable quanta from a quantum graph file.
`to_old_quantum_graph`()	Transform into an old `QuantumGraph` instance.
`write_configs`(butler[, compare_existing])	Write the config datasets for all tasks in the quantum graph.
`write_init_outputs`(butler[, skip_existing])	Write the init-output datasets for all tasks in the quantum graph.
`write_packages`(butler[, compare_existing])	Write the 'packages' dataset for the currently-active software versions.

Attributes Documentation

bipartite_xgraph¶

A directed acyclic graph with quantum and dataset nodes.

This graph never includes init-input and init-output datasets.

Notes

Node keys are quantum or dataset UUIDs. Nodes for quanta are present if the thin_graph component is loaded (all nodes) or if the quantum_datasets component is loaded (just loaded quanta). Edges and dataset nodes are only present for quanta whose quantum_datasets were loaded.

Node state dictionaries are described by the PredictedQuantumInfo and PredictedDatasetInfo types.

The returned object is a read-only view of an internal one.

datasets_by_type¶

A nested mapping of all datasets, keyed first by dataset type name and then by data ID.

Notes

This is populated only by the quantum_datasets and init_quanta components, and only datasets referenced by loaded quanta are present. All dataset types in the pipeline graph are included, even if none of their datasets were loaded (i.e. nested mappings may be empty).

The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.

dimension_data¶

All dimension records needed to expand the data IDS in the graph.

This may be None if the dimension data was not loaded. If all execution quanta have been built, all records are guaranteed to have been deserialized and the records attribute is complete. In other cases some records may still only be present in the deserializers attribute.

quanta_by_task¶

A nested mapping of all quanta, keyed first by task name and then by data ID.

Notes

This is populated by the thin_graph component (all quanta are added) and the quantum_datasets` component (only loaded quanta are added). All tasks in the pipeline graph are included, even if none of their quanta were loaded (i.e. nested mappings may be empty).

The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.

quantum_only_xgraph¶

A directed acyclic graph with quanta as nodes and datasets elided.

Notes

Node keys are quantum UUIDs, and are populated by the thin_graph component (all nodes and edges) and quantum_datasets component (only those that were loaded).

Node state dictionaries are described by the PredictedQuantumInfo type.

The returned object is a read-only view of an internal one.

Methods Documentation

build_execution_quanta(quantum_ids: Iterable[UUID] | None = None, task_label: str | None = None) → dict[uuid.UUID, lsst.daf.butler._quantum.Quantum]¶

Build lsst.daf.butler.Quantum objects suitable for executing tasks.

In addition to returning the quantum objects directly, this also causes the quantum_only_xgraph and bipartite_xgraph graphs to include a quantum attribute for the affected quanta.

Parameters:

quantum_idsIterable [ uuid.UUID ], optional: IDs of all quanta to return. If not provided, all quanta for the given task label (if given) or graph are returned.
task_labelstr, optional: Task label whose quanta should be generated. Ignored if quantum_ids is not None.

Returns:

quantadict [ uuid.UUID, lsst.daf.butler.Quantum ]: Mapping of quanta, keyed by UUID. All dataset types are adapted to the task’s storage class declarations and inputs may be components. All data IDs have dimension records attached.

classmethod from_old_quantum_graph(old_quantum_graph: QuantumGraph) → PredictedQuantumGraph¶

Construct from an old QuantumGraph instance.

Parameters:

old_quantum_graphQuantumGraph: Quantum graph to transform.

Returns:

predicted_quantum_graphPredictedQuantumGraph: A new predicted quantum graph.

get_init_inputs(task_label: str) → dict[str, lsst.daf.butler._dataset_ref.DatasetRef]¶

Return the init-input datasets for the given task.

Parameters:

task_labelstr: Label of the task.

Returns:

init_inputsdict [ str, lsst.daf.butler.DatasetRef ]: Dataset references for init-input datasets, keyed by connection name. Dataset types storage classes match the task connection declarations, not necessarily the data repository, and may be components.

get_init_outputs(task_label: str) → dict[str, lsst.daf.butler._dataset_ref.DatasetRef]¶

Return the init-output datasets for the given task.

Parameters:

task_labelstr: Label of the task. "" may be used to get global init-outputs.

Returns:

init_outputsdict [ str, lsst.daf.butler.DatasetRef ]: Dataset references for init-outputs datasets, keyed by connection name. Dataset types storage classes match the task connection declarations, not necessarily the data repository.

init_output_run(butler: LimitedButler, existing: bool = True) → None¶

Initialize a new output RUN collection by writing init-output datasets (including configs and packages).

Parameters:

butlerlsst.daf.butler.LimitedButler: A limited butler data repository client.
existingbool, optional: If True check or ignore outputs that already exist. If False, always raise if an output dataset already exists.

Raises:

lsst.daf.butler.registry.ConflictingDefinitionError: Raised if there are existing init output datasets, and either existing=False or their contents are not compatible with this graph.

Construct an quantum-backed butler suitable for reading and writing init input and init output datasets, respectively.

This only requires the init_quanta component to have been loaded.

Parameters:

butler_configConfig or ResourcePathExpression: A butler repository root, configuration filename, or configuration instance.
config_search_pathsIterable [ str ], optional: Additional search paths for butler configuration.

Returns:

qbbQuantumBackedButler: A limited butler that can get init-input datasets and put init-output datasets.

classmethod open(uri: str | ParseResult | ResourcePath | Path, page_size: int = 5000000, import_mode: TaskImportMode = TaskImportMode.ASSUME_CONSISTENT_EDGES) → AbstractContextManager[PredictedQuantumGraphReader]¶

Open a quantum graph and return a reader to load from it.

Parameters:

uriconvertible to lsst.resources.ResourcePath: URI to open. Should have a .qg extension.
page_sizeint, optional: Approximate number of bytes to read at once from address files. Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads.
import_modepipeline_graph.TaskImportMode, optional: How to handle importing the task classes referenced in the pipeline graph.

Returns:

readercontextlib.AbstractContextManager [ PredictedQuantumGraphReader ]: A context manager that returns the reader when entered.

classmethod read_execution_quanta(uri: str | ParseResult | ResourcePath | Path, quantum_ids: Iterable[UUID] | None = None, page_size: int = 5000000) → PredictedQuantumGraph¶

Read one or more executable quanta from a quantum graph file.

Parameters:

uriconvertible to lsst.resources.ResourcePath: URI to open. Should have a .qg extension for new quantum graph files, or .qgraph for the old format.
quantum_idsIterable [ uuid.UUID ], optional: Iterable of quantum IDs to load. If not provided, all quanta will be loaded. The UUIDs of special init quanta will be ignored.
page_sizeint, optional: Approximate number of bytes to read at once from address files. Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads.

Returns:

quantum_graphPredictedQuantumGraph ]: A quantum graph that can build execution quanta for all of the given IDs.

to_old_quantum_graph() → QuantumGraph¶

Transform into an old QuantumGraph instance.

Returns:

old_quantum_graphQuantumGraph: Old quantum graph.

Notes

This can only be called on graphs that have loaded all quantum datasets, init datasets, and dimension records.

write_configs(butler: LimitedButler, compare_existing: bool = True) → None¶

Write the config datasets for all tasks in the quantum graph.

Parameters:

butlerlsst.daf.butler.LimitedButler: A limited butler data repository client.
compare_existingbool, optional: If True check configs that already exist for consistency. If False, always raise if configs already exist.

Raises:

lsst.daf.butler.registry.ConflictingDefinitionError: Raised if an config dataset already exists and compare_existing=False, or if the existing config is not consistent with the config in the quantum graph.

write_init_outputs(butler: LimitedButler, skip_existing: bool = True) → None¶

Write the init-output datasets for all tasks in the quantum graph.

This only requires the init_quanta component to have been loaded.

Parameters:

butlerlsst.daf.butler.LimitedButler: A limited butler data repository client.
skip_existingbool, optional: If True (default) ignore init-outputs that already exist. If False, raise.

Raises:

lsst.daf.butler.registry.ConflictingDefinitionError: Raised if an init-output dataset already exists and skip_existing=False.

write_packages(butler: LimitedButler, compare_existing: bool = True) → None¶

Write the ‘packages’ dataset for the currently-active software versions.

Parameters:

butlerlsst.daf.butler.LimitedButler: A limited butler data repository client.
compare_existingbool, optional: If True check packages that already exist for consistency. If False, always raise if the packages dataset already exists.

Raises:

lsst.daf.butler.registry.ConflictingDefinitionError: Raised if the packages dataset already exists and is not consistent with the current packages.

Navigation

PredictedQuantumGraph¶