PredictedQuantumGraphComponents#

class lsst.pipe.base.quantum_graph.PredictedQuantumGraphComponents(*, header: ~lsst.pipe.base.quantum_graph._common.HeaderModel = <factory>, pipeline_graph: ~lsst.pipe.base.pipeline_graph._pipeline_graph.PipelineGraph, dimension_data: ~lsst.daf.butler.dimensions._record_set.DimensionDataAttacher | None = None, init_quanta: ~lsst.pipe.base.quantum_graph._predicted.PredictedInitQuantaModel = <factory>, thin_graph: ~lsst.pipe.base.quantum_graph._predicted.PredictedThinGraphModel = <factory>, quantum_datasets: dict[~uuid.UUID, ~lsst.pipe.base.quantum_graph._predicted.PredictedQuantumDatasetsModel] = <factory>)#

Bases: object

A helper class for building and writing predicted quantum graphs.

Notes#

This class is a simple struct of model classes to allow different tools that build predicted quantum graphs to assemble them in whatever order they prefer. It does not enforce any internal invariants (e.g. the quantum and dataset counts in the header, different representations of quanta, internal ID sorting, etc.), but it does provide methods that can satisfy them.

Attributes Summary

`dimension_data`	Object that can attach dimension records to data IDs.
`header`	Basic metadata about the graph.
`init_quanta`	A list of special quanta that describe the init-inputs and init-outputs of the graph.
`pipeline_graph`	Description of the pipeline this graph runs, including all task label and dataset type definitions.
`quantum_datasets`	The full descriptions of all quanta, including input and output dataset, keyed by UUID.
`thin_graph`	A lightweight quantum-quantum DAG with task labels and data IDs only.

Methods Summary

`assemble`()	Construct a `PredictedQuantumGraph` from these components.
`from_old_quantum_graph`(old_quantum_graph)	Construct from an old `QuantumGraph` instance.
`make_dataset_ref`(predicted)	Make a `lsst.daf.butler.DatasetRef` from information in the predicted quantum graph.
`read_execution_quanta`(uri[, quantum_ids, ...])	Read one or more executable quanta from a quantum graph file.
`set_header_counts`()	Populate the quantum and dataset counts in the header from the `thin_graph`, `init_quanta`, and `quantum_datasets` components.
`set_thin_graph`()	Populate the `thin_graph` component from the `pipeline_graph`, `quantum_datasets` components (which must be complete).
`update_output_run`(output_run)	Update the output `RUN` collection name in all datasets and regenerate all output dataset and quantum UUIDs.
`write`(uri, *[, zstd_level, zstd_dict_size, ...])	Write the graph to a file.

Attributes Documentation

dimension_data: DimensionDataAttacher | None = None#: Object that can attach dimension records to data IDs.

header: HeaderModel = <dataclasses._MISSING_TYPE object>#: Basic metadata about the graph.

init_quanta: PredictedInitQuantaModel = <dataclasses._MISSING_TYPE object>#

A list of special quanta that describe the init-inputs and init-outputs of the graph.

Tasks that are included in the pipeline graph but do not have any quanta may or may not have an init quantum, but tasks that do have regular quanta always have an init quantum as well.

When used to construct a PredictedQuantumGraph, this must have either zero entries or all tasks in the pipeline.

pipeline_graph: PipelineGraph = <dataclasses._MISSING_TYPE object>#

Description of the pipeline this graph runs, including all task label and dataset type definitions.

This may include tasks that do not have any quanta (e.g. due to skipping already-executed tasks).

This also includes the dimension universe used to construct the graph.

quantum_datasets: dict[UUID, PredictedQuantumDatasetsModel] = <dataclasses._MISSING_TYPE object>#

The full descriptions of all quanta, including input and output dataset, keyed by UUID.

When used to construct a PredictedQuantumGraph, this need not have all entries.

This does not include special “init” quanta.

thin_graph: PredictedThinGraphModel = <dataclasses._MISSING_TYPE object>#

A lightweight quantum-quantum DAG with task labels and data IDs only.

This does not include the special “init” quanta.

Methods Documentation

assemble() → PredictedQuantumGraph#: Construct a PredictedQuantumGraph from these components.

classmethod from_old_quantum_graph(old_quantum_graph: QuantumGraph) → PredictedQuantumGraphComponents#

Construct from an old QuantumGraph instance.

Parameters#

old_quantum_graphQuantumGraph: Quantum graph to transform.

Returns#

componentsPredictedQuantumGraphComponents: Components for a new predicted quantum graph.

make_dataset_ref(predicted: PredictedDatasetModel) → DatasetRef#

Make a lsst.daf.butler.DatasetRef from information in the predicted quantum graph.

Parameters#

predictedPredictedDatasetModel: Model for the dataset in the predicted graph.

Returns#

reflsst.daf.butler.DatasetRef: A dataset reference. Data ID will be expanded if and only if the dimension data has been loaded.

classmethod read_execution_quanta(uri: str | ParseResult | ResourcePath | Path, quantum_ids: Iterable[UUID] | None = None, page_size: int = 5000000) → PredictedQuantumGraphComponents#

Read one or more executable quanta from a quantum graph file.

Parameters#

uriconvertible to lsst.resources.ResourcePath: URI to open. Should have a .qg extension for new quantum graph files, or .qgraph for the old format.
quantum_idsIterable [ uuid.UUID ], optional: Iterable of quantum IDs to load. If not provided, all quanta will be loaded. The UUIDs of special init quanta will be ignored.
page_sizeint, optional: Approximate number of bytes to read at once from address files. Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads.

Returns#

componentsPredictedQuantumGraphComponents ]: Components for quantum graph that can build execution quanta for all of the given IDs.

set_header_counts() → None#: Populate the quantum and dataset counts in the header from the thin_graph, init_quanta, and quantum_datasets components.

set_thin_graph() → None#: Populate the thin_graph component from the pipeline_graph, quantum_datasets components (which must be complete).

update_output_run(output_run: str) → None#

Update the output RUN collection name in all datasets and regenerate all output dataset and quantum UUIDs.

Parameters#

output_runstr: New output RUN collection name.

write(uri: str | ParseResult | ResourcePath | Path, *, zstd_level: int = 10, zstd_dict_size: int = 32768, zstd_dict_n_inputs: int = 512) → None#

Write the graph to a file.

Parameters#

uriconvertible to lsst.resources.ResourcePath: Path to write to. Should have a .qg extension, or .qgraph to force writing the old format.
zstd_levelint, optional: ZStandard compression level to use on JSON blocks.
zstd_dict_sizeint, optional: Size of a ZStandard dictionary that shares compression information across components. Set to zero to disable the dictionary. Dictionary compression is automatically disabled if the number of quanta is smaller than zstd_dict_n_inputs.
zstd_dict_n_inputsint, optional: Maximum number of PredictedQuantumDatasetsModel JSON representations to feed the ZStandard dictionary training routine.

Notes#

Only a complete predicted quantum graph with all components fully populated should be written.