PredictedQuantumGraph¶
- class lsst.pipe.base.quantum_graph.PredictedQuantumGraph(components: PredictedQuantumGraphComponents)¶
Bases:
BaseQuantumGraph
A directed acyclic graph that predicts a processing run and supports it during execution.
- Parameters:
- components
PredictedQuantumGraphComponents
A struct of components used to construct the graph.
- components
Notes
Iteration over a
PredictedQuantumGraph
yields loaded quantum IDs in deterministic topological order (but the tiebreaker is unspecified). Thelen
of aPredictedQuantumGraph
is the number of loaded non-init quanta, i.e. the same as the number of quanta iterated over.Attributes Summary
A directed acyclic graph with quantum and dataset nodes.
A nested mapping of all datasets, keyed first by dataset type name and then by data ID.
All dimension records needed to expand the data IDS in the graph.
A nested mapping of all quanta, keyed first by task name and then by data ID.
A directed acyclic graph with quanta as nodes and datasets elided.
Methods Summary
build_execution_quanta
([quantum_ids, task_label])Build
lsst.daf.butler.Quantum
objects suitable for executing tasks.from_old_quantum_graph
(old_quantum_graph)Construct from an old
QuantumGraph
instance.get_init_inputs
(task_label)Return the init-input datasets for the given task.
get_init_outputs
(task_label)Return the init-output datasets for the given task.
init_output_run
(butler[, existing])Initialize a new output RUN collection by writing init-output datasets (including configs and packages).
make_init_qbb
(butler_config, *[, ...])Construct an quantum-backed butler suitable for reading and writing init input and init output datasets, respectively.
open
(uri[, page_size, import_mode])Open a quantum graph and return a reader to load from it.
read_execution_quanta
(uri[, quantum_ids, ...])Read one or more executable quanta from a quantum graph file.
Transform into an old
QuantumGraph
instance.write_configs
(butler[, compare_existing])Write the config datasets for all tasks in the quantum graph.
write_init_outputs
(butler[, skip_existing])Write the init-output datasets for all tasks in the quantum graph.
write_packages
(butler[, compare_existing])Write the 'packages' dataset for the currently-active software versions.
Attributes Documentation
- bipartite_xgraph¶
A directed acyclic graph with quantum and dataset nodes.
This graph never includes init-input and init-output datasets.
Notes
Node keys are quantum or dataset UUIDs. Nodes for quanta are present if the
thin_graph
component is loaded (all nodes) or if thequantum_datasets
component is loaded (just loaded quanta). Edges and dataset nodes are only present for quanta whosequantum_datasets
were loaded.Node state dictionaries are described by the
PredictedQuantumInfo
andPredictedDatasetInfo
types.The returned object is a read-only view of an internal one.
- datasets_by_type¶
A nested mapping of all datasets, keyed first by dataset type name and then by data ID.
Notes
This is populated only by the
quantum_datasets
andinit_quanta
components, and only datasets referenced by loaded quanta are present. All dataset types in the pipeline graph are included, even if none of their datasets were loaded (i.e. nested mappings may be empty).The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.
- dimension_data¶
All dimension records needed to expand the data IDS in the graph.
This may be
None
if the dimension data was not loaded. If all execution quanta have been built, all records are guaranteed to have been deserialized and therecords
attribute is complete. In other cases some records may still only be present in thedeserializers
attribute.
- quanta_by_task¶
A nested mapping of all quanta, keyed first by task name and then by data ID.
Notes
This is populated by the
thin_graph
component (all quanta are added) and thequantum_datasets`
component (only loaded quanta are added). All tasks in the pipeline graph are included, even if none of their quanta were loaded (i.e. nested mappings may be empty).The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.
- quantum_only_xgraph¶
A directed acyclic graph with quanta as nodes and datasets elided.
Notes
Node keys are quantum UUIDs, and are populated by the
thin_graph
component (all nodes and edges) andquantum_datasets
component (only those that were loaded).Node state dictionaries are described by the
PredictedQuantumInfo
type.The returned object is a read-only view of an internal one.
Methods Documentation
- build_execution_quanta(quantum_ids: Iterable[UUID] | None = None, task_label: str | None = None) dict[uuid.UUID, lsst.daf.butler._quantum.Quantum] ¶
Build
lsst.daf.butler.Quantum
objects suitable for executing tasks.In addition to returning the quantum objects directly, this also causes the
quantum_only_xgraph
andbipartite_xgraph
graphs to include aquantum
attribute for the affected quanta.- Parameters:
- Returns:
- quanta
dict
[uuid.UUID
,lsst.daf.butler.Quantum
] Mapping of quanta, keyed by UUID. All dataset types are adapted to the task’s storage class declarations and inputs may be components. All data IDs have dimension records attached.
- quanta
- classmethod from_old_quantum_graph(old_quantum_graph: QuantumGraph) PredictedQuantumGraph ¶
Construct from an old
QuantumGraph
instance.- Parameters:
- old_quantum_graph
QuantumGraph
Quantum graph to transform.
- old_quantum_graph
- Returns:
- predicted_quantum_graph
PredictedQuantumGraph
A new predicted quantum graph.
- predicted_quantum_graph
- get_init_inputs(task_label: str) dict[str, lsst.daf.butler._dataset_ref.DatasetRef] ¶
Return the init-input datasets for the given task.
- Parameters:
- task_label
str
Label of the task.
- task_label
- Returns:
- init_inputs
dict
[str
,lsst.daf.butler.DatasetRef
] Dataset references for init-input datasets, keyed by connection name. Dataset types storage classes match the task connection declarations, not necessarily the data repository, and may be components.
- init_inputs
- get_init_outputs(task_label: str) dict[str, lsst.daf.butler._dataset_ref.DatasetRef] ¶
Return the init-output datasets for the given task.
- Parameters:
- task_label
str
Label of the task.
""
may be used to get global init-outputs.
- task_label
- Returns:
- init_outputs
dict
[str
,lsst.daf.butler.DatasetRef
] Dataset references for init-outputs datasets, keyed by connection name. Dataset types storage classes match the task connection declarations, not necessarily the data repository.
- init_outputs
- init_output_run(butler: LimitedButler, existing: bool = True) None ¶
Initialize a new output RUN collection by writing init-output datasets (including configs and packages).
- Parameters:
- butler
lsst.daf.butler.LimitedButler
A limited butler data repository client.
- existing
bool
, optional If
True
check or ignore outputs that already exist. IfFalse
, always raise if an output dataset already exists.
- butler
- Raises:
- lsst.daf.butler.registry.ConflictingDefinitionError
Raised if there are existing init output datasets, and either
existing=False
or their contents are not compatible with this graph.
- make_init_qbb(butler_config: Config | str | ParseResult | ResourcePath | Path, *, config_search_paths: Iterable[str] | None = None) QuantumBackedButler ¶
Construct an quantum-backed butler suitable for reading and writing init input and init output datasets, respectively.
This only requires the
init_quanta
component to have been loaded.- Parameters:
- Returns:
- qbb
QuantumBackedButler
A limited butler that can
get
init-input datasets andput
init-output datasets.
- qbb
- classmethod open(uri: str | ParseResult | ResourcePath | Path, page_size: int = 5000000, import_mode: TaskImportMode = TaskImportMode.ASSUME_CONSISTENT_EDGES) AbstractContextManager[PredictedQuantumGraphReader] ¶
Open a quantum graph and return a reader to load from it.
- Parameters:
- uriconvertible to
lsst.resources.ResourcePath
URI to open. Should have a
.qg
extension.- page_size
int
, optional Approximate number of bytes to read at once from address files. Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads.
- import_mode
pipeline_graph.TaskImportMode
, optional How to handle importing the task classes referenced in the pipeline graph.
- uriconvertible to
- Returns:
- reader
contextlib.AbstractContextManager
[PredictedQuantumGraphReader
] A context manager that returns the reader when entered.
- reader
- classmethod read_execution_quanta(uri: str | ParseResult | ResourcePath | Path, quantum_ids: Iterable[UUID] | None = None, page_size: int = 5000000) PredictedQuantumGraph ¶
Read one or more executable quanta from a quantum graph file.
- Parameters:
- uriconvertible to
lsst.resources.ResourcePath
URI to open. Should have a
.qg
extension for new quantum graph files, or.qgraph
for the old format.- quantum_ids
Iterable
[uuid.UUID
], optional Iterable of quantum IDs to load. If not provided, all quanta will be loaded. The UUIDs of special init quanta will be ignored.
- page_size
int
, optional Approximate number of bytes to read at once from address files. Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads.
- uriconvertible to
- Returns:
- quantum_graph
PredictedQuantumGraph
] A quantum graph that can build execution quanta for all of the given IDs.
- quantum_graph
- to_old_quantum_graph() QuantumGraph ¶
Transform into an old
QuantumGraph
instance.- Returns:
- old_quantum_graph
QuantumGraph
Old quantum graph.
- old_quantum_graph
Notes
This can only be called on graphs that have loaded all quantum datasets, init datasets, and dimension records.
- write_configs(butler: LimitedButler, compare_existing: bool = True) None ¶
Write the config datasets for all tasks in the quantum graph.
- Parameters:
- butler
lsst.daf.butler.LimitedButler
A limited butler data repository client.
- compare_existing
bool
, optional If
True
check configs that already exist for consistency. IfFalse
, always raise if configs already exist.
- butler
- Raises:
- lsst.daf.butler.registry.ConflictingDefinitionError
Raised if an config dataset already exists and
compare_existing=False
, or if the existing config is not consistent with the config in the quantum graph.
- write_init_outputs(butler: LimitedButler, skip_existing: bool = True) None ¶
Write the init-output datasets for all tasks in the quantum graph.
This only requires the
init_quanta
component to have been loaded.- Parameters:
- butler
lsst.daf.butler.LimitedButler
A limited butler data repository client.
- skip_existing
bool
, optional If
True
(default) ignore init-outputs that already exist. IfFalse
, raise.
- butler
- Raises:
- lsst.daf.butler.registry.ConflictingDefinitionError
Raised if an init-output dataset already exists and
skip_existing=False
.
- write_packages(butler: LimitedButler, compare_existing: bool = True) None ¶
Write the ‘packages’ dataset for the currently-active software versions.
- Parameters:
- butler
lsst.daf.butler.LimitedButler
A limited butler data repository client.
- compare_existing
bool
, optional If
True
check packages that already exist for consistency. IfFalse
, always raise if the packages dataset already exists.
- butler
- Raises:
- lsst.daf.butler.registry.ConflictingDefinitionError
Raised if the packages dataset already exists and is not consistent with the current packages.