ProvenanceQuantumGraph

class lsst.pipe.base.quantum_graph.ProvenanceQuantumGraph(header: HeaderModel, pipeline_graph: PipelineGraph)

Bases: BaseQuantumGraph

A quantum graph that represents processing that has already been executed.

Parameters:

headerHeaderModel

General metadata shared with other quantum graph types.

pipeline_graphpipeline_graph.PipelineGraph

Graph of tasks and dataset types. May contain a superset of the tasks and dataset types that actually have quanta and datasets in the quantum graph.

Notes

A provenance quantum graph is generally obtained via the ProvenanceQuantumGraphReader.graph attribute, which is updated in-place as information is read from disk.

Attributes Summary

bipartite_xgraph	A directed acyclic graph with quantum and dataset nodes.
datasets_by_type	A nested mapping of all datasets, keyed first by dataset type name and then by data ID.
init_quanta	A mapping from task label to the ID of the special init quantum for that task.
quanta_by_task	A nested mapping of all quanta, keyed first by task name and then by data ID.
quantum_only_xgraph	A directed acyclic graph with quanta as nodes (and datasets elided).

Methods Summary

make_exception_table()	Construct an astropy.table.Table with counts for each exception type raised by each task.
make_quantum_table()	Construct an astropy.table.Table with a tabular summary of the quanta.
make_task_resource_usage_table(task_label[, ...])	Make a table of resource usage for a single task.

Attributes Documentation

bipartite_xgraph

A directed acyclic graph with quantum and dataset nodes.

Notes

Node keys are quantum or dataset UUIDs, and are populated one quantum or dataset at a time as they are loaded. Loading quanta (via ProvenanceQuantumGraphReader.read_quanta) or datasets (via ProvenanceQuantumGraphReader.read_datasets) will load those nodes with full attributes and edges to adjacent nodes with no attributes. Loading quanta necessary to populate edge attributes.

Node attributes are described by the ProvenanceQuantumInfo, ProvenanceInitQuantumInfo, and ProvenanceDatasetInfo types.

This graph includes init-input and init-output datasets, but it does not reflect the dependency between each task’s special “init” quantum and its runtime quanta (as this would require edges between quanta, and that would break the “bipartite” property).

The returned object is a read-only view of an internal one.

datasets_by_type

A nested mapping of all datasets, keyed first by dataset type name and then by data ID.

Notes

This is populated one dataset at a time as they are read. All dataset types in the pipeline graph are included, even if none of their datasets were loaded (i.e. nested mappings may be empty).

The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.

init_quanta

A mapping from task label to the ID of the special init quantum for that task.

This is populated by the init_quanta component. Additional information about each init quantum can be found by using the ID to look up node attributes in the bipartite_xgraph, i.e.:

info: ProvenanceInitQuantumInfo = qg.bipartite_xgraph.nodes[id]

quanta_by_task

A nested mapping of all quanta, keyed first by task name and then by data ID.

Notes

This is populated one quantum at a time as they are read. All tasks in the pipeline graph are included, even if none of their quanta were loaded (i.e. nested mappings may be empty).

The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.

quantum_only_xgraph

A directed acyclic graph with quanta as nodes (and datasets elided).

Notes

Node keys are quantum UUIDs, and are populated one quantum at a time as they are loaded. Loading quanta (via ProvenanceQuantumGraphReader.read_quanta) will add the loaded nodes with full attributes and add edges to adjacent nodes with no attributes. Loading datasets (via ProvenanceQuantumGraphReader.read_datasets) will also add edges and nodes with no attributes.

Node attributes are described by the ProvenanceQuantumInfo types.

This graph does not include special “init” quanta.

The returned object is a read-only view of an internal one.

Methods Documentation

make_exception_table() → Table

Construct an astropy.table.Table with counts for each exception type raised by each task.

At present this only includes information from partial-outputs-error successes, since exception information for failures is not tracked. This may change in the future.

Returns:

tableastropy.table.Table

A table with columns for task label, exception type, and counts.

make_quantum_table() → Table

Construct an astropy.table.Table with a tabular summary of the quanta.

Returns:

tableastropy.table.Table

A table view of the quantum information. This only includes counts of status categories and caveats, not any per-data-ID detail.

Notes

Success caveats in the table are represented by their concise form, so when pretty-printing this table for users, the legend should generally be printed as well.

make_task_resource_usage_table(task_label: str, include_data_ids: bool = False) → Table

Make a table of resource usage for a single task.

Parameters:

task_labelstr

Label of the task to extract resource usage for.

include_data_idsbool, optional

Whether to also include data ID columns.

Returns:

tableastropy.table.Table

A table with columns for quantum ID and all fields in QuantumResourceUsage.