ProvenanceQuantumGraph

class lsst.pipe.base.quantum_graph.ProvenanceQuantumGraph(header: HeaderModel, pipeline_graph: PipelineGraph)

Bases: BaseQuantumGraph

A quantum graph that represents processing that has already been executed.

Parameters:

headerHeaderModel

General metadata shared with other quantum graph types.

pipeline_graphpipeline_graph.PipelineGraph

Graph of tasks and dataset types. May contain a superset of the tasks and dataset types that actually have quanta and datasets in the quantum graph.

Notes

A provenance quantum graph is generally obtained via the ProvenanceQuantumGraphReader.graph attribute, which is updated in-place as information is read from disk.

Attributes Summary

bipartite_xgraph	A directed acyclic graph with quantum and dataset nodes.
datasets_by_type	A nested mapping of all datasets, keyed first by dataset type name and then by data ID.
init_quanta	A mapping from task label to the ID of the special init quantum for that task.
quanta_by_task	A nested mapping of all quanta, keyed first by task name and then by data ID.
quantum_only_xgraph	A directed acyclic graph with quanta as nodes (and datasets elided).

Methods Summary

from_args(repo_or_filename, /[, collection, ...])	Construct a ProvenanceQuantumGraph fron CLI-friendly arguments for a file or butler-ingested graph dataset.
make_exception_table()	Construct an astropy.table.Table with counts for each exception type raised by each task.
make_many_reports([states, ...])	Write multiple reports.
make_quantum_table([drop_unused_columns])	Construct an astropy.table.Table with a tabular summary of the quanta.
make_status_report([states, also, ...])	Make a JSON- or YAML-friendly report of all quanta with the given states.
make_task_resource_usage_table(task_label[, ...])	Make a table of resource usage for a single task.

Attributes Documentation

bipartite_xgraph

A directed acyclic graph with quantum and dataset nodes.

Notes

Node keys are quantum or dataset UUIDs, and are populated one quantum or dataset at a time as they are loaded. Loading quanta (via ProvenanceQuantumGraphReader.read_quanta) or datasets (via ProvenanceQuantumGraphReader.read_datasets) will load those nodes with full attributes and edges to adjacent nodes with no attributes. Loading quanta is necessary to populate edge attributes. Reading a quantum also populates its log and metadata datasets.

Node attributes are described by the ProvenanceQuantumInfo, ProvenanceInitQuantumInfo, and ProvenanceDatasetInfo types.

This graph includes init-input and init-output datasets, but it does not reflect the dependency between each task’s special “init” quantum and its runtime quanta (as this would require edges between quanta, and that would break the “bipartite” property).

The returned object is a read-only view of an internal one.

datasets_by_type

A nested mapping of all datasets, keyed first by dataset type name and then by data ID.

Notes

This is populated one dataset at a time as they are read. All dataset types in the pipeline graph are included, even if none of their datasets were loaded (i.e. nested mappings may be empty).

Reading a quantum also populates its log and metadata datasets.

The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.

init_quanta

A mapping from task label to the ID of the special init quantum for that task.

This is populated by the init_quanta component. Additional information about each init quantum can be found by using the ID to look up node attributes in the bipartite_xgraph, i.e.:

info: ProvenanceInitQuantumInfo = qg.bipartite_xgraph.nodes[id]

quanta_by_task

A nested mapping of all quanta, keyed first by task name and then by data ID.

Notes

This is populated one quantum at a time as they are read. All tasks in the pipeline graph are included, even if none of their quanta were loaded (i.e. nested mappings may be empty).

The returned object may be an internal dictionary; as the type annotation indicates, it should not be modified in place.

quantum_only_xgraph

A directed acyclic graph with quanta as nodes (and datasets elided).

Notes

Node keys are quantum UUIDs, and are populated one quantum at a time as they are loaded. Loading quanta (via ProvenanceQuantumGraphReader.read_quanta) will add the loaded nodes with full attributes and add edges to adjacent nodes with no attributes. Loading datasets (via ProvenanceQuantumGraphReader.read_datasets) will also add edges and nodes with no attributes.

Node attributes are described by the ProvenanceQuantumInfo types.

This graph does not include special “init” quanta.

The returned object is a read-only view of an internal one.

Methods Documentation

classmethod from_args(repo_or_filename: str, /, collection: str | None = None, *, quanta: Iterable[UUID] | None = None, datasets: Iterable[UUID] | None = None, writeable: bool = False) → Iterator[tuple[lsst.pipe.base.quantum_graph._provenance.ProvenanceQuantumGraph, lsst.daf.butler._butler.Butler | None]]

Construct a ProvenanceQuantumGraph fron CLI-friendly arguments for a file or butler-ingested graph dataset.

Parameters:

repo_or_filenamestr

Either a provenance quantum graph filename or a butler repository path or alias.

collectionsIterable [ str ], optional

Collections to search; presence indicates that the first argument is a butler repository, not a filename.

quantaIterable [ str ] or None, optional

IDs of the quanta to load, or None to load all.

datasetsIterable [ str ], optional

IDs of the datasets to load, or None to load all.

writeablebool, optional

Whether the butler should be constructed with write support.

Returns:

contextcontextlib.AbstractContextManager

A context manager that yields a tuple of

• the ProvenanceQuantumGraph

• the Butler constructed (or None)

when entered.

make_exception_table() → Table

Construct an astropy.table.Table with counts for each exception type raised by each task.

Returns:

tableastropy.table.Table

A table with columns for task label, exception type, and counts.

make_many_reports(states: ~collections.abc.Iterable[~lsst.pipe.base._status.QuantumAttemptStatus] = (<QuantumAttemptStatus.FAILED: -1>, <QuantumAttemptStatus.ABORTED: -4>, <QuantumAttemptStatus.ABORTED_SUCCESS: -2>), *, status_report_file: str | ~urllib.parse.ParseResult | ~lsst.resources._resourcePath.ResourcePath | ~pathlib._local.Path | None = None, print_quantum_table: bool = False, print_exception_table: bool = False, also: ~lsst.pipe.base._status.QuantumAttemptStatus | ~collections.abc.Iterable[~lsst.pipe.base._status.QuantumAttemptStatus] = (), with_caveats: ~lsst.pipe.base._status.QuantumSuccessCaveats | None = None, data_id_table_dir: str | ~urllib.parse.ParseResult | ~lsst.resources._resourcePath.ResourcePath | ~pathlib._local.Path | None = None) → None

Write multiple reports.

Parameters:

statesIterable [QuantumAttemptStatus] or QuantumAttemptStatus, optional

A quantum is included in the status report and data ID tables if it has any of these states. Defaults to states that clearly represent problems.

status_report_fileconvertible to ResourcePath,

optional

Filename for the JSON status report (see make_status_report).

print_quantum_tablebool, optional

If True, print a quantum summary table (counts only) to STDOUT.

print_exception_tablebool, optional

If True, print an exception-type summary table (counts only) to STDOUT.

alsoIterable [QuantumAttemptStatus] or QuantumAttemptStatus, optional

Additional states to consider in the status report and data ID tables; unioned with states. This is provided so users can easily request additional states while also getting the defaults.

with_caveatsQuantumSuccessCaveats or None, optional

Only include quanta with these caveat flags in the status report and data ID tables. May be set to None to report on all successful quanta (an empty sequence reports on only quanta with no caveats). If provided, QuantumAttemptStatus.SUCCESSFUL is automatically included in states.

data_id_table_dirconvertible to ResourcePath, optional

If provided, a directory to write data ID tables (in ECSV format) with all of the data IDs with the given states, for use with the --data-id-tables argument to the quantum graph builder. Subdirectories for each task and status will created within this directory, with one file for each exception type (or UNKNOWN when there is no exception).

make_quantum_table(drop_unused_columns: bool = True) → Table

Construct an astropy.table.Table with a tabular summary of the quanta.

Parameters:

drop_unused_columnsbool, optional

Whether to drop columns for rare states that did not actually occur in this run.

Returns:

tableastropy.table.Table

A table view of the quantum information. This only includes counts of status categories and caveats, not any per-data-ID detail.

Notes

Success caveats in the table are represented by their concise form, so when pretty-printing this table for users, the legend should generally be printed as well.

make_status_report(states: ~collections.abc.Iterable[~lsst.pipe.base._status.QuantumAttemptStatus] = (<QuantumAttemptStatus.FAILED: -1>, <QuantumAttemptStatus.ABORTED: -4>, <QuantumAttemptStatus.ABORTED_SUCCESS: -2>), *, also: ~lsst.pipe.base._status.QuantumAttemptStatus | ~collections.abc.Iterable[~lsst.pipe.base._status.QuantumAttemptStatus] = (), with_caveats: ~lsst.pipe.base._status.QuantumSuccessCaveats | None = QuantumSuccessCaveats.PARTIAL_OUTPUTS_ERROR, data_id_table_dir: str | ~urllib.parse.ParseResult | ~lsst.resources._resourcePath.ResourcePath | ~pathlib._local.Path | None = None) → ProvenanceReport

Make a JSON- or YAML-friendly report of all quanta with the given states.

Parameters:

statesIterable [QuantumAttemptStatus] or QuantumAttemptStatus, optional

A quantum is included if it has any of these states. Defaults to states that clearly represent problems.

alsoIterable [QuantumAttemptStatus] or QuantumAttemptStatus, optional

Additional states to consider; unioned with states. This is provided so users can easily request additional states while also getting the defaults.

with_caveatsQuantumSuccessCaveats or None, optional

If QuantumAttemptStatus.SUCCESSFUL is in states, only include quanta with these caveat flags. May be set to None to report on all successful quanta.

data_id_table_dirconvertible to ResourcePath, optional

If provided, a directory to write data ID tables (in ECSV format) with all of the data IDs with the given states, for use with the --data-id-tables argument to the quantum graph builder. Subdirectories for each task and status will created within this directory, with one file for each exception type (or UNKNOWN when there is no exception).

Returns:

reportProvenanceModel

A Pydantic model that groups quanta by task label and exception type.

make_task_resource_usage_table(task_label: str, include_data_ids: bool = False) → Table

Make a table of resource usage for a single task.

Parameters:

task_labelstr

Label of the task to extract resource usage for.

include_data_idsbool, optional

Whether to also include data ID columns.

Returns:

tableastropy.table.Table

A table with columns for quantum ID and all fields in QuantumResourceUsage.