ProvenanceQuantumGraphReader

class lsst.pipe.base.quantum_graph.ProvenanceQuantumGraphReader(header: HeaderModel, pipeline_graph: PipelineGraph, zf: zipfile.ZipFile, decompressor: Decompressor, address_reader: AddressReader, page_size: int)

Bases: BaseQuantumGraphReader

A helper class for reading provenance quantum graphs.

Notes

The open context manager should be used to construct new instances. Instances cannot be used after the context manager exits, except to access the graph attribute`.

The various read_* methods in this class update the graph attribute in place.

Attributes Summary

address_reader	A helper object for reading addresses into the multi-block files.
decompressor	A decompressor for all compressed JSON blocks.
graph	Loaded provenance graph, populated in place as components are read.
header	Header metadata for the quantum graph.
page_size	Approximate number of bytes to read at a time.
pipeline_graph	Graph of tasks and dataset type names that appear in the quantum graph.
zf	The zip archive that represents the quantum graph on disk.

Methods Summary

fetch_logs(nodes)	Fetch log datasets.
fetch_metadata(nodes)	Fetch metadata datasets.
fetch_packages()	Fetch package version information.
open(uri, *[, page_size, import_mode])	Construct a reader from a URI.
read_datasets([datasets])	Read information about the given datasets.
read_full_graph()	Read all bipartite edges and all quantum and dataset node attributes, fully populating the graph attribute.
read_init_quanta()	Read the thin graph, with all edge information and categorization of quanta by task label.
read_quanta([quanta])	Read information about the given quanta.

Attributes Documentation

address_reader: AddressReader = <dataclasses._MISSING_TYPE object>

A helper object for reading addresses into the multi-block files.

decompressor: Decompressor = <dataclasses._MISSING_TYPE object>

A decompressor for all compressed JSON blocks.

graph: ProvenanceQuantumGraph = <dataclasses._MISSING_TYPE object>

Loaded provenance graph, populated in place as components are read.

header: HeaderModel = <dataclasses._MISSING_TYPE object>

Header metadata for the quantum graph.

page_size: int = <dataclasses._MISSING_TYPE object>

Approximate number of bytes to read at a time.

Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads.

pipeline_graph: PipelineGraph = <dataclasses._MISSING_TYPE object>

Graph of tasks and dataset type names that appear in the quantum graph.

zf: zipfile.ZipFile = <dataclasses._MISSING_TYPE object>

The zip archive that represents the quantum graph on disk.

Methods Documentation

fetch_logs(nodes: Iterable[UUID]) → dict[UUID, list[ButlerLogRecords | None]]

Fetch log datasets.

Parameters

nodesIterable [ uuid.UUID ]

UUIDs of the log datasets themselves or of the quanta they correspond to.

Returns

logsdict [ uuid.UUID, list [ lsst.daf.butler.ButlerLogRecords or None] ]

Logs for the given IDs. Each value is a list of lsst.daf.butler.ButlerLogRecords instances representing different execution attempts, ordered chronologically from first to last. Attempts where logs were missing will have None in this list.

fetch_metadata(nodes: Iterable[UUID]) → dict[UUID, list[TaskMetadata | None]]

Fetch metadata datasets.

Parameters

nodesIterable [ uuid.UUID ]

UUIDs of the metadata datasets themselves or of the quanta they correspond to.

Returns

metadatadict [ uuid.UUID, list [TaskMetadata] ]

Metadata for the given IDs. Each value is a list of TaskMetadata instances representing different execution attempts, ordered chronologically from first to last. Attempts where metadata was missing (not written even in the fallback extra provenance in the logs) will have None in this list.

fetch_packages() → Packages

Fetch package version information.

classmethod open(uri: str | ParseResult | ResourcePath | Path, *, page_size: int | None = None, import_mode: TaskImportMode = TaskImportMode.DO_NOT_IMPORT) → Iterator[ProvenanceQuantumGraphReader]

Construct a reader from a URI.

Parameters

uriconvertible to lsst.resources.ResourcePath

URI to open. Should have a .qg extension.

page_sizeint, optional

Approximate number of bytes to read at once from address files and multi-block files. Note that this does not set a page size for all reads, but it does affect the smallest, most numerous reads. Can also be set via the LSST_QG_PAGE_SIZE environment variable.

import_modepipeline_graph.TaskImportMode, optional

How to handle importing the task classes referenced in the pipeline graph.

Returns

readercontextlib.AbstractContextManager [ ProvenanceQuantumGraphReader ]

A context manager that returns the reader when entered.

read_datasets(datasets: Iterable[UUID] | None = None) → None

Read information about the given datasets.

Parameters

datasetsIterable [uuid.UUID], optional

Iterable of dataset IDs to load. If not provided, all datasets will be loaded. The UUIDs and indices of quanta will be ignored.

read_full_graph() → None

Read all bipartite edges and all quantum and dataset node attributes, fully populating the graph attribute.

Notes

This does not read logs, metadata, or packages ; those must always be fetched explicitly.

read_init_quanta() → None

Read the thin graph, with all edge information and categorization of quanta by task label.

read_quanta(quanta: Iterable[UUID] | None = None) → None

Read information about the given quanta.

Parameters

quantaIterable [uuid.UUID], optional

Iterable of quantum IDs to load. If not provided, all quanta will be loaded. The UUIDs and indices of datasets and special init quanta will be ignored.