QuantumProvenanceData¶
- class lsst.daf.butler.QuantumProvenanceData(*, predicted_inputs: set[uuid.UUID], available_inputs: set[uuid.UUID], actual_inputs: set[uuid.UUID], predicted_outputs: set[uuid.UUID], actual_outputs: set[uuid.UUID], datastore_records: dict[str, lsst.daf.butler.datastore.record_data.SerializedDatastoreRecordData])¶
Bases:
BaseModel
A serializable struct for per-quantum provenance information and datastore records.
Notes
This class slightly duplicates information from the
Quantum
class itself (thepredicted_inputs
andpredicted_outputs
sets should have the same IDs present inQuantum.inputs
andQuantum.outputs
), but overall it assumes the originalQuantum
is also available to reconstruct the complete provenance (e.g. by associating dataset IDs with data IDs, dataset types, andRUN
names.Note that
pydantic
methodparse_raw()
is not going to work correctly for this class, usedirect
method instead.Attributes Summary
Configuration for the model, should be a dictionary conforming to [
ConfigDict
][pydantic.config.ConfigDict].Metadata about the fields defined on the model, mapping of field names to [
FieldInfo
][pydantic.fields.FieldInfo].Methods Summary
collect_and_transfer
(butler, quanta, provenance)Transfer output datasets from multiple quanta to a more permanent
Butler
repository.direct
(*, predicted_inputs, ...)Construct an instance directly without validators.
parse_raw
(*args, **kwargs)Attributes Documentation
- model_config: ClassVar[ConfigDict] = {}¶
Configuration for the model, should be a dictionary conforming to [
ConfigDict
][pydantic.config.ConfigDict].
- model_fields: ClassVar[dict[str, FieldInfo]] = {'actual_inputs': FieldInfo(annotation=set[UUID], required=True), 'actual_outputs': FieldInfo(annotation=set[UUID], required=True), 'available_inputs': FieldInfo(annotation=set[UUID], required=True), 'datastore_records': FieldInfo(annotation=dict[str, SerializedDatastoreRecordData], required=True), 'predicted_inputs': FieldInfo(annotation=set[UUID], required=True), 'predicted_outputs': FieldInfo(annotation=set[UUID], required=True)}¶
Metadata about the fields defined on the model, mapping of field names to [
FieldInfo
][pydantic.fields.FieldInfo].This replaces
Model.__fields__
from Pydantic V1.
Methods Documentation
- static collect_and_transfer(butler: Butler, quanta: Iterable[Quantum], provenance: Iterable[QuantumProvenanceData]) None ¶
Transfer output datasets from multiple quanta to a more permanent
Butler
repository.- Parameters:
- butler
Butler
Full butler representing the data repository to transfer datasets to.
- quanta
Iterable
[Quantum
] Iterable of
Quantum
objects that carry information about predicted outputs. May be a single-pass iterator.- provenance
Iterable
[QuantumProvenanceData
] Provenance and datastore data for each of the given quanta, in the same order. May be a single-pass iterator.
- butler
Notes
Input-output provenance data is not actually transferred yet, because
Registry
has no place to store it.This method probably works most efficiently if run on all quanta for a single task label at once, because this will gather all datasets of a particular type together into a single vectorized
Registry
import. It should still behave correctly if run on smaller groups of quanta or even quanta from multiple tasks.Currently this method transfers datastore record data unchanged, with no possibility of actually moving (e.g.) files. Datastores that are present only in execution or only in the more permanent butler are ignored.
- classmethod direct(*, predicted_inputs: Iterable[str | UUID], available_inputs: Iterable[str | UUID], actual_inputs: Iterable[str | UUID], predicted_outputs: Iterable[str | UUID], actual_outputs: Iterable[str | UUID], datastore_records: Mapping[str, Mapping]) QuantumProvenanceData ¶
Construct an instance directly without validators.
- Parameters:
- predicted_inputs
Iterable
ofstr
oruuid.UUID
The predicted inputs.
- available_inputs
Iterable
ofstr
oruuid.UUID
The available inputs.
- actual_inputs
Iterable
ofstr
oruuid.UUID
The actual inputs.
- predicted_outputs
Iterable
ofstr
oruuid.UUID
The predicted outputs.
- actual_outputs
Iterable
ofstr
oruuid.UUID
The actual outputs.
- datastore_records
Mapping
[str
,Mapping
] The datastore records.
- predicted_inputs
- Returns:
- provenance
QuantumProvenanceData
Serializable model of the quantum provenance.
- provenance
Notes
This differs from the Pydantic “construct” method in that the arguments are explicitly what the model requires, and it will recurse through members, constructing them from their corresponding
direct
methods.This method should only be called when the inputs are trusted.
- classmethod parse_raw(*args: Any, **kwargs: Any) QuantumProvenanceData ¶