QuantumProvenanceData

class lsst.daf.butler.QuantumProvenanceData

Bases: pydantic.main.BaseModel

A serializable struct for per-quantum provenance information and datastore records.

Notes

This class slightly duplicates information from the Quantum class itself (the predicted_inputs and predicted_outputs sets should have the same IDs present in Quantum.inputs and Quantum.outputs), but overall it assumes the original Quantum is also available to reconstruct the complete provenance (e.g. by associating dataset IDs with data IDs, dataset types, and RUN names.

Note that pydantic method parse_raw() is not going to work correctly for this class, use direct method instead.

Attributes Summary

copy	Duplicate a model, optionally choose which fields to include, exclude and change.
dict	Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
json	Generate a JSON representation of the model, include and exclude arguments as per dict().

Methods Summary

collect_and_transfer(butler, quanta, provenance)	Transfer output datasets from multiple quanta to a more permantent Butler repository.
construct(_fields_set, None] = None, **values)	Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
direct(*, predicted_inputs, uuid.UUID]], …)	Construct an instance directly without validators.
from_orm(obj)
parse_file(path, pathlib.Path], *, …)
parse_obj(obj)
parse_raw(*args, **kwargs)
schema(by_alias, ref_template)
schema_json(*, by_alias, ref_template, …)
update_forward_refs(**localns)	Try to update ForwardRefs on fields based on this Model, globalns and localns.
validate(value)

Attributes Documentation

copy

Duplicate a model, optionally choose which fields to include, exclude and change.

Parameters:

• include – fields to include in new model
• exclude – fields to exclude from new model, as with values this takes precedence over include
• update – values to change/add in the new model. Note: the data is not validated before creating the new model: you should trust this data
• deep – set to True to make a deep copy of the model

Returns:

new model instance

dict

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

json

Generate a JSON representation of the model, include and exclude arguments as per dict().

encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().

Methods Documentation

static collect_and_transfer(butler: Butler, quanta: Iterable[Quantum], provenance: Iterable[QuantumProvenanceData]) → None

Transfer output datasets from multiple quanta to a more permantent Butler repository.

Parameters:

butler : Butler

Full butler representing the data repository to transfer datasets to.

quanta : Iterable [ Quantum ]

Iterable of Quantum objects that carry information about predicted outputs. May be a single-pass iterator.

provenance : Iterable [ QuantumProvenanceData ]

Provenance and datastore data for each of the given quanta, in the same order. May be a single-pass iterator.

Notes

Input-output provenance data is not actually transferred yet, because Registry has no place to store it.

This method probably works most efficiently if run on all quanta for a single task label at once, because this will gather all datasets of a particular type together into a single vectorized Registry import. It should still behave correctly if run on smaller groups of quanta or even quanta from multiple tasks.

Currently this method transfers datastore record data unchanged, with no possibility of actually moving (e.g.) files. Datastores that are present only in execution or only in the more permanent butler are ignored.

classmethod construct(_fields_set: Optional[SetStr, None] = None, **values) → Model

Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed. Behaves as if Config.extra = 'allow' was set since it adds all passed values

classmethod direct(*, predicted_inputs: Iterable[Union[str, uuid.UUID]], available_inputs: Iterable[Union[str, uuid.UUID]], actual_inputs: Iterable[Union[str, uuid.UUID]], predicted_outputs: Iterable[Union[str, uuid.UUID]], actual_outputs: Iterable[Union[str, uuid.UUID]], datastore_records: Mapping[str, Mapping]) → lsst.daf.butler._quantum_backed.QuantumProvenanceData

Construct an instance directly without validators.

This differs from the pydantic “construct” method in that the arguments are explicitly what the model requires, and it will recurse through members, constructing them from their corresponding direct methods.

This method should only be called when the inputs are trusted.

classmethod from_orm(obj: Any) → Model

classmethod parse_file(path: Union[str, pathlib.Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: pydantic.parse.Protocol = None, allow_pickle: bool = False) → Model

classmethod parse_obj(obj: Any) → Model

classmethod parse_raw(*args, **kwargs) → lsst.daf.butler._quantum_backed.QuantumProvenanceData

classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny

classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs) → unicode

classmethod update_forward_refs(**localns) → None

Try to update ForwardRefs on fields based on this Model, globalns and localns.

classmethod validate(value: Any) → Model