PersistenceContextVars

class lsst.daf.butler.persistence_context.PersistenceContextVars

Bases: object

Helper class for deserializing butler data structures.

When serializing various butler data structures nested dataset types get serialized independently. This means what were multiple references to the same object in memory are all duplicated in the serialization process.

Upon deserialization multiple independent data structures are created to represent the same logical bit of data.

This class can be used to remove this duplication by caching objects as they are created and returning a reference to that object. This is done in concert with direct and from_simple methods on the various butler dataset structures.

This class utilizes class level variables as a form of global state. Each of the various data structures can look to see if these global caches has been initialized as a cache (a dictionary) or is in the default None state.

Users of this class are intended to create an instance, and then call the run method, supplying a callable function, and passing any required arguments. The run method then creates a specific execution context, initializing the caches, and then runs the supplied function. Upon completion of the function call, the caches are cleared and returned to the default state.

This process is thread safe.

Notes

Caches of SerializedDatasetRefs are intentionally left out. It was discovered that these caused excessive python memory allocations which though cleaned up upon completion, left the process using more memory than it otherwise needed as python does not return allocated memory to the OS until process completion. It was determined the runtime cost of recreating the SerializedDatasetRefs was worth the memory savings.

Attributes Summary

dataCoordinates	A cache of DataCoordinates.
dataStoreRecords	A cache of DatastoreRecordData objects.
datasetRefs	A cache of DatasetRefs.
dimensionRecords	A cache of DimensionRecords.
loadedTypes	A cache of DatasetTypes.
serializedDataCoordinateMapping	A cache of SerializedDataCoordinates.
serializedDatasetTypeMapping	A cache of SerializedDatasetTypes.
serializedDimensionRecordMapping	A cache of SerializedDimensionRecords.

Methods Summary

run(function, *args, **kwargs)

Execute the supplied function inside context specific caches.

Attributes Documentation

dataCoordinates: ContextVar[dict[tuple[frozenset, bool], DataCoordinate] | None] = <ContextVar name='dataCoordinates' default=None>

A cache of DataCoordinates.

dataStoreRecords: ContextVar[dict[frozenset[str | uuid.UUID], DatastoreRecordData] | None] = <ContextVar name='dataStoreRecords' default=None>

A cache of DatastoreRecordData objects.

datasetRefs: ContextVar[dict[int, DatasetRef] | None] = <ContextVar name='datasetRefs' default=None>

A cache of DatasetRefs.

Keys are UUID converted to int, but only refs of parent dataset types are cached AND THE STORAGE CLASS IS UNSPECIFIED; consumers of this cache must call overrideStorageClass on the result.

dimensionRecords: ContextVar[dict[Hashable, DimensionRecord] | None] = <ContextVar name='dimensionRecords' default=None>

A cache of DimensionRecords.

loadedTypes: ContextVar[dict[tuple[str, str], DatasetType] | None] = <ContextVar name='loadedTypes' default=None>

A cache of DatasetTypes.

serializedDataCoordinateMapping: ContextVar[dict[tuple[frozenset, bool], SerializedDataCoordinate] | None] = <ContextVar name='serializedDataCoordinateMapping' default=None>

A cache of SerializedDataCoordinates.

serializedDatasetTypeMapping: ContextVar[dict[tuple[str, str], SerializedDatasetType] | None] = <ContextVar name='serializedDatasetTypeMapping' default=None>

A cache of SerializedDatasetTypes.

serializedDimensionRecordMapping: ContextVar[dict[tuple[str, frozenset] | tuple[int, DataCoordinate], SerializedDimensionRecord] | None] = <ContextVar name='serializedDimensionRecordMapping' default=None>

A cache of SerializedDimensionRecords.

Methods Documentation

run(function: ~collections.abc.Callable[[~_Q], ~lsst.daf.butler.persistence_context._T], *args: ~typing.~_Q, **kwargs: ~typing.~_Q) → _T

Execute the supplied function inside context specific caches.

Parameters:

functionCallable

A callable which is to be executed inside a specific context.

*argstuple

Positional arguments which are to be passed to the Callable

**kwargs: dict, optional

Extra key word arguments which are to be passed to the Callable

Returns:

resultAny

The result returned by executing the supplied Callable