AbstractDatastoreCacheManager

class lsst.daf.butler.AbstractDatastoreCacheManager(config: Union[str, DatastoreCacheManagerConfig], universe: DimensionUniverse)

Bases: abc.ABC

An abstract base class for managing caching in a Datastore.

Parameters:

config : str or DatastoreCacheManagerConfig

Configuration to control caching.

universe : DimensionUniverse

Set of all known dimensions, used to expand and validate any used in lookup keys.

Attributes Summary

cache_size	Size of the cache in bytes.
file_count	Return number of cached files tracked by registry.

Methods Summary

find_in_cache(ref, extension)	Look for a dataset in the cache and return its location.
known_to_cache(ref, extension)	Report if the dataset is known to the cache.
move_to_cache(uri, ref)	Move a file to the cache.
remove_from_cache(ref, …)	Remove the specified datasets from the cache.
should_be_cached(entity, DatasetType, …)	Indicate whether the entity should be added to the cache.

Attributes Documentation

cache_size

Size of the cache in bytes.

file_count

Return number of cached files tracked by registry.

Methods Documentation

find_in_cache(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, extension: str) → Iterator[Optional[lsst.resources._resourcePath.ResourcePath]]

Look for a dataset in the cache and return its location.

Parameters:

ref : DatasetRef

Dataset to locate in the cache.

extension : str

File extension expected. Should include the leading “.”.

Yields:

uri : lsst.resources.ResourcePath or None

The URI to the cached file, or None if the file has not been cached.

Notes

Should be used as a context manager in order to prevent this file from being removed from the cache for that context.

known_to_cache(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, extension: Optional[str] = None) → bool

Report if the dataset is known to the cache.

Parameters:

ref : DatasetRef

Dataset to check for in the cache.

extension : str, optional

File extension expected. Should include the leading “.”. If None the extension is ignored and the dataset ID alone is used to check in the cache. The extension must be defined if a specific component is being checked.

Returns:

known : bool

Returns True if the dataset is currently known to the cache and False otherwise.

Notes

This method can only report if the dataset is known to the cache in this specific instant and does not indicate whether the file can be read from the cache later. find_in_cache() should be called if the cached file is to be used.

move_to_cache(uri: lsst.resources._resourcePath.ResourcePath, ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → Optional[lsst.resources._resourcePath.ResourcePath]

Move a file to the cache.

Move the given file into the cache, using the supplied DatasetRef for naming. A call is made to should_be_cached() and if the DatasetRef should not be accepted None will be returned.

Cache expiry can occur during this.

Parameters:

uri : lsst.resources.ResourcePath

Location of the file to be relocated to the cache. Will be moved.

ref : DatasetRef

Ref associated with this file. Will be used to determine the name of the file within the cache.

Returns:

new : lsst.resources.ResourcePath or None

URI to the file within the cache, or None if the dataset was not accepted by the cache.

remove_from_cache(ref: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]]) → None

Remove the specified datasets from the cache.

It is not an error for these datasets to be missing from the cache.

Parameters:

ref : DatasetRef or iterable of DatasetRef

The datasets to remove from the cache.

should_be_cached(entity: Union[DatasetRef, DatasetType, StorageClass]) → bool

Indicate whether the entity should be added to the cache.

This is relevant when reading or writing.

Parameters:

entity : StorageClass or DatasetType or DatasetRef

Thing to test against the configuration. The name property is used to determine a match. A DatasetType will first check its name, before checking its StorageClass. If there are no matches the default will be returned.

Returns:

should_cache : bool

Returns True if the dataset should be cached; False otherwise.