DatastoreCacheManager

class lsst.daf.butler.DatastoreCacheManager(config: Union[str, DatastoreCacheManagerConfig], universe: DimensionUniverse)

Bases: lsst.daf.butler.AbstractDatastoreCacheManager

A class for managing caching in a Datastore using local files.

Parameters:
config : str or DatastoreCacheManagerConfig

Configuration to control caching.

universe : DimensionUniverse

Set of all known dimensions, used to expand and validate any used in lookup keys.

Notes

Two environment variables can be used to override the cache directory and expiration configuration:

  • $DAF_BUTLER_CACHE_DIRECTORY
  • $DAF_BUTLER_CACHE_EXPIRATION_MODE

The expiration mode should take the form mode=threshold so for example to configure expiration to limit the cache directory to 5 datasets the value would be datasets=5.

Additionally the $DAF_BUTLER_CACHE_DIRECTORY_IF_UNSET environment variable can be used to indicate that this directory should be used if no explicit directory has been specified from configuration or from the $DAF_BUTLER_CACHE_DIRECTORY environment variable.

Attributes Summary

cache_directory
cache_size Size of the cache in bytes.
file_count Return number of cached files tracked by registry.

Methods Summary

find_in_cache(ref, extension) Look for a dataset in the cache and return its location.
known_to_cache(ref, extension, None] = None) Report if the dataset is known to the cache.
move_to_cache(uri, ref) Move a file to the cache.
remove_from_cache(refs, …) Remove the specified datasets from the cache.
scan_cache() Scan the cache directory and record information about files.
set_fallback_cache_directory_if_unset() Defines a fallback cache directory if a fallback not set already.
should_be_cached(entity, DatasetType, …) Indicate whether the entity should be added to the cache.

Attributes Documentation

cache_directory
cache_size

Size of the cache in bytes.

file_count

Return number of cached files tracked by registry.

Methods Documentation

find_in_cache(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, extension: str) → Iterator[Optional[lsst.resources._resourcePath.ResourcePath, None]]

Look for a dataset in the cache and return its location.

Parameters:
ref : DatasetRef

Dataset to locate in the cache.

extension : str

File extension expected. Should include the leading “.”.
Yields:
uri : lsst.resources.ResourcePath or None

The URI to the cached file, or None if the file has not been cached.

Notes

Should be used as a context manager in order to prevent this file from being removed from the cache for that context.

known_to_cache(ref: lsst.daf.butler.core.datasets.ref.DatasetRef, extension: Optional[str, None] = None) → bool

Report if the dataset is known to the cache.

Parameters:
ref : DatasetRef

Dataset to check for in the cache.

extension : str, optional

File extension expected. Should include the leading “.”. If None the extension is ignored and the dataset ID alone is used to check in the cache. The extension must be defined if a specific component is being checked.
Returns:
known : bool

Returns True if the dataset is currently known to the cache and False otherwise. If the dataset refers to a component and an extension is given then only that component is checked.

Notes

This method can only report if the dataset is known to the cache in this specific instant and does not indicate whether the file can be read from the cache later. find_in_cache() should be called if the cached file is to be used.

This method does not force the cache to be re-scanned and so can miss cached datasets that have recently been written by other processes.

move_to_cache(uri: lsst.resources._resourcePath.ResourcePath, ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → Optional[lsst.resources._resourcePath.ResourcePath, None]

Move a file to the cache.

Move the given file into the cache, using the supplied DatasetRef for naming. A call is made to should_be_cached() and if the DatasetRef should not be accepted None will be returned.

Cache expiry can occur during this.

Parameters:
uri : lsst.resources.ResourcePath

Location of the file to be relocated to the cache. Will be moved.

ref : DatasetRef

Ref associated with this file. Will be used to determine the name of the file within the cache.
Returns:
new : lsst.resources.ResourcePath or None

URI to the file within the cache, or None if the dataset was not accepted by the cache.
remove_from_cache(refs: Union[lsst.daf.butler.core.datasets.ref.DatasetRef, Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]]) → None

Remove the specified datasets from the cache.

It is not an error for these datasets to be missing from the cache.

Parameters:
ref : DatasetRef or iterable of DatasetRef

The datasets to remove from the cache.
scan_cache() → None

Scan the cache directory and record information about files.

classmethod set_fallback_cache_directory_if_unset() → tuple

Defines a fallback cache directory if a fallback not set already.

Returns:
defined : bool

True if the fallback directory was newly-defined in this method. False if it had already been set.

cache_dir : str

Returns the path to the cache directory that will be used if it’s needed. This can allow the caller to run a directory cleanup when it’s no longer needed (something that the cache manager can not do because forks should not clean up directories defined by the parent process).

Notes

The fallback directory will not be defined if one has already been defined. This method sets the DAF_BUTLER_CACHE_DIRECTORY_IF_UNSET environment variable only if a value has not previously been stored in that environment variable. Setting the environment variable allows this value to survive into spawned subprocesses. Calling this method will lead to all subsequently created cache managers sharing the same cache.

should_be_cached(entity: Union[DatasetRef, DatasetType, StorageClass]) → bool

Indicate whether the entity should be added to the cache.

This is relevant when reading or writing.

Parameters:
entity : StorageClass or DatasetType or DatasetRef

Thing to test against the configuration. The name property is used to determine a match. A DatasetType will first check its name, before checking its StorageClass. If there are no matches the default will be returned.
Returns:
should_cache : bool

Returns True if the dataset should be cached; False otherwise.