FileTransferSource

class lsst.daf.butler.datastore.FileTransferSource(*args, **kwargs)

Bases: Protocol

Protocol for an object that can return information about files that need to be transferred to copy datasets from one Butler repository to another.

Methods Summary

get_file_info_for_transfer(dataset_ids)	Given a list of dataset IDs, return all file information associated with the datasets that can be determined without searching the filesystem.
locate_missing_files_for_transfer(refs, ...)	Given a list of DatasetRef, search the filesystem to locate artifacts associated with the dataset.
mexists(refs, artifact_existence)	Scan the filesystem to determine whether the given datasets exist on disk.

Methods Documentation

get_file_info_for_transfer(dataset_ids: Iterable[UUID]) → dict[uuid.UUID, list[lsst.daf.butler.datastore._transfer.FileTransferRecord]]

Given a list of dataset IDs, return all file information associated with the datasets that can be determined without searching the filesystem.

Parameters:

dataset_idsIterable [ DatasetId ]

List of dataset IDs for which we will retrieve file information.

locate_missing_files_for_transfer(refs: Iterable[DatasetRef], artifact_existence: dict[lsst.resources._resourcePath.ResourcePath, bool]) → dict[uuid.UUID, list[lsst.daf.butler.datastore._transfer.FileTransferRecord]]

Given a list of DatasetRef, search the filesystem to locate artifacts associated with the dataset.

Parameters:

refsiterable of DatasetRef

The datasets to be checked.

artifact_existencedict [lsst.resources.ResourcePath, bool]

Optional mapping of datastore artifact to existence. Updated by this method with details of all artifacts tested.

Raises:

ValueError

If this file transfer source cannot locate artifacts by searching the filesystem.

mexists(refs: Iterable[DatasetRef], artifact_existence: dict[lsst.resources._resourcePath.ResourcePath, bool]) → dict[lsst.daf.butler._dataset_ref.DatasetRef, bool]

Scan the filesystem to determine whether the given datasets exist on disk.

Parameters:

refsiterable of DatasetRef

The datasets to be checked.

artifact_existencedict [lsst.resources.ResourcePath, bool]

Optional mapping of datastore artifact to existence. Updated by this method with details of all artifacts tested.

Returns:

existencedict of [DatasetRef, bool]

Mapping from dataset to boolean indicating existence.