htc_backup_files

lsst.ctrl.bps.htcondor.htc_backup_files(wms_path, subdir=None, limit=100)

Backup select HTCondor files in the submit directory.

Files will be saved in separate subdirectories which will be created in the submit directory where the files are located. These subdirectories will be consecutive, zero-padded integers. Their values will correspond to the number of HTCondor rescue DAGs in the submit directory.

Hence, with the default settings, copies after the initial failed run will be placed in ‘001’ subdirectory, ‘002’ after the first restart, and so on until the limit of backups is reached. If there’s no rescue DAG yet, files will be copied to ‘000’ subdirectory.

Parameters:

wms_path : str or pathlib.Path

Path to the submit directory either absolute or relative.

subdir : str or pathlib.Path, optional

A path, relative to the submit directory, where all subdirectories with backup files will be kept. Defaults to None which means that the backup subdirectories will be placed directly in the submit directory.

limit : int, optional

Maximal number of backups. If the number of backups reaches the limit, the last backup files will be overwritten. The default value is 100 to match the default value of HTCondor’s DAGMAN_MAX_RESCUE_NUM in version 8.8+.

Raises:

FileNotFoundError

If the submit directory or the file that needs to be backed up does not exist.

OSError

If the submit directory cannot be accessed or backing up a file failed either due to permission or filesystem related issues.

Notes

This is not a generic function for making backups. It is intended to be used once, just before a restart, to make snapshots of files which will be overwritten by HTCondor after during the next run.