WorkQueue

class lsst.ctrl.bps.parsl.sites.WorkQueue(*args, **kwargs)

Bases: SiteConfig

Base class configuraton for parsl.executors.WorkQueueExecutor.

Subclasses must provide implementations for .get_executors and .select_executor. In .get_executors, the site-specific ExecutionProvider must be defined.

Parameters

*argsAny

Parameters forwarded to base class constructor.

**kwargsAny

Keyword arguments passed to base class constructor, augmented by the resource_list argument.

Notes

The following BPS configuration parameters are recognized, overriding the defaults:

• port (int): The port used by work_queue. Default: 9000.

• worker_options (`str`): Extra options to pass to work_queue workers. A typical option specifies the memory available per worker, e.g., ``"--memory=90000", which sets the available memory to 90 GB. Default: ""

• wq_max_retries (int): The number of retries that work_queue will make in case of task failures. Set to None to have work_queue retry forever; set to 1 to have retries managed by Parsl. Default: 1

Methods Summary

make_executor(label, provider, *[, port, ...])

Return a parsl.executors.WorkQueueExecutor.

Methods Documentation

make_executor(label: str, provider: ExecutionProvider, *, port: int = 0, worker_options: str = '', wq_max_retries: int = 1) → ParslExecutor

Return a parsl.executors.WorkQueueExecutor. The provider contains the site-specific configuration.

Parameters

labelstr

Label for executor.

providerparsl.providers.base.ExecutionProvider

Parsl execution provider, e.g., parsl.providers.SlurmProvider.

portint, optional

Port used by work_queue. Default: 0. For a value of 0, parsl will allocate a port number automatically.

worker_optionsstr, optional

Extra options to pass to work_queue workers, e.g., "--memory=90000". Default: "".

wq_max_retriesint, optional

Number of retries for work_queue to attempt per job. Set to None to have it try indefinitely; set to 1 to have Parsl control the number of retries. Default: 1.