WorkQueue

class lsst.ctrl.bps.parsl.sites.WorkQueue(*args, **kwargs)

Bases: SiteConfig

Base class configuraton for WorkQueueExecutor.

Subclasses must provide implementations for .get_executors and .select_executor. In .get_executors, the site-specific ExecutionProvider must be defined.

Parameters

*argsAny

Parameters forwarded to base class constructor.

**kwargsAny

Keyword arguments passed to base class constructor, augmented by the resource_list argument.

Notes

The following BPS configuration parameters are recognized, overriding the defaults:

• port (int): The port used by work_queue. Default: 9000.

• worker_options (`str`): Extra options to pass to work_queue workers. A typical option specifies the memory available per worker, e.g., ``"--memory=90000", which sets the available memory to 90 GB. Default: ""

• wq_max_retries (int): The number of retries that work_queue will make in case of task failures. Set to None to have work_queue retry forever; set to 1 to have retries managed by Parsl. Default: 1

Methods Summary

make_executor(label, provider, *[, port, ...])

Return a WorkQueueExecutor.

Methods Documentation

make_executor(label: str, provider: ExecutionProvider, *, port: int = 9000, worker_options: str = '', wq_max_retries: int = 1) → ParslExecutor

Return a WorkQueueExecutor. The provider contains the site-specific configuration.

Parameters

labelstr

Label for executor.

providerExecutionProvider

Parsl execution provider, e.g., SlurmProvider.

portint, optional

Port used by work_queue. Default: 9000.

worker_optionsstr, optional

Extra options to pass to work_queue workers, e.g., "--memory=90000". Default: "".

wq_max_retriesint, optional

Number of retries for work_queue to attempt per job. Set to None to have it try indefinitely; set to 1 to have Parsl control the number of retries. Default: 1.