Slurm#

class lsst.ctrl.bps.parsl.sites.Slurm(*args, **kwargs)#

Bases: SiteConfig

Configuration for generic Slurm cluster.

Parameters#

*argsAny

Parameters forwarded to base class constructor.

**kwargsAny

Keyword arguments passed to base class constructor, augmented by the resource_list argument.

Notes#

This can be used directly as the site configuration for a Slurm cluster by setting the BPS config, e.g.:

computeSite: slurm
site:
  slurm:
    class: lsst.ctrl.bps.parsl.sites.Slurm
    nodes: 3
    cores_per_node: 20
    walltime: "00:59:00"  # Note: always quote walltime in YAML

Alternatively, it can be used as a base class for Slurm cluster configurations.

The following BPS configuration parameters are recognised (and required unless there is a default mentioned here, or provided by a subclass):

  • nodes (int): number of nodes for each Slurm job.

  • cores_per_node (int): number of cores per node for each Slurm job; by default we use all cores on the node.

  • walltime (str): time limit for each Slurm job.

  • mem_per_node (int): memory per node (GB) for each Slurm job; by default we use whatever Slurm gives us.

  • qos (str): quality of service to request for each Slurm job; by default we use whatever Slurm gives us.

  • singleton (bool): allow only one job to run at a time; by default False.

  • account (str): account to use for Slurm jobs.

  • scheduler_options (str): text to prepend to the Slurm submission script (each line usually starting with #SBATCH).

Methods Summary

get_executors()

Get a list of executors to be used in processing.

make_executor(label, *[, nodes, ...])

Return an executor for running on a Slurm cluster.

select_executor(job)

Get the label of the executor to use to execute a job.

Methods Documentation

get_executors() → list[ParslExecutor]#

Get a list of executors to be used in processing.

Each executor should have a unique label.

make_executor(label: str, *, nodes: int | None = None, cores_per_node: int | None = None, walltime: str | None = None, mem_per_node: int | None = None, mem_per_worker: float | None = None, qos: str | None = None, constraint: str | None = None, singleton: bool = False, exclusive: bool = False, scheduler_options: str | None = None, provider_options: dict[str, Any] | None = None, executor_options: dict[str, Any] | None = None) → ParslExecutor#

Return an executor for running on a Slurm cluster.

Parameters#

labelstr

Label for executor.

nodesint, optional

Default number of nodes for each Slurm job.

cores_per_nodeint, optional

Default number of cores per node for each Slurm job.

walltimestr, optional

Default time limit for each Slurm job.

mem_per_nodefloat, optional

Memory per node (GB) to request for each Slurm job.

mem_per_workerfloat, optional

Minimum memory per worker (GB), limited by the executor.

qosstr, optional

Quality of service for each Slurm job.

constraintstr, optional

Node feature(s) to require for each Slurm job.

singletonbool, optional

Whether to allow only a single Slurm job to run at a time.

exclusivebool, optional

Flag to specify exclusive nodes.

scheduler_optionsstr, optional

#SBATCH directives to prepend to the Slurm submission script.

provider_optionsdict, optional

Additional arguments for parsl.providers.SlurmProvider constructor.

executor_optionsdict, optional

Additional arguments for parsl.executors.HighThroughputExecutor constructor.

Returns#

executorparsl.executors.HighThroughputExecutor

Executor for Slurm jobs.

select_executor(job: ParslJob) → str#

Get the label of the executor to use to execute a job.

Parameters#

joblsst.ctrl.bps.parsl.ParslJob

Job to be executed.

Returns#

labelstr

Label of executor to use to execute job.