Torque¶

class lsst.ctrl.bps.parsl.sites.Torque(config: BpsConfig, add_resources: bool = False)¶

Bases: SiteConfig

Configuration for generic Torque cluster.

This can be used directly as the site configuration for a Torque cluster by setting the BPS config, e.g.:

computeSite: torque
site:
  torque:
    class: lsst.ctrl.bps.parsl.sites.Torque
    nodes: 4
    tasks_per_node: 20
    walltime: "00:59:00"  # Note: always quote walltime in YAML

Alternatively, it can be used as a base class for Torque cluster configurations.

The following BPS configuration parameters are recognised (and required unless there is a default mentioned here, or provided by a subclass):

queue (int): Queue for the Torque job.
nodes (int): number of nodes for each Torque job.
tasks_per_node (int): number of cores per node for each Torque job; by default we use all cores on the node.
walltime (str): time limit for each Torque job.
scheduler_options (str): text to prepend to the Torque submission script (each line usually starting with #PBS).

Methods Summary

`from_config`(config)	Get the site configuration nominated in the BPS config.
`get_address`()	Return the IP address of the machine hosting the driver/submission.
`get_command_prefix`()	Return command(s) to add before each job command.
`get_executors`()	Get a list of executors to be used in processing.
`get_monitor`()	Get parsl monitor.
`get_parsl_config`()	Get Parsl configuration for this site.
`get_site_subconfig`(config)	Get BPS configuration for the site of interest.
`make_executor`(label, *[, queue, nodes, ...])	Return an executor for running on a Torque cluster.
`select_executor`(job)	Get the `label` of the executor to use to execute a job.

Methods Documentation

classmethod from_config(config: BpsConfig) → SiteConfig¶

Get the site configuration nominated in the BPS config.

The computeSite (str) value in the BPS configuration is used to select a site configuration. The site configuration class to use is specified by the BPS configuration as site.<computeSite>.class (str), which should be the fully-qualified name of a python class that inherits from SiteConfig.

Parameters:

configBpsConfig: BPS configuration.

Returns:

site_configsubclass of SiteConfig: Site configuration.

get_address() → str¶

Return the IP address of the machine hosting the driver/submission.

This address should be accessible from the workers. This should generally by the return value of one of the functions in parsl.addresses.

This is used by the default implementation of get_monitor, but will generally be used by get_executors too.

This default implementation gets the address from the hostname, but that will not work if the workers don’t access the driver/submission node by that address.

get_command_prefix() → str¶

Return command(s) to add before each job command.

These may be used to configure the environment for the job.

This default implementation respects the BPS configuration elements:

site.<computeSite>.commandPrefix (str): command(s) to use as a prefix to executing a job command on a worker.
site.<computeSite>.environment (bool): add bash commands that replicate the environment on the driver/submit machine?

get_executors() → list[parsl.executors.base.ParslExecutor]¶

Get a list of executors to be used in processing.

Each executor should have a unique label.

get_monitor() → MonitoringHub | None¶

Get parsl monitor.

The parsl monitor provides a database that tracks the progress of the workflow and the use of resources on the workers.

This implementation respects the BPS configuration elements:

site.<computeSite>.monitorEnable (bool): enable monitor?
site.<computeSite>.monitorInterval (float): time interval (sec) between logging of resource usage.
site.<computeSite>.monitorFilename (str): name of file to use for the monitor sqlite database.

Returns:

monitorMonitoringHub or None: Parsl monitor, or None for no monitor.

get_parsl_config() → Config¶

Get Parsl configuration for this site.

Subclasses can overwrite this method to build a more specific Parsl configuration, if required.

The retries are set from the site.<computeSite>.retries value found in the BPS configuration file.

Returns:

configparsl.config.Config: The configuration to be used for Parsl.

static get_site_subconfig(config: BpsConfig) → BpsConfig¶

Get BPS configuration for the site of interest.

We return the BPS sub-configuration for the site indicated by the computeSite value, which is site.<computeSite>.

Parameters:

configBpsConfig: BPS configuration.

Returns:

siteBpsConfig: Site sub-configuration.

Return an executor for running on a Torque cluster.

Parameters:

labelstr: Label for executor.
queuestr, optional: Queue for the Torque job.
nodesint, optional: Default number of nodes for each Torque job.
tasks_per_nodeint, optional: Default number of cores per node for each Torque job.
walltimestr, optional: Default time limit for each Torque job.
mem_per_workerfloat, optional: Minimum memory per worker (GB), limited by the executor.
worker_initstr, optional: Environment initiation command
scheduler_optionsstr, optional: #SBATCH directives to prepend to the Torque submission script.
provider_optionsdict, optional: Additional arguments for TorqueProvider constructor.
executor_optionsdict, optional: Additional arguments for HighThroughputExecutor constructor.

Returns:

executorHighThroughputExecutor: Executor for Torque jobs.

select_executor(job: ParslJob) → str¶

Get the label of the executor to use to execute a job.

Parameters:

jobParslJob: Job to be executed.

Returns:

labelstr: Label of executor to use to execute job.

Navigation

Torque¶