Ccin2p3¶

class lsst.ctrl.bps.parsl.sites.ccin2p3.Ccin2p3(*args, **kwargs)¶

Bases: SiteConfig

Configuration for running Parsl jobs in CC-IN2P3 Slurm batch farm.

This class provides 4 job slot sizes with different requirements, in particular in terms of memory. Those slot sizes are named “small”, “medium”, “large” and “xlarge”.

Sensible default values for those requirements are provided for each kind of job but you can specify different values either in the the BPS submission file or in a site configuration file that you include in your BPS submission file.

This is an example of how to modify the specifications for those job slot sizes in the BPS submission file:

wmsServiceClass: lsst.ctrl.bps.parsl.ParslService
computeSite: ccin2p3

site:
  ccin2p3:
    class: lsst.ctrl.bps.parsl.sites.Ccin2p3
    walltime: "72:00:00"
    qos: "normal"
    small:
        memory: 4
        partition: "flash"
    medium:
        memory: 10
        partition: "lsst,htc"
    large:
        memory: 50
    xlarge:
        memory: 150
        partition: "lsst"

At the level of ‘site:’ entry in the BPS submission file, the following configuration parameters are accepted, which apply to all slot sizes:

partition (str): name of the one or more configured partitions. If
more than one, separate them with comma (‘,’). (Default: “lsst,htc”)
qos (str): quality of service to use (Default: “normal”)
walltime (str): walltime to require for the job (Default: “72:00:00”)

For each kind of job slot (i.e. “small”, “medium”, etc.) you can specify the parameters above as well as:

max_blocks (int): maximum number of Slurm jobs that your workflow can
simultaneously use.
memory (int): required amount of memory in Gigabytes.

as shown in the example above.

If you don’t need to modify those values and use the default configuration for all the job slot sizes use:

wmsServiceClass: lsst.ctrl.bps.parsl.ParslService
computeSite: ccin2p3

site:
ccin2p3:
    class: lsst.ctrl.bps.parsl.sites.Ccin2p3

Parameters:

*argsoptional: Arguments to initialize the super-class.
**kwargsoptional: Keyword arguments to initialize the super-class.

Returns:

Ccin2p3SiteConfig: Concrete instance of a SiteConfig specific for the CC-IN2P3 Slurm farm.

Methods Summary

`from_config`(config)	Get the site configuration nominated in the BPS config.
`get_address`()	Return the IP address of the machine hosting the driver/submission.
`get_command_prefix`()	Return command(s) to add before each job command.
`get_executors`()	Get a list of executors to be used for processing a workflow.
`get_monitor`()	Get parsl monitor.
`get_parsl_config`()	Get Parsl configuration for using CC-IN2P3 Slurm farm as a Parsl execution site.
`get_site_subconfig`(config)	Get BPS configuration for the site of interest.
`select_executor`(job)	Get the `label` of the executor to use to execute `job`.

Methods Documentation

classmethod from_config(config: BpsConfig) → SiteConfig¶

Get the site configuration nominated in the BPS config.

The computeSite (str) value in the BPS configuration is used to select a site configuration. The site configuration class to use is specified by the BPS configuration as site.<computeSite>.class (str), which should be the fully-qualified name of a python class that inherits from SiteConfig.

Parameters:

configBpsConfig: BPS configuration.

Returns:

site_configsubclass of SiteConfig: Site configuration.

get_address() → str¶

Return the IP address of the machine hosting the driver/submission.

This address should be accessible from the workers. This should generally by the return value of one of the functions in parsl.addresses.

This is used by the default implementation of get_monitor, but will generally be used by get_executors too.

This default implementation gets the address from the hostname, but that will not work if the workers don’t access the driver/submission node by that address.

get_command_prefix() → str¶

Return command(s) to add before each job command.

These may be used to configure the environment for the job.

This default implementation respects the BPS configuration elements:

site.<computeSite>.commandPrefix (str): command(s) to use as a prefix to executing a job command on a worker.
site.<computeSite>.environment (bool): add bash commands that replicate the environment on the driver/submit machine?

get_executors() → List[ParslExecutor]¶: Get a list of executors to be used for processing a workflow. Each executor must have a unique label.

get_monitor() → MonitoringHub | None¶

Get parsl monitor.

The parsl monitor provides a database that tracks the progress of the workflow and the use of resources on the workers.

This implementation respects the BPS configuration elements:

site.<computeSite>.monitorEnable (bool): enable monitor?
site.<computeSite>.monitorInterval (float): time interval (sec) between logging of resource usage.
site.<computeSite>.monitorFilename (str): name of file to use for the monitor sqlite database.

Returns:

monitorMonitoringHub or None: Parsl monitor, or None for no monitor.

get_parsl_config() → Config¶

Get Parsl configuration for using CC-IN2P3 Slurm farm as a Parsl execution site.

Returns:

configparsl.config.Config: The configuration to be used to initialize Parsl for this site.

static get_site_subconfig(config: BpsConfig) → BpsConfig¶

Get BPS configuration for the site of interest.

We return the BPS sub-configuration for the site indicated by the computeSite value, which is site.<computeSite>.

Parameters:

configBpsConfig: BPS configuration.

Returns:

siteBpsConfig: Site sub-configuration.

select_executor(job: ParslJob) → str¶

Get the label of the executor to use to execute job.

Parameters:

jobParslJob: Job to be executed.

Returns:

labelstr: Label of executor to use to execute job.

Navigation

Ccin2p3¶