Ccin2p3¶
- class lsst.ctrl.bps.parsl.sites.ccin2p3.Ccin2p3(*args, **kwargs)¶
Bases:
SiteConfig
Configuration for executing Parsl jobs in CC-IN2P3 Slurm batch farm.
This class provides four job slot sizes each with its specific requirements, in particular in terms of memory. Those slot sizes are named “small”, “medium”, “large” and “xlarge”.
Sensible default values for those requirements are provided for each job slot but you can overwrite those defaults either in the the BPS submission file or in a site configuration file that you include in your BPS submission file.
If you don’t need to modify the default requirements for the job slot sizes, use the site specification below in your BPS configuration file:
wmsServiceClass: lsst.ctrl.bps.parsl.ParslService computeSite: ccin2p3 site: ccin2p3: class: lsst.ctrl.bps.parsl.sites.ccin2p3.Ccin2p3
If you do need to modify those defaults, you can overwrite them for all job slots or for specific each job slots. Requirements specified for a job slot take priority over those specified for all job slots at the level of entry ‘.site.ccin2p3:’.
This is an example of how to overwrite selected requirements in your BPS submission file:
wmsServiceClass: lsst.ctrl.bps.parsl.ParslService computeSite: ccin2p3 site: ccin2p3: class: lsst.ctrl.bps.parsl.sites.ccin2p3.Ccin2p3 walltime: "72:00:00" scheduler_options: - "--licenses=sps" - "--qos=normal" small: memory: 6 partition: "flash" medium: memory: 10 partition: "lsst,htc" large: memory: 80 xlarge: memory: 180 partition: "lsst" scheduler_options: - "--constraint=el7" - "--licenses=my_product" - "--reservation=my_reservation"
At the level of entry ‘site.ccin2p3:’ in the BPS submission file, the following configuration parameters are accepted, which apply to all slot sizes:
partition
(str
): name of the one or more configured partitions. Ifmore than one, separate them with comma (‘,’). (Default: “lsst,htc”)
walltime
(str
): walltime to require for the job (Default: “72:00:00”)
In addition, as shown in the previous example, for each job slot (i.e. “small”, “medium”, etc.) you can specify the requirements above as well as the following:
max_blocks
(int
): maximum number of Slurm jobs that your workflow cansimultaneously use.
memory
(int
): required amount of memory for each job, in Gigabytes.(Defaults: 4 for “small”, 10 for “medium”, 50 fo “large” and 150 for “xlarge”).
- Parameters:
- *argsoptional
Arguments to initialize the super-class.
- **kwargsoptional
Keyword arguments to initialize the super-class.
- Returns:
- Ccin2p3
SiteConfig
Concrete instance of a
SiteConfig
specific for the CC-IN2P3 Slurm farm.
- Ccin2p3
Attributes Summary
Methods Summary
from_config
(config)Get the site configuration nominated in the BPS config.
Return the IP address of the machine hosting the driver/submission.
Return command(s) to add before each job command.
Get a list of Parsl executors that can be used for processing a workflow.
Get parsl monitor.
Get Parsl configuration for using CC-IN2P3 Slurm farm as a Parsl execution site.
get_site_subconfig
(config)Get BPS configuration for the site of interest.
select_executor
(job)Get the
label
of the executor to use to executejob
.Attributes Documentation
Methods Documentation
- classmethod from_config(config: BpsConfig) SiteConfig ¶
Get the site configuration nominated in the BPS config.
The
computeSite
(str
) value in the BPS configuration is used to select a site configuration. The site configuration class to use is specified by the BPS configuration assite.<computeSite>.class
(str
), which should be the fully-qualified name of a python class that inherits fromSiteConfig
.- Parameters:
- config
BpsConfig
BPS configuration.
- config
- Returns:
- site_configsubclass of
SiteConfig
Site configuration.
- site_configsubclass of
- get_address() str ¶
Return the IP address of the machine hosting the driver/submission.
This address should be accessible from the workers. This should generally by the return value of one of the functions in
parsl.addresses
.This is used by the default implementation of
get_monitor
, but will generally be used byget_executors
too.This default implementation gets the address from the hostname, but that will not work if the workers don’t access the driver/submission node by that address.
- get_command_prefix() str ¶
Return command(s) to add before each job command.
These may be used to configure the environment for the job.
This default implementation respects the BPS configuration elements:
- get_executors() list[parsl.executors.base.ParslExecutor] ¶
Get a list of Parsl executors that can be used for processing a workflow.
Each executor must have a unique
label
.
- get_monitor() MonitoringHub | None ¶
Get parsl monitor.
The parsl monitor provides a database that tracks the progress of the workflow and the use of resources on the workers.
This implementation respects the BPS configuration elements:
- get_parsl_config() Config ¶
Get Parsl configuration for using CC-IN2P3 Slurm farm as a Parsl execution site.
- Returns:
- config
parsl.config.Config
The configuration to be used to initialize Parsl for this site.
- config
- static get_site_subconfig(config: BpsConfig) BpsConfig ¶
Get BPS configuration for the site of interest.
We return the BPS sub-configuration for the site indicated by the
computeSite
value, which issite.<computeSite>
.- Parameters:
- config
BpsConfig
BPS configuration.
- config
- Returns:
- site
BpsConfig
Site sub-configuration.
- site