lsst.ctrl.bps.parsl

Changes

Introduction

This package is a Parsl-based plugin for the LSST Batch Production Service (BPS) execution framework (lsst.ctrl.bps). It is intended to support running LSST PipelineTask jobs on high-performance computing (HPC) clusters. Parsl includes execution providers that allow operation on batch systems typically used by HPC clusters, e.g., Slurm, PBS/Torque and LSF. Parsl can also be configured to run on a single node using a thread pool, which is useful for testing and development.

Note that while Parsl does provide means for staging files for workers, these are not currently used by the ctrl_bps_parsl plugin. Use of this plugin, therefore, is currently limited to environments where workers will have direct access to the butler files being processed. If there is sufficient interest, this might change in the future, opening up the possibility of using this plugin to process data with cloud nodes, e.g., AWS and GoogleCloud.

Software

This package is based on the Parsl plugin developed for the LSST-DESC Data Challenge 2 by Jim Chiang. The plugin is now suitable for use in a variety of cluster environments and will be used by other projects, including the Subaru Prime Focus Spectrograph and the Merian survey.

This package is open source software, released under the GNU Public License (GPL). Contributions, including bug reports and feature requests, are welcome: please open an issue on GitHub.

Installation

Use

Python API reference

Functions

get_bps_config_value()

Get a value from the BPS configuration.

get_file_paths(workflow, name)

Extract file paths for a job.

get_parsl_config(config)

Construct parsl configuration from BPS configuration.

get_workflow_filename(out_prefix)

Get filename for persisting workflow.

get_workflow_name(config)

Get name of this workflow.

set_parsl_logging(config)

Set parsl logging levels.

Classes

ParslJob(generic, config, file_paths)

Job to execute with parsl.

ParslService(config)

Parsl-based implementation for the WMS interface.

ParslWorkflow(name, config, path, jobs, ...)

Parsl-based workflow object to manage execution of workflow.

SiteConfig(config[, add_resources])

Base class for site configuration.

Base and general-purpose sites

The following are base classes that can also be used as general-purpose site configurations.

Classes

Local(config[, add_resources])

Configuration for running jobs on the local machine.

LocalSrunWorkQueue(*args, **kwargs)

Configuration for a WorkQueueExecutor that uses a LocalProvider to manage resources.

Slurm(config[, add_resources])

Configuration for generic Slurm cluster.

TripleSlurm(*args, **kwargs)

Configuration for running jobs on a Slurm cluster with three levels.

WorkQueue(*args, **kwargs)

Base class configuraton for WorkQueueExecutor.

Class Inheritance Diagram

digraph inheritancedde74a2ce3 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "Local" [URL="../../../py-api/lsst.ctrl.bps.parsl.sites.Local.html#lsst.ctrl.bps.parsl.sites.Local",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration for running jobs on the local machine."]; "SiteConfig" -> "Local" [arrowsize=0.5,style="setlinewidth(0.5)"]; "LocalSrunWorkQueue" [URL="../../../py-api/lsst.ctrl.bps.parsl.sites.LocalSrunWorkQueue.html#lsst.ctrl.bps.parsl.sites.LocalSrunWorkQueue",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration for a `WorkQueueExecutor` that uses a `LocalProvider`"]; "WorkQueue" -> "LocalSrunWorkQueue" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SiteConfig" [URL="../../../py-api/lsst.ctrl.bps.parsl.SiteConfig.html#lsst.ctrl.bps.parsl.SiteConfig",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for site configuration."]; "ABC" -> "SiteConfig" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Slurm" [URL="../../../py-api/lsst.ctrl.bps.parsl.sites.Slurm.html#lsst.ctrl.bps.parsl.sites.Slurm",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration for generic Slurm cluster."]; "SiteConfig" -> "Slurm" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TripleSlurm" [URL="../../../py-api/lsst.ctrl.bps.parsl.sites.TripleSlurm.html#lsst.ctrl.bps.parsl.sites.TripleSlurm",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration for running jobs on a Slurm cluster with three levels."]; "Slurm" -> "TripleSlurm" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WorkQueue" [URL="../../../py-api/lsst.ctrl.bps.parsl.sites.WorkQueue.html#lsst.ctrl.bps.parsl.sites.WorkQueue",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class configuraton for `WorkQueueExecutor`."]; "SiteConfig" -> "WorkQueue" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Specific sites

The following are intended to support operations at specific sites or by specific groups.

Classes

CoriKnl(*args, **kwargs)

Configuration for running jobs on the NERSC Cori-KNL cluster.

Classes

Tiger(config[, add_resources])

Configuration for running jobs on Princeton's Tiger cluster.

Classes

Sdf(config[, add_resources])

Slurm-based configuration for running jobs on SLAC's Shared Data Facility cluster using the rubin partition.

Classes

Ccin2p3(*args, **kwargs)

Configuration for running Parsl jobs in CC-IN2P3 Slurm batch farm.

Environment utility functions

Functions

export_environment()

Generate bash script to regenerate the current environment.