PanDAService

class lsst.ctrl.bps.panda.PanDAService(config)

Bases: lsst.ctrl.bps.BaseWmsService

PanDA version of WMS service

Methods Summary

add_decoder_prefix(cmd_line, …)	Compose the command line sent to the pilot from the functional part (the actual SW running) and the middleware part (containers invocation)
cancel(wms_id[, pass_thru])	Cancel submitted workflows/jobs.
convert_exec_string_to_hex(cmdline)	Convert the command line into hex representation.
copy_files_for_distribution(tasks, …)	Brings locally generated files into Cloud for further utilization them on the edge nodes.
get_idds_client()	Get the idds client
get_idds_result(ret)	Parse the results returned from iDDS.
list_submitted_jobs([wms_id, user, …])	Query WMS for list of submitted WMS workflows/jobs.
ping([pass_thru])	Checks whether PanDA WMS services are up, reachable,
prepare(config, generic_workflow[, out_prefix])	Convert generic workflow to an PanDA iDDS ready for submission
report([wms_workflow_id, user, hist, …])	Stub for future implementation of the report method Expected to return run information based upon given constraints.
restart(wms_workflow_id)	Restart a workflow from the point of failure.
run_submission_checks()	Checks to run at start if running WMS specific submission steps.
submit(workflow)	Submit a single PanDA iDDS workflow

Methods Documentation

add_decoder_prefix(cmd_line, distribution_path, files)

Compose the command line sent to the pilot from the functional part (the actual SW running) and the middleware part (containers invocation)

Parameters:

cmd_line : str

UTF-8 based functional part of the command line

distribution_path : str

URI of path where all files are located for distribution

files `list` [`str`]

File names needed for a task

Returns:

decoder_prefix : str

Full command line to be executed on the edge node

cancel(wms_id, pass_thru=None)

Cancel submitted workflows/jobs.

Parameters:

wms_id : str

ID or path of job that should be canceled.

pass_thru : str, optional

Information to pass through to WMS.

Returns:

deleted : bool

Whether successful deletion or not. Currently, if any doubt or any individual jobs not deleted, return False.

message : str

Any message from WMS (e.g., error details).

convert_exec_string_to_hex(cmdline)

Convert the command line into hex representation.

This step is currently involved because large blocks of command lines including special symbols passed to the pilot/container. To make sure the 1 to 1 matching and pass by the special symbol stripping performed by the Pilot we applied the hexing.

Parameters:

cmdline : str

UTF-8 command line string

Returns:

hex : str

Hex representation of string

static copy_files_for_distribution(tasks, file_distribution_uri)

Brings locally generated files into Cloud for further utilization them on the edge nodes.

Parameters:

local_pfns: `list` of `tasks`

Tasks that input files needs to be placed for distribution

file_distribution_uri: `str`

Path on the edge node accessed storage, including access protocol, bucket name to place files

Returns:

files_plc_hldr, direct_IO_files : dict [str, str], set of str

First parameters is key values pairs of file placeholder - file name Second parameter is set of files which will be directly accessed.

get_idds_client()

Get the idds client

Returns:

idds_client: idds.client.clientmanager.ClientManager

iDDS ClientManager object.

get_idds_result(ret)

Parse the results returned from iDDS.

Parameters:

ret: `tuple` of (`int`, (`bool`, payload)).

The first part ret[0] is the status of PanDA relay service. The part of ret[1][0] is the status of iDDS service. The part of ret[1][1] is the returned payload. If ret[1][0] is False, ret[1][1] can be error messages.

Returns:

status: bool

The status of iDDS calls.

result: int or list or dict

The result returned from iDDS.

error: str

Error messages.

list_submitted_jobs(wms_id=None, user=None, require_bps=True, pass_thru=None, is_global=False)

Query WMS for list of submitted WMS workflows/jobs.

This should be a quick lookup function to create list of jobs for other functions.

Parameters:

wms_id : int or str, optional

Id or path that can be used by WMS service to look up job.

user : str, optional

User whose submitted jobs should be listed.

require_bps : bool, optional

Whether to require jobs returned in list to be bps-submitted jobs.

pass_thru : str, optional

Information to pass through to WMS.

is_global : bool, optional

If set, all available job queues will be queried for job information. Defaults to False which means that only a local job queue will be queried for information.

Only applicable in the context of a WMS using distributed job queues (e.g., HTCondor). A WMS with a centralized job queue (e.g. PanDA) can safely ignore it.

Returns:

req_ids : list [Any]

Only job ids to be used by cancel and other functions. Typically this means top-level jobs (i.e., not children jobs).

ping(pass_thru=None)

Checks whether PanDA WMS services are up, reachable,

and can authenticate if authentication is required.

The services to be checked are those needed for submit, report, cancel, restart, but ping cannot guarantee whether jobs would actually run successfully. Any messages should be sent directly to the logger.

Parameters:

pass_thru : str, optional

Information to pass through to WMS.

Returns:

status : int

0 for success, non-zero for failure

message : str

Any message from WMS (e.g., error details).

prepare(config, generic_workflow, out_prefix=None)

Convert generic workflow to an PanDA iDDS ready for submission

Parameters:

config : lsst.ctrl.bps.BpsConfig

BPS configuration that includes necessary submit/runtime information.

generic_workflow : lsst.ctrl.bps.GenericWorkflow

out_prefix : str

The root directory into which all WMS-specific files are written

Returns:

workflow : lsst.ctrl.bps.panda.PandaBpsWmsWorkflow

PanDA workflow ready to be run.

report(wms_workflow_id=None, user=None, hist=0, pass_thru=None, is_global=False)

Stub for future implementation of the report method Expected to return run information based upon given constraints.

Parameters:

wms_workflow_id : int or str

Limit to specific run based on id.

user : str

Limit results to runs for this user.

hist : float

Limit history search to this many days.

pass_thru : str

Constraints to pass through to HTCondor.

is_global : bool, optional

If set, all available job queues will be queried for job information. Defaults to False which means that only a local job queue will be queried for information.

Returns:

runs : list [lsst.ctrl.bps.WmsRunReport]

Information about runs from given job information.

message : str

Extra message for report command to print. This could be pointers to documentation or to WMS specific commands.

restart(wms_workflow_id)

Restart a workflow from the point of failure.

Parameters:

wms_workflow_id : str

Id that can be used by WMS service to identify workflow that need to be restarted.

Returns:

wms_id : str

Id of the restarted workflow. If restart failed, it will be set to None.

run_name : str

Name of the restarted workflow. If restart failed, it will be set to None.

message : str

A message describing any issues encountered during the restart. If there were no issue, an empty string is returned.

run_submission_checks()

Checks to run at start if running WMS specific submission steps.

Any exception other than NotImplementedError will halt submission. Submit directory may not yet exist when this is called.

submit(workflow)

Submit a single PanDA iDDS workflow

Parameters:

workflow : lsst.ctrl.bps.BaseWorkflow

A single PanDA iDDS workflow to submit