PanDAService¶
- class lsst.ctrl.bps.panda.PanDAService(config)¶
Bases:
BaseWmsService
PanDA version of WMS service
Methods Summary
add_decoder_prefix
(cmd_line, ...)Compose the command line sent to the pilot from the functional part (the actual SW running) and the middleware part (containers invocation)
cancel
(wms_id[, pass_thru])Cancel submitted workflows/jobs.
convert_exec_string_to_hex
(cmdline)Convert the command line into hex representation.
copy_files_for_distribution
(tasks, ...)Brings locally generated files into Cloud for further utilization them on the edge nodes.
list_submitted_jobs
([wms_id, user, ...])Query WMS for list of submitted WMS workflows/jobs.
ping
(pass_thru)Checks whether WMS services are up, reachable, and can authenticate if authentication is required.
prepare
(config, generic_workflow[, out_prefix])Convert generic workflow to an PanDA iDDS ready for submission
report
([wms_workflow_id, user, hist, ...])Stub for future implementation of the report method Expected to return run information based upon given constraints.
restart
(wms_workflow_id)Restart a workflow from the point of failure.
Checks to run at start if running WMS specific submission steps.
submit
(workflow)Submit a single PanDA iDDS workflow
Methods Documentation
- add_decoder_prefix(cmd_line, distribution_path, files)¶
Compose the command line sent to the pilot from the functional part (the actual SW running) and the middleware part (containers invocation)
- cancel(wms_id, pass_thru=None)¶
Cancel submitted workflows/jobs.
- Parameters:
- Returns:
- convert_exec_string_to_hex(cmdline)¶
Convert the command line into hex representation.
This step is currently involved because large blocks of command lines including special symbols passed to the pilot/container. To make sure the 1 to 1 matching and pass by the special symbol stripping performed by the Pilot we applied the hexing.
- static copy_files_for_distribution(tasks, file_distribution_uri)¶
Brings locally generated files into Cloud for further utilization them on the edge nodes.
- Parameters:
- local_pfns: `list` of `tasks`
Tasks that input files needs to be placed for distribution
- file_distribution_uri: `str`
Path on the edge node accessed storage, including access protocol, bucket name to place files
- Returns:
- list_submitted_jobs(wms_id=None, user=None, require_bps=True, pass_thru=None, is_global=False)¶
Query WMS for list of submitted WMS workflows/jobs.
This should be a quick lookup function to create list of jobs for other functions.
- Parameters:
- wms_id
int
orstr
, optional Id or path that can be used by WMS service to look up job.
- user
str
, optional User whose submitted jobs should be listed.
- require_bps
bool
, optional Whether to require jobs returned in list to be bps-submitted jobs.
- pass_thru
str
, optional Information to pass through to WMS.
- is_global
bool
, optional If set, all available job queues will be queried for job information. Defaults to False which means that only a local job queue will be queried for information.
Only applicable in the context of a WMS using distributed job queues (e.g., HTCondor). A WMS with a centralized job queue (e.g. PanDA) can safely ignore it.
- wms_id
- Returns:
- job_ids
list
[Any
] Only job ids to be used by cancel and other functions. Typically this means top-level jobs (i.e., not children jobs).
- job_ids
- ping(pass_thru)¶
Checks whether WMS services are up, reachable, and can authenticate if authentication is required.
The services to be checked are those needed for submit, report, cancel, restart, but ping cannot guarantee whether jobs would actually run successfully.
- prepare(config, generic_workflow, out_prefix=None)¶
Convert generic workflow to an PanDA iDDS ready for submission
- Parameters:
- config
lsst.ctrl.bps.BpsConfig
BPS configuration that includes necessary submit/runtime information.
- generic_workflow
lsst.ctrl.bps.GenericWorkflow
- out_prefix
str
The root directory into which all WMS-specific files are written
- config
- Returns:
- workflow
lsst.ctrl.bps.panda.panda_service.PandaBpsWmsWorkflow
PanDA workflow ready to be run.
- workflow
- report(wms_workflow_id=None, user=None, hist=0, pass_thru=None, is_global=False)¶
Stub for future implementation of the report method Expected to return run information based upon given constraints.
- Parameters:
- wms_workflow_id
int
orstr
Limit to specific run based on id.
- user
str
Limit results to runs for this user.
- hist
float
Limit history search to this many days.
- pass_thru
str
Constraints to pass through to HTCondor.
- is_global
bool
, optional If set, all available job queues will be queried for job information. Defaults to False which means that only a local job queue will be queried for information.
- wms_workflow_id
- Returns:
- runs
list
[lsst.ctrl.bps.WmsRunReport
] Information about runs from given job information.
- message
str
Extra message for report command to print. This could be pointers to documentation or to WMS specific commands.
- runs
- restart(wms_workflow_id)¶
Restart a workflow from the point of failure.
- Parameters:
- wms_workflow_id
str
Id that can be used by WMS service to identify workflow that need to be restarted.
- wms_workflow_id
- Returns:
- run_submission_checks()¶
Checks to run at start if running WMS specific submission steps.
Any exception other than NotImplementedError will halt submission. Submit directory may not yet exist when this is called.
- submit(workflow)¶
Submit a single PanDA iDDS workflow
- Parameters:
- workflow
lsst.ctrl.bps.BaseWorkflow
A single PanDA iDDS workflow to submit
- workflow