lsst-ctrl-bps-panda v28.0.0 (2024-11-21)

New Features

  • Updated to support event service which allows multiple short payload jobs to run in a single PanDA job for efficiency. (DM-38177)

Other Changes and Additions

  • Removed requestMemory, memoryMultipler and numberOfRetries from since those are now defined in ctrl_bps (DM-44513)

  • Fixed pandaDistributionEndpoint to support different protocols (DM-45631)

  • Removed support for execution butler for remote job submission (DM-46845)

lsst-ctrl-bps-panda v27.0.0 (2024-06-04)

New Features

  • Mapped bps memory boosting attributes to PanDA memory boosting attributes (DM-36981)

  • Updated the open-source license to allow for the code to be distributed with either GPLv3 or BSD 3-clause license. (DM-37231)

  • Updated panda-service.py to return exit code information. (DM-41543)

Bug Fixes

  • Restored error message about too long pseudo filename while preparing workflow. (DM-40699)

  • Fixed a typo in the reference of a job attribute. (DM-42090)

  • Fixed report function during checking of request status. (DM-42528)

    • Fixed bps panda report function when no tasks are created (when no tasks are created, a SQL outer join will create an empty row which seems like a task is created).

    • Fixed bps panda report function when a build task fails and doesn’t report any file information. (DM-42788)

Other Changes and Additions

  • Removed error message about too long pseudo filename while preparing workflow as the length is now checked by iDDS. (DM-41545)

lsst-ctrl-bps-panda v26.0.0 (2023-09-25)

New Features

  • Enabled saving of iDDS client workflow objects at prepare bps submission stage. (DM-34915)

  • Updated BPS PanDA plugin to work with quantum-backed butler. (DM-39553)

Bug Fixes

  • Fixed the execution butler transfer bug when period in run collection. (DM-37843)

  • Fixed PanDA task chunking bug that caused assertion error during submission for really large QuantumGraphs. (DM-38101)

  • Handled dependency issues when preparing rescue workflows. (DM-38377)

Other Changes and Additions

  • Now print out pseudo_file_name in the bps PanDA plugin to simplify debugging if there is a problem with it being too long. (DM-37352)

  • Included butler repo URL in log message labels. (DM-37961)

  • Modified PanDA task chunking to be evenly divided. (DM-38101)

  • Updated ctrl_bps_panda/config/bps_usdf.yaml to allow for local custom setup (DM-38142)

  • Updated some default YAML values to more easily allow parts to be modified as well as provided values to go with the updated bps default YAML (e.g., no longer need runQuantumCommands). (DM-38307)

  • Moved fileDistributionEndPoint from lustre to weka in USDF configuration. (DM-39334)

lsst-ctrl-bps-panda v25.0.0 (2023-03-02)

New Features

  • Added cancel, restart, report and ping functions in bps panda plugin. (DM-34964)

  • Added setupLSSTEnv in bps_usdf.yaml which can be updated to setup developer lsst pipelines stack. (DM-36376)

Bug Fixes

  • Fixed the bug that bps-panda reports success when there is an authentication permission error. (DM-35364)

  • Fixed the setting of number_of_retries to None in idds_tasks. (DM-35508)

  • Fixed the bug that iDDS results can be something other than a string. (DM-35964)

Other Changes and Additions

  • Added the ability to prioritize dev/test tasks * prodSourceLabel: it can be configured in the submission yaml, by default it is ‘managed’ * priority: it can be set in the submission yaml, by default it is 500 (DM-36375)

lsst-ctrl-bps-panda v24.0.0 (2022-08-29)

New Features

  • This package has been extracted from lsst_ctrl_bps into a standalone package to make it easier to manage development of the PanDA plugin. (DM-33521)

  • Introduced a new parameter dockerImageLocation in the PanDA IDF configuration yaml file to pull lsst release containers from GAR (Google Artifact Registry). This parameter is trailed with '/', so it could be used in sw_image path in the following example. And the sw_image will still refer to the Docker hub, if the parameter dockerImageLocation is empty or not defined, to make the sw_image backward compatible with previous PanDA IDF configuration yaml files.

    In the user bps submission yaml file, just prepend this parameter to the sw_image path, that is:

    sw_image: "{dockerImageLocation}lsstsqre/centos:7-stack-lsst_distrib-w_2022_05"

    Please note that there is no extra character(s) between {dockerImageLocation} and lsstsqre.

    In case you have to use images from the Docker hub instead, you just take out the prefix {dockerImageLocation} in the path, that is:

    sw_image: "lsstsqre/centos:7-stack-lsst_distrib-w_2022_05" (`DM-32992 <https://rubinobs.atlassian.net/browse/DM-32992>`_)

Bug Fixes

  • Update the path to the command line decoder in the config file and the documentation. (DM-34574)

Other Changes and Additions

  • Changed the parameter runnerCommand in the PanDA IDF example yaml file, to start prmon to monitor the memory usage of the payload job. This executable prmon is only available in releases after w_2022_05. (DM-32579)

  • Make the PanDA example config more easily runnable from data-int RSP (DM-32695)

    • PanDA cloud was mapped from BPS compute site, fixed it.

    • Pass BPS cloud to PanDA cloud.

    • Add supports for task priority, vo, working group, prodSourceLabel. (DM-33889)

  • Remove iddsServer from bps_idf.yml, to use the iDDS server defined in the PanDA relay service.

    Remove IDDS_CONFIG requirements (requiring idds version 0.10.6 and later). (DM-34106)

  • Add missing __all__ statement to make the documentation render properly at https://pipelines.lsst.io. (DM-34921)

ctrl_bps v23.0.1 (2022-02-02)

New Features

    • Large tasks (> 30k jobs) splitted into chunks

    • Updated iDDS API usage for the most recent version

    • Updated iDDS API initialization to force PanDA proxy using the IAM user name for submitted workflow

    • Added limit on number of characters in the task pseudo inputs (DM-32675)

    • New panda_auth command for handling PanDA authentication token. Includes status, reset, and clean capabilities.

    • Added early check of PanDA authentication token in submission process. (DM-32830)

Other Changes and Additions

    • Changed printing of submit directory early.

    • Changed PanDA plugin to only print the numeric id when outputing the request/run id.

    • Set maximum number of jobs in a PanDA task (maxJobsPerTask) to 70000 in config/bps_idf.yaml. (DM-32830)

ctrl_bps v23.0.0 (2021-12-10)

Other Changes and Additions

  • Provide a cleaned up version of default config yaml for PanDA-plugin on IDF (DM-31476)