lsst-ctrl-mpexec v24.0.0 (2022-08-26) ===================================== New Features ------------ - Added ``--dataset-query-constraint`` option to ``pipetask qgraph`` command (and thus downstream commands) that allows a user to control how `~lsst.pipe.base.QuantumGraph` creation is constrained by dataset existence. (`DM-31769 <https://jira.lsstcorp.org/browse/DM-31769>`_) - Builds using ``setuptools`` now calculate versions from the Git repository, including the use of alpha releases for those associated with weekly tags. (`DM-32408 <https://jira.lsstcorp.org/browse/DM-32408>`_) - Added ``--summary`` option to ``pipetask run`` command, it produces JSON report for execution status of the whole process and individual quanta. (`DM-33481 <https://jira.lsstcorp.org/browse/DM-33481>`_) - Added ``pipetask`` CLI commands ``purge`` and ``cleanup``. (`DM-33634 <https://jira.lsstcorp.org/browse/DM-33634>`_) - Removed dependency on the ``obs_base`` and ``afw`` packages. Now only depends on ``pipe_base`` and ``daf_butler`` (along with ``pex_config`` and ``utils``). (`DM-34105 <https://jira.lsstcorp.org/browse/DM-34105>`_) - Replaced the unused ``--do-raise`` option with ``--pdb``, which drops the user into the debugger (``pdb`` by default, but ``--pdb=ipdb`` also works if you have ``ipdb`` installed) on an exception. (`DM-34215 <https://jira.lsstcorp.org/browse/DM-34215>`_) Bug Fixes --------- - The `click.Path` API should mostly be used with keyword arguments, changed from ordered arguments to keyword arguments when calling it. (`DM-34261 <https://jira.lsstcorp.org/browse/DM-34261>`_) - Fixed a bug where dot graphs of pipelines did not correctly render edges between composite and component dataset types. (`DM-34811 <https://jira.lsstcorp.org/browse/DM-34811>`_) Other Changes and Additions --------------------------- - Added `lsst.ctrl.mpexec.SimplePipelineExecutor`, a minimal high-level Python interface for `~lsst.pipe.base.PipelineTask` execution intended primarily for unit testing. (`DM-31966 <https://jira.lsstcorp.org/browse/DM-31966>`_) lsst-ctrl-mpexec v23.0.1 (2022-02-02) ===================================== Miscellaneous Changes of Minor Interest --------------------------------------- - Allow ``pipetask run`` execution to continue in single-process mode after failure of one or more tasks. Previously execution stopped on an exception from any task. (`DM-33339 <https://jira.lsstcorp.org/browse/DM-33339>`_) lsst-ctrl-mpexec v23.0.0 (2021-12-10) ===================================== New Features ------------ - Several improvements in ``pipetask`` execution options: - New option ``--skip-existing-in`` which takes collection names(s), if output datasets already exist in those collections corresponding quanta is skipped. - A ``--skip-existing`` option is now equivalent to appending output run collection to the ``--skip-existing-in`` list. - An ``--extend-run`` option implicitly enables ``--skip-existing`` option. - A ``--prune-replaced=unstore`` option only removes regular output datasets; InitOutputs, task configs, and package versions are not removed. (`DM-27492 <https://jira.lsstcorp.org/browse/DM-27492>`_) - GraphViz dot files generated by pipetask now include more information (``RUN`` collection for datasets, dimensions for tasks, data IDs for quanta). (`DM-28111 <https://jira.lsstcorp.org/browse/DM-28111>`_) - ``pipetask qgraph`` can now generate a standalone "execution butler" which is a SQLite registry with all the expected outputs pre-filled in registry. Using this registry allow ``pipetask run`` to execute without touching the main registry whilst still writing file artifacts to the standard location. It is not yet configured to allow a completely detached processing using a local datastore but this can be changed manually after creation to use a chained datastore. (`DM-28649 <https://jira.lsstcorp.org/browse/DM-28649>`_) - * Log messages issued during quantum execution are now collected and stored in butler as ``tasklabel_log`` dataset types. * New command line options for logging have been added to ``pipetask``. These include ``--log-file`` to write log messages to a file and ``--no-log-tty`` to disable log output to the terminal. (`DM-30977 <https://jira.lsstcorp.org/browse/DM-30977>`_) - * Add the output run to the log record. * Add ``--log-label`` option to ``pipetask`` command to allow extra information to be injected into the log record. (`DM-31884 <https://jira.lsstcorp.org/browse/DM-31884>`_) Bug Fixes --------- - Improve exception handling in ``ExecFixupDataId`` (`DM-29384 <https://jira.lsstcorp.org/browse/DM-29384>`_) - Fix config comparison with ``--skip-existing``. (`DM-29580 <https://jira.lsstcorp.org/browse/DM-29580>`_) - Include output collection in call to ``buildExecutionButler``. (`DM-31691 <https://jira.lsstcorp.org/browse/DM-31691>`_) - Fix call to ``buildExecutionButler`` when chained input collection. (`DM-31711 <https://jira.lsstcorp.org/browse/DM-31711>`_) Miscellaneous Changes of Minor Interest --------------------------------------- - Add some of the pipetask command line options to QuantumGraph metadata (`DM-30702 <https://jira.lsstcorp.org/browse/DM-30702>`_) lsst-ctrl-mpexec v22.0 (2021-04-01) =================================== New Features ------------ * ``pipetask run`` can now execute a subset of a graph. This allows a single graph file to be created with an entire workflow and then only part of it to be executed. This is very important for large scale workflow execution. [DM-27667] Performance Enhancement ----------------------- * Multi-processing execution performance has been significantly improved for large graphs. [DM-28418] Other ----- * Ignore ``--input`` instead of rejecting it if it hasn't changed. [DM-28101] * The graph file format has been changed from a pickle file to a form that can efficiently be accessed from an object store. This new format has a ``.qgraph`` file extension. [DM-27784] * A full URI can now be used to specify the location of the quantum graph. [DM-27682]