Multi-Processing Executor v23.0.1 2022-02-02
============================================

Miscellaneous Changes of Minor Interest
---------------------------------------

- Allow ``pipetask run`` execution to continue in single-process mode after failure of one or more tasks. Previously execution stopped on an exception from any task. (`DM-33339 <https://jira.lsstcorp.org/browse/DM-33339>`_)


Multi-Processing Executor v23.0.0 2021-12-10
============================================

New Features
------------

- Several improvements in ``pipetask`` execution options:
  - New option ``--skip-existing-in`` which takes collection names(s), if output
    datasets already exist in those collections corresponding quanta is skipped.
  - A ``--skip-existing`` option is now equivalent to appending output run
    collection to the ``--skip-existing-in`` list.
  - An ``--extend-run`` option implicitly enables ``--skip-existing`` option.
  - A ``--prune-replaced=unstore`` option only removes regular output datasets;
    InitOutputs, task configs, and package versions are not removed. (`DM-27492 <https://jira.lsstcorp.org/browse/DM-27492>`_)
- GraphViz dot files generated by pipetask now include more information (``RUN`` collection for datasets, dimensions for tasks, data IDs for quanta). (`DM-28111 <https://jira.lsstcorp.org/browse/DM-28111>`_)
- ``pipetask qgraph`` can now generate a standalone "execution butler" which is a SQLite registry with all the expected outputs pre-filled in registry.  Using this registry allow ``pipetask run`` to execute without touching the main registry whilst still writing file artifacts to the standard location.  It is not yet configured to allow a completely detached processing using a local datastore but this can be changed manually after creation to use a chained datastore. (`DM-28649 <https://jira.lsstcorp.org/browse/DM-28649>`_)
- * Log messages issued during quantum execution are now collected and stored in butler as ``tasklabel_log`` dataset types.
  * New command line options for logging have been added to ``pipetask``. These include ``--log-file`` to write log messages to a file and ``--no-log-tty`` to disable log output to the terminal. (`DM-30977 <https://jira.lsstcorp.org/browse/DM-30977>`_)
- * Add the output run to the log record.
  * Add ``--log-label`` option to ``pipetask`` command to allow extra information to be injected into the log record. (`DM-31884 <https://jira.lsstcorp.org/browse/DM-31884>`_)


Bug Fixes
---------

- Improve exception handling in ``ExecFixupDataId`` (`DM-29384 <https://jira.lsstcorp.org/browse/DM-29384>`_)
- Fix config comparison with ``--skip-existing``. (`DM-29580 <https://jira.lsstcorp.org/browse/DM-29580>`_)
- Include output collection in call to ``buildExecutionButler``. (`DM-31691 <https://jira.lsstcorp.org/browse/DM-31691>`_)
- Fix call to ``buildExecutionButler`` when chained input collection. (`DM-31711 <https://jira.lsstcorp.org/browse/DM-31711>`_)


Miscellaneous Changes of Minor Interest
---------------------------------------

- Add some of the pipetask command line options to QuantumGraph metadata (`DM-30702 <https://jira.lsstcorp.org/browse/DM-30702>`_)


Multi-Processing Executor v22.0 2021-04-01
==========================================

New Features
------------

* ``pipetask run`` can now execute a subset of a graph. This allows a single graph file to be created with an entire workflow and then only part of it to be executed. This is very important for large scale workflow execution. [DM-27667]

Performance Enhancement
-----------------------

* Multi-processing execution performance has been significantly improved for large graphs. [DM-28418]

Other
-----

* Ignore ``--input`` instead of rejecting it if it hasn't changed. [DM-28101]
* The graph file format has been changed from a pickle file to a form that can efficiently be accessed from an object store. This new format has a ``.qgraph`` file extension. [DM-27784]
* A full URI can now be used to specify the location of the quantum graph. [DM-27682]