Multi-Processing Executor v23.0.0 2021-12-10

New Features

  • Several improvements in pipetask execution options: - New option --skip-existing-in which takes collection names(s), if output

    datasets already exist in those collections corresponding quanta is skipped.

    • A --skip-existing option is now equivalent to appending output run collection to the --skip-existing-in list.
    • An --extend-run option implicitly enables --skip-existing option.
    • A --prune-replaced=unstore option only removes regular output datasets; InitOutputs, task configs, and package versions are not removed. (DM-27492)

  • GraphViz dot files generated by pipetask now include more information (RUN collection for datasets, dimensions for tasks, data IDs for quanta). (DM-28111)

  • pipetask qgraph can now generate a standalone “execution butler” which is a SQLite registry with all the expected outputs pre-filled in registry. Using this registry allow pipetask run to execute without touching the main registry whilst still writing file artifacts to the standard location. It is not yet configured to allow a completely detached processing using a local datastore but this can be changed manually after creation to use a chained datastore. (DM-28649)

    • Log messages issued during quantum execution are now collected and stored in butler as tasklabel_log dataset types.
    • New command line options for logging have been added to pipetask. These include --log-file to write log messages to a file and --no-log-tty to disable log output to the terminal. (DM-30977)
    • Add the output run to the log record.
    • Add --log-label option to pipetask command to allow extra information to be injected into the log record. (DM-31884)

Bug Fixes

  • Improve exception handling in ExecFixupDataId (DM-29384)
  • Fix config comparison with --skip-existing. (DM-29580)
  • Include output collection in call to buildExecutionButler. (DM-31691)
  • Fix call to buildExecutionButler when chained input collection. (DM-31711)

Miscellaneous Changes of Minor Interest

  • Add some of the pipetask command line options to QuantumGraph metadata (DM-30702)

Multi-Processing Executor v22.0 2021-04-01

New Features

  • pipetask run can now execute a subset of a graph. This allows a single graph file to be created with an entire workflow and then only part of it to be executed. This is very important for large scale workflow execution. [DM-27667]

Performance Enhancement

  • Multi-processing execution performance has been significantly improved for large graphs. [DM-28418]

Other

  • Ignore --input instead of rejecting it if it hasn’t changed. [DM-28101]
  • The graph file format has been changed from a pickle file to a form that can efficiently be accessed from an object store. This new format has a .qgraph file extension. [DM-27784]
  • A full URI can now be used to specify the location of the quantum graph. [DM-27682]