lsst-pipe-base v25.0.0 (2023-02-28)

This is the first release without any support for the Generation 2 middleware.

New Features

  • Added PipelineStepTester class, to enable testing that multi-step pipelines are able to run without error. (DM-33779)

  • QuantumGraph now saves the DimensionUniverse it was created with when it is persisted. This removes the need to explicitly pass the DimensionUniverse when loading a saved graph. (DM-35082)

    • Added support for transferring files into execution butler. (DM-35494)

  • A new class InMemoryDatasetHandle is now available. This class provides a variant of lsst.daf.butler.DeferredDatasetHandle that does not require a butler and lets you store your in-memory objects in something that looks like one and so can be passed to Task.run() methods that expect to be able to do deferred loading. (DM-35741)

    • Add unit test to cover the new getNumberOfQuantaForTask method.

    • Add graph interface, getNumberOfQuantaForTask, to determine number of quanta associated with a given taskDef.

    • Modifications to getQuantaForTask to support showing added additional quanta information in the logger. (DM-36145)

  • Allow PipelineTasks to provide defaults for the --dataset-query-constraints option for the pipetask tool. (DM-37786)

API Changes

  • ButlerQuantumContext.get method can accept None as a reference and returns None as a result object. (DM-35752)

  • GraphBuilder.makeGraph method adds bind parameter for bind values to use with the user expression. (DM-36487)

  • InMemoryDatasetHandle now supports storage class conversion on get(). (DM-4551)

Bug Fixes

  • lsst.pipe.base.testUtils.makeQuantum no longer crashes if given a connection that is set to a dataset component. (DM-35721)

  • Ensure QuantumGraphs are given a DimensionUniverse at construction.

    This fixes a mostly-spurious dimension universe inconsistency warning when reading QuantumGraphs, introduced on DM-35082. (DM-35681)

  • Fixed an error message that says that repository state has changed during QuantumGraph generation when init input datasets are just missing. (DM-37786)

Other Changes and Additions

  • Make diagnostic logging for empty QuantumGraphs harder to ignore.

    Log messages have been upgraded from WARNING to FATAL, and an exception traceback that tends to hide them has been removed. (DM-36360)

An API Removal or Deprecation

  • Removed the Task.getSchemaCatalogs and Task.getAllSchemaCatalogs APIs. These were used by CmdLineTask but are no longer used in the current middleware. (DM-2850)

  • Relocated lsst.pipe.base.cmdLineTask.profile to lsst.utils.timer.profile. This was relocated as part of the Gen2 removal that includes the removal of CmdLineTask. (DM-35697)

    • ArgumentParser, CmdLineTask, and TaskRunner classes have been removed and associated gen2 documentation.

    • The PipelineIR.from_file() method has been removed.

    • The getTaskLogger function has been removed. (DM-35917)

  • Replaced CmdLineTask and ArgumentParser with non-functioning stubs, disabling all Gen2 functionality. A deprecation message is now issued but the classes do nothing. (DM-35675)

lsst-pipe-base v24.0.0 (2022-08-26)

New Features

  • Add the ability for user control over dataset constraints in QuantumGraph creation. (DM-31769)

  • Builds using setuptools now calculate versions from the Git repository, including the use of alpha releases for those associated with weekly tags. (DM-32408)

  • Improve diagnostics for empty QuantumGraph. (DM-32459)

  • A new class has been written for handling Task metadata. lsst.pipe.base.TaskMetadata will in future become the default metadata class for Task, replacing lsst.daf.base.PropertySet. The new metadata class is not yet enabled by default. (DM-32682)

    • Add TaskMetadata.to_dict() method (this is now used by the lsst.daf.base.PropertySet.from_mapping() method and triggered by the Butler if type conversion is needed).

    • Use the existing metadata storage class definition if one already exists in a repository.

    • Switch Task to use TaskMetadata for storing task metadata, rather than lsst.daf.base.PropertySet. This removes a C++ dependency from the middleware. (DM-33155)

    • Added lsst.pipe.base.Instrument to represent an instrument in Butler registry.

    • Added butler register-instrument command (relocated from obs_base).

    • Added a formatter for pex_config Config objects. (DM-34105)

Bug Fixes

  • Fixed a bug where imported pipeline parameters were taking preference over “top-level” preferences (DM-32080)

Other Changes and Additions

  • If a PipelineTask has connections that have a different storage class for a dataset type than the one defined in registry, this will now be allowed if the storage classes are compatible. The Task run() method will be given the Python type it expects and can return the Python type it has declared it returns. The Butler will do the type conversion automatically. (DM-33303)

  • Topological sorting of pipelines on write has been disabled; the order in which the pipeline tasks were read/added is preserved instead. This makes it unnecessary to import all tasks referenced by the pipeline in order to write it. (DM-34155)

lsst-pipe-base v23.0.1 (2022-02-02)

Miscellaneous Changes of Minor Interest

  • Execution butler creation time has been reduced significantly by avoiding unnecessary checks for existence of files in the datastore. (DM-33345)

lsst-pipe-base v23.0.0 (2021-12-10)

New Features

  • Added a new facility for creating “lightweight” (execution) butlers that pre-fills a local SQLite registry. This can allow a pipeline to be executed without talking to the main registry. (DM-28646)

  • Allow PipelineTasks inputs and outputs to be optional under certain conditions, so tasks with no work to do can be skipped without blocking downstream tasks from running. (DM-30649)

  • Log diagnostic information when QuantumGraphs are empty because the initial query yielded no results.

    At present, these diagnostics only cover missing input datasets, which is a common way to get an empty QuantumGraph, but not the only way. (DM-31583)

API Changes

  • GraphBuilder constructor boolean argument skipExisting is replaced with skipExistingIn which accepts collections to check for existing quantum outputs. (DM-27492)

Other Changes and Additions

  • The logger associated with Task is now derived from a Python logging.Logger and not lsst.log.Log. This logger includes a new verbose() log method as an intermediate between INFO and DEBUG. (DM-30301)

  • Added metadata to QuantumGraphs. This changed the on disk save format, but is backwards compatible with graphs saved with previous versions of the QuantumGraph code. (DM-30702)

  • All Doxygen documentation has been removed and replaced by Sphinx. (DM-23330)

  • New documentation on writing pipelines has been added. (DM-27416)

lsst-pipe-base v22.0 (2021-04-01)

New Features

  • Add ways to test a PipelineTask’s init inputs/outputs [DM-23156]

  • Pipelines can now support URIs [DM-28036]

  • Graph files can now be loaded and saved via URIs [DM-27682]

  • A new format for saving graphs has been developed (with a .qgraph extension). This format supports the ability to read a subset of a graph from an object store. [DM-27784]

  • Graph building with a pipeline that specifies an instrument no longer needs an explicit instrument to be given. [DM-27985]

  • A parameters section has been added to pipeline definitions. [DM-27633]