lsst-pipe-base v24.1.0 (2023-02-23)¶

New Features¶

QuantumGraph now saves the DimensionUniverse it was created with when it is persisted. This removes the need to explicitly pass the DimensionUniverse when loading a saved graph. (DM-35082)
Allow PipelineTasks to provide defaults for the --dataset-query-constraints option for the pipetask tool. (DM-37786)

Ensure QuantumGraphs are given a DimensionUniverse at construction.

This fixes a mostly-spurious dimension universe inconsistency warning when reading QuantumGraphs, introduced on DM-35082. (DM-35681)
Fix an error message that says that repository state has changed during QuantumGraph generation when init input datasets are just missing. (DM-37786)

Add the ability for user control over dataset constraints in QuantumGraph creation. (DM-31769)
Builds using setuptools now calculate versions from the Git repository, including the use of alpha releases for those associated with weekly tags. (DM-32408)
Improve diagnostics for empty QuantumGraph. (DM-32459)
A new class has been written for handling Task metadata. lsst.pipe.base.TaskMetadata will in future become the default metadata class for Task, replacing lsst.daf.base.PropertySet. The new metadata class is not yet enabled by default. (DM-32682)
- Add TaskMetadata.to_dict() method (this is now used by the lsst.daf.base.PropertySet.from_mapping() method and triggered by the Butler if type conversion is needed).
- Use the existing metadata storage class definition if one already exists in a repository.
- Switch Task to use TaskMetadata for storing task metadata, rather than lsst.daf.base.PropertySet. This removes a C++ dependency from the middleware. (DM-33155)
- Added lsst.pipe.base.Instrument to represent an instrument in Butler registry.
- Added butler register-instrument command (relocated from obs_base).
- Added a formatter for pex_config Config objects. (DM-34105)

Fixed a bug where imported pipeline parameters were taking preference over “top-level” preferences (DM-32080)

If a PipelineTask has connections that have a different storage class for a dataset type than the one defined in registry, this will now be allowed if the storage classes are compatible. The Task run() method will be given the Python type it expects and can return the Python type it has declared it returns. The Butler will do the type conversion automatically. (DM-33303)
Topological sorting of pipelines on write has been disabled; the order in which the pipeline tasks were read/added is preserved instead. This makes it unnecessary to import all tasks referenced by the pipeline in order to write it. (DM-34155)

Execution butler creation time has been reduced significantly by avoiding unnecessary checks for existence of files in the datastore. (DM-33345)

Added a new facility for creating “lightweight” (execution) butlers that pre-fills a local SQLite registry. This can allow a pipeline to be executed without talking to the main registry. (DM-28646)
Allow PipelineTasks inputs and outputs to be optional under certain conditions, so tasks with no work to do can be skipped without blocking downstream tasks from running. (DM-30649)
Log diagnostic information when QuantumGraphs are empty because the initial query yielded no results.

At present, these diagnostics only cover missing input datasets, which is a common way to get an empty QuantumGraph, but not the only way. (DM-31583)

GraphBuilder constructor boolean argument skipExisting is replaced with skipExistingIn which accepts collections to check for existing quantum outputs. (DM-27492)

The logger associated with Task is now derived from a Python logging.Logger and not lsst.log.Log. This logger includes a new verbose() log method as an intermediate between INFO and DEBUG. (DM-30301)
Added metadata to QuantumGraphs. This changed the on disk save format, but is backwards compatible with graphs saved with previous versions of the QuantumGraph code. (DM-30702)
All Doxygen documentation has been removed and replaced by Sphinx. (DM-23330)
New documentation on writing pipelines has been added. (DM-27416)

Add ways to test a PipelineTask’s init inputs/outputs [DM-23156]
Pipelines can now support URIs [DM-28036]
Graph files can now be loaded and saved via URIs [DM-27682]
A new format for saving graphs has been developed (with a .qgraph extension). This format supports the ability to read a subset of a graph from an object store. [DM-27784]
Graph building with a pipeline that specifies an instrument no longer needs an explicit instrument to be given. [DM-27985]
A parameters section has been added to pipeline definitions. [DM-27633]