Release 22.0.0 (2021-07-09)

Source

Identifier

Git tag

22.0.0

EUPS distrib

v22_0_0

This release is based on the w_2021_14 weekly build. The bug fix on [DM-29907] was backported.

The notes below highlight significant technical changes to the Science Pipelines codebase in this release. For a complete list of changes made, see Tickets Addressed in Release 22.0.0.

The Characterization Metric Report (DMTR-311) describes the scientific performance of this release in terms of scientific performance metrics.

If you have questions or comments about this release, visit our community forum for advice.

See also:

Major New Features

Generation 3 Middleware

Major improvements have been made to the Generation 3 middleware. Some highlights include:

  • Usability improvements to PipelineTask including URI support.

  • Additional butler command-line commands, including

    • butler prune-datasets,

    • butler query-dimension-records,

    • and butler associate.

  • Allow lsst.daf.butler.Butler.get to support dimension record values such as exposure observing day or detector name in the dataID.

  • pipetask run can now execute a subset of a graph. This allows a single graph file to be created with an entire workflow and then only part of it to be executed, necessary for large-scale workflow execution.

At the time of this release, Gen 2 middleware is now deprecated.

Initial integration of Piff

Piff has been added to rubin-env as a third-party package. A new package, meas_extension_piff, (included in lsst_distrib) integrates Piff with the pipelines. Tasks can be configured to use Piff PSF models instead of the current default PSFex. [RFC-755]

Coadd measurement now uses the Scarlet deblender by default

Scarlet is now the default deblender used for measurements on coadds. The configuration for single-frame measurement has not changed. Scarlet produces additional flags for filtering duplicates, as it handles isolated sources differently: deblend_parentNPeaks, deblend_parentNChild. The flag detect_isPrimary is still populated and is the recommended column. See the community.lsst.org post for more information. [RFC-745]

Dust Maps

The dustmaps_cachedata package has been added to the pipelines, and the dustmaps module is a third-party dependency. [RFC-752]

Faro package for scientific performance metrics

A package for measuring scientific performance metrics, faro, was added. It implements the Gen3 versions of the tasks in validate_drp. See Removal of validate_drp. This package was used to generate the Characterization Report for this release. [RFC-753, DM-28351]

Significant Interface Changes

Replace afw.image.Filter with FilterLabel

Filter information is now stored as a FilterLabel, replacing the existing lsst.afw.image.Filter. This class replaces Filter’s system of names, canonical names, and aliases with just two names: a band (e.g., “g” or “r”) and a physical filter (e.g., “g DECam SDSS c0001 4720.0 1520.0” or “HSC-R2”). Note that not all FilterLabel objects have both a band and a physical filter, especially during the transition period, so please program defensively. See community.lsst.org for details. [RFC-730]

Removal of the obs_ctio0m9 package

The obs_ctio0m9 camera package has been removed. [RFC-729, DM-26867, DM-26868]

Removal of metric configurations in commonMetric.py

The metric field has been replaced by connections.package and connections.metric.

Pending Deprecations

These Tasks will be deprecated in the next major release.

Deprecated DecamRawIngestTask and MegaPrimeRawIngestTask

MegaPrime and DECam no longer require a specialist Gen3 ingest task. Please use the default RawIngestTask. Both DecamRawIngestTask and MegaPrimeRawIngestTask will be removed after v23.

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Deprecated Generation 2 Middleware

Generation 2 middleware (Gen2) is no longer being developed and should not be used for new code. Gen2 infrastructure code (within e.g., pipe_tasks, pipe_base, obs_base, daf_persistence, and obs packages) will no longer be maintained after January 1 2022, and may be removed at any point afterward. The CI package ci_hsc_gen2 will continue to be run, and the tasks it checks will be maintained until we remove the Gen2 infrastructure code in 2022. Following this release, we will begin to drop gen2 pipelines from our verification packages (e.g., ap_verify), where a functional and validated gen3 pipeline exists.

Removal of lsst.afw.Filter

The lsst.afw.Filter class has been replaced with filterLabel, marked as deprecated, and will be removed after this release. Method names that contain Filter, such as getFilter, have been replaced e.g., with .getFilterLabel. [RFC-730]

Removal of validate_drp

The algorithms implemented in validate_drp were ported as-is to run in faro. All future development of scientific performance metrics will be carried out in faro. validate_drp will be removed after this release.

Deprecated doApplyUberCal in forcedPhotCcd.py

The field doApplyUberCal is deprecated. Use doApplyExternalPhotoCalib and doApplyExternalSkyWcs instead. It will be removed before the 23.0.0 release. [DM-23352]

Deprecated configurations in isrTask.py

The following configurations will be removed before the 23.0.0 release:

  • overscanFitType

  • overscanOrder

  • overscanNumSigmaClip

  • overscanIsInt

Please configure overscan via the OverscanCorrectionConfig interface. [DM-23396]

Deprecated configurations in fgcmFitCycle.py

The following configurations are no longer used and will be removed before the 23.0.0 release:

  • fitFlag

  • requiredFlag

  • superStarSubCcd

  • ccdGraySubCcd

  • expGrayPhotometricCut

  • expGrayHighCut

  • expVarGrayPhotometricCut

  • aperCorrInputSlopes

  • sedFudgeFactors, use sedSlopeMap instead

  • sigFgcmMaxEGray, use sigFgcmMaxEGrayDict instead

  • approxThroughput, use approxThroughputDict instead

  • colorSplitIndices, use colorSplitBands instead

  • useRepeatabilityForExpGrayCuts, use useRepeatabilityForExpGrayCutsDict instead

[DM-23699]

Deprecated configurations in psfexStarSelector.py

The following configurations are no longer used and will be removed before the 23.0.0 release:

  • maxbad

  • maxbadflag