Release 21.0.0 (2020-12-08)

Source Identifier
Git tag 21.0.0
EUPS distrib v21_0_0

This release is based on the w_2020_46 weekly build.

The notes below highlight significant technical changes to the Science Pipelines codebase in this release. For a complete list of changes made, see Tickets Addressed in Release 21.0.0.

The Characterization Metric Report (DMTR-281) describes the scientific performance of this release in terms of scientific performance metrics, while Changes to Data Products in Release 21.0.0 covers key issues likely to impact scientists consuming Rubin data products.

If you have questions or comments about this release, visit our community forum for advice.

See also:

Major New Features

Beta release of Generation 3 Middleware

Reaching schema stability in this release, the Gen3 butler can now support Science Pipelines development. See Removal of “Generation 2” Middleware for more information on the planned deprecation of Generation 2 middleware. Interested parties may find more info at daf_butler and pipe_base. A general release of Generation 3 Middleware is expected with the next release v22.0.0.

Task to mask streak-like artifacts

This release adds the MaskStreaksTask to pipe_tasks, which finds and masks streaks and other straight lines in image data. It uses the Kernel Hough Transform (KHT) (Fernandes and Oliveira 2007) implemented in the kht, package which was also added to the Science Pipelines. It can be turned on during coaddition with CompareWarpAssembleCoaddTask with the config parameter doFilterMorphological.

Inject arbitrary input images

The task to inject simulated sources into data release processing, InsertFakesTask, can now accept arbitrary input images. [DM-25747]

Proper motions and parallaxes applied to reference sources

jointcal now can apply proper motions and parallaxes to reference sources during astrometric fitting. [DM-24277]

Addition of ci_cpp

The ci_cpp metapackage for the ci_cpp_gen2 and ci_cpp_gen3 packages provides continuous integration tests for calibration production code. This supplements unit tests in cp_pipe as it can do more computationally expensive tests.

For more information, refer to RFC-718.

Significant Interface Changes

Rename the cat package to sdm_schemas

The package cat was renamed to sdm_schemas, which contains database schemas for output tables.

For more information, refer to RFC-672.

Addition of the ctrl_bps package

The ctrl_bps package provides a PipelineTask execution framework for multi-node processing for the LSST Batch Production Service (BPS).

For more information, refer to RFC-736.

Addition of the pipelines_check package

The new pipelines_check package replaces lsst_dm_stack_demo, which relied on the now release-v21-0-0-remove-obs_sdss. This new package demonstrates the processing of a single CCD with the Science Pipelines. It contains input files and astrometric reference catalogs needed to process the data from a single HSC detector. Its main purpose is to check that the pipeline infrastructure is working correctly.

Removal of the obs_sdss package

The obs_sdss camera package has been removed from the Science Pipelines before the release of version 21.0.0. Note that this package was previously slated for removal before the release of version 20.0.0, but this has been delayed. The lsst_dm_stack_demo package, which makes use of obs_sdss was replaced with a package providing similar functionality, Addition of the pipelines_check package. [RFC-645, DM-22578].

Removal of the Synpipe package

The Synpipe package is no longer included as part of the Science Pipelines. Tasks for inserting simulated sources are now available as part of pipe_tasks; see community.lsst.org for details. [RFC-661, DM-23344]

Removal of the pex_policy package

The obsolete pex_policy package has been removed from the Science Pipelines before the release of version 21.0.0. Note that this package was previously slated for removal before the release of version 20.0.0, but this has been delayed. Calls to lsst.pex.policy.makePolicy should be modernized to become calls to lsst.pex.config.makePropertySet. [RFC-626, DM-23374]

Modernization of the C++ API in lsst.afw

As announced in v20.0, a number of deprecated C++ APIs in lsst.afw have been removed. Third-party calls to most of these APIs will simply fail to compile. However, calls to afw::math::convolve that previously passed individual bool values for their final argument(s) instead of using ConvolutionControl may inadvertently call the version with ConvolutionControl, leading to subtle bugs or segmentation faults. [RFC-687, DM-17566, DM-22814, DM-23076, DM-24565]

Pending Deprecations

These packages/functions will be deprecated in the next major release.

Removal of “Generation 2” Middleware

The “Generation 3” middleware first included in the 17.0.0 release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. When the new middleware is stable and has reached feature parity with the old code, deprecation of the Generation 2 system will be announced. At that time, guidance for transitioning to the new system will be made available. This will include moving migrating from lsst.daf.persistence to lsst.daf.butler, and lsst.pipe.base.CmdLineTask will be replaced by lsst.pipe.base.PipelineTask. Progress towards the deprecation of Generation 2 may be tracked on DM-20572.

Removal of MetricTask

The MetricTask is replaced by lsst.verify.tasks.MetricTask. It will be removed along with daf_persistence. [DM-21910]

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Removal of the obs_ctio0m9 package

The obs_ctio0m9 camera package will be removed from the Science Pipelines before the release of version 22.0.0. [RFC-729, DM-26867, DM-26868]

Deprecated configurations in isrTask.py

The following configurations will be removed before the 22.0.0 release:

  • overscanFitType
  • overscanOrder
  • overscanNumSigmaClip
  • overscanIsInt

Please configure overscan via the OverscanCorrectionConfig interface.

Deprecated metric configurations in commonMetric.py

The metric field has been replaced by connections.package and connections.metric. It will be removed along with daf_persistence.

Deprecated doApplyUberCal in forcedPhotCcd.py

The field doApplyUberCal is deprecated. Use doApplyExternalPhotoCalib and code:doApplyExternalSkyWcs instead. It will be removed before the 22.0.0 release. [DM-23352]

Deprecated configurations in fgcmFitCycle.py

The following configurations are no longer used and will be removed before the 22.0.0 release:

  • fitFlag
  • requiredFlag
  • superStarSubCcd
  • ccdGraySubCcd
  • expGrayPhotometricCut
  • expGrayHighCut
  • expVarGrayPhotometricCut
  • aperCorrInputSlopes
  • sedFudgeFactors, use sedSlopeMap instead
  • sigFgcmMaxEGray, use sigFgcmMaxEGrayDict instead
  • approxThroughput, use approxThroughputDict instead
  • colorSplitIndices, use colorSplitBands instead
  • useRepeatabilityForExpGrayCuts, use useRepeatabilityForExpGrayCutsDict instead

[DM-23699]

Deprecated configurations in psfexStarSelector.py

The following configurations are no longer used and will be removed before the 22.0.0 release:

  • maxbad
  • maxbadflag