Release 19.0.0 (2019-12-05)

Source Identifier
Git tag 19.0.0
EUPS distrib v19_0_0

This release is based on the w_2019_46 weekly build.

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see Tickets Addressed in Release 19.0.0.

If you have questions or comments about this release, visit our community forum for advice.

See also:

Major Functionality Changes

Addition of meas_extensions_scarlet

This releases introduces the new meas_extensions_scarlet package. This package provides integration between the SCARLET deblender system and the LSST codebase.

Upgrades to lsst.afw.image.ExposureInfo

As part of an ongoing project to make ExposureInfo more flexible, a number of new methods have been made available that make it possible to attach almost arbitrary objects to an ExposureInfo instance. This makes it much more convenient to extend ExposureInfo to address new use cases. This change is fully backwards compatible with the old ExposureInfo interface. For more details and usage instructions, refer to this community.lsst.org post.

Performance improvements to DCR model generation

The generation of Differential Chromatic Refraction (DCR) models in lsst.pipe.tasks.DcrAssembleCoaddTask is now tens of percent faster than in the previous release.

Mechanism for deprecating configuration fields

A new system for marking configuration fields (instances of lsst.pex.config.ConfigField) as deprecated has been added to the codebase. In the same way as API changes, configuration fields will not be removed until they have been marked as deprecated for at least one major release. For technical details of the mechanism refer to the Developer Guide.

Reference catalog creation now includes parallaxes

The code used for incorporating external reference catalogs into the LSST system now also ingests parallaxes, where available. For more information, refer to the documentation on reference catalog creation.

Task for identifying defects in CCD data

lsst.cp.pipe.FindDefectsTask provides a mechanism to identify defects in CCD images by identifying outlying bright and/or dark pixels in dark frames and flat field images.

New approach to the WCS of raw exposures

Previously, when reading raw data to create an Exposure, a simple world coordinate system (WCS) was provided based on the standard FITS header keys provided with the data (CRVAL, CRPIX, CDX_Y).

In the new system, when raw data is read, an SkyWcs instance is generated based on the combination of the known telescope boresight and rotation angle, the positions of the detectors on the focal plane, and an optical distortion model.

For more information, refer to RFC-616.

Tract-based FGCM

LSST’s implementation of the Forward Global Calibration Method (FGCM; Burke et al., 2018) has been upgraded by the addition of “tract mode”. This enables FGCM to process a single tract of data at a time — refer to lsst.skymap for the definition of a tract — rather than attempting to calibrate the entire survey. FGCM also internally calculates aperture corrections when operating in this mode (or, if there is insufficient data for it to determine the aperture corrections itself, it falls back to externally-provided values).

Plugin-based system for calculating DIAObject properties

DIAObjects, corresponding to astronomical objects detected on difference images, are accompanied by a suite of summary values derived from their constituent DIASources — average positions, fluxes, variability characteristics, and so on. The suite of quantities calculated is expected to increase over the remainder of construction, in particular in response to input from the scientific community. To support this, a “plugin” system has been implemented which makes it easy to add new quantities without disrupting the existing codebase.

Add lsst.afw.math.PixelAreaBoundedField

The new PixelAreaBoundedField class provides a convenient mechanism for evaluating the pixel area of an SkyWcs. It is typically used to move an image or source flux between surface brightness and fluence. It is intended to replace lsst.afw.math.PixelScaleBoundedField, which applies a per-CCD normalization that introduces discontinuities which make it awkward to use. PixelScaleBoundedField is now deprecated, and will be removed in the next release. Refer to RFC-644 for further details.

Significant Interface Changes

“Gen 3”-driven changes to loading reference catalogs

The “Generation 3” middleware included in the 17.0.0 release is undergoing rapid development, and some API instability is to be expected: these notes will not attempt to produce a comprehensive guide to all of the changes. However, it is worth nothing that, as of DM-17023, tasks which derive from both CmdLineTask and PipelineTask now have separate configuration options for specifying reference catalog configuration depending on whether they are being executed by “Gen 2” or “Gen 3” middleware. Further, these options must be manually kept synchronized. For more details and instructions, refer to this community.lsst.org post.

pipetask interface changes

The pipetask command is used to invoke processing based on the PipelineTask middleware. Its command-line interface has been overhauled in this release, adding a number of new capabilities and changing the ordering of sub-commands and some options. These changes are described in this community.lsst.org post.

Replacement of functions in lsst.ip.isr

As announced in the 18.0.0 release notes, a number of functions formerly provided in lsst.ip.isr have been removed. They are replaced by functionality in lsst.meas.algorithms.Defects. Specifically:

  • defectListFromFootprintList is replaced by lsst.meas.algorithms.Defects.fromFootPrintList
  • transposeDefectList is replaced by lsst.meas.algorithms.Dfects.transpose
  • maskPixelsFromDefectList is replaced by lsst.meas.algorithms.Dfects.maskPixels
  • getDefectListFromMask is replaced by lsst.meas.algorithms.Dfects.fromMask

Removal of lsst.afw.image.Calib compatibility API

As announced in the 18.0.0 release notes, the compatibility layer for the now-removed Calib API has been removed.

Removal of the --silent command-line option from ap_verify

The --silent option to ap_verify did nothing, and was maintained only for backwards compatibility reasons. As announced in the 18.0.0 release notes, it has now been removed.

Removal of deprecated test utility functions

The following functions have been removed from the lsst.utils package:

  • lsst.utils.tests.run
  • lsst.utils.tests.assertRaisesLsstCpp
  • lsst.utils.tests.assertClose
  • lsst.utils.tests.assertNotClose

The class lsst.utils.tests.TestDeprecations has also been removed. All of this code had been deprecated several releases in the past.

Removal of the pytest EUPS package

The pytest unit testing framework is no longer installed by EUPS as part of the Science Pipelines software distribution. It is still required, and must be provided through the environment. The standard Science Pipelines Conda environment (as installed by e.g. newinstall.sh with the default options) provides a suitable version of pytest.

Removal of meas_extensions_astrometryNet

The meas_extensions_astrometryNet package was unused in the Science Pipelines codebase, and was included only through a vestigial dependency. This has now been dropped, and hence meas_extensions_astrometryNet does not appear in this release.

Removal of meas_mosaic

Jointcal now provides all the features necessary to replace meas_mosaic, and has been successfully tested in at-scale data processing. The meas_mosaic package has therefore been removed from the Science Pipelines codebase, and does not appear in this release.

Removal of lsst::daf::base::Citizen

Citizen, an obsolete system for debugging memory leaks in C++ code, has been removed per RFC-280.

Pending Deprecations

These packages/functions will be deprecated in the next major release.

Upcoming removal of “Generation 2” Middleware

The “Generation 3” middleware included in the 17.0.0 release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality early in calendar year 2020. Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:

  • The daf_persistence package, to be replaced by daf_butler;
  • lsst.pipe.base.CmdLineTask, to be replaced by lsst.pipe.base.PipelineTask;
  • The pipe_drivers and ctrl_pool packages, for which replacements are still in development.

Upcoming removal of lsst.afw.geom.wcsUtils.makeDistortedTanWcs

Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering makeDistortedTanWcs obsolete. The function makeDistortedTanWcs will be removed after release 20.

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Removal of the obs_lsstSim package

The obs_lsst package, included in the release 17, obviates the need for the obs_lsstSim package. This package will be removed before the release of version 20.0.0.

Removal of lsst.afw.geom classes that have been relocated to lsst.geom

As announced in v16.0 (Reworked “geom” package, replacing much of “afw.geom”) some primitives have been moved from afw.geom to geom. We currently provide aliases for compatibility purposes, but new all code should use the geom package directly. The aliases will be removed before the release of version 20.0.0.

Removal of lsst.afw.math.PixelScaleBoundedField

PixelAreaBoundedField, added in this release, is intended to replace PixelScaleBoundedField. The latter will be removed before the release of version 20.0.0.

Removal of lsst.obs.base.CameraMapper._extractAmpId

This method is no longer used and will be removed before the release of version 20.0.0.

Removal of pex_policy

The pex_policy package is obsolete, and will be removed in version 20.0.0. Calls to lsst.pex.policy.makePolicy should be modernized to become calls to lsst.pex.config.makePropertySet. See also RFC-626.

Removal of astrometry_net_data

The astrometry_net_data package is obsolete, and will be removed in version 20.0.0.

Removal of lsst.afw.cameraGeom.Amplifier.getHasRawInfo

getHasRawInfo is redundant, since amplifier objects always have raw information. It will be removed before the release of version 20.0.0

Removal of lsst.fgcmcal.FgcmFitCycleConfg.pixelScale

This configuration field is no longer used, and therefore will be removed before the release of version 20.0.0.

Removal of obs_sdss

Following the approval of RFC-645, the obs_sdss package will be removed from the Science Pipelines before the release of version 20.0.0.