Release 19.0.0 (2019-12-05)¶
This release is based on the
w_2019_46 weekly build.
These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see Tickets Addressed in Release 19.0.0.
If you have questions or comments about this release, visit our community forum for advice.
Major Functionality Changes¶
Addition of meas_extensions_scarlet¶
This releases introduces the new meas_extensions_scarlet package. This package provides integration between the SCARLET deblender system and the LSST codebase.
As part of an ongoing project to make
ExposureInfo more flexible, a number of new methods have been made available that make it possible to attach almost arbitrary objects to an
This makes it much more convenient to extend
ExposureInfo to address new use cases.
This change is fully backwards compatible with the old
For more details and usage instructions, refer to this community.lsst.org post.
Performance improvements to DCR model generation¶
The generation of Differential Chromatic Refraction (DCR) models in
lsst.pipe.tasks.DcrAssembleCoaddTask is now tens of percent faster than in the previous release.
Mechanism for deprecating configuration fields¶
A new system for marking configuration fields (instances of
lsst.pex.config.ConfigField) as deprecated has been added to the codebase.
In the same way as API changes, configuration fields will not be removed until they have been marked as deprecated for at least one major release.
For technical details of the mechanism refer to the Developer Guide.
Reference catalog creation now includes parallaxes¶
The code used for incorporating external reference catalogs into the LSST system now also ingests parallaxes, where available. For more information, refer to the documentation on reference catalog creation.
Task for identifying defects in CCD data¶
lsst.cp.pipe.FindDefectsTask provides a mechanism to identify defects in CCD images by identifying outlying bright and/or dark pixels in dark frames and flat field images.
New approach to the WCS of raw exposures¶
Previously, when reading raw data to create an
Exposure, a simple world coordinate system (WCS) was provided based on the standard FITS header keys provided with the data (
In the new system, when raw data is read, an
SkyWcs instance is generated based on the combination of the known telescope boresight and rotation angle, the positions of the detectors on the focal plane, and an optical distortion model.
For more information, refer to RFC-616.
LSST’s implementation of the Forward Global Calibration Method (FGCM; Burke et al., 2018) has been upgraded by the addition of “tract mode”. This enables FGCM to process a single tract of data at a time — refer to lsst.skymap for the definition of a tract — rather than attempting to calibrate the entire survey. FGCM also internally calculates aperture corrections when operating in this mode (or, if there is insufficient data for it to determine the aperture corrections itself, it falls back to externally-provided values).
Plugin-based system for calculating DIAObject properties¶
DIAObjects, corresponding to astronomical objects detected on difference images, are accompanied by a suite of summary values derived from their constituent DIASources — average positions, fluxes, variability characteristics, and so on. The suite of quantities calculated is expected to increase over the remainder of construction, in particular in response to input from the scientific community. To support this, a “plugin” system has been implemented which makes it easy to add new quantities without disrupting the existing codebase.
PixelAreaBoundedField class provides a convenient mechanism for evaluating the pixel area of an
It is typically used to move an image or source flux between surface brightness and fluence.
It is intended to replace
lsst.afw.math.PixelScaleBoundedField, which applies a per-CCD normalization that introduces discontinuities which make it awkward to use.
PixelScaleBoundedField is now deprecated, and will be removed in the next release.
Refer to RFC-644 for further details.
Significant Interface Changes¶
“Gen 3”-driven changes to loading reference catalogs¶
The “Generation 3” middleware included in the 17.0.0 release is undergoing rapid development, and some API instability is to be expected: these notes will not attempt to produce a comprehensive guide to all of the changes.
However, it is worth nothing that, as of DM-17023, tasks which derive from both
PipelineTask now have separate configuration options for specifying reference catalog configuration depending on whether they are being executed by “Gen 2” or “Gen 3” middleware.
Further, these options must be manually kept synchronized.
For more details and instructions, refer to this community.lsst.org post.
pipetask interface changes¶
pipetask command is used to invoke processing based on the
Its command-line interface has been overhauled in this release, adding a number of new capabilities and changing the ordering of sub-commands and some options.
These changes are described in this community.lsst.org post.
Replacement of functions in
As announced in the 18.0.0 release notes, a number of functions formerly provided in
lsst.ip.isr have been removed.
They are replaced by functionality in
defectListFromFootprintListis replaced by
transposeDefectListis replaced by
maskPixelsFromDefectListis replaced by
getDefectListFromMaskis replaced by
lsst.afw.image.Calib compatibility API¶
announced in the 18.0.0 release notes, the compatibility layer for the now-removed
Calib API has been removed.
Removal of the
--silent command-line option from ap_verify¶
--silent option to ap_verify did nothing, and was maintained only for backwards compatibility reasons.
announced in the 18.0.0 release notes, it has now been removed.
Removal of deprecated test utility functions¶
The following functions have been removed from the
lsst.utils.tests.TestDeprecations has also been removed.
All of this code had been deprecated several releases in the past.
Removal of the pytest EUPS package¶
The pytest unit testing framework is no longer installed by EUPS as part of the Science Pipelines software distribution. It is still required, and must be provided through the environment. The standard Science Pipelines Conda environment (as installed by e.g. newinstall.sh with the default options) provides a suitable version of pytest.
Removal of meas_extensions_astrometryNet¶
The meas_extensions_astrometryNet package was unused in the Science Pipelines codebase, and was included only through a vestigial dependency. This has now been dropped, and hence meas_extensions_astrometryNet does not appear in this release.
Removal of meas_mosaic¶
Jointcal now provides all the features necessary to replace meas_mosaic, and has been successfully tested in at-scale data processing. The meas_mosaic package has therefore been removed from the Science Pipelines codebase, and does not appear in this release.
Citizen, an obsolete system for debugging memory leaks in C++ code, has been removed per RFC-280.
These packages/functions will be deprecated in the next major release.
Upcoming removal of “Generation 2” Middleware¶
The “Generation 3” middleware included in the 17.0.0 release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality early in calendar year 2020. Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:
The daf_persistence package, to be replaced by daf_butler;
lsst.pipe.base.CmdLineTask, to be replaced by
The pipe_drivers and ctrl_pool packages, for which replacements are still in development.
Upcoming removal of
Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering
makeDistortedTanWcs will be removed after release 20.
These packages/functions are deprecated and will not be available in the next major release.
Removal of the obs_lsstSim package¶
The obs_lsst package, included in the release 17, obviates the need for the obs_lsstSim package. This package will be removed before the release of version 20.0.0.
lsst.afw.geom classes that have been relocated to
As announced in v16.0 (Reworked “geom” package, replacing much of “afw.geom”) some primitives have been moved from
We currently provide aliases for compatibility purposes, but new all code should use the
geom package directly.
The aliases will be removed before the release of version 20.0.0.
This method is no longer used and will be removed before the release of version 20.0.0.
The pex_policy package is obsolete, and will be removed in version 20.0.0.
lsst.pex.policy.makePolicy should be modernized to become calls to
See also RFC-626.
Removal of astrometry_net_data¶
The astrometry_net_data package is obsolete, and will be removed in version 20.0.0.
getHasRawInfo is redundant, since amplifier objects always have raw information.
It will be removed before the release of version 20.0.0
This configuration field is no longer used, and therefore will be removed before the release of version 20.0.0.
Removal of obs_sdss¶
Following the approval of RFC-645, the obs_sdss package will be removed from the Science Pipelines before the release of version 20.0.0.