Release 19.0.0 (2019-12-05)¶
Source | Identifier |
---|---|
Git tag | 19.0.0 |
EUPS distrib | v19_0_0 |
This release is based on the w_2019_46
weekly build.
These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see Tickets Addressed in Release 19.0.0.
If you have questions or comments about this release, visit our community forum for advice.
See also:
- Installation instructions
- Known issues
- Characterization Metric Report (DMTR-191)
- Doxygen Documentation
Major Functionality Changes¶
- Addition of meas_extensions_scarlet
- Upgrades to lsst.afw.image.ExposureInfo
- Performance improvements to DCR model generation
- Mechanism for deprecating configuration fields
- Reference catalog creation now includes parallaxes
- Task for identifying defects in CCD data
- New approach to the WCS of raw exposures
- Tract-based FGCM
- Plugin-based system for calculating DIAObject properties
- Add lsst.afw.math.PixelAreaBoundedField
Addition of meas_extensions_scarlet¶
This releases introduces the new meas_extensions_scarlet package. This package provides integration between the SCARLET deblender system and the LSST codebase.
Upgrades to lsst.afw.image.ExposureInfo
¶
As part of an ongoing project to make ExposureInfo
more flexible, a number of new methods have been made available that make it possible to attach almost arbitrary objects to an ExposureInfo
instance.
This makes it much more convenient to extend ExposureInfo
to address new use cases.
This change is fully backwards compatible with the old ExposureInfo
interface.
For more details and usage instructions, refer to this community.lsst.org post.
Performance improvements to DCR model generation¶
The generation of Differential Chromatic Refraction (DCR) models in lsst.pipe.tasks.DcrAssembleCoaddTask
is now tens of percent faster than in the previous release.
Mechanism for deprecating configuration fields¶
A new system for marking configuration fields (instances of lsst.pex.config.ConfigField
) as deprecated has been added to the codebase.
In the same way as API changes, configuration fields will not be removed until they have been marked as deprecated for at least one major release.
For technical details of the mechanism refer to the Developer Guide.
Reference catalog creation now includes parallaxes¶
The code used for incorporating external reference catalogs into the LSST system now also ingests parallaxes, where available. For more information, refer to the documentation on reference catalog creation.
Task for identifying defects in CCD data¶
lsst.cp.pipe.FindDefectsTask
provides a mechanism to identify defects in CCD images by identifying outlying bright and/or dark pixels in dark frames and flat field images.
New approach to the WCS of raw exposures¶
Previously, when reading raw data to create an Exposure
, a simple world coordinate system (WCS) was provided based on the standard FITS header keys provided with the data (CRVAL
, CRPIX
, CDX_Y
).
In the new system, when raw data is read, an SkyWcs
instance is generated based on the combination of the known telescope boresight and rotation angle, the positions of the detectors on the focal plane, and an optical distortion model.
For more information, refer to RFC-616.
Tract-based FGCM¶
LSST’s implementation of the Forward Global Calibration Method (FGCM; Burke et al., 2018) has been upgraded by the addition of “tract mode”. This enables FGCM to process a single tract of data at a time — refer to lsst.skymap for the definition of a tract — rather than attempting to calibrate the entire survey. FGCM also internally calculates aperture corrections when operating in this mode (or, if there is insufficient data for it to determine the aperture corrections itself, it falls back to externally-provided values).
Plugin-based system for calculating DIAObject properties¶
DIAObjects, corresponding to astronomical objects detected on difference images, are accompanied by a suite of summary values derived from their constituent DIASources — average positions, fluxes, variability characteristics, and so on. The suite of quantities calculated is expected to increase over the remainder of construction, in particular in response to input from the scientific community. To support this, a “plugin” system has been implemented which makes it easy to add new quantities without disrupting the existing codebase.
Add lsst.afw.math.PixelAreaBoundedField
¶
The new PixelAreaBoundedField
class provides a convenient mechanism for evaluating the pixel area of an SkyWcs
.
It is typically used to move an image or source flux between surface brightness and fluence.
It is intended to replace lsst.afw.math.PixelScaleBoundedField
, which applies a per-CCD normalization that introduces discontinuities which make it awkward to use.
PixelScaleBoundedField
is now deprecated, and will be removed in the next release.
Refer to RFC-644 for further details.
Significant Interface Changes¶
- “Gen 3”-driven changes to loading reference catalogs
- pipetask interface changes
- Replacement of functions in lsst.ip.isr
- Removal of lsst.afw.image.Calib compatibility API
- Removal of the --silent command-line option from ap_verify
- Removal of deprecated test utility functions
- Removal of the pytest EUPS package
- Removal of meas_extensions_astrometryNet
- Removal of meas_mosaic
- Removal of lsst::daf::base::Citizen
“Gen 3”-driven changes to loading reference catalogs¶
The “Generation 3” middleware included in the 17.0.0 release is undergoing rapid development, and some API instability is to be expected: these notes will not attempt to produce a comprehensive guide to all of the changes.
However, it is worth nothing that, as of DM-17023, tasks which derive from both CmdLineTask
and PipelineTask
now have separate configuration options for specifying reference catalog configuration depending on whether they are being executed by “Gen 2” or “Gen 3” middleware.
Further, these options must be manually kept synchronized.
For more details and instructions, refer to this community.lsst.org post.
pipetask
interface changes¶
The pipetask
command is used to invoke processing based on the PipelineTask
middleware.
Its command-line interface has been overhauled in this release, adding a number of new capabilities and changing the ordering of sub-commands and some options.
These changes are described in this community.lsst.org post.
Replacement of functions in lsst.ip.isr
¶
As announced in the 18.0.0 release notes, a number of functions formerly provided in lsst.ip.isr
have been removed.
They are replaced by functionality in lsst.meas.algorithms.Defects
.
Specifically:
defectListFromFootprintList
is replaced bylsst.meas.algorithms.Defects.fromFootPrintList
transposeDefectList
is replaced bylsst.meas.algorithms.Dfects.transpose
maskPixelsFromDefectList
is replaced bylsst.meas.algorithms.Dfects.maskPixels
getDefectListFromMask
is replaced bylsst.meas.algorithms.Dfects.fromMask
Removal of lsst.afw.image.Calib
compatibility API¶
As announced in the 18.0.0 release notes
, the compatibility layer for the now-removed Calib
API has been removed.
Removal of the --silent
command-line option from ap_verify¶
The --silent
option to ap_verify did nothing, and was maintained only for backwards compatibility reasons.
As announced in the 18.0.0 release notes
, it has now been removed.
Removal of deprecated test utility functions¶
The following functions have been removed from the lsst.utils
package:
lsst.utils.tests.run
lsst.utils.tests.assertRaisesLsstCpp
lsst.utils.tests.assertClose
lsst.utils.tests.assertNotClose
The class lsst.utils.tests.TestDeprecations
has also been removed.
All of this code had been deprecated several releases in the past.
Removal of the pytest EUPS package¶
The pytest unit testing framework is no longer installed by EUPS as part of the Science Pipelines software distribution. It is still required, and must be provided through the environment. The standard Science Pipelines Conda environment (as installed by e.g. newinstall.sh with the default options) provides a suitable version of pytest.
Removal of meas_extensions_astrometryNet¶
The meas_extensions_astrometryNet package was unused in the Science Pipelines codebase, and was included only through a vestigial dependency. This has now been dropped, and hence meas_extensions_astrometryNet does not appear in this release.
Pending Deprecations¶
These packages/functions will be deprecated in the next major release.
- Upcoming removal of “Generation 2” Middleware
- Upcoming removal of lsst.afw.geom.wcsUtils.makeDistortedTanWcs
Upcoming removal of “Generation 2” Middleware¶
The “Generation 3” middleware included in the 17.0.0 release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality early in calendar year 2020. Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:
- The daf_persistence package, to be replaced by daf_butler;
lsst.pipe.base.CmdLineTask
, to be replaced bylsst.pipe.base.PipelineTask
;- The pipe_drivers and ctrl_pool packages, for which replacements are still in development.
Upcoming removal of lsst.afw.geom.wcsUtils.makeDistortedTanWcs
¶
Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering makeDistortedTanWcs
obsolete.
The function makeDistortedTanWcs
will be removed after release 20.
Deprecations¶
These packages/functions are deprecated and will not be available in the next major release.
- Removal of the obs_lsstSim package
- Removal of lsst.afw.geom classes that have been relocated to lsst.geom
- Removal of lsst.afw.math.PixelScaleBoundedField
- Removal of lsst.ip.isr.addDistortionModel and related configuration classes
- Removal of lsst.obs.base.CameraMapper._extractAmpId
- Removal of pex_policy
- Removal of astrometry_net_data
- Removal of lsst.afw.cameraGeom.Amplifier.getHasRawInfo
- Removal of lsst.fgcmcal.FgcmFitCycleConfg.pixelScale
- Removal of obs_sdss
Removal of the obs_lsstSim package¶
The obs_lsst package, included in the release 17, obviates the need for the obs_lsstSim package. This package will be removed before the release of version 20.0.0.
Removal of lsst.afw.geom
classes that have been relocated to lsst.geom
¶
As announced in v16.0 (Reworked “geom” package, replacing much of “afw.geom”) some primitives have been moved from afw.geom
to geom
.
We currently provide aliases for compatibility purposes, but new all code should use the geom
package directly.
The aliases will be removed before the release of version 20.0.0.
Removal of lsst.afw.math.PixelScaleBoundedField
¶
PixelAreaBoundedField
, added in this release, is intended to replace PixelScaleBoundedField
.
The latter will be removed before the release of version 20.0.0.
Removal of lsst.obs.base.CameraMapper._extractAmpId
¶
This method is no longer used and will be removed before the release of version 20.0.0.
Removal of pex_policy
¶
The pex_policy package is obsolete, and will be removed in version 20.0.0.
Calls to lsst.pex.policy.makePolicy
should be modernized to become calls to lsst.pex.config.makePropertySet
.
See also RFC-626.
Removal of astrometry_net_data¶
The astrometry_net_data package is obsolete, and will be removed in version 20.0.0.
Removal of lsst.afw.cameraGeom.Amplifier.getHasRawInfo
¶
getHasRawInfo
is redundant, since amplifier objects always have raw information.
It will be removed before the release of version 20.0.0
Removal of lsst.fgcmcal.FgcmFitCycleConfg.pixelScale
¶
This configuration field is no longer used, and therefore will be removed before the release of version 20.0.0.