.. _release-v19-0-0: ########################### Release 19.0.0 (2019-12-05) ########################### .. toctree:: :hidden: tickets/v19_0_0 +-------------------------------------------+------------+ | Source | Identifier | +===========================================+============+ | Git tag | 19.0.0 | +-------------------------------------------+------------+ | :doc:`EUPS distrib ` | v19\_0\_0 | +-------------------------------------------+------------+ This release is based on the ``w_2019_46`` weekly build. These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see :doc:`tickets/v19_0_0`. If you have questions or comments about this release, visit our `community forum `_ for advice. - :ref:`release-v19-0-0-functionality` - :ref:`release-v19-0-0-interface` - :ref:`release-v19-0-0-pending-deprecations` - :ref:`release-v19-0-0-deprecations` *See also:* - :doc:`Installation instructions <../install/index>` - :doc:`Known issues ` - `Characterization Metric Report (DMTR-191) `_ - `Doxygen Documentation`__ __ http://doxygen.lsst.codes/stack/doxygen/xlink_master_2019_11_16_09.13.30/ .. _release-v19-0-0-functionality: Major Functionality Changes =========================== - :ref:`release-v19-0-0-scarlet` - :ref:`release-v19-0-0-exposureinfo` - :ref:`release-v19-0-0-dcr` - :ref:`release-v19-0-0-config-deprecation` - :ref:`release-v19-0-0-parallax` - :ref:`release-v19-0-0-defects` - :ref:`release-v19-0-0-wcs` - :ref:`release-v19-0-0-fgcm-tract` - :ref:`release-v19-0-0-diaobject-property` - :ref:`release-v19-0-0-PixelAreaBoundedField` .. _release-v19-0-0-scarlet: Addition of meas_extensions_scarlet ----------------------------------- This releases introduces the new meas_extensions_scarlet package. This package provides integration between the `SCARLET`_ deblender system and the LSST codebase. .. _SCARLET: https://pmelchior.github.io/scarlet/ .. _release-v19-0-0-exposureinfo: Upgrades to `lsst.afw.image.ExposureInfo` ----------------------------------------- As part of an ongoing project to make `~lsst.afw.image.ExposureInfo` more flexible, a number of new methods have been made available that make it possible to attach almost arbitrary objects to an `~lsst.afw.image.ExposureInfo` instance. This makes it much more convenient to extend `~lsst.afw.image.ExposureInfo` to address new use cases. This change is fully backwards compatible with the old `~lsst.afw.image.ExposureInfo` interface. For more details and usage instructions, refer to `this community.lsst.org post`__. __ https://community.lsst.org/t/changes-to-exposureinfo/3788 .. _release-v19-0-0-dcr: Performance improvements to DCR model generation ------------------------------------------------ The generation of Differential Chromatic Refraction (DCR) models in `lsst.pipe.tasks.DcrAssembleCoaddTask` is now tens of percent faster than in the previous release. .. _release-v19-0-0-config-deprecation: Mechanism for deprecating configuration fields ---------------------------------------------- A new system for marking configuration fields (instances of `lsst.pex.config.ConfigField`) as deprecated has been added to the codebase. In the same way as API changes, configuration fields will not be removed until they have been marked as deprecated for at least one major release. For technical details of the mechanism refer to the `Developer Guide `_. .. _release-v19-0-0-parallax: Reference catalog creation now includes parallaxes -------------------------------------------------- The code used for incorporating external reference catalogs into the LSST system now also ingests parallaxes, where available. For more information, refer to the :ref:`documentation on reference catalog creation `. .. _release-v19-0-0-defects: Task for identifying defects in CCD data ---------------------------------------- `lsst.cp.pipe.FindDefectsTask` provides a mechanism to identify defects in CCD images by identifying outlying bright and/or dark pixels in dark frames and flat field images. .. _release-v19-0-0-wcs: New approach to the WCS of raw exposures ---------------------------------------- Previously, when reading raw data to create an `~lsst.afw.image.Exposure`, a simple world coordinate system (WCS) was provided based on the standard FITS header keys provided with the data (``CRVAL``, ``CRPIX``, ``CDX_Y``). In the new system, when raw data is read, an `~lsst.afw.geom.SkyWcs` instance is generated based on the combination of the known telescope boresight and rotation angle, the positions of the detectors on the focal plane, and an optical distortion model. For more information, refer to :jira:`RFC-616`. .. _release-v19-0-0-fgcm-tract: Tract-based FGCM ---------------- LSST's implementation of the Forward Global Calibration Method (FGCM; `Burke et al., 2018`_) has been upgraded by the addition of “tract mode”. This enables FGCM to process a single tract of data at a time — refer to :ref:`lsst.skymap` for the definition of a tract — rather than attempting to calibrate the entire survey. FGCM also internally calculates aperture corrections when operating in this mode (or, if there is insufficient data for it to determine the aperture corrections itself, it falls back to externally-provided values). .. _Burke et al., 2018: http://adsabs.harvard.edu/abs/2018AJ....155...41B .. _release-v19-0-0-diaobject-property: Plugin-based system for calculating DIAObject properties -------------------------------------------------------- DIAObjects, corresponding to astronomical objects detected on difference images, are accompanied by a suite of summary values derived from their constituent DIASources — average positions, fluxes, variability characteristics, and so on. The suite of quantities calculated is expected to increase over the remainder of construction, in particular in response to input from the scientific community. To support this, a “plugin” system has been implemented which makes it easy to add new quantities without disrupting the existing codebase. .. _release-v19-0-0-PixelAreaBoundedField: Add `lsst.afw.math.PixelAreaBoundedField` ----------------------------------------- The new `~lsst.afw.math.PixelAreaBoundedField` class provides a convenient mechanism for evaluating the pixel area of an `~lsst.afw.geom.SkyWcs`. It is typically used to move an image or source flux between surface brightness and fluence. It is intended to replace `lsst.afw.math.PixelScaleBoundedField`, which applies a per-CCD normalization that introduces discontinuities which make it awkward to use. `~lsst.afw.math.PixelScaleBoundedField` is now :ref:`deprecated `, and will be removed in the next release. Refer to :jira:`RFC-644` for further details. .. _release-v19-0-0-interface: Significant Interface Changes ============================= - :ref:`release-v19-0-0-gen3-api` - :ref:`release-v19-0-0-pipetask` - :ref:`release-v19-0-0-isr` - :ref:`release-v19-0-0-calib-compat` - :ref:`release-v19-0-0-ap-silent` - :ref:`release-v19-0-0-test-util` - :ref:`release-v19-0-0-pytest` - :ref:`release-v19-0-0-adn` - :ref:`release-v19-0-0-mosaic` - :ref:`release-v19-0-0-citizen` .. _release-v19-0-0-gen3-api: “Gen 3”-driven changes to loading reference catalogs ---------------------------------------------------- The “Generation 3” middleware :ref:`included in the 17.0.0 release ` is undergoing rapid development, and some API instability is to be expected: these notes will not attempt to produce a comprehensive guide to all of the changes. However, it is worth nothing that, as of :jira:`DM-17023`, tasks which derive from *both* `~lsst.pipe.base.CmdLineTask` and `~lsst.pipe.base.PipelineTask` now have *separate* configuration options for specifying reference catalog configuration depending on whether they are being executed by “Gen 2” or “Gen 3” middleware. Further, these options must be manually kept synchronized. For more details and instructions, refer to `this community.lsst.org post`__. __ https://community.lsst.org/t/gen3-api-changes-and-configuring-reference-catalogs/3854/3 .. _release-v19-0-0-pipetask: ``pipetask`` interface changes ------------------------------ The ``pipetask`` command is used to invoke processing based on the `~lsst.pipe.base.PipelineTask` middleware. Its command-line interface has been overhauled in this release, adding a number of new capabilities and changing the ordering of sub-commands and some options. These changes are described in `this community.lsst.org post`__. __ https://community.lsst.org/t/pipetask-command-line-interface-changes/3923 .. _release-v19-0-0-isr: Replacement of functions in `lsst.ip.isr` ----------------------------------------- As :ref:`announced in the 18.0.0 release notes `, a number of functions formerly provided in `lsst.ip.isr` have been removed. They are replaced by functionality in `lsst.meas.algorithms.Defects`. Specifically: - ``defectListFromFootprintList`` is replaced by `lsst.meas.algorithms.Defects.fromFootPrintList` - ``transposeDefectList`` is replaced by `lsst.meas.algorithms.Dfects.transpose` - ``maskPixelsFromDefectList`` is replaced by `lsst.meas.algorithms.Dfects.maskPixels` - ``getDefectListFromMask`` is replaced by `lsst.meas.algorithms.Dfects.fromMask` .. _release-v19-0-0-calib-compat: Removal of `lsst.afw.image.Calib` compatibility API --------------------------------------------------- As `announced in the 18.0.0 release notes `, the compatibility layer for the now-removed `~lsst.afw.image.Calib` API has been removed. .. _release-v19-0-0-ap-silent: Removal of the ``--silent`` command-line option from ap_verify -------------------------------------------------------------- The ``--silent`` option to ap_verify did nothing, and was maintained only for backwards compatibility reasons. As `announced in the 18.0.0 release notes `, it has now been removed. .. _release-v19-0-0-test-util: Removal of deprecated test utility functions -------------------------------------------- The following functions have been removed from the `lsst.utils` package: - `lsst.utils.tests.run` - `lsst.utils.tests.assertRaisesLsstCpp` - `lsst.utils.tests.assertClose` - `lsst.utils.tests.assertNotClose` The class `lsst.utils.tests.TestDeprecations` has also been removed. All of this code had been deprecated several releases in the past. .. _release-v19-0-0-pytest: Removal of the pytest EUPS package ---------------------------------- The `pytest `_ unit testing framework is no longer installed by EUPS as part of the Science Pipelines software distribution. It is still required, and must be provided through the environment. The standard Science Pipelines Conda environment (as installed by e.g. :ref:`newinstall.sh ` with the default options) provides a suitable version of pytest. .. _release-v19-0-0-adn: Removal of meas_extensions_astrometryNet ---------------------------------------- The meas_extensions_astrometryNet package was unused in the Science Pipelines codebase, and was included only through a vestigial dependency. This has now been dropped, and hence meas_extensions_astrometryNet does not appear in this release. .. _release-v19-0-0-mosaic: Removal of meas_mosaic ---------------------- :ref:`Jointcal ` now provides all the features necessary to replace meas_mosaic, and has been successfully tested in at-scale data processing. The meas_mosaic package has therefore been removed from the Science Pipelines codebase, and does not appear in this release. .. _release-v19-0-0-citizen: Removal of :cpp:class:`lsst::daf::base::Citizen` ------------------------------------------------ :cpp:class:`~lsst::daf::base::Citizen`, an obsolete system for debugging memory leaks in C++ code, has been removed per :jira:`RFC-280`. .. _release-v19-0-0-pending-deprecations: Pending Deprecations ==================== These packages/functions will be deprecated in the next major release. - :ref:`release-v19-0-0-deprecate-gen2` - :ref:`release-v19-0-0-deprecate-makeDistortedTanWcs` .. _release-v19-0-0-deprecate-gen2: Upcoming removal of “Generation 2” Middleware --------------------------------------------- The “Generation 3” middleware :ref:`included in the 17.0.0 release ` is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality early in calendar year 2020. Following that release, the “Generation 2” middleware will be removed from the codebase. This will include: - The daf_persistence package, to be replaced by daf_butler; - `lsst.pipe.base.CmdLineTask`, to be replaced by `lsst.pipe.base.PipelineTask`; - The pipe_drivers and ctrl_pool packages, for which replacements are still in development. .. _release-v19-0-0-deprecate-makeDistortedTanWcs: Upcoming removal of `lsst.afw.geom.wcsUtils.makeDistortedTanWcs` ---------------------------------------------------------------- Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering `~lsst.afw.geom.wcsUtils.makeDistortedTanWcs` obsolete. The function ``makeDistortedTanWcs`` will be removed after release 20. .. _release-v19-0-0-deprecations: Deprecations ============ These packages/functions are deprecated and will not be available in the next major release. - :ref:`release-v19-0-0-deprecate-lsstsim` - :ref:`release-v19-0-0-deprecate-afwGeom` - :ref:`release-v19-0-0-deprecate-PixelScaleBoundedField` - :ref:`release-v19-0-0-deprecate-addDistortionModel` - :ref:`release-v19-0-0-deprecate-_extractAmpId` - :ref:`release-v19-0-0-deprecate-pex_policy` - :ref:`release-v19-0-0-deprecate-astrometry_net_data` - :ref:`release-v19-0-0-deprecate-getHasRawInfo` - :ref:`release-v19-0-0-deprecate-pixelScale` - :ref:`release-v19-0-0-deprecate-obs_sdss` .. _release-v19-0-0-deprecate-lsstSim: Removal of the obs_lsstSim package ---------------------------------- The obs_lsst package, :ref:`included in the release 17 `, obviates the need for the obs_lsstSim package. This package will be removed before the release of version 20.0.0. .. _release-v19-0-0-deprecate-afwGeom: Removal of `lsst.afw.geom` classes that have been relocated to `lsst.geom` -------------------------------------------------------------------------- As announced in v16.0 (:ref:`release-v16-0-new-geom`) some primitives have been moved from `afw.geom` to `geom`. We currently provide aliases for compatibility purposes, but new all code should use the `geom` package directly. The aliases will be removed before the release of version 20.0.0. .. _release-v19-0-0-deprecate-PixelScaleBoundedField: Removal of `lsst.afw.math.PixelScaleBoundedField` ------------------------------------------------- `~lsst.afw.math.PixelAreaBoundedField`, :ref:`added in this release `, is intended to replace `~lsst.afw.math.PixelScaleBoundedField`. The latter will be removed before the release of version 20.0.0. .. _release-v19-0-0-deprecate-addDistortionModel: Removal of `lsst.ip.isr.addDistortionModel` and related configuration classes ----------------------------------------------------------------------------- Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering this functionality obsolete. It will be removed before the release of version 20.0.0. .. _release-v19-0-0-deprecate-_extractAmpId: Removal of `lsst.obs.base.CameraMapper._extractAmpId` ----------------------------------------------------- This method is no longer used and will be removed before the release of version 20.0.0. .. _release-v19-0-0-deprecate-pex_policy: Removal of `pex_policy` ----------------------- The pex_policy package is obsolete, and will be removed in version 20.0.0. Calls to `lsst.pex.policy.makePolicy` should be modernized to become calls to `lsst.pex.config.makePropertySet`. See also :jira:`RFC-626`. .. _release-v19-0-0-deprecate-astrometry_net_data: Removal of astrometry_net_data ------------------------------ The astrometry_net_data package is obsolete, and will be removed in version 20.0.0. .. _release-v19-0-0-deprecate-getHasRawInfo: Removal of `lsst.afw.cameraGeom.Amplifier.getHasRawInfo` -------------------------------------------------------- `~lsst.afw.cameraGeom.Amplifier.getHasRawInfo` is redundant, since amplifier objects *always* have raw information. It will be removed before the release of version 20.0.0 .. _release-v19-0-0-deprecate-pixelScale: Removal of `lsst.fgcmcal.FgcmFitCycleConfg.pixelScale` ------------------------------------------------------ This configuration field is no longer used, and therefore will be removed before the release of version 20.0.0. .. _release-v19-0-0-deprecate-obs_sdss: Removal of obs_sdss ------------------- Following the approval of :jira:`RFC-645`, the obs_sdss package will be removed from the Science Pipelines before the release of version 20.0.0.