.. _release-v20-0-0: ########################### Release 20.0.0 (2020-06-24) ########################### .. toctree:: :hidden: data-products/v20_0_0 tickets/v20_0_0 +-------------------------------------------+------------+ | Source | Identifier | +===========================================+============+ | Git tag | 20.0.0 | +-------------------------------------------+------------+ | :doc:`EUPS distrib ` | v20\_0\_0 | +-------------------------------------------+------------+ This release is based on the ``w_2020_22`` weekly build. The notes below highlight significant technical changes to the Science Pipelines codebase in this release. For a complete list of changes made, see :doc:`tickets/v20_0_0`. The `Characterization Metric Report (DMTR-251) `_ describes the scientific performance of this release in terms of scientific performance metrics, while :doc:`data-products/v20_0_0` covers key issues likely to impact scientists consuming Rubin data products. If you have questions or comments about this release, visit our `community forum `_ for advice. - :ref:`release-v20-0-0-functionality` - :ref:`release-v20-0-0-interface` - :ref:`release-v20-0-0-pending-deprecations` - :ref:`release-v20-0-0-deprecations` *See also:* - :doc:`Installation instructions <../install/index>` - :doc:`Known issues ` - `Doxygen Documentation`__ __ http://doxygen.lsst.codes/stack/doxygen/xlink_master_2020_06_24_18.09.15/ .. _release-v20-0-0-functionality: Major New Features ================== - :ref:`release-v20-0-0-cbp` - :ref:`release-v20-0-0-alert-gen` - :ref:`release-v20-0-0-linearity` - :ref:`release-v20-0-0-sky-sources` - :ref:`release-v20-0-0-fgcmcal-docs` - :ref:`release-v20-0-0-fgcmcal-bg` - :ref:`release-v20-0-0-gen2to3` .. _release-v20-0-0-cbp: Collimated beam projector support --------------------------------- This release adds the :ref:`cbp ` package to the Science Pipelines. This provides codes for working with the `collimated beam projector `_ (CBP). In particular, cbp can compute the telescope and CBP pointing that will produce a desired beam arrangement, and can compute the hole positions for a CBP mask. :jirab:`RFC-658` .. _release-v20-0-0-alert-gen: Alert packets are now generated during Alert Production Pipeline execution -------------------------------------------------------------------------- The Alert Production system can now generate schema-compliant alerts in `Apache Avro`_ format. Note that these alerts do not yet provide all contents specified by :lse:`163` (the Data Products Definition Document) — in particular, they do not include cut-out images — and are currently written to disk, rather than broadcast over the Alert Distribution system. In support of this effort, the :ref:`alert_packet ` package was added to the Science Pipelines. This provides a number of utility routines for manipulating alert packets. :jirab:`DM-24324` .. _Apache Avro: https://avro.apache.org. .. _release-v20-0-0-linearity: Improved linearity correction ----------------------------- The mechanisms for correcting for detector linearity have been substantially improved. In particular, the new system makes it easy for user-supplied linearity corrections to override the defaults, and a new generic polynomial linearity correction type has been added. Refer to `community.lsst.org `_ and :jira:`RFC-665` for details. :jirab:`DM-23023` .. _release-v20-0-0-sky-sources: Sky sources added to single frame processing -------------------------------------------- “Sky sources” — corresponding to measurements made on empty patches of sky — may now be inserted into single-frame processing. These provide an important source of QA data. See `community.lsst.org `__ for details. :jirab:`DM-23078` .. _release-v20-0-0-fgcmcal-docs: Documentation for fgcmcal ------------------------- Documentation for :ref:`fgcmcal `, including reference material on all included tasks and a “cookbook” providing a tutorial-style introduction, is now available. This package runs the Forward Global Calibration Method to perform global photometric survey calibration; refer to `Burke, Rykoff, et al. 2018 `_ for details. :jirab:`DM-22469` .. _release-v20-0-0-fgcmcal-bg: fgcmcal calibration on local background-corrected fluxes -------------------------------------------------------- The :ref:`fgcmcal ` tool can now correct for the local background on aperture fluxes during calibration. This brings faint stars more into line with their bright counterparts, and thereby improves the resultant calibration. :jirab:`DM-23036` .. _release-v20-0-0-gen2to3: Conversion of Generation 2 to Generation 3 data repositories ------------------------------------------------------------ It is now possible to convert Generation 2 to Generation 3 data repositories using the :command:`butler convert` command. For details, refer to the :ref:`obs_base ` :ref:`lsst.obs.base-cli`. :jirab:`DM-22655` .. _release-v20-0-0-interface: Significant Interface Changes ============================= - :ref:`release-v20-0-0-conda` - :ref:`release-v20-0-0-fits-headers` - :ref:`release-v20-0-0-collections` - :ref:`release-v20-0-0-lsstsw-envconfig` - :ref:`release-v20-0-0-jointcal-gaia` - :ref:`release-v20-0-0-remove-obs_lsstSim` - :ref:`release-v20-0-0-remove-afwGeom` - :ref:`release-v20-0-0-remove-PixelScaleBoundedField` - :ref:`release-v20-0-0-remove-addDistortionModel` - :ref:`release-v20-0-0-remove-_extractAmpId` - :ref:`release-v20-0-0-remove-astrometry_net_data` - :ref:`release-v20-0-0-remove-getHasRawInfo` - :ref:`release-v20-0-0-remove-pixelScale` .. _release-v20-0-0-conda: Third party packages managed through Conda ------------------------------------------ The `Conda `_ package management system, together with the `Conda-Forge `_ package collection, is now used to provide all compilers and most third-party dependencies. This substantially simplifies :ref:`part-installation` and makes it easier to integrate the Science Pipelines with other tools, while also facilitating quicker updates to third-party code during development. For more information, refer to `community.lsst.org `__. :jirab:`RFC-679, DM-24232` .. _release-v20-0-0-fits-headers: All FITS headers are now stored in uppercase -------------------------------------------- The version of `CFITSIO`_ used in this release requires that all header keys be written in uppercase to better comply with the `FITS standard`_. Before this release, some Science Pipelines code would store information in FITS headers in lowercase. This has the consequence that: - If user code attempts to write a lowercase header to FITS, it will be automatically converted to uppercase, and a warning will be issued; - If user code attempts to read FITS data which includes lowercase headers (for example, because it was written by an earlier Pipelines release), they will automatically be converted to uppercase, and an informational message may be logged. :jirab:`RFC-640, RFC-686, DM-21989, DM-21991, DM-24376, DM-24570, DM-24618` .. _CFITSIO: https://heasarc.gsfc.nasa.gov/fitsio/ .. _FITS standard: https://fits.gsfc.nasa.gov/fits_standard.html .. _release-v20-0-0-collections: Changes to collections, runs, and ``pipetask`` arguments in Generation 3 middleware ----------------------------------------------------------------------------------- The way in which collections behave in the Generation 3 middleware has been substantially overhauled. In particular: - a “run” is now a special type of collection, rather than an entity that is associated with a collection; - the only kind of collection we had before is now called a “tagged” collection; - we now also have “chained” collections, which are simply an ordered list of other collections to be searched. This changes some prominent interfaces, including both the arguments used to construct a `~lsst.daf.butler.Butler` and the command-line arguments to the ``pipetask`` tool. For more information, see `this community.lsst.org posting `_ and the :ref:`daf_butler ` documentation. :jirab:`RFC-663, DM-21849` .. _release-v20-0-0-lsstsw-envconfig: Changes to building using lsstsw -------------------------------- The initialization scripts used by the :ref:`lsstsw ` system, used by some developers to install and update the Science Pipelines, have been renamed. Where previously you sourced :file:`setup.sh` (or :file:`setup.csh`), you should now source :file:`envconfig.sh` (or :file:`envconfig.csh`). This helps distinguish this script from the :command:`setup` command provided by `EUPS `_. :jirab:`DM-23098` .. _release-v20-0-0-jointcal-gaia: Jointcal now defaults to using Gaia and PanSTARRS1 reference catalogs --------------------------------------------------------------------- By default, Jointcal will attempt to use reference catalogs from Gaia (``gaia_dr2_20200414``) and PanSTARRS1 (``ps1_pv3_3pi_20170110``) for astrometry and photometry respectively. If these catalogs are not available, you must specify alternatives by setting ``astrometryRefObjLoader.ref_dataset_name`` and/or ``photometryRefObjLoader.ref_dataset_name`` within :lsst-config:`lsst.jointcal.JointcalConfig`. :jirab:`DM-17597` .. _release-v20-0-0-remove-obs_lsstSim: Removal of the obs_lsstSim package ---------------------------------- The :ref:`obs_lsst package `, :ref:`included in release 17 `, obviates the need for the obs_lsstSim package, which has now been removed. :jirab:`RFC-533, DM-20573, DM-24716` .. _release-v20-0-0-remove-afwGeom: Removal of `lsst.afw.geom` aliases for code relocated to `lsst.geom` -------------------------------------------------------------------- As announced in v16.0 (:ref:`release-v16-0-new-geom`) some primitives have been moved from `afw.geom` to `geom`. Compatibility aliases making them accessible under their old names have now been removed, and all code must use the `geom` package directly. :jirab:`DM-20565` .. _release-v20-0-0-remove-PixelScaleBoundedField: Removal of `lsst.afw.math.PixelScaleBoundedField` ------------------------------------------------- `~lsst.afw.math.PixelAreaBoundedField`, :ref:`added in release 19.0.0 `, replaced `~lsst.afw.math.PixelScaleBoundedField`. The latter has now been removed. :jirab:`RFC-644, DM-22192` .. _release-v20-0-0-remove-addDistortionModel: Removal of `lsst.ip.isr.addDistortionModel` and related configuration classes ----------------------------------------------------------------------------- Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering this functionality obsolete. It has therefore been removed. :jirab:`RFC-616, DM-23369` .. _release-v20-0-0-remove-_extractAmpId: Removal of `lsst.obs.base.CameraMapper._extractAmpId` ----------------------------------------------------- This obsolete method has been unused since release 11.0 (September 2015). It has therefore been removed. :jirab:`DM-23371` .. _release-v20-0-0-remove-astrometry_net_data: Removal of astrometry_net_data ------------------------------ The obsolete astrometry_net_data package has been removed. :jirab:`DM-22233` .. _release-v20-0-0-remove-getHasRawInfo: Removal of `lsst.afw.cameraGeom.Amplifier.getHasRawInfo` -------------------------------------------------------- `~lsst.afw.cameraGeom.Amplifier.getHasRawInfo` was redundant, since amplifier objects *always* have raw information. It has now been removed. :jirab:`RFC-585, RFC-632, DM-21711` .. _release-v20-0-0-remove-pixelScale: Removal of `lsst.fgcmcal.FgcmFitCycleConfig.pixelScale` ------------------------------------------------------- This configuration field was no longer used, and has therefore been removed. :jirab:`DM-23380` .. _release-v20-0-0-pending-deprecations: Pending Deprecations ==================== These packages/functions will be deprecated in the next major release. Removal of “Generation 2” Middleware ------------------------------------ The “Generation 3” middleware :ref:`included in the 17.0.0 release ` is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. When the new middleware is stable and has reached feature parity with the old code, deprecation of the Generation 2 system will be announced. At that time, guidance for transitioning to the new system will be made available. This will include moving migrating from `lsst.daf.persistence` to to `lsst.daf.butler`, and from `lsst.pipe.base.CmdLineTask`, to be replaced by `lsst.pipe.base.PipelineTask`. Progress towards the deprecation of Generation 2 may be tracked on :jira:`DM-20572`. .. _release-v20-0-0-deprecations: Deprecations ============ These packages/functions are deprecated and will not be available in the next major release. - :ref:`release-v20-0-0-deprecate-makeDistortedTanWcs` - :ref:`release-v20-0-0-deprecate-afw` - :ref:`release-v20-0-0-deprecate-synpipe` - :ref:`release-v20-0-0-deprecate-SourceDetectionTask-makeSourceCatalog` - :ref:`release-v20-0-0-deprecate-obs_sdss` - :ref:`release-v20-0-0-deprecate-pex_policy` .. _release-v20-0-0-deprecate-makeDistortedTanWcs: Removal of `lsst.afw.geom.wcsUtils.makeDistortedTanWcs` ------------------------------------------------------- Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering `~lsst.afw.geom.wcsUtils.makeDistortedTanWcs` obsolete. The function ``makeDistortedTanWcs`` will be removed before the release of version 21.0.0 :jirab:`RFC-616, DM-20937, DM-22114` .. _release-v20-0-0-deprecate-afw: Modernization of the C++ API in `lsst.afw` ------------------------------------------ A number of long-discouraged C++ APIs in `lsst.afw` have been marked as deprecated and will be removed before the release of version 21.0.0. :jirab:`RFC-687, DM-17566, DM-22814, DM-23076, DM-24565` .. _release-v20-0-0-deprecate-synpipe: Removal of the Synpipe package ------------------------------ The Synpipe package for inserting fake galaxies and stars during pipeline execution has been rendered obsolete by the tasks for fake source insertion which were :ref:`added to pipe_tasks in release 18.0.0 `. It will therefore be removed before the release of version 21.0.0. :jirab:`RFC-661, DM-23343` .. _release-v20-0-0-deprecate-SourceDetectionTask-makeSourceCatalog: Removal of `lsst.meas.algorithms.SourceDetectionTask.makeSourceCatalog` ----------------------------------------------------------------------- This method is an alias for `~lsst.meas.algorithms.SourceDetectionTask.run` which exists only for backwards compatibility purposes. It will be removed before the release of version 21.0.0. :jirab:`DM-23085` .. _release-v20-0-0-deprecate-obs_sdss: Removal of the obs_sdss package ------------------------------- The obs_sdss camera package will be removed from the Science Pipelines before the release of version 21.0.0. Note that this package was previously slated for removal before the release of version 20.0.0, but this has been delayed. The lsst_dm_stack_demo package, which makes use of obs_sdss, will be replaced with a new package providing similar functionality. :jirab:`RFC-645, DM-22578`. .. _release-v20-0-0-deprecate-pex_policy: Removal of the pex_policy package --------------------------------- The obsolete pex_policy package will be removed from the Science Pipelines before the release of version 21.0.0. Note that this package was previously slated for removal before the release of version 20.0.0, but this has been delayed. Calls to `lsst.pex.policy.makePolicy` should be modernized to become calls to `lsst.pex.config.makePropertySet`. :jirab:`RFC-626, DM-23374`