Release 20.0.0 (2020-06-24)

Source Identifier
Git tag 20.0.0
EUPS distrib v20_0_0

This release is based on the w_2020_22 weekly build.

The notes below highlight significant technical changes to the Science Pipelines codebase in this release. For a complete list of changes made, see Tickets Addressed in the Spring 2020 release (v20_0_0).

The Characterization Metric Report (DMTR-251) describes the scientific performance of this release in terms of scientific performance metrics, while Changes to Data Products in Release 20.0.0 covers key issues likely to impact scientists consuming Rubin data products.

If you have questions or comments about this release, visit our community forum for advice.

See also:

Major New Features

Collimated beam projector support

This release adds the cbp package to the Science Pipelines. This provides codes for working with the collimated beam projector (CBP). In particular, cbp can compute the telescope and CBP pointing that will produce a desired beam arrangement, and can compute the hole positions for a CBP mask. [RFC-658]

Alert packets are now generated during Alert Production Pipeline execution

The Alert Production system can now generate schema-compliant alerts in Apache Avro format. Note that these alerts do not yet provide all contents specified by LSE-163 (the Data Products Definition Document) — in particular, they do not include cut-out images — and are currently written to disk, rather than broadcast over the Alert Distribution system. In support of this effort, the alert_packet package was added to the Science Pipelines. This provides a number of utility routines for manipulating alert packets. [DM-24324]

Improved linearity correction

The mechanisms for correcting for detector linearity have been substantially improved. In particular, the new system makes it easy for user-supplied linearity corrections to override the defaults, and a new generic polynomial linearity correction type has been added. Refer to community.lsst.org and RFC-665 for details. [DM-23023]

Sky sources added to single frame processing

“Sky sources” — corresponding to measurements made on empty patches of sky — may now be inserted into single-frame processing. These provide an important source of QA data. See community.lsst.org for details. [DM-23078]

Documentation for fgcmcal

Documentation for fgcmcal, including reference material on all included tasks and a “cookbook” providing a tutorial-style introduction, is now available. This package runs the Forward Global Calibration Method to perform global photometric survey calibration; refer to Burke, Rykoff, et al. 2018 for details. [DM-22469]

fgcmcal calibration on local background-corrected fluxes

The fgcmcal tool can now correct for the local background on aperture fluxes during calibration. This brings faint stars more into line with their bright counterparts, and thereby improves the resultant calibration. [DM-23036]

Conversion of Generation 2 to Generation 3 data repositories

It is now possible to convert Generation 2 to Generation 3 data repositories using the butler convert command. For details, refer to the obs_base Command Line Interface. [DM-22655]

Significant Interface Changes

Third party packages managed through Conda

The Conda package management system, together with the Conda-Forge package collection, is now used to provide all compilers and most third-party dependencies. This substantially simplifies Installation and makes it easier to integrate the Science Pipelines with other tools, while also facilitating quicker updates to third-party code during development.

For more information, refer to community.lsst.org.

[RFC-679, DM-24232]

All FITS headers are now stored in uppercase

The version of CFITSIO used in this release requires that all header keys be written in uppercase to better comply with the FITS standard. Before this release, some Science Pipelines code would store information in FITS headers in lowercase. This has the consequence that:

  • If user code attempts to write a lowercase header to FITS, it will be automatically converted to uppercase, and a warning will be issued;
  • If user code attempts to read FITS data which includes lowercase headers (for example, because it was written by an earlier Pipelines release), they will automatically be converted to uppercase, and an informational message may be logged.

[RFC-640, RFC-686, DM-21989, DM-21991, DM-24376, DM-24570, DM-24618]

Changes to collections, runs, and pipetask arguments in Generation 3 middleware

The way in which collections behave in the Generation 3 middleware has been substantially overhauled. In particular:

  • a “run” is now a special type of collection, rather than an entity that is associated with a collection;
  • the only kind of collection we had before is now called a “tagged” collection;
  • we now also have “chained” collections, which are simply an ordered list of other collections to be searched.

This changes some prominent interfaces, including both the arguments used to construct a Butler and the command-line arguments to the pipetask tool.

For more information, see this community.lsst.org posting and the daf_butler documentation.

[RFC-663, DM-21849]

Changes to building using lsstsw

The initialization scripts used by the lsstsw system, used by some developers to install and update the Science Pipelines, have been renamed. Where previously you sourced setup.sh (or setup.csh), you should now source envconfig.sh (or envconfig.csh). This helps distinguish this script from the setup command provided by EUPS. [DM-23098]

Jointcal now defaults to using Gaia and PanSTARRS1 reference catalogs

By default, Jointcal will attempt to use reference catalogs from Gaia (gaia_dr2_20200414) and PanSTARRS1 (ps1_pv3_3pi_20170110) for astrometry and photometry respectively. If these catalogs are not available, you must specify alternatives by setting astrometryRefObjLoader.ref_dataset_name and/or photometryRefObjLoader.ref_dataset_name within lsst.jointcal.JointcalConfig. [DM-17597]

Removal of the obs_lsstSim package

The obs_lsst package, included in release 17, obviates the need for the obs_lsstSim package, which has now been removed. [RFC-533, DM-20573, DM-24716]

Removal of lsst.afw.geom aliases for code relocated to lsst.geom

As announced in v16.0 (Reworked “geom” package, replacing much of “afw.geom”) some primitives have been moved from afw.geom to geom. Compatibility aliases making them accessible under their old names have now been removed, and all code must use the geom package directly. [DM-20565]

Removal of lsst.afw.math.PixelScaleBoundedField

PixelAreaBoundedField, added in release 19.0.0, replaced PixelScaleBoundedField. The latter has now been removed. [RFC-644, DM-22192]

Removal of lsst.obs.base.CameraMapper._extractAmpId

This obsolete method has been unused since release 11.0 (September 2015). It has therefore been removed. [DM-23371]

Removal of astrometry_net_data

The obsolete astrometry_net_data package has been removed. [DM-22233]

Removal of lsst.afw.cameraGeom.Amplifier.getHasRawInfo

getHasRawInfo was redundant, since amplifier objects always have raw information. It has now been removed. [RFC-585, RFC-632, DM-21711]

Removal of lsst.fgcmcal.FgcmFitCycleConfig.pixelScale

This configuration field was no longer used, and has therefore been removed. [DM-23380]

Pending Deprecations

These packages/functions will be deprecated in the next major release.

Removal of “Generation 2” Middleware

The “Generation 3” middleware included in the 17.0.0 release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. When the new middleware is stable and has reached feature parity with the old code, deprecation of the Generation 2 system will be announced. At that time, guidance for transitioning to the new system will be made available. This will include moving migrating from lsst.daf.persistence to to lsst.daf.butler, and from lsst.pipe.base.CmdLineTask, to be replaced by lsst.pipe.base.PipelineTask. Progress towards the deprecation of Generation 2 may be tracked on DM-20572.

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Removal of lsst.afw.geom.wcsUtils.makeDistortedTanWcs

Camera geometry-based distortions are now automatically incorporated into the WCS automatically when reading raw data, rendering makeDistortedTanWcs obsolete. The function makeDistortedTanWcs will be removed before the release of version 21.0.0 [RFC-616, DM-20937, DM-22114]

Modernization of the C++ API in lsst.afw

A number of long-discouraged C++ APIs in lsst.afw have been marked as deprecated and will be removed before the release of version 21.0.0. [RFC-687, DM-17566, DM-22814, DM-23076, DM-24565]

Removal of the Synpipe package

The Synpipe package for inserting fake galaxies and stars during pipeline execution has been rendered obsolete by the tasks for fake source insertion which were added to pipe_tasks in release 18.0.0. It will therefore be removed before the release of version 21.0.0. [RFC-661, DM-23343]

Removal of lsst.meas.algorithms.SourceDetectionTask.makeSourceCatalog

This method is an alias for run which exists only for backwards compatibility purposes. It will be removed before the release of version 21.0.0. [DM-23085]

Removal of the obs_sdss package

The obs_sdss camera package will be removed from the Science Pipelines before the release of version 21.0.0. Note that this package was previously slated for removal before the release of version 20.0.0, but this has been delayed. The lsst_dm_stack_demo package, which makes use of obs_sdss, will be replaced with a new package providing similar functionality. [RFC-645, DM-22578].

Removal of the pex_policy package

The obsolete pex_policy package will be removed from the Science Pipelines before the release of version 21.0.0. Note that this package was previously slated for removal before the release of version 20.0.0, but this has been delayed. Calls to lsst.pex.policy.makePolicy should be modernized to become calls to lsst.pex.config.makePropertySet. [RFC-626, DM-23374]