Release 18.0.0 (2019-07-09)

Source

Identifier

Git tag

18.0.0

EUPS distrib

v18_0_0

This release is based on the w_2019_23 weekly build.

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see Tickets Addressed in Release 18.0.0.

As of this release, we have adopted the “Semantic Versioning” system, version 2.0.0. In this system, releases are described by a three part number representing major (18 in this case), minor (0), and patch (0) version. The major version is incremented on incompatible API changes; the minor version when new functionality is added; and the patch version when bug fixes are made. For further details, refer to DMTN-106.

If you have questions or comments about this release, visit our Community Forum for advice.

See also:

Major Functionality Changes

Improved integration with the ngmix package

meas_extensions_ngmix, which provides an interface between the LSST codebase and Erin Sheldon’s ngmix package, has been substantially rewritten. This new package exposes more ngmix functionality, and may act as the basis for future development of the LSST measurement framework. For more information, refer to DM-16268.

New tasks for inserting “fake” sources

New command-line tasks have been added for inserting simulated sources into single epoch (processFakes.py) and coadded (insertFakes.py) data. For more information, refer to this community.lsst.org post.

FCGM can now use reference stars

By default, fgcmcal — LSST’s implementation of the Forward Global Calibration Method (Burke et al., 2018) — does not use an external reference catalog, but solves internally to produce a relative calibration. It has now been upgraded to optionally load a set of reference stars, and use them to produce an absolute calibration. For more information, refer to DM-16702.

DIAObjects now include basic lightcurve characterization

DIAObjects generated by the Alert Production system now include a basic set of features which extracted from the object’s light-curve (mean fluxes, best-fit slope, etc). While it is hoped that these features are intrinsically useful, they also serve to exercise the machinery which will be used to calculate the full set of features which is currently being developed in conjunction with the relevant science collaborations. For more details about this functionality, refer to DM-18318; for information about the ultimate set of features which will be computed, refer to DM-11962 and LSE-163.

Significant Interface Changes

Python interface to PropertySet and PropertyList changed

__getitem__, update, and get methods have been added to the Python interfaces of PropertySet and PropertyList. This means that they can be used in the same way as native Python dictionaries. It is also possible to store the value None in a PropertySet. For more information, refer to this community.lsst.org post and RFC-596.

Alert Production Pipeline command-line interface changed

The ap_pipe.py command will no longer try to create a “prompt products” database when it is executed. This change makes the pipeline better able to run in different environments, including large-scale testing and operations. An appropriate database should now be created in advance, either by using the make_ppdb.py command, or by configuring the pipeline to use an externally-provided database. For more information, refer to this community.lsst.org post and RFC-587.

New configurations for AstrometryTask source and reference selectors

The configuration options for single-frame astrometry (as implemented in lsst.meas.astrom.AstrometryTask) have changed. This fixes various bugs where (a) selections of the reference catalog were performed only in some modes of operation; and (b) multiple conflicting selections of the source catalog could be performed in some modes of operation. All obs package defaults have been updated to reflect the new changes; you need only worry about these changes if you have overridden the obs package defaults. For more information, refer to this community.lsst.org post and RFC-589.

lsst.afw.image.Calib removed

Calib, which provided only a photometric zeropoint per CCD, has been replaced with PhotoCalib, which provides a spatially-varying photometric model. Some Calib interfaces are supported by PhotoCalib, but full API compatibility is not possible; using the old-style interfaces is deprecated, and they will be removed following this release. PhotoCalib is able to read files persisted with Calib objects, so backwards compatibility of on-disk data is maintained. For more information, refer to RFC-289 and RFC-573.

Pending Deprecations

These packages/functions will be deprecated in the next major release.

Upcoming removal of “Generation 2” Middleware

The “Generation 3” middleware included in the previous release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality in late calendar year 2019 (likely version 19.0.0, but that remains to be confirmed). Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:

  • The daf_persistence package, to be replaced by daf_butler;

  • lsst.pipe.base.CmdLineTask, to be replaced by lsst.pipe.base.PipelineTask;

  • The pipe_drivers and ctrl_pool packages, for which replacements are still in development.

Upcoming removal of the obs_lsstSim package

The obs_lsst package, included in the previous release, obviates the need for the obs_lsstSim package. All LSST code is expected to transition to the new system later in summer 2019. Some work will be required to update old data repositories to the new system. After that, a final release will be made containing obs_lsstSim in late 2019, after which the package will be retired.

Upcoming removal of lsst.afw.geom classes that have been relocated to lsst.geom

As announced in v16.0 (Reworked “geom” package, replacing much of “afw.geom”) some primitives have been moved from afw.geom to geom. We currently provide aliases for compatibility purposes, but new code should use the geom package. These aliases will be removed after the version 19.0.0 will be released.

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Upcoming removal of lsst.afw.image.Calib compatibility API

This release includes (partial) backwards compatibility with the now removed (see above) Calib API. This will be removed before the next release.

Upcoming removal of the --silent argument to ap_verify.py

The --silent argument used to disable upload of metrics from ap_verify.py to SQuaSH. The capability to upload metrics has been removed from ap_verify.py (see DM-16536), but --silent has been retained as a no-op for compatibility reasons. It will be removed before the next release.

Upcoming removal of ip_isr functions from isrFunctions.py

These functions are replaced by functionality in lsst.meas.algorithms.Defects:

  • defectListFromFootprintList replaced by Defects.fromFootPrintList()

  • transposeDefectList replaced by Defects.transpose()

  • maskPixelsFromDefectList replaced by Defects.maskPixels()

  • getDefectListFromMask replaced by Defects.fromMask()

Upcoming removal of meas_algorithms functions from defects.py

  • policyToBadRegionList, policy defect files no longer supported.