Release 17.0 (2019-02-15)

Source Identifier
Git tag 17.0
EUPS distrib v17_0

This release is based on the w_2019_07 weekly build.

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see Tickets Addressed in Release 17.0.

If you have questions or comments about this release, visit our Community Forum for advice.

See also:

Major Functionality Changes

Proper motion support added to reference catalogs

Reference catalogs now support providing proper motions and parallaxes. If proper motions are available, then it will be applied to the specified epoch when the catalog is read. No correction for parallax is currently made. For more information, refer to this community.lsst.org post and RFC-368.

Parallelized image ingest

The new ingestDriver.py command makes it possible to ingest images to a data repository in parallel, providing significant speed-ups. For more information, refer to this community.lsst.org post.

Stand-alone instrument signature removal

The new runIsr.py command makes it possible to run instrument signature removal (ISR) from the command line. For more information, refer to this community.lsst.org post.

Alert production capabilities

The prototype Alert Production Pipeline is now included in the Science Pipelines software distribution. This comprises three new software packages:

  • dax_ppdb implements the Prompt Products Database;
  • ap_association implements multiple DIASource detections to compose DIAObjects;
  • ap_pipe provides command-line tasks to orchestrate the pipeline, running CCD processing, image differencing, and source association.

For more information, refer to this community.lsst.org post and RFC-521.

obs_lsst camera package

This release provides obs_lsst, a new camera description and configuration package which will provide support for data from the LSST Camera, ComCam, the Auxiliary Telescope camera, camera test stand 8, ImSim, and PhoSim. This package is expected to be updated and maintained for the indefinite future. Previous packages which provide some of the same functionality (e.g. obs_lsstSim) are provided with this release, but will ultimately be deprecated. For more information, refer to the documentation for lsst.obs.lsst.

DCR-corrected template construction

This release contains code for generating image differencing templates corrected for the effects of differential chromatic refraction (DCR). For information about the technique adopted, refer to DMTN-037; for documentation on the code, see lsst.ip.diffim.DcrModel.

The “Pessimistic Pattern Matcher b” is now the default astrometric matcher

This new matcher works consistently over the large range of stellar densities which LSST will observe. The algorithm and associated validation testing are described in DMTN-031; for more information, refer to this community.lsst.org post.

“Generation 3” middleware

This release contains early versions of the “Generation 3” Data Butler and `PipelineTask` middleware, which are being developed to satisify the requirements described in LDM-556. This software is still under heavy development and is not currently deployed for regular data processing by the Data Management team; it should be regarded here as an unsupported preview. For more information, refer to the documentation for lsst.pipe.base and lsst.daf.butler.

Significant Interface Changes

Reworked Task entrypoints

All Tasks now provide a consistent interface to their primary functionality. The Task provides a run() method, which takes as explicit arguments everything the task needs to get its job done. Tasks may also provide a runDataRef() method, which takes a Butler “dataRef” as its argument, unpacks it, and calls run(). Command line task runners call runDataRef(). For further details, refer to this community.lsst.org post and RFC-352.

Updates to output table field names

All table fields which previously had the suffix Sigma now have the suffix Err. This reflects usage in the Data Products Definition Document (DPDD; LSE-163). For more information, refer to this community.lsst.org post and RFC-333.

Further, source catalogs have been updated to refer to fluxes with the suffix instFlux (or “instrumental flux”), rather than simply flux (which is taken to refer to a calibrated flux). The table formerly known as InstFlux is now called GaussianFlux, which more correctly represents it contents. For more information, refer to this community.lsst.org post, RFC-322 and RFC-510.

Use inverted to get an inverse transform

Previously, the objects in the codebase representing coordinate transforms (e.g. lsst.geom.LinearTransform, astshim.Mapping) supported a heterogeneous mix of APIs for retrieving the inverse transform. These have now all been standardized to use the method inverted(). For more information, refer to this community.lsst.org post and RFC-500.

Changes to calibration flags

The following changes were made to table field names describing sources used for calibration:

  • calib_psfCandidate was renamed calib_psf_candidate;
  • calib_psfUsed was renamed calib_psf_used;
  • calib_astrometryUsed was renamed calib_astrometry_used.

For more information, refer to this community.lsst.org post and RFC-498.

Changes to blendedness flags

The following changes were made to table field names describing source blendedness:

  • base_Blendedness_raw_instFlux was renamed to base_Blendedness_raw;
  • base_Blendedness_abs_instFlux was renamed to base_Blendedness_abs;
  • base_Blendedness_raw_instFlux_child was renamed to base_Blendedness_raw_child_instFlux;
  • base_Blendedness_raw_instFlux_parent was renamed to base_Blendedness_raw_parent_instFlux;
  • base_Blendedness_abs_instFlux_child was renamed to base_Blendedness_abs_child_instFlux;
  • base_Blendedness_abs_instFlux_parent was renamed to base_Blendedness_abs_parent_instFlux;
  • deblend_psfFlux was renamed to deblend_psf_instFlux.

For more information, refer top this community.lsst.org post and RFC-534.

Task metadata is now written as YAML

Previously, metadata was stored as a mix of YAML and Boost.serialization; now, only the former is used. For more information, refer to this community.lsst.org post.

Changes to wrapping C++ code into Python

DM uses pybind11 to expose functionality written in C++ to the Python level. This release rethinks the way pybind11 is used, to increase robustness and reduce the size of compiled binaries. Although it is possible to mix the old and the new techniques within the same software stack, packages must be updated to the new system atomically. For more information, refer to this community.lsst.org post and the pybind11 section in the Developer Guide.

Changes to metadata extraction from file headers

The new astro_metadata_translator package provides a unified interface for extraction of standardized information from image file headers. All camera support (“obs”) packages are being updated to use this new system, which will be required for the next generation of DM middleware. For more information, refer to this community.lsst.org post.

Revised logging system

Log messages generated by the standard Python logging module during the execution of command-line tasks are now forwarded to the lsst.log system. Furthermore, utility functions have been added to make it easy to check that the expected log messages have been generated during testing. For more information, refer to this community.lsst.org post.

Pending Deprecations

Upcoming removal of lsst.afw.image.Calib

This is the final major release which provides the lsst.afw.image.Calib photometric calibration object. Begining with 18.0, Calib, which provides only a photometric zeropoint per CCD, will be replaced with PhotoCalib, which provides a spatially-varying photometric model. Some Calib interfaces are supported by PhotoCalib, but full API compatibility is not possible. By the time this transition is made, PhotoCalib will be able to read files persisted with Calib objects, so backwards compatibility of on-disk data will be maintained. For more information, refer to RFC-289 and RFC-573.

Upcoming removal of “Generation 2” Middleware

The “Generation 3” middleware included in this release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality in late calendar year 2019. Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:

Upcoming removal of the obs_lsstSim package

With the introduction of the new obs_lsst package, there is no longer a need for the obs_lsstSim package. Current work should move to using obs_lsst as soon as possible. Repositories using obs_lsstSim will need to be reprocessed using obs_lsst. For the most part this transition will be transparent to end users. The only code updates needed are in the simulations related packages. The porting work for simulations is planned for the Summer of 2019. We believe that obs_lsstSim can be archived after the release in late calendar year 2019.

17.0.1 patch release

This patch fixes an issue that affects users of the lsst.display.firefly display backend on the Notebook Aspect of the LSST Science Platform. The viewer ID was not being set. See DM-18167 for additional information.