Changes to Data Products in Release 20.0.0

The notes below highlight significant changes to data products generated by release 20.0.0 the Science Pipelines when processing DECam (HiTS; Förster et al., 2016 ) and/or Hyper Suprime-Cam data using the default configuration. Results obtained in practice will vary with the pipeline configuration.

For a more detailed technical discussion of this release, refer to the release notes. Some items described in those notes may be missing here because they have not yet attained the level of vetting or integration necessary to include them in our standard processing, or simply because they have little or no effect on data products from the perspective of science users consuming them.

Alert Production Data Products

Serialized Alert Packets

ap_assocation now serializes alert packets in Apache Avro format to disk. As of this release, these alerts do not yet provide all contents specified by LSE-163 (the Data Products Definition Document) — in particular, they do not include cut-out images. The alert_packet package contains the alert schemas as well as a number of utility routines for manipulating alert packets. [DM-24324]

Fixes to Alard and Lupton Decorrelation

Some bugs in the Discrete Fourier Transform implementation of the Alard and Lupton decorrelation (DMTN-021) were fixed. The corresponding improvements to detection thresholding will be most noticeable when processing datasets where the noise in the co-added template is not negligible. [DM-24371]

Rescale Template Variances during Difference Imaging

Detection thresholds for difference imaging will be incorrect if the variances of the input images are incorrect. Pixel covariances introduced to template images during the coaddition and warping process are one source of frequent variance inaccuracy. A doScaleTemplateVariance configuration option has been added to lsst.pipe.tasks.imageDifference.ImageDifferenceTask. If enabled, lsst.pipe.tasks.scaleVariance.ScaleVarianceTask is invoked by lsst.ip.diffim.imageDifference.ImageDifferenceTask to empirically rescale the template variance before PSF matching and subtraction. [DM-20558]

Data Release Data Products

Standardized Source Tables

The src catalogs produced by single-frame processing are now rewritten with standardized columns and calibrations applied, producing the new sourceTable (one per visit+detector) and sourceTable_visit (one per visit) butler datasets. These should have a more stable and well-documented schema than the src catalogs they are derived from, and will evolve towards the complete Source table described in LSE-163. By using a columnar data format (Apache Parquet), access to just a few columns is also considerably faster.

Similar Parquet tables for objects (objectTable and objectTable_tract) have existed since the last major release, based on the lower-level deepCoadd_meas, deepCoadd_ref, and deepCoadd_forced_src datasets.

[DM-24062]

Fixes to Defect Masking

Large-area sensor defects were not being masked or interpolated properly in the previous release, because the convolution-like correction for the brighter-fatter effect could expand the affected region beyond our original masks. This has been addressed by growing the mask region. [DM-23083]

Sky Sources in Single-Frame Measurement

Single-frame processing catalogs (including src and the new sourceTable and sourceTable_visit) now include “sky sources” – catalog entries that represent random patches of approximately blank sky (defined as not overlapping any detection footprint) that are measured with exactly the same algorithms as real detections. These can be used for diagnostics on background subtraction, detection thresholds, noise propagation, and probably many aspects of processing we haven’t considered, but it’s also easy to accidentally include them in analyses where only real detections should be used. This can be avoided by filtering rows where the sky_source flag field is True.

See community.lsst.org for more information.

[DM-23078]

Mitigations for Background-Subtraction Problems in Aperture Fluxes

The quality of background subtractions has long been recognized as a problem in our Hyper Suprime-Cam processing, due to a combination of depth, competing science goals, and poorly understood instrumental features. One impact of this that we had not appreciated until recently was the degree to which this affected our aperture photometry, especially on single-epoch processing (which happens before some of our more sophisticated background subtraction steps).

This release includes two changes to mitigate this problem:

  • Aperture corrections are now derived from a brighter sample of stars, for which the errors due to bad backgrounds are relatively smaller. [DM-23071]

  • When using fgcmcal to perform relative photometric calibration, an analytic correction for the background bias is applied to all input photometry, including the quality of the overall calibration. [DM-23036]