Release 23.0.4 (2023-03-12)

Source

Identifier

Git tag

23.0.4

EUPS distrib

v23_0_4

rubin-env version

0.8.5

This release is based on the w_2021_40 weekly build (September 30, 2021), with 63 tickets backported to 23.0.4 to support the Data Preview 0.2 Data Release (DP0.2).

For the production of DP0.2, 23.0.0 was used for single-frame processing (the Tasks within the Pipeline subsets step1 and step2), and 23.0.1 was used for coadd-processing (Tasks within the Pipeline subset step3). 23.0.2 was used for associating DIA Objects, their forced photometry, recalibrating the Source Tables, aggregating global property maps and parquet tables (Tasks within the Pipeline subsets step5, step6, and step7).

Rerunning a data release with the final minor release will produce the same results as the DP0.2. In other words, none of the updates to 23.0.1 affect single-frame processing, and none of the updates to 23.0.2 affect single-frame processing, coadd-processing, image-differencing, and forced-photometry.

The notes below highlight significant technical changes to the Science Pipelines codebase in this release. For a complete list of changes made, see Tickets Addressed in Release 23.0.4.

The Characterization Metric Report (DMTR-351) describes the scientific performance of this release in terms of scientific performance metrics.

If you have questions or comments about this release, visit our community forum for advice.

See also:

Major New Features

Generation 3 Middleware

Generation 3 Middleware is being used for the DP0.2 data release production. This is the last release with functional generation 2 middleware. See Deprecated Generation 2 Middleware. Some highlights include:

  • Added a new facility for creating lightweight butlers that pre-fills a local SQLite registry. This “execution butler” can allow a pipeline to be executed without talking to the main registry, reducing contention for connections to shared registries. [DM-28646]

  • Log diagnostic information when QuantumGraphs are empty because the initial query yielded no results. At present, these diagnostics only cover missing input datasets, which is a common way to get an empty QuantumGraph, but not the only way. [DM-31583]

  • Add ability to cache datasets locally when using a remote file store. This can significantly improve performance when retrieving components from a dataset. [DM-13365]

For more information, see detailed notes on changes to the following middleware packages on this release:

Measurement Plugin for Gaussian Aperture and PSF Photometry

A Measurement plugin implementing Gaussian Aperture and PSF Photometry (GAaP) for measuring galaxy colors is now available via the new package meas_extensions_gaap included in lsst_distrib. GAaP columns are included in the Object Tables produced in default HSC and DC2 data releases. See Kuijken+15 and DMTN-190 for more information on the algorithm and the implementation. Preliminary results show colors recovered on simulated (DC2 ImSim) red cluster galaxies are improved compared with CModel colors.

Measurement Plugin for Characterizing Trailed Sources

A measurement plugin that estimates endpoints, length, and angle parameters for fast-moving trailed sources (e.g., NEOs) is now available in the Science Pipelines via the new package meas_extensions_trailedSources. [RFC-768]

Addition of cp_verify

The cp_verify package provides tasks to test the quality of calibration products.

For more information, refer to RFC-791.

Addition of ci_imsim

The ci_imsim package provides continuous integration tests for the DC2 data release pipelines. It relies on testdata_ci_imsim and ci_builder.

Use the C++17 Standard

As approved in [RFC-759], the Science Pipelines C++ code now uses C++17.

Significant Interface Changes

Object Table column names and sentinel values

Column names in the parquet Object Table have been renamed for consistency with other tables. The main differences are that the band is prefixed with an underscore: g_psfFlux instead of gPsFlux. See RFC-807 for more information. Per RFC-808, when data for a band is unavailable, non-floating point fields are now filled with sentinel values.

Pending Deprecations

These Tasks or methods will be deprecated in the next major release.

Deprecate PSF evaluation without position argument

PSF models are spatially-varying. Evaluation of these models with methods such as psf.computeImage() without a position argument will return the realization of the PSF at the average position. A position argument will be required after v24. The current behavior can be replicated by explicitly providing the average position like psf.computeImage(psf.getAveragePosition()).

For more information, refer to RFC-796.

Deprecate f-string log messages

f-string log messages are now deprecated to match Python logging convention.

For more information, refer to RFC-789.

Deprecate ap.verify.Dataset.getSupportedDatasets

The concept of ‘supported’ datasets is deprecated. The getSupportedDatasets method will be removed after v24.

Deprecated Task Configurations:

The following configurations are no longer used and will be removed after the 24.0.0 release:

In PsfWcsSelectImagesTask, the following configuration parameters and functionality have been moved to ComputeExposureSummaryStatsTask:

  • starSelection

  • starShape

  • psfShape

The configuration parameter doLegacyStarSelectionComputation has been added to provide backwards compatibility for the duration of two releases.

In lsst.pipe.tasks.InsertFakesTask:

  • raColName: Use ra_col instead.

  • decColName: Use dec_col instead.

  • diskHLR, aDisk, bDisk, paDisk: Use disk_semimajor_col, disk_axis_ratio_col, and disk_pa_col to specify disk half-light ellipse.

  • nDisk: Use disk_n instead.

  • bulgeHLR, bulge_pa_col, aBulge, bBulge, paBulge: Use bulge_semimajor_col, bulge_axis_ratio_col, and to specify disk half-light ellipse.

  • nBulge: Use bulge_n instead.

  • magVar: Use mag_col instead.

  • sourceSelectionColName: Use select_col instead.

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Deprecated Generation 2 Middleware

Generation 2 middleware (Gen2) is no longer being developed and should not be used for new code. Gen2 infrastructure code (within e.g., lsst.pipe.tasks, lsst.pipe.base, lsst.obs.base, lsst.daf.persistence, and obs packages) will no longer be maintained after January 1, 2022, and may be removed at any point afterward. The CI package ci_hsc_gen2 will continue to be run, and the tasks it checks will be maintained until we remove the Gen2 infrastructure code in 2022. Following this release, we will remove Gen2 infrastructure code and the command-line task entry points, runDataRef, to the Tasks.

Deprecated DecamRawIngestTask and MegaPrimeRawIngestTask

MegaPrime and DECam no longer require a specialist Gen3 ingest task. Please use the default RawIngestTask. Both DecamRawIngestTask and MegaPrimeRawIngestTask will be removed after v23.

Removal of lsst.afw.Filter

The lsst.afw.Filter class has been replaced with filterLabel, is deprecated, and will be removed after this release. Method names that contain Filter, such as getFilter, have been replaced e.g., with .getFilterLabel. [RFC-730]

Removal of validate_drp

The algorithms implemented in lsst.validate.drp were ported as-is to run in lsst.faro. All future development of scientific performance metrics will be carried out in faro. The validate_drp EUPS package will be removed after this release.

Deprecated doApplyUberCal in ForcedPhotCcdTask

The field doApplyUberCal is deprecated. Use doApplyExternalPhotoCalib and doApplyExternalSkyWcs instead. It will be removed after this release. [DM-23352]

Deprecated configurations in IsrTask

The following configurations are no longer used and will be removed after this release:

  • overscanFitType

  • overscanOrder

  • overscanNumSigmaClip

  • overscanIsInt

  • useFilterAliases

Please configure overscan via the OverscanCorrectionConfig interface. [DM-23396]

Deprecated configurations in PsfexStarSelectorTask

The following configurations are no longer used and will be removed after this release:

  • maxbad

  • maxbadflag

[DM-5684]