Release 23.0.1 (2022-04-06)¶
Source | Identifier |
---|---|
Git tag | 23.0.1 |
EUPS distrib | v23_0_1 |
This release is based on the w_2021_40
weekly build (September 30, 2021), with 42 tickets backported to 23.0.1 to support the Data Preview 0.2 Data Release (DP0.2).
For the production of DP0.2, 23.0.0 was used for single-frame processing (the Tasks within the Pipeline subsets step1
and step2
), and 23.0.1 was used for coadd-processing (Tasks within the Pipeline subset step3
).
None of the updates to 23.0.1 affect single-frame processing, such that rerunning a data release with the final minor release will produce the same results as the DP0.2.
The notes below highlight significant technical changes to the Science Pipelines codebase in this release. For a complete list of changes made, see Tickets Addressed in Release 23.0.1. The 13 tickets that were backported after 23.0.0 (released December 21, 2021) are listed first.
The Characterization Metric Report (DMTR-351) describes the scientific performance of this release in terms of scientific performance metrics.
If you have questions or comments about this release, visit our community forum for advice.
See also:
Major New Features¶
- Generation 3 Middleware
- Measurement Plugin for Gaussian Aperture and PSF Photometry
- Measurement Plugin for Characterizing Trailed Sources
- Addition of cp_verify
- Addition of ci_imsim
Generation 3 Middleware¶
Generation 3 Middleware is being used for the DP0.2 data release production. This is the last release with functional generation 2 middleware. See Deprecated Generation 2 Middleware. Some highlights include:
- Added a new facility for creating lightweight butlers that pre-fills a local SQLite registry. This “execution butler” can allow a pipeline to be executed without talking to the main registry, reducing contention for connections to shared registries. [DM-28646]
- Log diagnostic information when QuantumGraphs are empty because the initial query yielded no results. At present, these diagnostics only cover missing input datasets, which is a common way to get an empty QuantumGraph, but not the only way. [DM-31583]
- Add ability to cache datasets locally when using a remote file store. This can significantly improve performance when retrieving components from a dataset. [DM-13365]
For more information, see detailed notes on changes to the following middleware packages on this release:
Measurement Plugin for Gaussian Aperture and PSF Photometry¶
A Measurement plugin implementing Gaussian Aperture and PSF Photometry (GAaP) for measuring galaxy colors is now available via the new package meas_extensions_gaap included in lsst_distrib. GAaP columns are included in the Object Tables produced in default HSC and DC2 data releases. See Kuijken+15 and DMTN-190 for more information on the algorithm and the implementation. Preliminary results show colors recovered on simulated (DC2 ImSim) red cluster galaxies are improved compared with CModel colors.
Measurement Plugin for Characterizing Trailed Sources¶
A measurement plugin that estimates endpoints, length, and angle parameters for fast-moving trailed sources (e.g., NEOs) is now available in the Science Pipelines via the new package meas_extensions_trailedSources. [RFC-768]
Addition of cp_verify¶
The cp_verify package provides tasks to test the quality of calibration products.
For more information, refer to RFC-791.
Addition of ci_imsim¶
The ci_imsim package provides continuous integration tests for the DC2 data release pipelines. It relies on testdata_ci_imsim and ci_builder.
Use the C++17 Standard¶
As approved in [RFC-759], the Science Pipelines C++ code now uses C++17.
Significant Interface Changes¶
Object Table column names and sentinel values¶
Column names in the parquet Object Table have been renamed for consistency with other tables. The main differences are that the band is prefixed with an underscore: g_psfFlux
instead of gPsFlux
. See RFC-807 for more information. Per RFC-808, when data for a band is unavailable, non-floating point fields are now filled with sentinel values.
Pending Deprecations¶
These Tasks or methods will be deprecated in the next major release.
- Deprecate PSF evaluation without position argument
- Deprecate f-string log messages
- Deprecate ap.verify.Dataset.getSupportedDatasets
- Deprecated Task Configurations:
Deprecate PSF evaluation without position argument¶
PSF models are spatially-varying. Evaluation of these models with methods such as psf.computeImage()
without a position argument will return the realization of the PSF at the average position. A position argument will be required after v24. The current behavior can be replicated by explicitly providing the average position like psf.computeImage(psf.getAveragePosition())
.
For more information, refer to RFC-796.
Deprecate f-string log messages¶
f-string log messages are now deprecated to match Python logging convention.
For more information, refer to RFC-789.
Deprecate ap.verify.Dataset.getSupportedDatasets¶
The concept of ‘supported’ datasets is deprecated.
The getSupportedDatasets
method will be removed after v24.
Deprecated Task Configurations:¶
The following configurations are no longer used and will be removed after the 24.0.0 release:
In PsfWcsSelectImagesTask
, the following configuration parameters and functionality have been moved to ComputeExposureSummaryStatsTask
:
starSelection
starShape
psfShape
The configuration parameter doLegacyStarSelectionComputation
has been added to provide backwards compatibility for the duration of two releases.
In lsst.pipe.tasks.InsertFakesTask
:
raColName
: Usera_col
instead.decColName
: Usedec_col
instead.diskHLR
,aDisk
,bDisk
,paDisk
: Usedisk_semimajor_col
,disk_axis_ratio_col
, anddisk_pa_col
to specify disk half-light ellipse.nDisk
: Usedisk_n
instead.bulgeHLR
,bulge_pa_col
,aBulge
,bBulge
,paBulge
: Usebulge_semimajor_col
,bulge_axis_ratio_col
, and to specify disk half-light ellipse.nBulge
: Usebulge_n
instead.magVar
: Usemag_col
instead.sourceSelectionColName
: Useselect_col
instead.
Deprecations¶
These packages/functions are deprecated and will not be available in the next major release.
- Deprecated Generation 2 Middleware
- Deprecated DecamRawIngestTask and MegaPrimeRawIngestTask
- Removal of lsst.afw.Filter
- Removal of validate_drp
- Deprecated doApplyUberCal in ForcedPhotCcdTask
- Deprecated configurations in IsrTask
- Deprecated configurations in PsfexStarSelectorTask
Deprecated Generation 2 Middleware¶
Generation 2 middleware (Gen2) is no longer being developed and should not be used for new code.
Gen2 infrastructure code (within e.g., lsst.pipe.tasks, lsst.pipe.base, lsst.obs.base, lsst.daf.persistence, and obs packages) will no longer be maintained after January 1, 2022, and may be removed at any point afterward.
The CI package ci_hsc_gen2
will continue to be run, and the tasks it checks will be maintained until we remove the Gen2 infrastructure code in 2022.
Following this release, we will remove Gen2 infrastructure code and the command-line task entry points, runDataRef
, to the Tasks.
Deprecated DecamRawIngestTask and MegaPrimeRawIngestTask¶
MegaPrime and DECam no longer require a specialist Gen3 ingest task.
Please use the default RawIngestTask
.
Both DecamRawIngestTask
and MegaPrimeRawIngestTask
will be removed after v23.
Removal of lsst.afw.Filter
¶
The lsst.afw.Filter
class has been replaced with filterLabel, is deprecated, and will be removed after this release.
Method names that contain Filter
, such as getFilter
, have been replaced e.g., with .getFilterLabel
.
[RFC-730]
Removal of validate_drp¶
The algorithms implemented in lsst.validate.drp were ported as-is to run in lsst.faro.
All future development of scientific performance metrics will be carried out in faro.
The validate_drp
EUPS package will be removed after this release.
Deprecated doApplyUberCal in ForcedPhotCcdTask¶
The field doApplyUberCal
is deprecated.
Use doApplyExternalPhotoCalib
and doApplyExternalSkyWcs
instead.
It will be removed after this release.
[DM-23352]