Release 14.0 (2017-10-23)¶
| Source | Identifier |
|---|---|
| Git tag | 14.0 |
| EUPS distrib | v14_0 |
This release is based on the w_2017_33 weekly build.
These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes made, see Tickets Addressed in Release 15.0.
If you have questions or comments about this release, visit our Community Forum for advice.
See also:
Major Functionality and Interface Changes¶
- Detection: only add flags.negative to catalog schema if needed
- New Footprint system
- Change of argument order in single frame measurement
- More statistical operations possible for assembling coadds
- Hooks for injecting simulated sources
- Increased mask plane depth
- Option to disable template smoothing in deblender
- New matcher algorithm
- AST-backed WCS class
- C++ code is now exposed to Python using Pybind11
- PSF-matched coadds are an independent data product
- Flag sources used in calibration
- Convolved flux measurement
- Include pupil information in camera geometry
- Joint calibration
- FITS HDUs are now 0-indexed
- Blendedness calculation is run by default
- Spatially-varying Alard & Lupton image decorrelation
- Useful string representation of tabular data
Detection: only add flags.negative to catalog schema if needed¶
Rather than always including this flag in the results of source measurement, it is now only provided if the thresholdPolarity argument is set to both.
Otherwise, this flag is omitted altogether.
[DM-903]
New Footprint system¶
Footprints are used to represent contiguous areas of pixels on an image, corresponding e.g. to source detections.
The Footprint system has been comprehensively rewritten for performance, flexibility and ease of use in this release.
This change involves significant changes to the Footprint API: see community.lsst.org for a description of the changes and a migration guide.
[DM-3559]
Change of argument order in single frame measurement¶
For consistency with other code and documentation, the order of the arguments of the run() method of SingleFrameMeasurementTask has been reversed.
More statistical operations possible for assembling coadds¶
Rather than simply taking the (possibly clipped) mean, it is now possible to specify other operations, such as the median, to be used for combining pixel data when assembling coadds.
[DM-4158]
Hooks for injecting simulated sources¶
Hooks are now available in ProcessCcdTask for inserting simulated sources into a processing run.
These hooks are designed to enable the easy integration of external packages supporting this functionality (e.g. SynPipe).
Increased mask plane depth¶
It is now possible to include 32 separate mask planes with each image.
[DM-7477]
Option to disable template smoothing in deblender¶
Under some circumstances, smoothing of deblend templates can have a detrimental effect on output quality. This smoothing is now optional.
New matcher algorithm¶
A new algorithm — “match pessimistic B” — for matching sources to reference catalog objects, typically used when performing photometry or astrometry, has been added to the release. This matcher is not currently enabled by default: it should be regarded at this stage as a technology preview.
[DM-8110, DM-8111, DM-8112, DM-8113, DM-8645, DM-9623, DM-9751]
AST-backed WCS class¶
A new representation of the celestial world coordinate system based on the Starlink AST library has been introduced. This will ultimately replace our previous WCS implementation; the transition is not yet complete at the time of this release.
[DM-8440]
C++ code is now exposed to Python using Pybind11¶
The technology used to wrap our low-level C++ functionality into Python has been switched from Swig to Pybind11. Pybind11 provides a more convenient environment for developers and enables several API improvements in our Python code. A summary of the changes is available on community.lsst.org
[DM-8467]
PSF-matched coadds are an independent data product¶
That is, rather than generating a single “coadd” data product which may be configured to be either direct or PSF-matched, it is now possible to generate both PSF-matched and direct coadds in a single pipeline execution.
Flag sources used in calibration¶
We now store flags indicating which sources were used in astrometric and/or photometric calibration. This is analogous to the flags used to indicate sources used in PSF modeling.
[DM-9050]
Convolved flux measurement¶
The meas_extensions_convolved package has been added to the lsst_distrib distribution. This convolves images to a specified PSF before measuring aperture fluxes. This may be useful when, for example:
- Measuring the flux that would be obtained with a fiber spectrograph, by using an aperture equal to the size of the spectrograph fiber and measuring with multiple target seeings to allow interpolation of the flux at the seeing of the spectroscopic observation.
- Measuring the color of galaxies in variable seeing, by using a common aperture and target seeing in each band.
Include pupil information in camera geometry¶
The afw.cameraGeom.Camera object now provides an API for accessing a model of the pupil obscuration.
Joint calibration¶
The Jointcal package, which provides for simultaneous photometric and astrometric calibration over multiple exposures, has been added to the lsst_distrib distribution. While we believe that Jointcal works well, note that it is still under heavy development and is not currently run as part of regular processing by the LSST development team: regard this package as a technology preview.
[DM-9752]
FITS HDUs are now 0-indexed¶
The previous LSST convention was to use 1-based indexing for HDUs.
That is, previously, the first HDU in the file was referred to as HDU 1, with 0 being a special value taken to mean “the first non-empty HDU”.
Now the first HDU in the file is HDU 0; the special value INT_MIN is used to refer to the first non-empty HDU.
Blendedness calculation is run by default¶
The default configuration has been updated so that “blendedness” is always calculated unless it has been specifically disabled by the user.
[DM-10506]
Spatially-varying Alard & Lupton image decorrelation¶
It is now possible to perform spatially-varying decorrelation when performing Alard & Lupton (1998) style image differencing. See DMTN-021 for details.
[DM-10804]
Bug Fixes¶
- Command line tasks now provide standard Unix return codes
- Deterministic package version checking
- Correct CModel priors
- Correct blendedness debiasing calculation
- Fixed order of operations when using temporary local backgrounds in detection
- Correct number of bands used for peak culling
Command line tasks now provide standard Unix return codes¶
When a command line task fails, it will now return a non-0 exit code to the calling shell, in line with standard Unix conventions.
[DM-4141]
Deterministic package version checking¶
The codebase keeps track of which software versions were used to calculate a particular set of outputs as a means of tracking data provenance. However, in some circumstances this list was being stored inconsistently, leading to version mismatches being reported where non existed. This has now been resolved.
[DM-9439]
Correct CModel priors¶
CModel priors are now weighted correctly relative to the likelihood.
[DM-9795]
Fixed order of operations when using temporary local backgrounds in detection¶
Resolved an issue which could result in missed isolated-object detections and incorrect Footprints for large objects.
[DM-10271]
Correct number of bands used for peak culling¶
During multi-band coadd processing, peaks may be rejected (or “culled”) if they have been detected in insufficiently many bands. A book-keeping error led to the number of bands in which a peak had been detected to be recorded incorrectly, and hence to the wrong peaks being culled. This has now been corrected.
[DM-11625]
Build and Code Improvements¶
- YAML-based policy
- Full support for Python 3.5
- More Robust Handling of Flagged Measurements
- Support for recent NumPy
- Automatic printing of C++ backtrace on segmentation faults
YAML-based policy¶
The “policy” files associated with standard stack cameras, which define where and how datasets are persisted by the Butler, were updated to use a new YAML-based syntax.
[DM-7363]
Full support for Python 3.5¶
All of the Science Pipelines code included in this release is tested with both Python 2.7 and Python 3.5. It should also support Python 3.6, although that has not been rigorously tested.
[DM-7756]
More Robust Handling of Flagged Measurements¶
The FlagHandler mechanism for setting flags indicating the status of particular measurements was enhanced to improve its robustness and ensure that flags are always set consistently.
[DM-9249]
Support for recent NumPy¶
Recent versions of NumPy have introduced a number of changes which were incompatible with usage in the Science Pipelines code. The codebase has now been adapted to work around these changes.