Release 13.0 (2017-02-28)¶
Source | Identifier |
---|---|
Git tag | 13.0 |
EUPS distrib | v13_0 |
This release is based on the w_2017_33
weekly build.
- Major Functionality and Interface Changes
- Bug Fixes
- Build and Code Improvements
- Qserv and Data Access
- Science User Interface
See also:
Major Functionality and Interface Changes¶
- Added a measurement extension which performs Kron photometry
- Added a package version tracking system (“stack provenance”)
- Multi-band coadd processing now saves metadata
- Disabled oversampling in PSFEx
- Added support for new Hyper Suprime-Cam filters
- Multi-band coadd processing no longer selects flagged bands as reference
- Renamed the “afterburner” system to CatalogCalculation
- Added sanity checking for source centroids
- Added ability to run ctrl_pool processes without a batch system
- Added ability to match against reference catalog without rotation, offset or WCS fitting
- Added functionality to show where configuration is being set and overridden
- frame now defaults to None when creating a new Display
- Attribute access to Display objects is now passed through to the underlying implementation
- Added a new fitter for astrometric distortions
- Unified exposure metadata
- Added convenient access to catalog metadata
- FITS headers are now read & returned in order
- Added functionality for performing measurement without deblending
- Corrected order of operation when construction PSF-matched coadds
- Reduced galaxy “shredding” when deblending
- Corrected CModel table schema units
- Added the SpherePoint class to represent positions on the sphere
- Added computeBBox method to all PSF classes
- PSF moments are now included in the SdssShape results
- Single frame processing no longer requires a PSF
- Added basic ISR capability for Hyper Suprime-Cam
Added a measurement extension which performs Kron photometry¶
The KronFluxAlgorithm
measures the flux in an elliptical aperture as defined by Kron (1980).
The plugin is not enabled by default, but may be switched on by requesting ext_photometryKron_KronFlux
in measurement configuration.
Added a package version tracking system (“stack provenance”)¶
A package version tracking and checking capability has been included in the stack and integrated with the CmdLineTask
system.
This keeps track of what versions of packages are being used in order to prevent the user from unwittingly processing data with multiple, subtly different, versions of the code.
This provides a provisional implementation of a provenance tracking system: it is essential for short-term use, but will be rendered obsolete before we move to operations.
A more detailed description of this system is available on the LSST Community Forum.
[DM-3372]
Multi-band coadd processing now saves metadata¶
Metadata describing the operation of algorithms within the multi-band coadd processing framework is now correctly persisted along with their outputs.
[DM-4991]
Disabled oversampling in PSFEx¶
Experiments with Hyper Suprime-Cam have established that oversampling does not work well in PSFEx, one of the PSF estimation algorithms available in the stack. It has therefore been disabled in the default configuration used for LSST.
[DM-6982]
Added support for new Hyper Suprime-Cam filters¶
The HSC I2, R2, NB0468 and NB0926 filters are now defined in the obs_subaru
camera package.
Multi-band coadd processing no longer selects flagged bands as reference¶
The algorithm used to select the reference band in multi-band coadd processing has been updated to avoid bands with certain user-specified flags set. By default, these flags include the source having an interpolated centre, or errors detected with PSF, Kron or CModel fluxes.
[DM-7044]
Renamed the “afterburner” system to CatalogCalculation
¶
The “afterburner” system was added to the stack in the previous (v12) release.
It provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not using pixel data).
To avoid ambiguity, this system has been renamed CatalogCalculation
.
Added sanity checking for source centroids¶
Centroid finding measurement algorithms were updated to ensure that the centroids they find fall within the footprint of the source being centroided. If not, the returned value is forced to correspond to the position of the most significant peak within the Footprint, and a flag is set to indicate the error.
[DM-4926]
Added ability to run ctrl_pool
processes without a batch system¶
The option --batch-type none
(or None
) runs the command in your process as normal.
This means that tools relying on ctrl_pool
, such as the pipe_drivers
scripts, need not rely on MPI.
Added ability to match against reference catalog without rotation, offset or WCS fitting¶
The new DirectMatchTask
matches sources against a reference catalog without performing any rotation, offset or WCS fitting.
This provides a simpler and less error-prone option for matching sources for quality assessment.
The MeasureMergedCoaddSourcesTask
has been modified to make use of this new DirectMatchTask
.
[DM-7117]
Added functionality to show where configuration is being set and overridden¶
You can now specify a command line argument of the form --show history=config.foo.bar
to see where configuration parameter foo.bar
is set and (if applicable) overridden (the leading config.
is optional).
N.B. pipe to cat
to turn of colourization (or investigate lsst.pex.config.history.Color
).
frame
now defaults to None
when creating a new Display
¶
When using the afw.display
system to create a new Display
object, the frame
argument is now optional: it will default to None
if not specified.
Attribute access to Display
objects is now passed through to the underlying implementation¶
Some afw.display
device types may provide implementation-specific functionality.
This may now be accessed directly through the Display
object, which will delegate to implementation-specific methods, if they exist, when a non-standard method is called on the Display
.
Added a new fitter for astrometric distortions¶
A new fitter task, meas.astrom.FitSipDistortionTask
has been added that will eventually replace the current fitter, meas.astrom.FitTanSipWcsTask
.
The new fitter should be more robust and certainly uses a better-motivated algorithm, but it has not yet received enough testing to be used as the default and hence must be enabled explicitly.
When running processCcd.py
, this can be done with the following config file:
from lsst.meas.astrom import FitSipDistortionTask
config.charImage.astrometry.wcsFitter.retarget(FitSipDistortionTask)
config.calibrate.astrometry.wcsFitter.retarget(FitSipDistortionTask)
The new fitter is not compatible with meas.astrom.ANetAstrometryTask
.
[DM-3549]
Unified exposure metadata¶
Exposures now contain a new VisitInfo
object that contains useful information about the overall visit (none of it specific to the individual CCD), including the exposure time, date and ID, boresight position and airmass.
See the community.lsst.org post on unified exposure metadata for more information.
Added convenient access to catalog metadata¶
It is now possible to read the metadata, length and schema of a catalog through a Butler interface without loading the whole catalog.
For example, given a particular data ID (dataID
) and catalog dataset type (someCatalog
), one can run:
butler.get("someCatalog_schema", dataId)
butler.get("someCatalog_len", dataId)
butler.get("someCatalog_md", dataId)
FITS headers are now read & returned in order¶
lsst.afw.image.readMetadata
now returns a PropertyList
, which maintains the intrinsic ordering of the header keywords in the FITS file it is reading.
Previous versions used a PropertySet
, which is unordered and hence lost this information.
[DM-8213]
Added functionality for performing measurement without deblending¶
The deblender sometimes runs into trouble with galaxies in clusters, leading to inaccurate fluxes. To mitigate, we can now provide measurements performed without any deblending having taken place. To activate this functionality, specify the measurement plugins to be run without deblending in the undeblended registry attached to the measurement class (this is analogous to the plugins registry used when performing measurement on deblended sources). By default, no undeblended measurement is performed. This functionality is available in both forced and single-frame measurement.
Corrected order of operation when construction PSF-matched coadds¶
WarpAndPsfMatchTask
has been updated so that, by default, it will first warp inputs to the coadd projection and then perform PSF matching; this is the reverse of its previous behaviour.
The new behaviour is preferable, but a configuration option makes it possible to revert to the old method on request.
[DM-8088]
Reduced galaxy “shredding” when deblending¶
If galaxies have substructure, such as face-on spirals, the process of identifying peaks can “shred” the galaxy into many pieces.
The templates of shredded galaxies are typically quite similar because they represent the same galaxy.
We try to identify these “degenerate” peaks by looking at the inner product (in pixel space) of pairs of templates.
If they are nearly parallel, we only keep one of the peaks an reject the other.
This behaviour is enabled by the removeDegenerateTemplates
option in SourceDeblendConfig
; by default, it is disabled.
Note that it is not safe to enable removeDegenerateTemplates
when using the multi-band coadd processing framework: it should only be used when calling SourceDeblendTask
directly.
[DM-4550]
Corrected CModel table schema units¶
Units for the time field changed from seconds to second to comply with the Astropy convention.
[DM-6282]
Added the SpherePoint
class to represent positions on the sphere¶
The SpherePoint
class allows locations on the celestial (or any other) sphere to be represented and manipulated without the coordinate-system complications associated with Coord
.
[DM-5529]
Added computeBBox
method to all PSF classes¶
Calling computeBBox
on a PSF returns its bounding box. It is available for all PSF classes distributed with the stack.
[DM-8175]
PSF moments are now included in the SdssShape
results¶
The base_SdssShape
measurement algorithm has been updated to record the adaptive moments of the PSF in its outputs.
This functionality may be disabled by setting the configuration parameter doMeasurePsf
to False
.
Single frame processing no longer requires a PSF¶
When ProcessCcdTask
is used to process an exposure which does not have an associated Psf
and PSF measurement is disabled (i.e. the doMeasurePsf
configuration option is False
) we now assume a default PSF rather than raising an error.
[DM-6045]
Added basic ISR capability for Hyper Suprime-Cam¶
Normally, instrument signature removal (ISR) is carried out during pipeline processing as part of ProcessCcdTask
.
Occasionally, it is convenient to perform ISR without full pipeline processing.
To make this more convenient, hscIsr.py
provides a simple front-end which simply performs ISR on HSC data without any further processing.
[DM-9186]
Bug Fixes¶
- Corrected configuration dictionary comparisons
- Restored capability to calculate temporary local backgrounds in source detection
- Resolved a memory error in convolution
- Patched GalSim to raise an exception in some error conditions
- Ensured star selector results are reproducible
- Fixed single frame measurement so it can exploit multiple CPU cores
- Fixed typo in CoaddSrcTransformTask
- Stop writing duplicate AR_NAME headers when persisting data
- Added more rigor about units angle persistence
- Resolved infinite recursion in afw.table
- Fixed a division by zero error in measurement of aperture corrections
- Fixed dataset lookup in multibandDriver.py
- Fixed incorrect definition of mask when building fringe frames
- Fixed failures in MPI-based processing due to large “pickles”
- Fixed failure to retrieve optimizer state when fitting DoubleShapeletPsf
- Resolved an issue with failed persistence of ConfigurableField
- Resolved an error with schema persistence in forced photometry
- Resolved issues due to scaling by non-finite dark time
- Resolved an issue which could cause unnecessary re-linking during builds
Corrected configuration dictionary comparisons¶
Resolved a bug whereby comparison of (intrinsically unordered) configuration dictionaries incorrectly compared their ordering rather than just their contents. This lead to unexpected failure of re-runs due to incorrectly identified “mismatched configurations”.
[DM-6661]
Restored capability to calculate temporary local backgrounds in source detection¶
Our SourceDetectionTask
provides the optional capability of fitting and temporarily removing a local background before detecting sources.
This capability was accidentally disabled in the X/W16 release (v12); it has now been restored.
It is disabled by default.
[DM-5323]
Resolved a memory error in convolution¶
A memory handling error could cause our convolution code to generate incorrect results on some platforms (problems observed on macOS when the code is built without optimization; no errors seen on Linux, or when building with -O2 or -O3 on macOS). This appears to originate in the Boost Generic Image Library, an external dependency of the LSST stack. We have been unable to isolate the problem in Boost, but have worked around it for our particular use case.
[DM-5822]
Patched GalSim to raise an exception in some error conditions¶
On certain error conditions, GalSim, a third party package used by the LSST stack, generates an assertion failure which is hard for our code to handle appropriately. LSST’s version of GalSim has been patched to raise a more appropriate exception instead. This change has now been included in GalSim itself.
[DM-7114]
Ensured star selector results are reproducible¶
Resolved an issue whereby the star selection could change depending on the number of CPU cores in use.
[DM-7040]
Fixed single frame measurement so it can exploit multiple CPU cores¶
Resolved an issue whereby the SingleFrameDriverTask
failed to use more than one CPU core regardless of configuration.
[DM-7134]
Fixed typo in CoaddSrcTransformTask
¶
A typo which rendered CoaddSrcTransformTask inoperable was resolved. In the process, the tests for this task were significantly upgraded to avoid a repeat of this, or other, problems.
[DM-6635]
Stop writing duplicate AR_NAME
headers when persisting data¶
When persisting objects to FITS, we were incorrectly writing duplicate AR_NAME
headers.
This has been resolved.
[DM-7221]
Added more rigor about units angle persistence¶
We now correctly write TUNIT
headers describing the units used when storing angles, and are more careful about checking units when reading them.
[DM-7221]
Resolved infinite recursion in afw.table
¶
A bug which could lead to infinite recursion has been fixed has been discovered in the afw.table
library.
Fixed a division by zero error in measurement of aperture corrections¶
Attempting to calculate an aperture correction for a measurement of zero flux would previously result in an unhandled division by zero. This has been resolved.
[DM-7180]
Fixed dataset lookup in multibandDriver.py
¶
The multiband driver script was previously checking for the existence of the pre-calexp version of the coadd being processed.
This is not guaranteed to exist if the coadds were built using the driver scripts (where there is no need to persist both the pre-calexp and calexp versions), and it is only the calexp coadds that are used in the multiband tasks, so it is the calexp version whose existence should be checked for in multibandDriver.py
.
[DM-7716]
Fixed incorrect definition of mask when building fringe frames¶
The code in pipe_drivers to construct fringe frames, FringeTask
, was incorrectly using a mask plane when in fact a mask bit was required.
This has now been corrected.
[DM-8033]
Fixed failures in MPI-based processing due to large “pickles”¶
Resolved an issue whereby the ctrl_pool
distributed processing middleware was unable to transmit large volumes of data over MPI due to a limitation of the Python 2.x “pickle” system.
[DM-8021]
Fixed failure to retrieve optimizer state when fitting DoubleShapeletPsf
¶
Resolved an issue whereby the optimizer state was being read incorrectly, which could lead to fitting failures.
[DM-6562]
Resolved an issue with failed persistence of ConfigurableField
¶
An error was resolved which could cause ConfigurableField
s to be improperly persisted and hence render the configuration impossible to load.
[DM-8273]
Resolved an error with schema persistence in forced photometry¶
When performing forced photometry an incorrect catalog schema, which did not match the data, was being persisted to the Butler. This has been fixed.
[DM-8221]
Resolved issues due to scaling by non-finite dark time¶
If not explicitly set in the camera package, the dark time defaults to NaN
.
When the dark time is used to scale an image, this would render the image unusable.
We now check for scaling by non-finite dark time and raise an appropriate error.
[DM-9004]
Build and Code Improvements¶
- Ongoing migration from Boost to standard library equivalents
- Warnings in ObjectSizeStarSelector are now properly logged
- Added unit tests for colorterm framework
- Made FlagHandler accessible from Python
- Added exact image comparison utility code
- CatalogCalculation is now used to handle aperture corrections
- End-user builds are now optimized by default
- Fixed a build failure of mpi4py on some versions of macOS
- New obs_base package supersedes daf_butlerUtils
- Centralized Butler dataset definitions
- Order in which SConscript files are read is configurable
- Made ndarray an external package
- Added backwards compatibility to CoaddPsf
- Changed API in SpatialCellCandidate hierarchy
- Python 3 support
- SpanSet added
- Logging migration
- Deblender API refactored
- Removed old example & experimental code
- Made the simpleShape algorithm less verbose
- Made the pipe_drivers package less verbose
- Added “reduce” functions to ctrl_pool
Ongoing migration from Boost to standard library equivalents¶
Continuing work started in the previous release, we have audited the stack for use of the Boost libraries. Where possible, Boost has been replaced with C++11 standard library equivalents.
[DM-5880, DM-6089, DM-6090, DM-6091, DM-6092, DM-6093, DM-6094, DM-6095, DM-6096, DM-6097, DM-6325]
Warnings in ObjectSizeStarSelector
are now properly logged¶
Previously, non-fatal NumPy warnings of the form RuntimeWarning: invalid value encountered in less
could appear on the console, but not be caught and logged anywhere.
It was therefore easy to miss problems arising when running the algorithm.
These warnings are now ingested into the LSST logging system, and are hence recorded properly.
[DM-5428]
Added unit tests for colorterm framework¶
A set of unit tests have been written which properly exercise the colorterm framework.
[DM-2837]
Made FlagHandler
accessible from Python¶
The FlagHandler
system for efficiently and safely setting flags corresponding to measurement algorithm failures on source records is now accessible from Python.
Previously, it was only available for use in C++ code.
[DM-4009]
Added exact image comparison utility code¶
The assertImagesEqual
and assertMaskedImagesEqual
methods have been added to added to LSST’s testing framework.
These provide explicit mechanisms for checking for exact equality of image data.
[DM-6641]
CatalogCalculation
is now used to handle aperture corrections¶
The CatalogCalculation
facility was added to the stack in the previous (v12) release (where it was referred to as “afterburners”).
It provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not pixel data).
This has now been used to clean up the logic used to apply aperture corrections within our regular data processing flow.
This old method for handling aperture corrections was a persistent source of bugs and confusion: these should be avoided by the new approach.
[DM-5877]
End-user builds are now optimized by default¶
Following an RFC, sconsUtils
was updated to enable optimization by default when compiling individual stack packages.
(Full-stack installations using a supported mechanism have always been optimized.)
Fixed a build failure of mpi4py on some versions of macOS¶
Added extra logic to ensure we target a version of the macOS SDK which is compatible both with the version of macOS installed and the Python interpreter in use.
[DM-6133]
New obs_base
package supersedes daf_butlerUtils
¶
A new obs_base
package has been added to the distribution.
This contains base classes used by the various obs_
camera packages.
The contents of the old daf_butlerUtils
package has been migrated to obs_base
, and the old package has been retired.
Centralized Butler dataset definitions¶
Butler dataset definitions which are common to multiple cameras have been consolidated in obs_base
rather than being repeated in each camera package.
Order in which SConscript files are read is configurable¶
The sconsUtils
build system now reads SConscript
files, which describe the build, in user-defined order.
This makes it possible to specify that some components of a package must be built before others.
[DM-7179]
Made ndarray
an external package¶
The ndarray
package in the LSST stack now installs the upstream version 1.3.0.
[DM-2005]
Added backwards compatibility to CoaddPsf
¶
Due to changes in the way CoaddPsf
and ExposureCatalog
were being persisted, it was no longer possible to load old data into a current version of the stack.
A compatibility layer has now been added.
[DM-8147]
Changed API in SpatialCellCandidate
hierarchy¶
Backwards incompatible API changes were made to the SpatialCellCandidate
class hierarchy to resolve build problems in some environments.
Details of the changes are available on RFC-250.
Python 3 support¶
Almost all of the Science Pipelines packages included in this release are compatible with both Python 2 and Python 3.
The notable exception is pipe_drivers
, which currently only works with Python 2.
[DM-6322, DM-7021, DM-6320, DM-7036, DM-7028, DM-7066, DM-7047, DM-7188, DM-7138, DM-7069, DM-7152, DM-7246, DM-7243, DM-7247, DM-7244, DM-7256, DM-7249, DM-7245, DM-7250, DM-7258, DM-7264, DM-7132, DM-7262, DM-7289, DM-7300, DM-7301, DM-7302, DM-7303, DM-7304, DM-7288, DM-7293, DM-7292, DM-7261, DM-7294, DM-7295, DM-7296, DM-7297, DM-7298, DM-7305, DM-7306, DM-7307, DM-7308, DM-7309, DM-7328, DM-7299, DM-7311, DM-7391, DM-8030]
SpanSet
added¶
The SpanSet
class provides a compact representation of a collection of pixels using run-length encoding.
It is designed to serve as the back-end of an upgrade Footprint
system which will be provided in a future release.
Logging migration¶
All code has been migrated from the old pex_logging
system to the new lsst.log
framework.
[DM-6984, DM-6985, DM-6986, DM-6999, DM-7000, DM-7502, DM-7670, DM-7741, DM-8357, DM-8359]
Deblender API refactored¶
The meas_deblender
API has been refactored to make it easier to swap in alternative deblending algorithms for experimentation.
There should be no end-user visible effects of this change.
[DM-8694]
Removed old example & experimental code¶
Some example code and experiments with GPU-based processing that was being distributed with the stack had not been kept up to date with API changes. It has been removed.
Made the simpleShape
algorithm less verbose¶
Changed the type of exception thrown by measurement errors in the meas_extensions_simpleShape
package.
The new exception type results in the error being appropriately recorded, but does not flood the log with errors.
This is consistent with the behaviour of other measurement algorithms.
[DM-9011]
Made the pipe_drivers
package less verbose¶
The various scripts in the pipe_drivers
package were configured to print information about the current environment to standard output on startup.
This has now been made optional, and defaults to off.
Use the --batch-verbose
command line option to re-enable it.
[DM-9016]
Added “reduce” functions to ctrl_pool
¶
The ctrl_pool
multiprocessing framework now supports “reduce” functionality: the master node will scatter work to slaves and then combine their results.
The master is able to reduce data which has been returned from the slaves without waiting for all of the slaves to complete their tasks.
This is more efficient than the “map” functionality which was previously being used for many of the same tasks.
[DM-5989]