Release 13.0 (2017-02-28)¶
This release is based on the
w_2017_33 weekly build.
Major Functionality and Interface Changes¶
Added a measurement extension which performs Kron photometry¶
KronFluxAlgorithm measures the flux in an elliptical aperture as defined by Kron (1980).
The plugin is not enabled by default, but may be switched on by requesting
ext_photometryKron_KronFlux in measurement configuration.
Added a package version tracking system (“stack provenance”)¶
A package version tracking and checking capability has been included in the stack and integrated with the
This keeps track of what versions of packages are being used in order to prevent the user from unwittingly processing data with multiple, subtly different, versions of the code.
This provides a provisional implementation of a provenance tracking system: it is essential for short-term use, but will be rendered obsolete before we move to operations.
A more detailed description of this system is available on the LSST Community Forum.
Multi-band coadd processing now saves metadata¶
Metadata describing the operation of algorithms within the multi-band coadd processing framework is now correctly persisted along with their outputs.
Disabled oversampling in PSFEx¶
Experiments with Hyper Suprime-Cam have established that oversampling does not work well in PSFEx, one of the PSF estimation algorithms available in the stack. It has therefore been disabled in the default configuration used for LSST.
Added support for new Hyper Suprime-Cam filters¶
The HSC I2, R2, NB0468 and NB0926 filters are now defined in the
obs_subaru camera package.
Multi-band coadd processing no longer selects flagged bands as reference¶
The algorithm used to select the reference band in multi-band coadd processing has been updated to avoid bands with certain user-specified flags set. By default, these flags include the source having an interpolated centre, or errors detected with PSF, Kron or CModel fluxes.
Renamed the “afterburner” system to
The “afterburner” system was added to the stack in the previous (v12) release.
It provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not using pixel data).
To avoid ambiguity, this system has been renamed
Added sanity checking for source centroids¶
Centroid finding measurement algorithms were updated to ensure that the centroids they find fall within the footprint of the source being centroided. If not, the returned value is forced to correspond to the position of the most significant peak within the Footprint, and a flag is set to indicate the error.
Added ability to run
ctrl_pool processes without a batch system¶
--batch-type none (or
None) runs the command in your process as normal.
This means that tools relying on
ctrl_pool, such as the
pipe_drivers scripts, need not rely on MPI.
Added ability to match against reference catalog without rotation, offset or WCS fitting¶
DirectMatchTask matches sources against a reference catalog without performing any rotation, offset or WCS fitting.
This provides a simpler and less error-prone option for matching sources for quality assessment.
MeasureMergedCoaddSourcesTask has been modified to make use of this new
Added functionality to show where configuration is being set and overridden¶
You can now specify a command line argument of the form
--show history=config.foo.bar to see where configuration parameter
foo.bar is set and (if applicable) overridden (the leading
config. is optional).
N.B. pipe to
cat to turn of colourization (or investigate
frame now defaults to
None when creating a new
When using the
afw.display system to create a new
Display object, the
frame argument is now optional: it will default to
None if not specified.
Attribute access to
Display objects is now passed through to the underlying implementation¶
afw.display device types may provide implementation-specific functionality.
This may now be accessed directly through the
Display object, which will delegate to implementation-specific methods, if they exist, when a non-standard method is called on the
Added a new fitter for astrometric distortions¶
A new fitter task,
meas.astrom.FitSipDistortionTask has been added that will eventually replace the current fitter,
The new fitter should be more robust and certainly uses a better-motivated algorithm, but it has not yet received enough testing to be used as the default and hence must be enabled explicitly.
processCcd.py, this can be done with the following config file:
from lsst.meas.astrom import FitSipDistortionTask config.charImage.astrometry.wcsFitter.retarget(FitSipDistortionTask) config.calibrate.astrometry.wcsFitter.retarget(FitSipDistortionTask)
The new fitter is not compatible with
Unified exposure metadata¶
Exposures now contain a new
VisitInfo object that contains useful information about the overall visit (none of it specific to the individual CCD), including the exposure time, date and ID, boresight position and airmass.
See the community.lsst.org post on unified exposure metadata for more information.
Added convenient access to catalog metadata¶
It is now possible to read the metadata, length and schema of a catalog through a Butler interface without loading the whole catalog.
For example, given a particular data ID (
dataID) and catalog dataset type (
someCatalog), one can run:
butler.get("someCatalog_schema", dataId) butler.get("someCatalog_len", dataId) butler.get("someCatalog_md", dataId)
FITS headers are now read & returned in order¶
lsst.afw.image.readMetadata now returns a
PropertyList, which maintains the intrinsic ordering of the header keywords in the FITS file it is reading.
Previous versions used a
PropertySet, which is unordered and hence lost this information.
Added functionality for performing measurement without deblending¶
The deblender sometimes runs into trouble with galaxies in clusters, leading to inaccurate fluxes. To mitigate, we can now provide measurements performed without any deblending having taken place. To activate this functionality, specify the measurement plugins to be run without deblending in the undeblended registry attached to the measurement class (this is analogous to the plugins registry used when performing measurement on deblended sources). By default, no undeblended measurement is performed. This functionality is available in both forced and single-frame measurement.
Corrected order of operation when construction PSF-matched coadds¶
WarpAndPsfMatchTask has been updated so that, by default, it will first warp inputs to the coadd projection and then perform PSF matching; this is the reverse of its previous behaviour.
The new behaviour is preferable, but a configuration option makes it possible to revert to the old method on request.
Reduced galaxy “shredding” when deblending¶
If galaxies have substructure, such as face-on spirals, the process of identifying peaks can “shred” the galaxy into many pieces.
The templates of shredded galaxies are typically quite similar because they represent the same galaxy.
We try to identify these “degenerate” peaks by looking at the inner product (in pixel space) of pairs of templates.
If they are nearly parallel, we only keep one of the peaks an reject the other.
This behaviour is enabled by the
removeDegenerateTemplates option in
SourceDeblendConfig; by default, it is disabled.
Note that it is not safe to enable
removeDegenerateTemplates when using the multi-band coadd processing framework: it should only be used when calling
Corrected CModel table schema units¶
Units for the time field changed from seconds to second to comply with the Astropy convention.
SpherePoint class to represent positions on the sphere¶
SpherePoint class allows locations on the celestial (or any other) sphere to be represented and manipulated without the coordinate-system complications associated with
computeBBox method to all PSF classes¶
computeBBox on a PSF returns its bounding box. It is available for all PSF classes distributed with the stack.
PSF moments are now included in the
base_SdssShape measurement algorithm has been updated to record the adaptive moments of the PSF in its outputs.
This functionality may be disabled by setting the configuration parameter
Single frame processing no longer requires a PSF¶
ProcessCcdTask is used to process an exposure which does not have an associated
Psf and PSF measurement is disabled (i.e. the
doMeasurePsf configuration option is
False) we now assume a default PSF rather than raising an error.
Added basic ISR capability for Hyper Suprime-Cam¶
Normally, instrument signature removal (ISR) is carried out during pipeline processing as part of
Occasionally, it is convenient to perform ISR without full pipeline processing.
To make this more convenient,
hscIsr.py provides a simple front-end which simply performs ISR on HSC data without any further processing.
Corrected configuration dictionary comparisons¶
Resolved a bug whereby comparison of (intrinsically unordered) configuration dictionaries incorrectly compared their ordering rather than just their contents. This lead to unexpected failure of re-runs due to incorrectly identified “mismatched configurations”.
Restored capability to calculate temporary local backgrounds in source detection¶
SourceDetectionTask provides the optional capability of fitting and temporarily removing a local background before detecting sources.
This capability was accidentally disabled in the X/W16 release (v12); it has now been restored.
It is disabled by default.
Resolved a memory error in convolution¶
A memory handling error could cause our convolution code to generate incorrect results on some platforms (problems observed on macOS when the code is built without optimization; no errors seen on Linux, or when building with -O2 or -O3 on macOS). This appears to originate in the Boost Generic Image Library, an external dependency of the LSST stack. We have been unable to isolate the problem in Boost, but have worked around it for our particular use case.
Patched GalSim to raise an exception in some error conditions¶
On certain error conditions, GalSim, a third party package used by the LSST stack, generates an assertion failure which is hard for our code to handle appropriately. LSST’s version of GalSim has been patched to raise a more appropriate exception instead. This change has now been included in GalSim itself.
Ensured star selector results are reproducible¶
Resolved an issue whereby the star selection could change depending on the number of CPU cores in use.
Fixed single frame measurement so it can exploit multiple CPU cores¶
Resolved an issue whereby the
SingleFrameDriverTask failed to use more than one CPU core regardless of configuration.
Fixed typo in
A typo which rendered CoaddSrcTransformTask inoperable was resolved. In the process, the tests for this task were significantly upgraded to avoid a repeat of this, or other, problems.
Stop writing duplicate
AR_NAME headers when persisting data¶
When persisting objects to FITS, we were incorrectly writing duplicate
This has been resolved.
Added more rigor about units angle persistence¶
We now correctly write
TUNIT headers describing the units used when storing angles, and are more careful about checking units when reading them.
Resolved infinite recursion in
A bug which could lead to infinite recursion has been fixed has been discovered in the
Fixed a division by zero error in measurement of aperture corrections¶
Attempting to calculate an aperture correction for a measurement of zero flux would previously result in an unhandled division by zero. This has been resolved.
Fixed dataset lookup in
The multiband driver script was previously checking for the existence of the pre-calexp version of the coadd being processed.
This is not guaranteed to exist if the coadds were built using the driver scripts (where there is no need to persist both the pre-calexp and calexp versions), and it is only the calexp coadds that are used in the multiband tasks, so it is the calexp version whose existence should be checked for in
Fixed incorrect definition of mask when building fringe frames¶
The code in pipe_drivers to construct fringe frames,
FringeTask, was incorrectly using a mask plane when in fact a mask bit was required.
This has now been corrected.
Fixed failures in MPI-based processing due to large “pickles”¶
Resolved an issue whereby the
ctrl_pool distributed processing middleware was unable to transmit large volumes of data over MPI due to a limitation of the Python 2.x “pickle” system.
Fixed failure to retrieve optimizer state when fitting
Resolved an issue whereby the optimizer state was being read incorrectly, which could lead to fitting failures.
Resolved an issue with failed persistence of
An error was resolved which could cause
ConfigurableFields to be improperly persisted and hence render the configuration impossible to load.
Resolved an error with schema persistence in forced photometry¶
When performing forced photometry an incorrect catalog schema, which did not match the data, was being persisted to the Butler. This has been fixed.
Resolved issues due to scaling by non-finite dark time¶
If not explicitly set in the camera package, the dark time defaults to
When the dark time is used to scale an image, this would render the image unusable.
We now check for scaling by non-finite dark time and raise an appropriate error.
Build and Code Improvements¶
Ongoing migration from Boost to standard library equivalents¶
Continuing work started in the previous release, we have audited the stack for use of the Boost libraries. Where possible, Boost has been replaced with C++11 standard library equivalents.
ObjectSizeStarSelector are now properly logged¶
Previously, non-fatal NumPy warnings of the form
RuntimeWarning: invalid value encountered in less could appear on the console, but not be caught and logged anywhere.
It was therefore easy to miss problems arising when running the algorithm.
These warnings are now ingested into the LSST logging system, and are hence recorded properly.
Added unit tests for colorterm framework¶
A set of unit tests have been written which properly exercise the colorterm framework.
FlagHandler accessible from Python¶
FlagHandler system for efficiently and safely setting flags corresponding to measurement algorithm failures on source records is now accessible from Python.
Previously, it was only available for use in C++ code.
Added exact image comparison utility code¶
assertMaskedImagesEqual methods have been added to added to LSST’s testing framework.
These provide explicit mechanisms for checking for exact equality of image data.
CatalogCalculation is now used to handle aperture corrections¶
CatalogCalculation facility was added to the stack in the previous (v12) release (where it was referred to as “afterburners”).
It provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not pixel data).
This has now been used to clean up the logic used to apply aperture corrections within our regular data processing flow.
This old method for handling aperture corrections was a persistent source of bugs and confusion: these should be avoided by the new approach.
End-user builds are now optimized by default¶
Following an RFC,
sconsUtils was updated to enable optimization by default when compiling individual stack packages.
(Full-stack installations using a supported mechanism have always been optimized.)
Fixed a build failure of mpi4py on some versions of macOS¶
Added extra logic to ensure we target a version of the macOS SDK which is compatible both with the version of macOS installed and the Python interpreter in use.
obs_base package supersedes
obs_base package has been added to the distribution.
This contains base classes used by the various
obs_ camera packages.
The contents of the old
daf_butlerUtils package has been migrated to
obs_base, and the old package has been retired.
Centralized Butler dataset definitions¶
Butler dataset definitions which are common to multiple cameras have been consolidated in
obs_base rather than being repeated in each camera package.
Order in which SConscript files are read is configurable¶
sconsUtils build system now reads
SConscript files, which describe the build, in user-defined order.
This makes it possible to specify that some components of a package must be built before others.
ndarray an external package¶
ndarray package in the LSST stack now installs the upstream version 1.3.0.
Added backwards compatibility to
Due to changes in the way
ExposureCatalog were being persisted, it was no longer possible to load old data into a current version of the stack.
A compatibility layer has now been added.
Changed API in
Backwards incompatible API changes were made to the
SpatialCellCandidate class hierarchy to resolve build problems in some environments.
Details of the changes are available on RFC-250.
Python 3 support¶
Almost all of the Science Pipelines packages included in this release are compatible with both Python 2 and Python 3.
The notable exception is
pipe_drivers, which currently only works with Python 2.
[DM-6322, DM-7021, DM-6320, DM-7036, DM-7028, DM-7066, DM-7047, DM-7188, DM-7138, DM-7069, DM-7152, DM-7246, DM-7243, DM-7247, DM-7244, DM-7256, DM-7249, DM-7245, DM-7250, DM-7258, DM-7264, DM-7132, DM-7262, DM-7289, DM-7300, DM-7301, DM-7302, DM-7303, DM-7304, DM-7288, DM-7293, DM-7292, DM-7261, DM-7294, DM-7295, DM-7296, DM-7297, DM-7298, DM-7305, DM-7306, DM-7307, DM-7308, DM-7309, DM-7328, DM-7299, DM-7311, DM-7391, DM-8030]
SpanSet class provides a compact representation of a collection of pixels using run-length encoding.
It is designed to serve as the back-end of an upgrade
Footprint system which will be provided in a future release.
All code has been migrated from the old
pex_logging system to the new
Deblender API refactored¶
meas_deblender API has been refactored to make it easier to swap in alternative deblending algorithms for experimentation.
There should be no end-user visible effects of this change.
Removed old example & experimental code¶
Some example code and experiments with GPU-based processing that was being distributed with the stack had not been kept up to date with API changes. It has been removed.
simpleShape algorithm less verbose¶
Changed the type of exception thrown by measurement errors in the
The new exception type results in the error being appropriately recorded, but does not flood the log with errors.
This is consistent with the behaviour of other measurement algorithms.
pipe_drivers package less verbose¶
The various scripts in the
pipe_drivers package were configured to print information about the current environment to standard output on startup.
This has now been made optional, and defaults to off.
--batch-verbose command line option to re-enable it.
Added “reduce” functions to
ctrl_pool multiprocessing framework now supports “reduce” functionality: the manager node will scatter work to workers and then combine their results.
The manager node is able to reduce data which has been returned from the workers without waiting for all of the workers to complete their tasks.
This is more efficient than the “map” functionality which was previously being used for many of the same tasks.