Release Notes

RFC-618 Release (v18_1_0)

Source Identifier
Git tag 18.1.0
EUPS distrib v18_1_0

This is a minor feature release, as requested by RFC-618.

It contains two changes relative to the Summer 2019 Release (v18_0_0):

  • lsst.daf.base.PropertySet now supports storing unsigned 64-bit integers. For more information, refer to DM-20506 and DM-20664.
  • The version number of the eigen package has changed from 3.3.4.lsst1-1-g221959b to 3.3.7. There are no changes to the contents of this package; the version included with release 18.0.0 was incorrectly labelled.

There are no changes to interfaces or deprecations in this release.

See also:

Summer 2019 Release (v18_0_0)

Source Identifier
Git tag 18.0.0
EUPS distrib v18_0_0

See also:

As of this release, we have adopted the “Semantic Versioning” system, verison 2.0.0. In this system, releases are described by a three part number representing major (18 in this case), minor (0), and patch (0) version. The major version is incremented on incompatible API changes; the minor version when new functionality is added; and the patch version when bug fixes are made. For further details, refer to DMTN-106.

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. In addition, a large number of minor upgrades and stability improvements have been made across the codebase; in particular, we expect Jointcal to be significantly more stable in this release. For a complete list of changes made, see Tickets Addressed in Summer 2019 Release (v18_0_0).

Major Functionality Changes

Improved integration with the ngmix package

meas_extensions_ngmix, which provides an interface between the LSST codebase and Erin Sheldon’s ngmix package, has been substantially rewritten. This new package exposes more ngmix functionality, and may act as the basis for future development of the LSST measurement framework. For more information, refer to DM-16268.

New tasks for inserting “fake” sources

New command-line tasks have been added for inserting simulated sources into single epoch (processFakes.py) and coadded (insertFakes.py) data. For more information, refer to this community.lsst.org post.

FCGM can now use reference stars

By default, fgcmcal — LSST’s implementation of the Forward Global Calibration Method (Burke et al., 2018) — does not use an external reference catalog, but solves internally to produce a relative calibration. It has now been upgraded to optionally load a set of reference stars, and use them to produce an absolute calibration. For more information, refer to DM-16702.

DIAObjects now include basic lightcurve characterization

DIAObjects generated by the Alert Production system now include a basic set of features which extracted from the object’s light-curve (mean fluxes, best-fit slope, etc). While it is hoped that these features are intrinsically useful, they also serve to exercise the machinery which will be used to calculate the full set of features which is currently being developed in conjunction with the relevant science collaborations. For more details about this functionality, refer to DM-18318; for information about the ultimate set of features which will be computed, refer to DM-11962 and LSE-163.

Significant Interface Changes

Python interface to PropertySet and PropertyList changed

__getitem__, update, and get methods have been added to the Python interfaces of PropertySet and PropertyList. This means that they can be used in the same way as native Python dictionaries. It is also possible to store the value None in a PropertySet. For more information, refer to this community.lsst.org post and RFC-596.

Alert Production Pipeline command-line interface changed

The ap_pipe.py command will no longer try to create a “prompt products” database when it is executed. This change makes the pipeline better able to run in different environments, including large-scale testing and operations. An appropriate database should now be created in advance, either by using the make_ppdb.py command, or by configuring the pipeline to use an externally-provided database. For more information, refer to this community.lsst.org post and RFC-587.

New configurations for AstrometryTask source and reference selectors

The configuration options for single-frame astrometry (as implemented in lsst.meas.astrom.AstrometryTask) have changed. This fixes various bugs where (a) selections of the reference catalog were performed only in some modes of operation; and (b) multiple conflicting selections of the source catalog could be performed in some modes of operation. All obs package defaults have been updated to reflect the new changes; you need only worry about these changes if you have overridden the obs package defaults. For more information, refer to this community.lsst.org post and RFC-589.

lsst.afw.image.Calib removed

Calib, which provided only a photometric zeropoint per CCD, has been replaced with PhotoCalib, which provides a spatially-varying photometric model. Some Calib interfaces are supported by PhotoCalib, but full API compatibility is not possible; using the old-style interfaces is deprecated, and they will be removed following this release. PhotoCalib is able to read files persisted with Calib objects, so backwards compatibility of on-disk data is maintained. For more information, refer to RFC-289 and RFC-573.

Pending Deprecations

These packages/functions will be deprecated in the next major release.

Upcoming removal of “Generation 2” Middleware

The “Generation 3” middleware included in the previous release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality in late calendar year 2019 (likely version 19.0.0, but that remains to be confirmed). Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:

Upcoming removal of the obs_lsstSim package

The obs_lsst package, included in the previous release, obviates the need for the obs_lsstSim package. All LSST code is expected to transition to the new system later in summer 2019. Some work will be required to update old data repositories to the new system. After that, a final release will be made containing obs_lsstSim in late 2019, after which the package will be retired.

Upcoming removal of lsst.afw.geom classes that have been relocated to lsst.geom

As announced in v16.0 (Reworked “geom” package, replacing much of “afw.geom”) some primitives have been moved from afw.geom to geom. We currently provide aliases for compatibility purposes, but new code should use the geom package. These aliases will be removed after the version 19.0.0 will be released.

Deprecations

These packages/functions are deprecated and will not be available in the next major release.

Upcoming removal of lsst.afw.image.Calib compatibility API

This release includes (partial) backwards compatibility with the now removed (see above) Calib API. This will be removed before the next release.

Upcoming removal of the --silent argument to ap_verify.py

The --silent argument used to disable upload of metrics from ap_verify.py to SQuaSH. The capability to upload metrics has been removed from ap_verify.py (see DM-16536), but --silent has been retained as a no-op for compatibility reasons. It will be removed before the next release.

Upcoming removal of ip_isr functions from isrFunctions.py

These functions are replaced by functionality in lsst.meas.algorithms.Defects:

  • defectListFromFootprintList replaced by Defects.fromFootPrintList()
  • transposeDefectList replaced by Defects.transpose()
  • maskPixelsFromDefectList replaced by Defects.maskPixels()
  • getDefectListFromMask replaced by Defects.fromMask()

Upcoming removal of meas_algorithms functions from defects.py

  • policyToBadRegionList, policy defect files no longer supported.

Spring 2019 Release (v17_0)

Source Identifier
Git tag 17.0
EUPS distrib v17_0

See also:

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes, see Tickets Addressed in Spring 2019 Release (v17_0).

Major Functionality Changes

Proper motion support added to reference catalogs

Reference catalogs now support providing proper motions and parallaxes. If proper motions are available, then it will be applied to the specified epoch when the catalog is read. No correction for parallax is currently made. For more information, refer to this community.lsst.org post and RFC-368.

Parallelized image ingest

The new ingestDriver.py command makes it possible to ingest images to a data repository in parallel, providing significant speed-ups. For more information, refer to this community.lsst.org post.

Stand-alone instrument signature removal

The new runIsr.py command makes it possible to run instrument signature removal (ISR) from the command line. For more information, refer to this community.lsst.org post.

Alert production capabilities

The prototype Alert Production Pipeline is now included in the Science Pipelines software distribution. This comprises three new software packages:

  • dax_ppdb implements the Prompt Products Database;
  • ap_association implements multiple DIASource detections to compose DIAObjects;
  • ap_pipe provides command-line tasks to orchestrate the pipeline, running CCD processing, image differencing, and source association.

For more information, refer to this community.lsst.org post and RFC-521.

obs_lsst camera package

This release provides obs_lsst, a new camera description and configuration package which will provide support for data from the LSST Camera, ComCam, the Auxiliary Telescope camera, camera test stand 8, ImSim, and PhoSim. This package is expected to be updated and maintained for the indefinite future. Previous packages which provide some of the same functionality (e.g. obs_lsstSim) are provided with this release, but will ultimately be deprecated. For more information, refer to the documentation for lsst.obs.lsst.

DCR-corrected template construction

This release contains code for generating image differencing templates corrected for the effects of differential chromatic refraction (DCR). For information about the technique adopted, refer to DMTN-037; for documentation on the code, see lsst.ip.diffim.DcrModel.

The “Pessimistic Pattern Matcher b” is now the default astrometric matcher

This new matcher works consistently over the large range of stellar densities which LSST will observe. The algorithm and associated validation testing are described in DMTN-031; for more information, refer to this community.lsst.org post.

“Generation 3” middleware

This release contains early versions of the “Generation 3” Data Butler and `PipelineTask` middleware, which are being developed to satisify the requirements described in LDM-556. This software is still under heavy development and is not currently deployed for regular data processing by the Data Management team; it should be regarded here as an unsupported preview. For more information, refer to the documentation for lsst.pipe.base and lsst.daf.butler.

Significant Interface Changes

Reworked Task entrypoints

All Tasks now provide a consistent interface to their primary functionality. The Task provides a run() method, which takes as explicit arguments everything the task needs to get its job done. Tasks may also provide a runDataRef() method, which takes a Butler “dataRef” as its argument, unpacks it, and calls run(). Command line task runners call runDataRef(). For further details, refer to this community.lsst.org post and RFC-352.

Updates to output table field names

All table fields which previously had the suffix Sigma now have the suffix Err. This reflects usage in the Data Products Definition Document (DPDD; LSE-163). For more information, refer to this community.lsst.org post and RFC-333.

Further, source catalogs have been updated to refer to fluxes with the suffix instFlux (or “instrumental flux”), rather than simply flux (which is taken to refer to a calibrated flux). The table formerly known as InstFlux is now called GaussianFlux, which more correctly represents it contents. For more information, refer to this community.lsst.org post, RFC-322 and RFC-510.

Use inverted to get an inverse transform

Previously, the objects in the codebase representing coordinate transforms (e.g. lsst.geom.LinearTransform, astshim.Mapping) supported a heterogeneous mix of APIs for retrieving the inverse transform. These have now all been standardized to use the method inverted(). For more information, refer to this community.lsst.org post and RFC-500.

Changes to calibration flags

The following changes were made to table field names describing sources used for calibration:

  • calib_psfCandidate was renamed calib_psf_candidate;
  • calib_psfUsed was renamed calib_psf_used;
  • calib_astrometryUsed was renamed calib_astrometry_used.

For more information, refer to this community.lsst.org post and RFC-498.

Changes to blendedness flags

The following changes were made to table field names describing source blendedness:

  • base_Blendedness_raw_instFlux was renamed to base_Blendedness_raw;
  • base_Blendedness_abs_instFlux was renamed to base_Blendedness_abs;
  • base_Blendedness_raw_instFlux_child was renamed to base_Blendedness_raw_child_instFlux;
  • base_Blendedness_raw_instFlux_parent was renamed to base_Blendedness_raw_parent_instFlux;
  • base_Blendedness_abs_instFlux_child was renamed to base_Blendedness_abs_child_instFlux;
  • base_Blendedness_abs_instFlux_parent was renamed to base_Blendedness_abs_parent_instFlux;
  • deblend_psfFlux was renamed to deblend_psf_instFlux.

For more information, refer top this community.lsst.org post and RFC-534.

Task metadata is now written as YAML

Previously, metadata was stored as a mix of YAML and Boost.serialization; now, only the former is used. For more information, refer to this community.lsst.org post.

Changes to wrapping C++ code into Python

DM uses pybind11 to expose functionality written in C++ to the Python level. This release rethinks the way pybind11 is used, to increase robustness and reduce the size of compiled binaries. Although it is possible to mix the old and the new techniques within the same software stack, packages must be updated to the new system atomically. For more information, refer to this community.lsst.org post and the pybind11 section in the Developer Guide.

Changes to metadata extraction from file headers

The new astro_metadata_translator package provides a unified interface for extraction of standardized information from image file headers. All camera support (“obs”) packages are being updated to use this new system, which will be required for the next generation of DM middleware. For more information, refer to this community.lsst.org post.

Revised logging system

Log messages generated by the standard Python logging module during the execution of command-line tasks are now forwarded to the lsst.log system. Furthermore, utility functions have been added to make it easy to check that the expected log messages have been generated during testing. For more information, refer to this community.lsst.org post.

Pending Deprecations

Upcoming removal of lsst.afw.image.Calib

This is the final major release which provides the lsst.afw.image.Calib photometric calibration object. Begining with 18.0, Calib, which provides only a photometric zeropoint per CCD, will be replaced with PhotoCalib, which provides a spatially-varying photometric model. Some Calib interfaces are supported by PhotoCalib, but full API compatibility is not possible. By the time this transition is made, PhotoCalib will be able to read files persisted with Calib objects, so backwards compatibility of on-disk data will be maintained. For more information, refer to RFC-289 and RFC-573.

Upcoming removal of “Generation 2” Middleware

The “Generation 3” middleware included in this release is ultimately intended to supplant the current (“Generation 2”) Data Butler and command-line task functionality. We expect to deliver a final major release supporting the Generation 2 functionality in late calendar year 2019. Following that release, the “Generation 2” middleware will be removed from the codebase. This will include:

Upcoming removal of the obs_lsstSim package

With the introduction of the new obs_lsst package, there is no longer a need for the obs_lsstSim package. Current work should move to using obs_lsst as soon as possible. Repositories using obs_lsstSim will need to be reprocessed using obs_lsst. For the most part this transition will be transparent to end users. The only code updates needed are in the simulations related packages. The porting work for simulations is planned for the Summer of 2019. We believe that obs_lsstSim can be archived after the release in late calendar year 2019.

17.0.1 patch release

This patch fixes an issue that affects users of the lsst.display.firefly display backend on the Notebook Aspect of the LSST Science Platform. The viewer ID was not being set. See DM-18167 for additional information.

Summer 2018 Release (v16_0)

Attention

This is the first major release that does not support Python 2.

Source Identifier
Git tag 16.0
EUPS distrib v16_0

See also:

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes, see Tickets Addressed in 2018 Summer Release (v16_0).

Major Functionality and Interface Changes

Reworked “geom” package, replacing much of “afw.geom”

The Angle, Point, Extent, SpherePoint, Box, LinearTransform and AffineTransform primitivies have been moved from afw.geom to geom. Aliases for compatibility purposes remain within afw for this release, but new code should use the geom package. For further details, refer to this community.lsst.org post.

DECam ingest now defaults to raw image

ingestImagesDecam.py (encapsulating DecamIngestTask) now ingests raw images by default instead of “instcals”. This change means that DecamIngestTask behaves more like IngestTask and other ingestion tasks. For further details, refer to this community.lsst.org post.

Python 2 is no longer supported

This release is no longer tested on or expected to work with Python 2. For further details, refer to this community.lsst.org post.

Improvements to the StarSelector interface

The call signatures for source and star selectors have been modernized and unified as described on RFC-198. Some legacy code has not yet adopted the new interface.

Automatically select the \(N\) images with the best seeing when building templates

The MaxPsfWcsSelectImagesTask can now automatically determine the thresholds needed to select images with an appropriate PSF when building templates, rather than having them specified by the user. For details, refer to DM-11953.

Spring 2018 Release (v15_0)

Attention

This will be the last major release that supports Python 2. From release v16.0 (Fall 2018) on, only Python 3 will be explicitly supported. Minor (bugfix) releases of v15 will continue to support Python 2.

Source Identifier
Git tag 15.0
EUPS distrib v15_0

See also:

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes, see Tickets Addressed in Spring 2018 Release (v15_0).

Major Functionality and Interface Changes

Image differencing algorithm improvements

Algorithms appropriate for differencing in a variety of contexts have been implemented as tasks in the LSST stack. These various algorithms and configurations have been compared. The results are written up in DMTN-061. This represents a significant step forward in being able to determine the baseline image differencing algorithm. Relevant code changes are in the ip_diffim package.

Performance improvements in coaddition

Now all coaddition algorithms have significantly reduced memory footprints.

Significantly improved artifact rejection in coaddition

Coaddition algorithms that do artifact clipping can now handle artifacts that overlap from epoch to epoch. SafeClipAssembleCoaddTask and CompareWarpAssembleCoaddTask are the two examples.

Full focal plane sky subtraction

There are now tasks to create and apply models of the sky that extend over the entire field of view. View notes in the LSST Community form post.

Replace all Wcs classes with the AST backed SkyWcs

The last release introduced a transform system backed by the AST package. Since that release the stack has been converted to using that system in all contexts where a world coordinate system is required.

Plotting frontend for Firefly

This release includes, for the first time, the package that allows the LSST plotting abstraction layer to plot directly in the Science Portal plotting tool, Firefly. View display_firefly on GitHub.

Lossless compression on by default

Lossless compression is turned on by default when persisting any image-like data product. Read the decision and view the notes in this LSST Community forum post and links therein.

Changes to command-line tasks

Command-line tasks now handle clobbering of versions, data, and configs in a more intuitive way. E.g. output repositories are now expected to differ from the input repository. This eliminates the need to explicitly turn on clobbering when making multiple runs to different outputs (reruns) when using the same inputs. Additional details are in an LSST Community forum post.

Updated pre-requisites

Pre-requisites for installing the science pipelines have been updated. Of note are that numpy 1.13, astropy 2.0 and matplotlib 2.0 are all required. The baseline version of Python is now Python 3.6. See the announcement in this post on the LSST Community forum.

Spring 2017 Release (v14_0)

Source Identifier
Git tag 14.0
EUPS distrib v14_0

See also:

These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. For a complete list of changes, see Tickets Addressed in Spring 2017 Release (v14_0).

Major Functionality and Interface Changes

Detection: only add flags.negative to catalog schema if needed

Rather than always including this flag in the results of source measurement, it is now only provided if the thresholdPolarity argument is set to both. Otherwise, this flag is omitted altogether.

[DM-903]

New Footprint system

Footprints are used to represent contiguous areas of pixels on an image, corresponding e.g. to source detections. The Footprint system has been comprehensively rewritten for performance, flexibility and ease of use in this release. This change involves significant changes to the Footprint API: see community.lsst.org for a description of the changes and a migration guide.

[DM-3559]

Change of argument order in single frame measurement

For consistency with other code and documentation, the order of the arguments of the run() method of SingleFrameMeasurementTask has been reversed.

[RFC-273, DM-3804]

More statistical operations possible for assembling coadds

Rather than simply taking the (possibly clipped) mean, it is now possible to specify other operations, such as the median, to be used for combining pixel data when assembling coadds.

[DM-4158]

Hooks for injecting simulated sources

Hooks are now available in ProcessCcdTask for inserting simulated sources into a processing run. These hooks are designed to enable the easy integration of external packages supporting this functionality (e.g. SynPipe).

[DM-5310, DM-9998]

Increased mask plane depth

It is now possible to include 32 separate mask planes with each image.

[DM-7477]

Option to disable template smoothing in deblender

Under some circumstances, smoothing of deblend templates can have a detrimental effect on output quality. This smoothing is now optional.

[DM-8059, DM-9856]

New matcher algorithm

A new algorithm — “match pessimistic B” — for matching sources to reference catalog objects, typically used when performing photometry or astrometry, has been added to the release. This matcher is not currently enabled by default: it should be regarded at this stage as a technology preview.

[DM-8110, DM-8111, DM-8112, DM-8113, DM-8645, DM-9623, DM-9751]

AST-backed WCS class

A new representation of the celestial world coordinate system based on the Starlink AST library has been introduced. This will ultimately replace our previous WCS implementation; the transition is not yet complete at the time of this release.

[DM-8440]

C++ code is now exposed to Python using Pybind11

The technology used to wrap our low-level C++ functionality into Python has been switched from Swig to Pybind11. Pybind11 provides a more convenient environment for developers and enables several API improvements in our Python code. A summary of the changes is available on community.lsst.org

[DM-8467]

PSF-matched coadds are an independent data product

That is, rather than generating a single “coadd” data product which may be configured to be either direct or PSF-matched, it is now possible to generate both PSF-matched and direct coadds in a single pipeline execution.

[DM-8491, DM-10004]

Flag sources used in calibration

We now store flags indicating which sources were used in astrometric and/or photometric calibration. This is analogous to the flags used to indicate sources used in PSF modeling.

[DM-9050]

Convolved flux measurement

The meas_extensions_convolved package has been added to the lsst_distrib distribution. This convolves images to a specified PSF before measuring aperture fluxes. This may be useful when, for example:

  1. Measuring the flux that would be obtained with a fiber spectrograph, by using an aperture equal to the size of the spectrograph fiber and measuring with multiple target seeings to allow interpolation of the flux at the seeing of the spectroscopic observation.
  2. Measuring the color of galaxies in variable seeing, by using a common aperture and target seeing in each band.

[RFC-294, DM-9394]

Include pupil information in camera geometry

The afw.cameraGeom.Camera object now provides an API for accessing a model of the pupil obscuration.

[RFC-320, DM-9572]

Joint calibration

The Jointcal package, which provides for simultaneous photometric and astrometric calibration over multiple exposures, has been added to the lsst_distrib distribution. While we believe that Jointcal works well, note that it is still under heavy development and is not currently run as part of regular processing by the LSST development team: regard this package as a technology preview.

[DM-9752]

FITS HDUs are now 0-indexed

The previous LSST convention was to use 1-based indexing for HDUs. That is, previously, the first HDU in the file was referred to as HDU 1, with 0 being a special value taken to mean “the first non-empty HDU”. Now the first HDU in the file is HDU 0; the special value INT_MIN is used to refer to the first non-empty HDU.

[RFC-304, DM-9952]

Blendedness calculation is run by default

The default configuration has been updated so that “blendedness” is always calculated unless it has been specifically disabled by the user.

[DM-10506]

Spatially-varying Alard & Lupton image decorrelation

It is now possible to perform spatially-varying decorrelation when performing Alard & Lupton (1998) style image differencing. See DMTN-021 for details.

[DM-10804]

Useful string representation of tabular data

afw::tables and the records contained within them now provide a useful string representation. This is particularly useful when, for example, working in an interactive Python environment.

[DM-6718]

Bug Fixes

Command line tasks now provide standard Unix return codes

When a command line task fails, it will now return a non-0 exit code to the calling shell, in line with standard Unix conventions.

[DM-4141]

Deterministic package version checking

The codebase keeps track of which software versions were used to calculate a particular set of outputs as a means of tracking data provenance. However, in some circumstances this list was being stored inconsistently, leading to version mismatches being reported where non existed. This has now been resolved.

[DM-9439]

Correct CModel priors

CModel priors are now weighted correctly relative to the likelihood.

[DM-9795]

Correct blendedness debiasing calculation

Resolved a mathematical error.

[DM-10237]

Fixed order of operations when using temporary local backgrounds in detection

Resolved an issue which could result in missed isolated-object detections and incorrect Footprints for large objects.

[DM-10271]

Correct number of bands used for peak culling

During multi-band coadd processing, peaks may be rejected (or “culled”) if they have been detected in insufficiently many bands. A book-keeping error led to the number of bands in which a peak had been detected to be recorded incorrectly, and hence to the wrong peaks being culled. This has now been corrected.

[DM-11625]

Build and Code Improvements

YAML-based policy

The “policy” files associated with standard stack cameras, which define where and how datasets are persisted by the Butler, were updated to use a new YAML-based syntax.

[DM-7363]

Full support for Python 3.5

All of the Science Pipelines code included in this release is tested with both Python 2.7 and Python 3.5. It should also support Python 3.6, although that has not been rigorously tested.

[DM-7756]

More Robust Handling of Flagged Measurements

The FlagHandler mechanism for setting flags indicating the status of particular measurements was enhanced to improve its robustness and ensure that flags are always set consistently.

[DM-9249]

Support for recent NumPy

Recent versions of NumPy have introduced a number of changes which were incompatible with usage in the Science Pipelines code. The codebase has now been adapted to work around these changes.

[DM-9316, DM-10926]

Automatic printing of C++ backtrace on segmentation faults

In the unfortunate event of a segmentation fault in pipelines code, a backtrace will now automatically be printed. This can be forwarded to the developers in the event of a problem.

[DM-10846]

Fall 2016 Release (v13_0)

Source Identifier
Git tag 13.0
EUPS distrib v13_0

See also:

Major Functionality and Interface Changes

Added a measurement extension which performs Kron photometry

The KronFluxAlgorithm measures the flux in an elliptical aperture as defined by Kron (1980). The plugin is not enabled by default, but may be switched on by requesting ext_photometryKron_KronFlux in measurement configuration.

[DM-3440, DM-6561]

Added a package version tracking system (“stack provenance”)

A package version tracking and checking capability has been included in the stack and integrated with the CmdLineTask system. This keeps track of what versions of packages are being used in order to prevent the user from unwittingly processing data with multiple, subtly different, versions of the code. This provides a provisional implementation of a provenance tracking system: it is essential for short-term use, but will be rendered obsolete before we move to operations. A more detailed description of this system is available on the LSST Community Forum.

[DM-3372]

Multi-band coadd processing now saves metadata

Metadata describing the operation of algorithms within the multi-band coadd processing framework is now correctly persisted along with their outputs.

[DM-4991]

Disabled oversampling in PSFEx

Experiments with Hyper Suprime-Cam have established that oversampling does not work well in PSFEx, one of the PSF estimation algorithms available in the stack. It has therefore been disabled in the default configuration used for LSST.

[DM-6982]

Added support for new Hyper Suprime-Cam filters

The HSC I2, R2, NB0468 and NB0926 filters are now defined in the obs_subaru camera package.

[DM-7045, DM-7510, DM-8913]

Multi-band coadd processing no longer selects flagged bands as reference

The algorithm used to select the reference band in multi-band coadd processing has been updated to avoid bands with certain user-specified flags set. By default, these flags include the source having an interpolated centre, or errors detected with PSF, Kron or CModel fluxes.

[DM-7044]

Renamed the “afterburner” system to CatalogCalculation

The “afterburner” system was added to the stack in the previous (v12) release. It provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not using pixel data). To avoid ambiguity, this system has been renamed CatalogCalculation.

[DM-6919, DM-7421]

Added sanity checking for source centroids

Centroid finding measurement algorithms were updated to ensure that the centroids they find fall within the footprint of the source being centroided. If not, the returned value is forced to correspond to the position of the most significant peak within the Footprint, and a flag is set to indicate the error.

[DM-4926]

Added ability to run ctrl_pool processes without a batch system

The option --batch-type none (or None) runs the command in your process as normal. This means that tools relying on ctrl_pool, such as the pipe_drivers scripts, need not rely on MPI.

[DM-7900, DM-7943]

Added ability to match against reference catalog without rotation, offset or WCS fitting

The new DirectMatchTask matches sources against a reference catalog without performing any rotation, offset or WCS fitting. This provides a simpler and less error-prone option for matching sources for quality assessment. The MeasureMergedCoaddSourcesTask has been modified to make use of this new DirectMatchTask.

[DM-7117]

Added functionality to show where configuration is being set and overridden

You can now specify a command line argument of the form --show history=config.foo.bar to see where configuration parameter foo.bar is set and (if applicable) overridden (the leading config. is optional). N.B. pipe to cat to turn of colourization (or investigate lsst.pex.config.history.Color).

[DM-5421, DM-7490]

frame now defaults to None when creating a new Display

When using the afw.display system to create a new Display object, the frame argument is now optional: it will default to None if not specified.

[RFC-241, DM-7848]

Attribute access to Display objects is now passed through to the underlying implementation

Some afw.display device types may provide implementation-specific functionality. This may now be accessed directly through the Display object, which will delegate to implementation-specific methods, if they exist, when a non-standard method is called on the Display.

[RFC-241, DM-7848]

Added a new fitter for astrometric distortions

A new fitter task, meas.astrom.FitSipDistortionTask has been added that will eventually replace the current fitter, meas.astrom.FitTanSipWcsTask. The new fitter should be more robust and certainly uses a better-motivated algorithm, but it has not yet received enough testing to be used as the default and hence must be enabled explicitly. When running processCcd.py, this can be done with the following config file:

from lsst.meas.astrom import FitSipDistortionTask
config.charImage.astrometry.wcsFitter.retarget(FitSipDistortionTask)
config.calibrate.astrometry.wcsFitter.retarget(FitSipDistortionTask)

The new fitter is not compatible with meas.astrom.ANetAstrometryTask.

[DM-3549]

Unified exposure metadata

Exposures now contain a new VisitInfo object that contains useful information about the overall visit (none of it specific to the individual CCD), including the exposure time, date and ID, boresight position and airmass. See the community.lsst.org post on unified exposure metadata for more information.

[RFC-199, DM-5503]

Added convenient access to catalog metadata

It is now possible to read the metadata, length and schema of a catalog through a Butler interface without loading the whole catalog. For example, given a particular data ID (dataID) and catalog dataset type (someCatalog), one can run:

butler.get("someCatalog_schema", dataId)
butler.get("someCatalog_len", dataId)
butler.get("someCatalog_md", dataId)

[DM-8211, DM-8212]

FITS headers are now read & returned in order

lsst.afw.image.readMetadata now returns a PropertyList, which maintains the intrinsic ordering of the header keywords in the FITS file it is reading. Previous versions used a PropertySet, which is unordered and hence lost this information.

[DM-8213]

Added functionality for performing measurement without deblending

The deblender sometimes runs into trouble with galaxies in clusters, leading to inaccurate fluxes. To mitigate, we can now provide measurements performed without any deblending having taken place. To activate this functionality, specify the measurement plugins to be run without deblending in the undeblended registry attached to the measurement class (this is analogous to the plugins registry used when performing measurement on deblended sources). By default, no undeblended measurement is performed. This functionality is available in both forced and single-frame measurement.

[DM-6785, DM-7889]

Corrected order of operation when construction PSF-matched coadds

WarpAndPsfMatchTask has been updated so that, by default, it will first warp inputs to the coadd projection and then perform PSF matching; this is the reverse of its previous behaviour. The new behaviour is preferable, but a configuration option makes it possible to revert to the old method on request.

[DM-8088]

Reduced galaxy “shredding” when deblending

If galaxies have substructure, such as face-on spirals, the process of identifying peaks can “shred” the galaxy into many pieces. The templates of shredded galaxies are typically quite similar because they represent the same galaxy. We try to identify these “degenerate” peaks by looking at the inner product (in pixel space) of pairs of templates. If they are nearly parallel, we only keep one of the peaks an reject the other. This behaviour is enabled by the removeDegenerateTemplates option in SourceDeblendConfig; by default, it is disabled. Note that it is not safe to enable removeDegenerateTemplates when using the multi-band coadd processing framework: it should only be used when calling SourceDeblendTask directly.

[DM-4550]

Corrected CModel table schema units

Units for the time field changed from seconds to second to comply with the Astropy convention.

[DM-6282]

Added the SpherePoint class to represent positions on the sphere

The SpherePoint class allows locations on the celestial (or any other) sphere to be represented and manipulated without the coordinate-system complications associated with Coord.

[DM-5529]

Added computeBBox method to all PSF classes

Calling computeBBox on a PSF returns its bounding box. It is available for all PSF classes distributed with the stack.

[DM-8175]

PSF moments are now included in the SdssShape results

The base_SdssShape measurement algorithm has been updated to record the adaptive moments of the PSF in its outputs. This functionality may be disabled by setting the configuration parameter doMeasurePsf to False.

[DM-6928, DM-8714]

Single frame processing no longer requires a PSF

When ProcessCcdTask is used to process an exposure which does not have an associated Psf and PSF measurement is disabled (i.e. the doMeasurePsf configuration option is False) we now assume a default PSF rather than raising an error.

[DM-6045]

Added basic ISR capability for Hyper Suprime-Cam

Normally, instrument signature removal (ISR) is carried out during pipeline processing as part of ProcessCcdTask. Occasionally, it is convenient to perform ISR without full pipeline processing. To make this more convenient, hscIsr.py provides a simple front-end which simply performs ISR on HSC data without any further processing.

[DM-9186]

Bug Fixes

Corrected configuration dictionary comparisons

Resolved a bug whereby comparison of (intrinsically unordered) configuration dictionaries incorrectly compared their ordering rather than just their contents. This lead to unexpected failure of re-runs due to incorrectly identified “mismatched configurations”.

[DM-6661]

Restored capability to calculate temporary local backgrounds in source detection

Our SourceDetectionTask provides the optional capability of fitting and temporarily removing a local background before detecting sources. This capability was accidentally disabled in the X/W16 release (v12); it has now been restored. It is disabled by default.

[DM-5323]

Resolved a memory error in convolution

A memory handling error could cause our convolution code to generate incorrect results on some platforms (problems observed on macOS when the code is built without optimization; no errors seen on Linux, or when building with -O2 or -O3 on macOS). This appears to originate in the Boost Generic Image Library, an external dependency of the LSST stack. We have been unable to isolate the problem in Boost, but have worked around it for our particular use case.

[DM-5822]

Patched GalSim to raise an exception in some error conditions

On certain error conditions, GalSim, a third party package used by the LSST stack, generates an assertion failure which is hard for our code to handle appropriately. LSST’s version of GalSim has been patched to raise a more appropriate exception instead. This change has now been included in GalSim itself.

[DM-7114]

Ensured star selector results are reproducible

Resolved an issue whereby the star selection could change depending on the number of CPU cores in use.

[DM-7040]

Fixed single frame measurement so it can exploit multiple CPU cores

Resolved an issue whereby the SingleFrameDriverTask failed to use more than one CPU core regardless of configuration.

[DM-7134]

Fixed typo in CoaddSrcTransformTask

A typo which rendered CoaddSrcTransformTask inoperable was resolved. In the process, the tests for this task were significantly upgraded to avoid a repeat of this, or other, problems.

[DM-6635]

Stop writing duplicate AR_NAME headers when persisting data

When persisting objects to FITS, we were incorrectly writing duplicate AR_NAME headers. This has been resolved.

[DM-7221]

Added more rigor about units angle persistence

We now correctly write TUNIT headers describing the units used when storing angles, and are more careful about checking units when reading them.

[DM-7221]

Resolved infinite recursion in afw.table

A bug which could lead to infinite recursion has been fixed has been discovered in the afw.table library.

[DM-7377, DM-7385]

Fixed a division by zero error in measurement of aperture corrections

Attempting to calculate an aperture correction for a measurement of zero flux would previously result in an unhandled division by zero. This has been resolved.

[DM-7180]

Fixed dataset lookup in multibandDriver.py

The multiband driver script was previously checking for the existence of the pre-calexp version of the coadd being processed. This is not guaranteed to exist if the coadds were built using the driver scripts (where there is no need to persist both the pre-calexp and calexp versions), and it is only the calexp coadds that are used in the multiband tasks, so it is the calexp version whose existence should be checked for in multibandDriver.py.

[DM-7716]

Fixed incorrect definition of mask when building fringe frames

The code in pipe_drivers to construct fringe frames, FringeTask, was incorrectly using a mask plane when in fact a mask bit was required. This has now been corrected.

[DM-8033]

Fixed failures in MPI-based processing due to large “pickles”

Resolved an issue whereby the ctrl_pool distributed processing middleware was unable to transmit large volumes of data over MPI due to a limitation of the Python 2.x “pickle” system.

[DM-8021]

Fixed failure to retrieve optimizer state when fitting DoubleShapeletPsf

Resolved an issue whereby the optimizer state was being read incorrectly, which could lead to fitting failures.

[DM-6562]

Resolved an issue with failed persistence of ConfigurableField

An error was resolved which could cause ConfigurableFields to be improperly persisted and hence render the configuration impossible to load.

[DM-8273]

Resolved an error with schema persistence in forced photometry

When performing forced photometry an incorrect catalog schema, which did not match the data, was being persisted to the Butler. This has been fixed.

[DM-8221]

Resolved issues due to scaling by non-finite dark time

If not explicitly set in the camera package, the dark time defaults to NaN. When the dark time is used to scale an image, this would render the image unusable. We now check for scaling by non-finite dark time and raise an appropriate error.

[DM-9004]

Resolved an issue which could cause unnecessary re-linking during builds

The build environment in the sconsUtils package now consistently sorts source files, eliminating unnecessary rebuilds.

[DM-8728]

Build and Code Improvements

Ongoing migration from Boost to standard library equivalents

Continuing work started in the previous release, we have audited the stack for use of the Boost libraries. Where possible, Boost has been replaced with C++11 standard library equivalents.

[DM-5880, DM-6089, DM-6090, DM-6091, DM-6092, DM-6093, DM-6094, DM-6095, DM-6096, DM-6097, DM-6325]

Warnings in ObjectSizeStarSelector are now properly logged

Previously, non-fatal NumPy warnings of the form RuntimeWarning: invalid value encountered in less could appear on the console, but not be caught and logged anywhere. It was therefore easy to miss problems arising when running the algorithm. These warnings are now ingested into the LSST logging system, and are hence recorded properly.

[DM-5428]

Added unit tests for colorterm framework

A set of unit tests have been written which properly exercise the colorterm framework.

[DM-2837]

Made FlagHandler accessible from Python

The FlagHandler system for efficiently and safely setting flags corresponding to measurement algorithm failures on source records is now accessible from Python. Previously, it was only available for use in C++ code.

[DM-4009]

Added exact image comparison utility code

The assertImagesEqual and assertMaskedImagesEqual methods have been added to added to LSST’s testing framework. These provide explicit mechanisms for checking for exact equality of image data.

[DM-6641]

CatalogCalculation is now used to handle aperture corrections

The CatalogCalculation facility was added to the stack in the previous (v12) release (where it was referred to as “afterburners”). It provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not pixel data). This has now been used to clean up the logic used to apply aperture corrections within our regular data processing flow. This old method for handling aperture corrections was a persistent source of bugs and confusion: these should be avoided by the new approach.

[DM-5877]

End-user builds are now optimized by default

Following an RFC, sconsUtils was updated to enable optimization by default when compiling individual stack packages. (Full-stack installations using a supported mechanism have always been optimized.)

[RFC-202, DM-7010]

Fixed a build failure of mpi4py on some versions of macOS

Added extra logic to ensure we target a version of the macOS SDK which is compatible both with the version of macOS installed and the Python interpreter in use.

[DM-6133]

New obs_base package supersedes daf_butlerUtils

A new obs_base package has been added to the distribution. This contains base classes used by the various obs_ camera packages. The contents of the old daf_butlerUtils package has been migrated to obs_base, and the old package has been retired.

Centralized Butler dataset definitions

Butler dataset definitions which are common to multiple cameras have been consolidated in obs_base rather than being repeated in each camera package.

[RFC-204, DM-6858, DM-7049, DM-7677]

Order in which SConscript files are read is configurable

The sconsUtils build system now reads SConscript files, which describe the build, in user-defined order. This makes it possible to specify that some components of a package must be built before others.

[DM-7179]

Made ndarray an external package

The ndarray package in the LSST stack now installs the upstream version 1.3.0.

[DM-2005]

Added backwards compatibility to CoaddPsf

Due to changes in the way CoaddPsf and ExposureCatalog were being persisted, it was no longer possible to load old data into a current version of the stack. A compatibility layer has now been added.

[DM-8147]

Changed API in SpatialCellCandidate hierarchy

Backwards incompatible API changes were made to the SpatialCellCandidate class hierarchy to resolve build problems in some environments. Details of the changes are available on RFC-250.

[RFC-250, DM-8029]

Python 3 support

Almost all of the Science Pipelines packages included in this release are compatible with both Python 2 and Python 3. The notable exception is pipe_drivers, which currently only works with Python 2.

[DM-6322, DM-7021, DM-6320, DM-7036, DM-7028, DM-7066, DM-7047, DM-7188, DM-7138, DM-7069, DM-7152, DM-7246, DM-7243, DM-7247, DM-7244, DM-7256, DM-7249, DM-7245, DM-7250, DM-7258, DM-7264, DM-7132, DM-7262, DM-7289, DM-7300, DM-7301, DM-7302, DM-7303, DM-7304, DM-7288, DM-7293, DM-7292, DM-7261, DM-7294, DM-7295, DM-7296, DM-7297, DM-7298, DM-7305, DM-7306, DM-7307, DM-7308, DM-7309, DM-7328, DM-7299, DM-7311, DM-7391, DM-8030]

SpanSet added

The SpanSet class provides a compact representation of a collection of pixels using run-length encoding. It is designed to serve as the back-end of an upgrade Footprint system which will be provided in a future release.

[DM-7170, DM-8102, DM-8608]

Logging migration

All code has been migrated from the old pex_logging system to the new lsst.log framework.

[DM-6984, DM-6985, DM-6986, DM-6999, DM-7000, DM-7502, DM-7670, DM-7741, DM-8357, DM-8359]

Deblender API refactored

The meas_deblender API has been refactored to make it easier to swap in alternative deblending algorithms for experimentation. There should be no end-user visible effects of this change.

[DM-8694]

Removed old example & experimental code

Some example code and experiments with GPU-based processing that was being distributed with the stack had not been kept up to date with API changes. It has been removed.

[DM-7669, DM-8390]

Made the simpleShape algorithm less verbose

Changed the type of exception thrown by measurement errors in the meas_extensions_simpleShape package. The new exception type results in the error being appropriately recorded, but does not flood the log with errors. This is consistent with the behaviour of other measurement algorithms.

[DM-9011]

Made the pipe_drivers package less verbose

The various scripts in the pipe_drivers package were configured to print information about the current environment to standard output on startup. This has now been made optional, and defaults to off. Use the --batch-verbose command line option to re-enable it.

[DM-9016]

Added “reduce” functions to ctrl_pool

The ctrl_pool multiprocessing framework now supports “reduce” functionality: the master node will scatter work to slaves and then combine their results. The master is able to reduce data which has been returned from the slaves without waiting for all of the slaves to complete their tasks. This is more efficient than the “map” functionality which was previously being used for many of the same tasks.

[DM-5989]

Winter 2016 & X2016 Release (v12_0)

Source Identifier
Git tag 12.0
EUPS distrib v12_0

See also:

Major Functionality and Interface Changes

Add the option of excluding mask planes before coaddition

The user-configurable parameter removeMaskPlanes has been added to AssembleCoaddConfig. This is a list of mask planes which will not be propagated to the coadd; by default, the CROSSTALK and NOT_DEBLENDED mask planes are removed. [DM-4866]

Dumping task configuration parameters now includes documentation

That is, running ${TASK} ... --show config displays not only the names and values of the configuration but also associated documentation. [DM-3811]

Clean up interpolation tasks and implement useFallbackValueAtEdge

The interpImageTask in pipe_tasks has been refactored to implement a single run function for interpolation over a list of defects in an image. This run function will accept an afw image type of either MaskedImage or Exposure. A defects list can be passed in directly or the name of mask plane can be passed from which a defects list will be created. If a PSF is attached to the image, it will be used as the (required) argument for the interpolateoverDefects function in meas_algorithms used for the interpolation. Otherwise a FWHM (in pixels) can be provided or the defaultFWHM value in meas_algorithms’ GaussianFactory is used. Note that while the PSF is a required argument for meas_algorithms’ interpolateOverDefects function it is currently not being used, so it is not necessary to pass in an accurate PSF.

The useFallbackValueAtEdge option is now implemented. This tapers the interpolation to a fallbackValue towards the image edge. The fallbackValue can be set via config parameters to be either computed as a statistical representation of the image data (MEAN, MEDIAN, or MEANCLIP) or set by providing a specific value. Allowance for a negative fallbackValue is also controlled though a config parameter.

[DM-3677]

HSC backport: Avoid I/O race conditions config write out

This adds functionality to avoid potential I/O race conditions when running multiple simultaneous process. This is accomplished by writing to temporary files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, …), a --no-backup-config option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run. [DM-3911]

HSC backport: Introduce SafeClipAssembleCoaddTask which extends AssembleCoaddTask to make clipping safer

SafeClipAssembleCoaddTask does the following,

  • build both clipped and unclipped coadds and difference them first
  • detect on the difference
  • identify difference footprints that overlap appreciably with only one input image
  • use identified difference footprints as a mask into a final coadd
  • set a “clipped” bit on the final coadd for any pixel that did not include all input frames within the valid polygons.

To support the new coadd task, the pixel flags measurement plugin has been modified to accept two new configuration parameters. These new parameters, named masksFpCenter and masksFpAnywhere, each accept a list of mask planes. When the pixel flags measurement plugin searches for mask planes to set corresponding flags, it will now additionally search the user supplied mask plane lists. The masksFpCenter parameter specifies mask planes that, if found within the center of a footprint, will have a corresponding pixel flag set. The masksFpCenter parameter specifies mask planes that, if found anywhere in a footprint, will have a corresponding pixel flag set. The masksFpAnywhere parameter now has it’s defaults set within the stack which specify the clipped mask plane created by SafeClipAssembleCoaddTask.

SafeClipAssebleCoaddTask is now the default method for building a coadd within the LSST Stack. The assembleCoadd.py CommandLineTask now supports the --legacy command line flag which will allow the original AssembleCoaddTask to be run. If the legacy task is run, the clipped mask plane must be removed from the masksFpAnywhere configuration parameter.

[DM-2915]

HSC backport: Allow for some fraction of PSF Candidates to be reserved from PSF fitting

This adds functionality that allows for the reservation of a fraction of the PSF Candidates from PSF fitting. The reserved fraction can then be used to test for over-fitting, do cross-validation, etc.. To support this functionality, the run methods of CalibrateTask and MeasurePsfTask now accept the additional keyword argument expId. Dummy versions of this keyword were added to the SdssCalibrateTask and CfhtCalibrateTask to maintain a consistent API. [DM-3692]

Updated pipeline configuration based on Hyper Suprime-Cam experience

Configuration defaults and metadata through the LSST single-processing pipeline have been updated to match the current best practice established on HSC. Major changes include:

  • Support for narrow band filters;
  • Updated CCD defect lists;
  • Optimized (in terms of CPU time) deblender settings;
  • Avoiding failure in certain corner cases (e.g. operating on zero-length arrays, taking log10 of zero).

[DM-3942]

Define polygon bounds for CCDs based on vignetted regions

This adds a function in ip_isr’s isrTask.py to set a “valid polygon” for a given CCD exposure as the intersection of a polygon defined in focal plane coordinates and the CCD corners. It is currently being used in obs_subaru’s isr.py to set the polygon bounds (added in DM-2981) for a CCD exposure to include the non-vignetted regions. The settings for the vignetted region is in a separate config file so that it can be used in different places in the code. [DM-3259]

Introduce --rerun option for command line tasks

This new command line option streamlines the process of specifying input and output repositories for command line tasks. In its simplest form, the --rerun option is a shorthand for specifying an output repository: output is written to a location relative to the input. Thus

CmdLineTask /path/to/inputroot --rerun useroutput

is equivalent to

CmdLineTask /path/to/inputroot --output /path/to/inputroot/rerun/useroutput

Often, one task will process the output of a previous rerun. For this situation, --rerun provides a two-valued form which specifies relative locations of both input and output. In this mode,

CmdLineTask /path/to/inputroot --rerun process1:useroutput

is equivalent to

CmdLineTask /path/to/inputroot --input /path/to/inputroot/rerun/process1 --output /path/to/inputroot/rerun/useroutput

[RFC-95, DM-3371]

Introduce framework for injecting fake sources into data processing

A new boolean option (doFakes) and a retargetable task (fakes) have been introduced into the ProcessCcdTask. This (along with a new class called BaseFakeSourcesTask) sets up a frame work that others may use to introduce known fake sources into the data processing stream. However, this framework itself does not actually insert any fake data itself, but provides an interface others may use to define their own fake source injection task. To implement a fake injection task one must create a child class of BaseFakeSourcesTask (located in lsst.pipe.tasks.fakes) and overload the run method to do the work of injecting the sources. Each source that is injected should have a corresponding bit set in the FAKE maskplane which can be accessed with the bitmask attribute of BaseFakeSourcesTask. Once a task has been created, the config field fakes in processCcd must be retargeted to point to the user created task. Additionally, the task will not be run unless the doFakes configuration option in processCcd is set to True. However, if this variable is set to True, and the task is not retargeted processCcd will fail. [DM-3380]

Add convenience routines for working with tracts

Added two new data id containers:

  1. PerTractCcdDataIdContainer: determines the set of tracts each visit touches and adds a data reference with those tracts for each of the visit components. This allows for the user to run a command line task forcedPhotCcd.py for a given visit without having to know which tracts overlap the visit. Note this will also be used by meas_mosaic if/when it gets incorporated into the LSST Stack.
  2. TractDataIdContainer: generates a list of data references for patches within a given tract (effectively a “data reference list” that points to the entire tract). Note that, at the time of writing, this is only being used by a QA analysis script currently under development.

[DM-4373]

Warp images based on an XYTransform

Added the ability to warp images using a transformation defined by an lsst::afw::geom::XYTransform. [DM-4162]

Add getCoordSystem to Coord and add UNKNOWN CoordSystem enum

Added method getCoordSystem to lsst::afw::coord::Coord. Also added UNKNOWN=-1 as a new CoordSystem enum (the existing enums retain their existing value). DM-4606

Adapt joinMatchListWithCatalog to facilitate and simplify denormalizing a match list

The match lists created when performing image calibration (astrometry and photometry) are normalized (i.e. stripped down to a list of the matched reference and source ids and their distance) prior to being persisted. The ability to denormalize a match catalog is very useful (for post QA analysis, for example). This can now be done using the joinMatchListWithCatalog function in meas_algorithms’s LoadReferenceObjectsTask. It has been moved from meas_astrom’s ANetBasicAstrometryTask so that it can be easily accessed (requiring only that a reference object loader be initiated) and to allow it to work with any kind of reference catalog (i.e. other than a_net). [DM-3633]

Add a script for visualizing skymaps and CCDs

The skymap package now contains the script showVisitSkyMap.py which provides a convenient way of visualizing the tracts, patches and CCDs contained in a set of visits. [DM-4095]

Add functions to generate “unpacked matches” to and from a catalog

Functions have been added to lsst::afw::catalogMatches to provide the ability to convert a match list into a catalog and vice versa (this can be useful for post-processing analyses; QA analysis, for example). [DM-4729]

Add a measurement algorithm which records the focal plane coordinates of sources

The SingleFrameFPPositionPlugin measurement plugin, available in meas_base, records the positions of source centroids in focal plane coordinates (which may be convenient for plotting). This plugin is not enabled by default, but may be switched on by requesting base_FPPosition in measurement configuration. [DM-4234]

Add a measurement algorithm which records the Jacobian at the positions of sources

The SingleFrameJacobianPlugin calculates the ratio between the nominal Jacobian determinant at the source centroid (as determined by a user-specified pixel scale) and the actual Jacobian determinant as derived from the astrometric solution. This plugin is not enabled by default, but may be switched on by requesting base_Jacobian in measurement configuration. [DM-4234]

Add a measurement algorithm which records the number of input images contributing to a coadd sources

When run on a source detected on a coadd, SingleFrameInputCountPlugin records the number of input images which were stacked to create the coadd at the position corresponding to the source centroid. The plugin is referred to as base_CountInputs, and is enabled by default when performing measurement on coadded images. It is not appropriate to enable this plugin when processing single visit (i.e., not coadded) images. [DM-4235]

Add a measurement algorithm which records the variance at the positions of sources

The SingleFrameVariancePlugin records the median variance in the background around the position of the source being measured. The plugin is referred to as base_Variance and is enabled by default when performing single frame measurement. [DM-4235, DM-5427]

Add a measurement algorithm which records source flux in an aperture scaled to the PSF

The ScaledApertureFluxAlgorithm measures the flux in a circular aperture with radius scaled to some user-specified multiple of the PSF FWHM. This plugin is not enabled by default, but may be switched on by requesting the base_ScaledApertureFlux in measurement configuration. [DM-3257]

Add a measurement algorithm which quantifies the amount of “blendedness” of an object

The BlendednessAlgorithm measures the amount to which an object is blended. Both the flux and shape of each child object are compared to measurements at the same point on the full image. The size of the weight function used on both images is determined from the child object. The blendedness metric implemented is defined as 1-childFlux/parentFlux. The plugin is referred to as base_Blendedness and is not enabled by default. [DM-4847]

Add a “simple” shape measurement algorithm

The SimpleShape algorithm, provided in the meas_extensions_simpleShape package, computes the non-adaptive elliptical Gaussian-weighted moments of an image. The plugin is referred to as ext_simpleShape_SimpleShape and is not enabled by default. [DM-5284]

Add Hirata-Seljak-Mandelbaum shape measurement algorithms

The meas_extensions_shapeHSM package has been added to the distribution. This provides a series of measurement algorithms based on the work by Hirata and Seljak (2003) and Mandelbaum et al (2005). Please cite those works if publishing results based on this code. These algorithms are disabled by default; they can be enabled by requesting ext_shapeHSM_HsmShapeBj, ext_shapeHSM_HsmShapeLinear, ext_shapeHSM_HsmShapeKsb, ext_shapeHSM_HsmShapeRegauss, ext_shapeHSM_HsmSourceMoments and/or ext_shapeHSM_HsmPsfMoments in the measurement configuration. [DM-2141, DM-3384, DM-4780]

Add option to temporarily remove an interpolated background prior to detection

This has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background. The extra subtracted interpolated background is added back in after detection. Currently, the default setting for the config parameter doTempLocalBackround is set to False. [DM-4821]

Add function to average coordinates

Added function lsst.afw.coord.averageCoord, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinates. [DM-4933]

Integrate support for Hyper Suprime-Cam

The obs_subaru camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST’s continuous integration system. It will now be included as part of the lsst_distrib release. Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present. [DM-3518, DONE DM-4358, DM-5007]

Reimplement PSF Shapelet approximations for CModel

A new algorithm for computing multi-shapelet approximations (DoubleShapeletPsfApprox) has been added to meas_modelfit. This is simpler and more robust than the old algorithm, which has been renamed to GeneralShapeletPsfApprox. The new algorithm is recommended for production use, and is now the default. [DM-5197]

Propagate flags from individual visits to coadds

A task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits. [DM-4878, DM-5084]

Prototype Brighter-Fatter correction

Code for correcting for the Brighter-Fatter effect on CCDs is now available in the stack. It is enabled using the doBrighterFatter configuration option to IsrTask. It requires a pre-generated correction kernel. Calculation of this kernel is not currently performed within the stack: a prototype exists, and will be merged to the Calibration Products Pipeline in a future cycle. [DM-4837, DM-5082, DM-5130]

Aperture correction on coadd measurements

Aperture corrections are now enabled for measurements performed on coadds. [DM-5086]

Return grown Footprints from detection

By default, Footprints returned by SourceDetectionTask are now grown by a multiple of the PSF size. [DM-4410]

Enable measurement of “sky objects” in coadd processing

Sky objects correspond to source properties measured at positions when no objects have been detected. This enables us to better characterize the depth of the survey. This functionality is enabled by default; it can be disabled by setting nSkySourcesPerPatch to zero in the configuration of MergeDetectionsTask. [DM-4840, DM-5288]

Specification of output directory is now mandatory

When running a command line task which produces output it is now mandatory to specify an output directory (previously, if no output location was specified, data products were written back to the input repository). Output locations may be specified with the --rerun or --output command line options. More information is available on community.lsst.org. [DM-4236]

Bright object masks

Given an input catalog listing the known positions and sizes of bright objects, a bit is set in the mask plane for every pixel lying within the object’s footprint. [DM-4831]

CModel fitting improvements

CModel is a model fitting approach in which a pure exponential and pure de Vaucouleur are each fit separately, and then their linear combination is fit while the ellipse parameters are held fixed. Improvements in this release make CModel fitting faster and improves results on objects which are detected with an unphysically large likelihood radius. This has been achieved in three ways:

  • The initial approximate exponential fit that is used to determine the starting parameters and pixel region to use for the exp and dev fit now uses per-pixel variances;
  • The method used to determine the pixel region used in fitting has been adjusted to make smarter choices, using fewer pixels on average for all objects and many fewer pixels for unphysically large objects;
  • A new semi-empirical Bayesian prior on radius and ellipticity based on COSMOS distributions has been introduced.

[DM-4768]

Astropy Table views into LSST Catalog objects

Astropy Table views into LSST catalog objects can now be created. These views share underlying data buffers (aside from flag fields), making them read-write, but rows and columns added on either side will not be visible in the other. Two equivalent interfaces are available:

astropy_table = lsst_catalog.asAstropy()

and (in Astropy >= v1.2):

astropy_table = astropy.table.Table(lsst_catalog)

QTable objects can also be used, resulting in columns that use Astropy’s units package to enforce unit correctness. These interfaces have multiple options to control the details of the view, including how to handle columns that require copies; see the Python on-line help for asAstropy for more information.

While LSST’s catalog objects have features that make them particularly useful in building pipelines, Astropy’s are much more appropriate for most analysis tasks, and we strongly recommend using them for any analysis tasks that need to add columns to tables or combine columns from multiple tables.

[DM-5641, DM-5642, DM-5643]

Add an “afterburner” measurement facility

This new functionality makes it possible to register plugins to calculate quantities based on the results of pixel measurement algorithms. This might include, for example, star-galaxy separation or applying aperture corrections. Afterburners of this type are run after measurement plugins, and do not have access to pixel data. [DM-4887]

Tasks can now be kept in registries

Related sets of tasks should now be kept in a registry as per RFC-183, with a common abstract base class.

Tasks can now use an lsst.pex.config.RegistryField config field to specify a subtask if that subtask is in a registry (DM-6074). The task is built and used the same way as if it was specified in an lsst.pex.config.ConfigurableField, but retargeting and overriding config parameters is different. See task documentation for more information. See also How to Write a Task for guidelines for choosing between using lsst.pex.config.RegistryField and lsst.pex.config.ConfigurableField to hold a subtask.

PSF determiners are now tasks that inherit from an abstract base class lsst.meas.algorithms.PsfDeterminerTask (DM-6077). However, the effect on existing code was negligible because they were already configurables used from a registry. The way you retarget PSF determiners and override their config parameters remains unchanged.

Reimplemented the registry for star selectors that was lost in DM-5532: lsst.meas.algorithms.starSelectorRegistry (DM-6474)

One backwards incompatible change: in [DM-6474] MeasurePsfTask and MeasureApCorrTask both now specify their star selectors using an lsst.pex.config.RegistryField. This means the format for retargeting star selectors and overriding their config parameters has changed. The config override files in the various obs_ packages are updated accordingly.

New test utilities in afw: BoxGrid and makeRampImage

lsst.afw.geom.testUtils.BoxGrid divides a bounding box into nCol x nRow equal sized sub-boxes (as equal sized as possible for integer boxes that do not divide evenly) that tile the larger box and have the same type.

lsst.afw.image.testUtils.makeRampImage makes an image (ImageX where X is any available type) with values that increase linearly between specified limits (linearly to the extent possible, for integer images).

[DM-5462]

Correcting non-linearity

Introduced a standard way to correct non-linearity (linearize data) as part of Instrument Signature Removal (ISR). Linearization is performed by new functors in ip_isr:

  • LinearizeBase is an abstract base class. It is called with an image and the detector information and the correction is performed in place (like all other ISR corrections in IsrTask).
  • LinearizeSquared performs a simple square correction: corrImage = uncorrImage + c0*uncorrImage^2 where c0 is the first coefficient in in the linearity coefficients of the amp into catalog. This is the model used by obs_subaru for SuprimeCam and HSC.
  • LinearizeLookupTable uses a lookup table to determine an offset (read the code doc string for details). The lookup table is saved with the linearizer, but the linearizer also performs a sanity check against the provided detector when called.
  • You can easily add other linearizers as desired.
  • Each linearizer has a class variable LinearizationType, a string whose value should be used as the linearization type in the amplifier info catalog. The linearizer checks this value when performing linearization.

All detector in a camera must use the same type of linearizer. However linearization can easily be disabled on a detector-by-detector basis by setting linearity type to lsst.afw.cameraGeom.NullLinearityType. For a camera that does not need linearization, do this for all detectors.

Linearizers are obtained from the butler, like any other calibration product.

  • For LinearizeSquared and other linearizers that get coefficients from the amplifier info catalog, only one instance is needed for all detectors. In that case the simplest technique is to define map_linearize and bypass_linearize methods on the camera mapper to return an instance. See the obs_subaru package for an example.
  • For LinearizeLookupTable and other linearizers that store detector-specific data, the obs_ package developer must pickle one linearizer for each detector and make them available as dataset type “linearizer”.
  • If the camera does not want linearization then no “linearizer” dataset type is required because IsrTask realizes linearization is not wanted before it tries to unpersist the linearizer. You may leave IsrConfig.doLinearize set to its default value of True without significant performance penalty.

[DM-5462, RFC-164]

Amplifier information catalogs have changed

The format of amplifier information catalogs has changed. Your versions of afw and associated obs_ packages must be compatible or else you will get errors when building a camera mapper (thus when running any nearly any command-line task).

Amplifier information catalogs have a new field as of DM-6147: suspect level. If the value is not nan then pixels whose values are above this level are masked as SUSPECT.

The only cameras that specify a suspect level, so far, are HSC and SuprimeCam. However, a value can be set for any camera, if desired. SUSPECT is intended to indicate pixels with doubtful values due to errors that are difficult to correct accurately, e.g. a regime where linearity correction is not very stable.

In addition, saturation level in the amplifier information catalog is now a floating point value (instead of an integer) and a value of nan means ‘do not mask pixels as SAT.’

[DM-6147]

Changes in how background subtraction is done

Background estimation in Python is now done using different routines in meas_algorithms.

There is a new task SubtractBackgroundTask, with full documentation and a working example.

The existing function getBackground (which fits a background) is replaced by SubtractBackgroundTask.fitBackground. Changes from getBackground are:

  • getBackground could return None if the fit failed; in that situation fitBackround will raise RuntimeError instead of returning None.
  • The argument image was renamed to maskedImage, for clarity.
  • The config is not passed as an argument.
  • The debug display code uses different keys and is updated to use afw.display.

The existing function estimateBackground (which subtract a background from an exposure) is replaced by SubtractBackgroundTask.run. Changes from estimateBackground are:

  • You may pass in a background model (an lsst.afw.math.BackgroundList).
  • It returns a struct containing the updated background model.
  • The config is not passed as an argument.
  • The debug display code displays the unsubtracted image and uses different keys and is updated to use afw.display.

The task’s config SubtractBackgroundConfig replaces the old BackgroundConfig. The field algorithm may no longer be None; you must use the string "NONE", instead. See discussion on Community for details.

[DM-5323, RFC-155]

Star selectors have changed

Star selectors are now tasks. They were already configurable and many added logs; now they are standard tasks.

The star selector registry starSelectorRegistry was gone for awhile. Now that it is back, using a registry field from that registry is the preferred way to specify a star selector as a subtask of another task.

Added BaseStarSelectorTask (but for awhile it was called StarSelectorTask) an abstract base class for star selectors with the following methods:

  • selectStars an abstract method that takes a catalog of sources and returns a catalog of stars.
  • makePsfCandidates a concrete method that takes a catalog of stars (as returned by selectStars and produces PSF candidates; it also returns a sub-catalog of those stars that were successfully turned into PSF candidates (which is usually all of them).
  • run a concrete method that selects stars, makes them into PSF candidates and optionally flags the stars.

[RFC-154, DM-5532]

Backward-incompatible changes to ProcessCcdTask and subtasks

Code changes

  • ProcessCoaddTask is gone, along with all bin scripts that run it. Use the new Multi-Band code, instead.

  • ProcessCcdTask has been rewritten, so its config has changed. Config override files will need to be updated. This will be done for the obs_ packages as part of the merge, but if you have personal config override files then you will probably need to update them.

  • Camera-specific variants of ProcessCcdTask are gone. You will run processCcdTask.py to process images for all cameras.

  • For DECam processCcdTask.py will use the LSST Stack’s ISR by default. To read instcal files from the DECam Community Pipeline, replace the ISR task with DecamNullIsrTask by using a config override file containing the following:

    from lsst.obs.decam.decamNullIsr import DecamNullIsrTask
config.isr.retarget(DecamNullIsrTask)

  • A new dynamic dataset type is available for adding data ID arguments to the argument parser for command-line tasks: ConfigDatasetType obtains the dataset type from a config parameter.

  • Various subtasks have changed, including:

    • New camera-specific ISR task variants for SDSS and DECam: SdssNullIsrTask and DecamNullIsrTask.
    • New task DetectAndMeasureTask detects and deblends sources and performs single-frame measurement.
    • New task CharacterizeImageTask measures PSF and aperture correction, among other things.
    • CalibrateTask has been rewritten. It now performs deep detection and measurement, astrometry and photometry.
    • Camera-specific variants of CalibrateTask are gone.
    • ProcessImageTask (formerly a base class for ProcessCcdTask and ProcessCoaddTask) is gone.
Data product changes
  • icSrc no longer includes RA/Dec coordinate data, because the fit WCS is not available when the catalog is constructed. You will have to set the coord field yourself if you need it.
  • icExp and icExpBackground can optionally be written by CharacterizeImageTask. They are so close to calexp and calexpBackground that they are not written by default.
  • icMatch is no longer being written.
Algorithm changes
  • PSF is fit somewhat differently. The new task fits the PSF in using a configurable number of iterations. By default each iteration starts with a simple Gaussian PSF whose sigma matches the PSF of the previous fit, but you can use the actual PSF each time. Using a Gaussian causes convergence in 2 iterations. Using the fit PSF slows convergence.
  • Sources in the icSrc catalog should have a more consistent minimum SNR for varying seeing. The old code detected once, using a Gaussian PSF with FWHM set by a config parameter. The new code performs detection using the PSF in the final PSF iteration.
  • The default star selector for MeasurePsfTask is objectSize rather than sizeMagnitude. The objectSize star selector is preferred and was already being specified as an override by HSC.
  • The icSrc catalog is not matched to an astrometric reference catalog unless the star selector used to measure PSF can use the matches (which is unusual).
  • The astrometric and photometric solution now use the deeper src catalog instead of the shallower icSrc catalog, though with a new SNR cutoff whose default provides a depth similar to the icSrc catalog.
  • Fake source handling is temporarily missing; it will be re-added in DM-5310.

[DM-4692, DM-5348]

Bug Fixes

Persist LTVn headers as floating point numbers

When persisting to a FITS file, these header cards were previously, incorrectly, stored as integers. [DM-4133]

Fix bug when identifying existing peaks in a merge

If two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixed. [DM-2978]

Fix situation in which the getChildren method of SourceCatalog may return the wrong information

The getChildren method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren method was updated to raise an exception if the precondition of sorting is not met. [DM-2976]

Fix warping when the WCS have different coordinate systems

Warping assumed that the sky representation of both WCS was identical. [DM-4162]

Correct bad default minInitialRadius for CModel

The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications. [DM-3821]

Correct algebraic error in CModel uncertainty calculation

There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude. [DM-3821]

NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot

Previously, initializing these algorithms was only possible if a centroid was already defined. That was not only unnecessary, but also made them more complicated to use, particularly in testing. [DM-3940]

Update (some) example code to run with recent stack versions

Changes in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API. [DM-4125]

Fix a failure to appropriately log failed task execution

When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved. [DM-4218]

Updates to Skymap packages

Add functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and tract,patch indices. Functions include findClosestTractPatchList, findAllTract, and findTractPatchList which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively. [DM-3775]

Fix variance in coadded images

Warping images in order to coadd them loses variance into covariance. This is mitigated by scaling the variance plane of the coadd. The scaling was being applied incorrectly in some cases. This has now been fixed. [DM-4798]

Fix variance in deblended sources

The deblender incorrectly scaled the variance plane in deblended sources with the fraction of the total flux assigned to the source. This has been corrected. [DM-4845]

Fix logic for applying aperture corrections

This fixes a bug whereby the aperture corrections were being applied only after all the measurement plugins had run through, independent of their execution order. This resulted in plugins whose measurements rely on aperture corrected fluxes (i.e. with execution order > APCORR_ORDER) being applied prior to the aperture correction, leading to erroneous results. The only plugin that was affected by this at this time was base_ClassificationExtendedness. [DM-4836]

More uniform support for assigning to catalog columns

Assignment of scalars or NumPy arrays to columns of afw.table.Catalog objects (e.g. catalog["column"] = value) is now more uniformly supported across types (support was inconsistent before, and never allowed scalar or augmented assignment). Flag columns still do not support column assignment, and Flag column access still returns a copy, not a view, because Flag values are stored internally as individual bits within a larger integer. [DM-4856]

Upgraded WCSLIB to version 5.13

Version 5.13 of WCSLIB resolves memory corruption errors that could crash the stack in some circumstances. [DM-4904, RFC-89, DM-4946, DM-3793]

Fix rotation for instrument signature removal in obs_subaru

Approximately half of the HSC CCDs are rotated 180 deg with respect to the others. Two others have 90 deg rotations and another two have 270 deg rotations (see HSC CCD layout). The raw images for the rotated CCDs thus need to be rotated to match the rotation of their associated calibration frames (in the context of how they have currently been ingested) prior to applying the corrections. This is accomplished by rotating the exposure using the rotated context manager function in obs_subaru’s isr.py and the nQuarter specification in the policy file for each CCD. Currently, rotated uses afw’s rotateImageBy90 (which apparently rotates in a counter-clockwise direction) to rotated the exposure by 4 - nQuarter turns. This turns out to be the wrong rotation for the odd nQuarter CCDs. This issue fixes this bug, leading to much improved processing of HSC CCD’s 100, 101, 102, and 103. Note that, in the future, the ingestion of the calibration data will be updated such that no rotations are necessary (so they will then be removed from obs_subaru accordingly). [DM-4998]

Fix a silent failure to apply config overrides

When applying a config override, using a variable which hadn’t been defined should throw a NameError, which ultimately propagates to the end user to notify them that something has gone awry. This warning was being incorrectly suppressed. [DM-5729]

Correctly set flags for bad SdssShape measurements.

The SdssShape algorithm provides both shape and flux measurements. In some cases, a failed shape measurement could go un-noticed, resulting in an incorrect and unflagged flux measurement being associated with that source. This is now checked for, and bad fluxes are appropriately flagged. [DM-3935]

Fix CONSTANT background interpolation of bad data

When performing interpolation over bad data (e.g. every pixel masked), all interpolation types other than CONSTANT would return NaNs; CONSTANT would throw. This has now been changed so that CONSTANT also returns NaNs. [DM-5797]

Accommodate pixel padding when unpersisting reference catalog matches

The reference object loader in meas_algorithm’s loadReferenceObjects.py grows the bbox by the config parameter pixelMargin (padding to add to 4 all edges of the bounding box [pixels]) when setting the radius of the sky circle to be searched in the reference catalog. This is set to 50 by default but was not reflected by the radius parameter set in the metadata, which left open the possibility that some matches could reside outside the circle searched within the unpersisted radius. Additionally, the match metadata was being set after the exposure’s WCS had been updated, also leading to an inconsistency with the sky circle that was actually searched. We now ensure that the actual sky circle that was searched for reference objects is the one set and persisted in the match metadata. [DM-5686]

Correct misleading filter fallback error message

When failing to load a calib, if fallbackFilterName was not set, a confusing and apparently unrelated error message would be generated (Unknown value type for filter: <type 'NoneType'>). This has been corrected to properly inform the user about the issue. [DM-6151]

Build and Code Improvements

Work-around incompatibilities with NumPy 1.10

NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match. [DM-4063, DM-4071, DM-4238].

When building boost warn user if user-config.jam or site-config.jam exists

Building boost can fail if a user-config.jam or site-config.jam exist and have options which conflict with the LSST build configuration process. Introduce a warning message if either of these files are found to notify the user. [DM-4198]

Remove deprecated Task.display() method

This method has been deprecated since release 9.2 (S14). It has been removed from the codebase, and all stack code updated to directly interface with afw.display or to use helper functions defined in meas_astrom. [DM-4428]

Efficiency improvement in converting Masks to DefectLists

The previous version of routine was extremely memory intensive when large numbers of pixels were masked. [DM-4800]

Add a new task parallelization framework

The ctrl_pool package has been added to the LSST stack. This is a high-level parallelization framework used for distributing Task execution across a cluster, based on an MPI process pool. It is based on work carried out on Hyper Suprime-Cam. It is not intended to be the long-term solution to parallelized processing in the LSST stack, but meets our data processing needs until the fully-fledged parallelization middleware is available. [DM-2983, DM-4835, DM-5409]

Add parallel-processing top level tasks

The new pipe_drivers package builds upon ctrl_pool, above, to provide command-line scripts which coordinate distributed execution of the single-frame, coaddition and multiband processing steps using either the Python multiprocessing module or with a SLURM batch scheduler on a cluster. [DM-3368, DM-3369, DM-3370]

Adjust test tolerances to be compatible with MKL-based NumPy

Anaconda 2.5 ships, by default, with a version of NumPy built against Intel MKL rather than OpenBLAS. This can change some numerical results slightly, necessitating a change to test tolerances. [DM-5108]

Now possible to directly get a Filter’s canonical name and aliases

Added the convenience methods getCanonicalName and getAliases to lsst.afw.image.Filter, accessible from both C++ and Python. These return the canonical name and the aliases, respectively, of the filter. This information was previously only available through an awkward sequence of method calls. [DM-4816]

Fix build issues with recent clang

Recent releases of the clang C/C++ compiler, as shipped with Apple XCode, caused build failures in the stack. Although we believe this may be a problem with clang, we have worked around it within the stack code. We hope to track down the source of the error and, if appropriate, report it to the clang developers in future. [DM-5590, DM-5609]

Fix incorrect linking against Anaconda-provided libraries when using CMake

Some external packages—mariadb and mariadbclient—use a CMake based build system. This can incorrectly link against some libraries bundled with the Anaconda Python distribution, rather than the system-provided equivalents, resulting in a build failure. We have adjusted the build process of the affected packages to work around this error. [DM-5595]

Execute afw test suite when afwdata is not available

Some tests in the afw package rely on data from the afwdata package. The test suite would search for afwdata, and skip all of the afw tests if afwdata is not available. This check has been made smarter, so that only tests which actually require afwdata are now skipped. [DM-609]

Disable implicit threading

Low-level threading packages (such as OpenBLAS or MKL) can implicitly use many threads. Since the LSST stack also parallelizes at a higher level (e.g. using Python’s multiprocessing module), this can cause undesirable contention. We now disable implicit threading when explicitly parallelizing at a higher level to protect the user from this. Implicit threading can be re-enabled by setting the LSST_ALLOW_IMPLICIT_THREADS environment variable. For more details, see this Community post. [DM-4719]

Migrate to standard smart pointers

C++11 introduced new smart pointer types (std::unique_ptr, std::shared_ptr and std::weak_ptr). We have migrated from the previously used Boost smart pointers to their standard equivalents. [DM-5879, DM-4008, RFC-100, DM-5966]

Science User Interface

During cycles winter and extra 2016, SUIT group has embarked on a major rewrite of the Firefly client side code, converting the Java/GWT based code to pure JavaScript (ES6) code, adopting React/Redux framework. The goal is to finish more than 90% of the client code conversion by the end of X16. We will make a release version of the code by end of September 2016.

Documentation improvements

A number of documentation improvements were made, three of which may be of general interest:

DM Developer Guide

New life has been breathed into the DM Developer Guide both on the surface and behind the scenes. While the primary audience for this are LSST developers, it can be a useful guide to anyone who wishes to participate in our open source development—all the fun without the paperwork! You can find it at developer.lsst.io

LSST Technotes

We have a lightweight bootstrap that allows contributors to produce ReStructuredText documents that are then published via our web services using Sphinx and some of our won sauce. You can see a rich example in Colin Slater’s technote, “DMTN-006: False Positive Rates in the LSST Image Differencing Pipeline,” at dmtn-006.lsst.io. A list of available technotes is currently curated on the Community forum.

If you have been thinking hard about an LSST-related technical or scientific issue that you feel is below your “worth a paper” threshold, why not contribute it as a technote? We have a lsst-technote-bootstrap project to get you get started.

Release Notes and Installation

You’re reading them now on pipelines.lsst.io! We have moved our release note and installation instructions to our Spinx-based platform as well. This means that if you find a deficiency with our notes that needs clarification, or discover a new issue, you are quite welcome to fork-and-PR them; the GitHub repository is pipelines_docs.

Summer 2015 Release (v11_0)

These release notes document notable changes since 10.1, which was the Winter 2015 release.

Source Identifier
Git tag 11.0
EUPS distrib v11_0

See also:

Major Functionality and Interface Changes

Improved semantics for loading Exposures and MaskedImages from arbitrary FITS files

The Exposure and MaskedImage represent image data with associated mask and variance information. When serialized to FITS, these are stored as three consecutive extensions in the FITS files. It is possible to load Exposures and MaskedImages from multi-extension FITS files which were not generated by LSST, but, due to the limitations of the FITS data model, it is not possible to ensure that the creator of the file adhered to the LSST convention: while an image object may be successfully instantiated, its contents may not be logically consistent.

We now go to greater lengths to check that the information in the file is consistent with the LSST standard, warning the user—and in some cases refusing to proceed—if it does not. [DM-2599]

Improved support for non-standard FITS headers

The LSST stack is now capable of loading FITS files which contain non-standard headers of the form PVi_nn (i=1..x, nn=5..16), as written by SCAMP, and EQUINOX headers with a “J” prefix, as written by SkyMapper. [DM-2883, DM-2924, DM-3196]

It is now possible to perform instrument signal removal on an Exposure which has no Detector

FakeAmp, a Detector-like object object which supports returning gain and saturation level, was added to make it possible to run updateVariance and saturationDetection if required. (DM-2890)

PVi_j header cards are correctly saved to FITS files

This makes it possible to round-trip TPV headers, for example. [DM-2926]

Changes to compound fields and delimiters in catalog schemas

In the older (“version 0”) approach to table schemas, we had several compound field types (Point, Moments, Covariance, Coord) which behaved differently from other field types - the square bracket [] operators could not be used to access them, and they could not be accessed as columns (though their scalar subfields – e.g. “x” and “y” for Point – could be). In version 0, we used periods to separate both words and namespace elements in field names, but converted periods to underscores and back when writing to FITS. These schemas were mostly produced by the old measurement framework in meas_algorithms’ SourceMeasurementTask, which was removed in the 10.1 release.

In the new (“version 1”) approach, compound objects are simply stored in catalogs as their constituent scalars, with helper classes called FunctorKeys provided to pack and unpack them from Records (the FunctorKeys that replace the old compound fields are all in afw/table/aggregates.h). Unlike the original compound fields, there’s no limit to how many types of FunctorKey we can have, or what package they can live in, making the system much more extensible. By making the constituent scalar objects what the Schema object knows about, it will be much easier to map a Schema to other table representations that don’t know about LSST classes (e.g. SQL or Pandas). Most FunctorKeys can be used anywhere a regular Key can be used. Also, in version 1, we use underscores as namespace separators, and CamelCase to separate words, eliminating some ambiguity between word and namespace boundaries. The new measurement framework in meas_base’s SingleFrameMeasurementTask and ForcedMeasurementTask uses version 1 tables exclusively.

In previous releases of the pipeline, version 0 schemas were deprecated but still supported. They have now been removed, but old catalogs saved as version 0 will still be readable - they will be converted to version 1 on read, with period delimiters converted to underscores, and all compound fields unpacked into scalar fields that can be used with a corresponding FunctorKey. This procedure obviously does not preserve field names, but all slot definitions will be preserved, so code that only relies on slot or minimal schema accessors (getCoord(), getCentroid(), getPsfFlux(), etc.) should not need to be modified. [DM-1766]

Allow for use of Approximate (Chebyshev) model in background estimation

In previous releases, the only method for background estimation was to use an interpolation scheme (constant, linear, or various splines). These schemes tend to lead to over-subtraction of the background near bright objects. The Approximate (Chebyshev) approach to background estimation greatly improves the background subtraction around bright objects. The relevant code to use this latter approach (including persistence and backwards compatibility issues) is now in place.

While the intention is to eventually set the Approximate background subtraction scheme as the default, there is some clean-up and restructuring that needs to be done before resetting the defaults (which may also require adjusting some defaults in the calibrate stage to be more appropriate for the approximation, as opposed to interpolation, scheme). Therefore, the default setting has not been changed (i.e. the default is still to use an interpolation scheme for background estimation). The Chebychev approximation can be selected for background estimation through configuration parameters in the obs_CAMERA packages, i.e. useApprox=True and, optionally, approxOrderX (approximation order in X for background Chebyshev), approxOrderX (approximation order in Y for background Chebyshev: currently approxOrderY must be equal to approxOrderX), weighting (if True, use inverse variance weighting in calculation). [DM-2778]

Multi-band processing for coadds

See the description of the multi-band coadd processing work performed in S15 for details. In short, four new command-line Tasks have been added for consistent multi-band coadd processing:

DetectCoaddSourcesTask
Detect sources (generate Footprints for parent sources) and model background for a single band.
MergeDetectionsTask
Merge Footprints and Peaks from all detection images into a single, consistent set of Footprints and Peaks.
MeasureMergedCoaddSourcesTask
Deblend and measure on per-band coadds, starting from consistent Footprints and Peaks for parent objects.
MergeMeasurementsTask
Combine separate measurements from different bands into a catalog suitable for driving forced photometry. Essentially, it must have a centroid, shape, and CModel fit for all objects, even for objects that were not detected on the canonical band. Will assume that all input catalogs already have consistent object lists.

[DM-1945, DM-3139]

Enable use of deblended HeavyFootprints in coadd forced photometry

Given the new multi-band processing for coadds (above), we now have a reference catalog that is consistent across all bands. This catalog allows the use of the source’s HeavyFootprints to replace neighbors with noise in forced photometry, thus providing deblended forced photometry and consistent deblending across all bands. This provides much better colors for blended objects as well as measurements for drop-out objects that do not get detected in the canonical band. This functionality has been enabled for forced coadd photometry.

See the description of the multi-band coadd processing work performed in S15 for further motivation of this change. [DM-1954]

Limited the fractional number of masked pixels per source

CModel has difficulties modelling backgrounds in vignetted regions of the focal plane, leading to a performance bottleneck. To mitigate the issue, if the fractional number of masked pixels in a particular source exceeds a given threshold, that source will be skipped. [DM-2914]

Peak culling around large objects

An excess of “junk” peaks may be observed around large objects. Given the new multi-band processing architecture (above), these must be consistently removed across bands. We therefore provide a method to consistently “cull” this peaks at an earlier stage, immediately after merging and sorting in MergeDetectionsTask. [DM-2914]

Parent Footprints are the union of their children

Parent Footprints are now trimmed so that they are strictly the union of their children: any pixels which are not assigned to a child are removed. This mitigates an issue whereby stray flux from the parent was not correctly assigned to the children. Note that this has the consequence that parent Footprints are not necessarily contiguous. [DM-2914]

Large Footprints may be skipped on initial processing

For practical processing purposes (specifically total processing time and memory limits due to current hardware limitations), we have the option to skip over objects with large Footprints during large-scale processing, with the intention to return to these objects to “reprocess” them using different hardware in future. The ability to properly record which objects have been skipped and require further processing has been implemented along with optimizations to the deblender configuration for the maximum number of Peaks per Footprint, and the size and area of Footprints. [DM-2914]

Command line tasks for measurement transformation

The measurement transformation framework provides a generic mechanism for transforming the outputs of measurement plugins in raw units, such as pixel positions or flux, to calibrated, physical units, such as celestial coordinates or magnitudes. Appropriate transformations are defined on a per-measurement-plugin basis, and may make use of the calibration information and WCS stored with the data.

This system is designed such that the transformation of a given catalog is performed by a command line task. Different catalog types (such as src, forced_src, etc) make use of separate command line tasks. In this release, we provide a variety of tasks to handle different source types.

(DM-2191, DM-3473, DM-3483)

Add NO_DATA mask plane

Previously, we have used the EDGE mask plane to indicate both pixels which are off-the-edge of the detector, and hence have no data available, and pixels near the edge which cannot therefore be properly searched for sources. Here, we introduce the NO_DATA plane to refer to the former case and now use EDGE strictly for the latter. [DM-3136]

Add slot for flux used in photometric calibration

We define a new slot, CalibFlux, on SourceRecords. This slot is used to record the flux used for photometric calibration, rather than hard-coding the name of a particular algorithm in the PhotoCal task. This slot defaults to a 12 pixel circular aperture flux, the previous default in PhotoCal. [DM-3106, DM-3108]

Table field prefix for aperture flux measurements changed

Our aperture flux measurement algorithms take a list of radii, in pixels, which define the radii over which measurements should be made. Previously, the names of the table fields produced by the algorithm were defined purely based on the position of the radius in that list (thus, the first radius listed would produce a flux field named PluginName_0_flux). This has been changed so that the fields are now named after the radius, regardless of its position in the list. Thus, a 12.5 pixel aperture will result in a field named PluginName_12_5_flux, regardless of its position in the list. [DM-3108]

Faster astrometry reference catalog loading

The reference catalog loading was optimised by caching HEALpix identifiers for the catalog files. This has been observed to speed up loading times from 144 sec to 12 sec.

The cache is saved as andCache.fits in the astrometry catalog directory. The use of the cache can be disabled through the andConfig.py file (or the AstrometryNetDataConfig) by setting allowCache to False. To prepare a cache, setup astrometry_net_data and use the generateANetCache.py script that now comes in meas_astrom. [DM-3142]

Bad pixels tracked when coadding images

When co-adding images, we now keep track of what fraction of the input data for a given pixel was masked. If the total masked data exceeds some user-configurable threshold, the mask is propagated to the coadd. [DM-3137]

Polygon masking in coadded PSFs

Polygonal masks are used to define the usable area of the focal plane; they can be used to, for example, exclude vignetted areas from coaddition. We now take account of these masks to determine which PSF images to included when building co-added PSFs. [DM-3243, DM-3528]

Scale counts to reflect CCD-specific zero-points when warping to create coadd inputs

[DM-2980]

Solving astrometry with distortions

The default astrometry matcher (matchOptimisticB) can now match stars against a reference catalog when the stars are distorted (e.g., at the outskirts of a wide field imager) if there is an estimate of the distortion available. [DM-3492]

Rejection iterations in astrometry fitting

Astrometric fitting (FitTanSipWcsTask) now includes support for iterative fitting with rejection. [DM-3492]

Inclusion of external package PSFEx as option for PSF determination

PSFEx is currently the state of the art external package for PSF determination, used in projects such as DES. LSST wrappers were created such that PSFEx could be used as a plugin in place of the built in PSF determiner. Tests with Hyper Supreme Camera data have shown that PSFEx out performs the built-in PSF determiner. [DM-2961]

Improvements to CModel magnitude measurement

This release includes many miscellaneous improvements and fixes resulting from testing on HSC data, including:

  • parameter tuning for computational performance improvement
  • correction to uncertainty estimation to account for extrapolation beyond the fit region
  • much more robust flagging of failure modes

Interface changes to forced measurement

The order of arguments to the forced measurement task was reversed, so that it takes a source catalog followed by an Exposure. This brings it into line with the single frame measurement interface. [DM-3459]

N-way spatial matching

A simple utility class for performing spatial matches between multiple catalogs with identical has been added as lsst.afw.table.multiMatch.MultiMatch. This is intended as a stop-gap measure until more flexible and efficient functionality becomes available, but is already usable. [DM-3490]

Display CCD data as laid out in the focal plane

It is now possible to use lsst.afw.cameraGeom.utils to display CCD data laid out in the focal plane. An example of how this functionality works in practice is available as an IPython notebook. [DM-2347]

Bug Fixes

The following fixes resolve problems visible to end users.

Doxygen documentation now correctly includes LaTeX formatting

Correctly referring to MathJax means that LaTeX markup in documentation is nicely formatted. [DM-2545]

Performance regression in Footprint dilation resolved

The previous release included improved algorithms for dilating Footprints. Unfortunately, in some circumstances (notably when dealing with particularly large Footprints) this code could actually perform more slowly than the previous implementation. This could have significant performance implications for many image processing operations. This regression has now been rectified, and the new dilation operations are significantly faster than the old ones in all circumstances tested. [DM-2787]

Footprint fixes

The following updates/fixes to Footprint handling have been made:

  • The default 32-bit heap space used to store FITS variable-length arrays isn’t large enough to store some of our extremely large HeavyFootprints. This persistence issue has been fixed the by switching to 64-bit heap descriptors, which is now supported by FITS.
  • Footprint::transform is now properly copying peaks over to the new footprint.
  • Footprint::clipTo is now properly removing those peaks lying outside the desired region.
  • Several parts of the pipeline assume peaks are sorted from most positive to most negative. We now ensure the cross-band merge code maintains this ordering as much as possible (even though the sorting may not be consistent across different bands).
  • The merging of a parent and its children’s Footprints was failing in cases where one or more child Footprints were themselves noncontiguous. This has been fixed by adapting the mergeFootprints code in afw such that it combines all the Footprints in the FootprintSet it uses in its implementation (instead of requiring that the FootprintSet have only one Footprint).

[DM-2606]

Fixed error in memory access in interpolation

An off-by-one error resulted in an attempt to read beyond the allocated memory. [DM-3112]

Fixed truncated write of certain WCS information to FITS

[DM-2931]

Use the correct weighting in photometric calibration

Previously, we were incorrectly weighting by errors, rather than inverse errors. [DM-2423]

Remove non-positive variance pixels in coadd creation

When interpolating variance maps we can produce negative values. These then cause failures when we try to take the square root. Ultimately, the means of creating variance maps needs to be fixed (which is DM-3201); as a temporary workaround, we replace negative variance values with infinity. [DM-2980]

Task defaults are set correctly for difference imaging

The DipoleMeasurementConfig.setDefaults method incorrectly contained a return that was executed before the defaults were actually applied. This has been corrected, and a number of tests updated to rely on those defaults. [DM-3159]

Build and code improvements

These improvements should not usually be visible to end users. They may be important for developers, however.

Backend-agnostic interface to displays

The image display code no longer makes the assumption that display is carried out using ds9. Rather, an API is available which is independent of the the particular image viewer is in use. A backwards compatibility layer ensures that display through ds9 is still supported, while other backends will be added in future.

[RFC-42, DM-2709, DM-2849, DM-2940, DM-3203, DM-3468]

Measurement framework compiler warnings resolved

The measurement framework was refactored to avoid a series of warnings produced by the clang compiler. [DM-2131]

Unsanctioned access to the display by tests suppressed

Some unit tests were attempting to write to a display, even when no display was available. On some systems, this directly caused test failures; on others, it could obscure the true cause of failures when a test did fail. [DM-2492, DM-2494]

Unused & obsolete code has been removed from the datarel package

This package is effectively obsolete, but is still used in documentation generation which makes removing it entirely complex. For now, therefore, it has simply been trimmed of all unused functionality; it may be removed entirely following DM-2948. [DM-2949]

Reduced verbosity of astrometry.net solver

A correction to the way that astrometry.net logging was propagated to the LSST logging system, together with reducing the priority of some messages, leads to a substantial reduction in chatter from astrometry. DM-3141

Ensure that slots are present before initializing algorithms that depend upon them

When initializing an algorithm that refers to a particular slot, we resolve the target of the slot and refer to that instead. That means that if the slot definition is changed after measurement has been performed, we are still pointing to the correct information. However, if the algorithm is initialized before the slot it depends on, this resolution could not take place and “circular” aliases could result. We now explicitly check for and throw an error in this case. [DM-3400]

Visualizations for astrometry.net solver

It is now possible to display the source positions, distorted source positions and reference positions to assist with debugging. [DM-3209]

Subaru support reinstated

The obs_subaru package, which provides packages and tasks specific to the Subaru telescope, has been brought up to date with recent changes to the LSST stack and improvements made during Hyper Suprime Cam development. [DM-1794, DM-3403]

Refactor & document coadd construction

A number of minor changes and documentation improvements were made to the CoaddBase, AssembleCoadd, CoaddInputRecorder and MakeCoaddTempExp tasks. These brought the structure of the code better into line with the state-of-the-art development on Hyper Suprime Cam. [DM-2980]

Properly handle masking NaN or saturated values in overscans

Resolved an issue where, in certain circumstances, flags in the mask plane for saturated and nan values in overscans were being improperly propagated to all amplifiers in an image. These flags are now applied to the amplifier where the bad values are seen. [DM-2923]

Deblender optimization

Several performance optimizations to the (C++) algorithms used in the deblender have been implemented, in particular those which identify objects with significant amounts of their flux attributed to edge pixels. In addition, memory usage was reduced by removing unused mask planes left over from debugging, not storing masks for deblending templates, and by clipping template images when their associated Footprints are clipped. [DM-2914]