.. _release-v12-0: ######################### Release 12.0 (2016-06-15) ######################### +-------------------------------------------+------------+ | Source | Identifier | +===========================================+============+ | Git tag | 12.0 | +-------------------------------------------+------------+ | :doc:`EUPS distrib ` | v12\_0 | +-------------------------------------------+------------+ Preamble ======== Major Feature Development ------------------------- With a :ref:`new release schedule `, this saw prolonged development. This release features a complete backport of functionality from the Hyper-Suprime Cam pipeline, in addition to many new LSST-driven developments. Even :ref:`Astropy views of LSST catalogs objects ` are available. New: Install with Anaconda -------------------------- In addition to the customary source installation, this release features two binary installation methods: - Anaconda - `CernVM FS `__ (contributed by Fabio Hernandez) If you are unsure which one to use, we recommend you try the conda binaries. This is the first conda binary release though and it is always possible we did not anticipate all the problems "in the wild;" if you experience difficulties, please `report them `__ so that we can improve our testing (and help you!). New Documentation at pipelines.lsst.io -------------------------------------- Our release documentation and installation instructions are no longer on the corporate wiki; you can find them at http://pipelines.lsst.io/. You are welcome and encouraged to contribute improvements to the repo on GitHub: https://github.com/lsst/pipelines_docs/. .. _v12-0-new-release-schedule: New Release Schedule -------------------- The timing of the formal releases (which mark the end of each DM planning cycle) has precessed from Winter/Summer to Spring/Fall. This has been done to avoid conflict between DM cycle planning activities and the LSST All-Hands Meetings in August and February. In the future, formal releases will be in June and December. In order to precess the cycles to the new schedule, an Extra 3-month cycle (aka X16) was added, so 12.0 is a "bumper" release containing approximately 9 months of activity. Join Us on the Community Forum ------------------------------ If you have questions or comments or comments about the LSST Science Pipelines, visit our new community forum for advice: https://community.lsst.org/. Contents ======== .. toctree:: :hidden: v12_0_qserv_dax - :ref:`release-v12-0-major-changes` - :ref:`release-v12-0-bug-fixes` - :ref:`release-v12-0-internal-improvements` - :ref:`release-v12-0-sui` - :ref:`Qserv and Data Access ` - :ref:`release-v12-0-doc` *See also:* - :doc:`Installation instructions <../install/index>` - :doc:`Known issues ` - `Characterization Metric Report (DMTR-14) `_ .. _release-v12-0-major-changes: Major Functionality and Interface Changes ========================================= - :ref:`release-v12-0-mask-planes-before-coaddition` - :ref:`release-v12-0-dumping-task-config-params-includes-docs` - :ref:`release-v12-0-clean-up-interpolation-tasks` - :ref:`release-v12-0-avoid-io-race-config-writes` - :ref:`release-v12-0-safeclipassemblecoaddtask` - :ref:`release-v12-0-reserve-psf-candidates-from-fitting` - :ref:`release-v12-0-update-pipeline-config` - :ref:`release-v12-0-vignetting-polygons` - :ref:`release-v12-0-rerun` - :ref:`release-v12-0-fakes` - :ref:`release-v12-0-tract-routines` - :ref:`release-v12-0-xytransform` - :ref:`release-v12-0-getcoordsystem` - :ref:`release-v12-0-jointmatchlistwithcatalog` - :ref:`release-v12-0-visualize-skymaps` - :ref:`release-v12-0-unpacked-matches` - :ref:`release-v12-0-focal-plane-coords` - :ref:`release-v12-0-jacobian-position-src` - :ref:`release-v12-0-record-images-contributing-to-coadds` - :ref:`release-v12-0-variance-at-source` - :ref:`release-v12-0-source-flux-in-ap` - :ref:`release-v12-0-blendedness` - :ref:`release-v12-0-simple-shape-meas` - :ref:`release-v12-0-mirata-seljak-mandelbaum` - :ref:`release-v12-0-interp-background` - :ref:`release-v12-0-averagecoord` - :ref:`release-v12-0-hsc-support` - :ref:`release-v12-0-psf-shapelet` - :ref:`release-v12-0-propagate-flags-to-coadds` - :ref:`release-v12-0-apcorr-coadd-meas` - :ref:`release-v12-0-grown-footprints` - :ref:`release-v12-0-meas-sky-objs` - :ref:`release-v12-0-specify-output-dir` - :ref:`release-v12-0-bright-object-masks` - :ref:`release-v12-0-cmodel-improvements` - :ref:`release-v12-0-astropy-table-views` - :ref:`release-v12-0-afterburner-measurements` - :ref:`release-v12-0-task-registry` - :ref:`release-v12-0-afw-test-utilities` - :ref:`release-v12-0-non-linearity-corrections` - :ref:`release-v12-0-amplifier-catalogs` - :ref:`release-v12-0-background-subtraction` - :ref:`release-v12-0-star-selectors` - :ref:`release-v12-0-processccdtask` .. _release-v12-0-mask-planes-before-coaddition: Add the option of excluding mask planes before coaddition --------------------------------------------------------- The user-configurable parameter ``removeMaskPlanes`` has been added to :lclass:`AssembleCoaddConfig`. This is a list of mask planes which will not be propagated to the coadd; by default, the ``CROSSTALK`` and ``NOT_DEBLENDED`` mask planes are removed. :jirab:`DM-4866` .. _release-v12-0-dumping-task-config-params-includes-docs: Dumping task configuration parameters now includes documentation ---------------------------------------------------------------- That is, running ``${TASK} ... --show config`` displays not only the names and values of the configuration but also associated documentation. :jirab:`DM-3811` .. _release-v12-0-clean-up-interpolation-tasks: Clean up interpolation tasks and implement useFallbackValueAtEdge ----------------------------------------------------------------- The ``interpImageTask`` in pipe_tasks has been refactored to implement a single run function for interpolation over a list of defects in an image. This run function will accept an :lmod:`afw` image type of either :lclass:`MaskedImage` or :lclass:`Exposure`. A defects list can be passed in directly *or* the name of mask plane can be passed from which a defects list will be created. If a PSF is attached to the image, it will be used as the (required) argument for the :lfunc:`interpolateoverDefects` function in ``meas_algorithms`` used for the interpolation. Otherwise a FWHM (in pixels) can be provided or the ``defaultFWHM`` value in ``meas_algorithms``\ ' :lclass:`GaussianFactory` is used. Note that while the PSF is a required argument for ``meas_algorithms``\ ' :lfunc:`interpolateOverDefects` function it is currently not being used, so it is not necessary to pass in an accurate PSF. The ``useFallbackValueAtEdge`` option is now implemented. This tapers the interpolation to a ``fallbackValue`` towards the image edge. The ``fallbackValue`` can be set via config parameters to be either computed as a statistical representation of the image data (``MEAN``, ``MEDIAN``, or ``MEANCLIP``) or set by providing a specific value. Allowance for a negative ``fallbackValue`` is also controlled though a config parameter. :jirab:`DM-3677` .. _release-v12-0-avoid-io-race-config-writes: HSC backport: Avoid I/O race conditions config write out -------------------------------------------------------- This adds functionality to avoid potential I/O race conditions when running multiple simultaneous process. This is accomplished by writing to temporary files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. :file:`config.py~1`, :file:`config.py~2`, …), a ``--no-backup-config`` option (to be used with ``--clobber-config``) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run. :jirab:`DM-3911` .. _release-v12-0-safeclipassemblecoaddtask: HSC backport: Introduce SafeClipAssembleCoaddTask which extends AssembleCoaddTask to make clipping safer -------------------------------------------------------------------------------------------------------- :lclass:`SafeClipAssembleCoaddTask` does the following, - build both clipped and unclipped coadds and difference them first - detect on the difference - identify difference footprints that overlap appreciably with only one input image - use identified difference footprints as a mask into a final coadd - set a "clipped" bit on the final coadd for any pixel that did not include all input frames within the valid polygons. To support the new coadd task, the pixel flags measurement plugin has been modified to accept two new configuration parameters. These new parameters, named ``masksFpCenter`` and ``masksFpAnywhere``, each accept a list of mask planes. When the pixel flags measurement plugin searches for mask planes to set corresponding flags, it will now additionally search the user supplied mask plane lists. The ``masksFpCenter`` parameter specifies mask planes that, if found within the center of a footprint, will have a corresponding pixel flag set. The ``masksFpCenter`` parameter specifies mask planes that, if found anywhere in a footprint, will have a corresponding pixel flag set. The ``masksFpAnywhere`` parameter now has it's defaults set within the stack which specify the clipped mask plane created by :lclass:`SafeClipAssembleCoaddTask`. :lclass:`SafeClipAssebleCoaddTask` is now the default method for building a coadd within the LSST Stack. The :command:`assembleCoadd.py` :lclass:`CommandLineTask` now supports the ``--legacy`` command line flag which will allow the original :lclass:`AssembleCoaddTask` to be run. If the legacy task is run, the clipped mask plane must be removed from the ``masksFpAnywhere`` configuration parameter. :jirab:`DM-2915` .. _release-v12-0-reserve-psf-candidates-from-fitting: HSC backport: Allow for some fraction of PSF Candidates to be reserved from PSF fitting --------------------------------------------------------------------------------------- This adds functionality that allows for the reservation of a fraction of the PSF Candidates from PSF fitting. The reserved fraction can then be used to test for over-fitting, do cross-validation, etc.. To support this functionality, the run methods of :lclass:`CalibrateTask` and :lclass:`MeasurePsfTask` now accept the additional keyword argument ``expId``. Dummy versions of this keyword were added to the :lclass:`SdssCalibrateTask` and :lclass:`CfhtCalibrateTask` to maintain a consistent API. :jirab:`DM-3692` .. _release-v12-0-update-pipeline-config: Updated pipeline configuration based on Hyper Suprime-Cam experience -------------------------------------------------------------------- Configuration defaults and metadata through the LSST single-processing pipeline have been updated to match the current best practice established on HSC. Major changes include: - Support for narrow band filters; - Updated CCD defect lists; - Optimized (in terms of CPU time) deblender settings; - Avoiding failure in certain corner cases (e.g. operating on zero-length arrays, taking log\ :sub:`10` of zero). :jirab:`DM-3942` .. _release-v12-0-vignetting-polygons: Define polygon bounds for CCDs based on vignetted regions --------------------------------------------------------- This adds a function in ip_isr's :command:`isrTask.py` to set a "valid polygon" for a given CCD exposure as the intersection of a polygon defined in focal plane coordinates and the CCD corners. It is currently being used in ``obs_subaru``\ 's :command:`isr.py` to set the polygon bounds (added in :jira:`DM-2981`) for a CCD exposure to include the non-vignetted regions. The settings for the vignetted region is in a separate config file so that it can be used in different places in the code. :jirab:`DM-3259` .. _release-v12-0-rerun: Introduce ``--rerun`` option for command line tasks --------------------------------------------------- This new command line option streamlines the process of specifying input and output repositories for command line tasks. In its simplest form, the ``--rerun`` option is a shorthand for specifying an output repository: output is written to a location relative to the input. Thus .. code-block:: bash CmdLineTask /path/to/inputroot --rerun useroutput is equivalent to .. code-block:: bash CmdLineTask /path/to/inputroot --output /path/to/inputroot/rerun/useroutput Often, one task will process the output of a previous rerun. For this situation, ``--rerun`` provides a two-valued form which specifies relative locations of both input and output. In this mode, .. code-block:: bash CmdLineTask /path/to/inputroot --rerun process1:useroutput is equivalent to .. code-block:: bash CmdLineTask /path/to/inputroot --input /path/to/inputroot/rerun/process1 --output /path/to/inputroot/rerun/useroutput :jirab:`RFC-95,DM-3371` .. _release-v12-0-fakes: Introduce framework for injecting fake sources into data processing ------------------------------------------------------------------- A new boolean option (``doFakes``) and a retargetable task (:command:`fakes`) have been introduced into the :lclass:`ProcessCcdTask`. This (along with a new class called :lclass:`BaseFakeSourcesTask`) sets up a frame work that others may use to introduce known fake sources into the data processing stream. However, this framework itself does not actually insert any fake data itself, but provides an interface others may use to define their own fake source injection task. To implement a fake injection task one must create a child class of :lclass:`BaseFakeSourcesTask` (located in :lmod:`lsst.pipe.tasks.fakes`) and overload the run method to do the work of injecting the sources. Each source that is injected should have a corresponding bit set in the ``FAKE`` maskplane which can be accessed with the ``bitmask`` attribute of :lclass:`BaseFakeSourcesTask`. Once a task has been created, the config field fakes in :command:`processCcd` must be retargeted to point to the user created task. Additionally, the task will not be run unless the ``doFakes`` configuration option in :command:`processCcd` is set to ``True``. However, if this variable is set to ``True``, and the task is not retargeted :command:`processCcd` will fail. :jirab:`DM-3380` .. _release-v12-0-tract-routines: Add convenience routines for working with tracts ------------------------------------------------ Added two new data id containers: 1. :lclass:`PerTractCcdDataIdContainer`: determines the set of tracts each visit touches and adds a data reference with those tracts for each of the visit components. This allows for the user to run a command line task :command:`forcedPhotCcd.py` for a given visit without having to know which tracts overlap the visit. *Note this will also be used by meas\_mosaic if/when it gets incorporated into the LSST Stack.* 2. :lclass:`TractDataIdContainer`: generates a list of data references for patches within a given tract (effectively a "data reference list" that points to the entire tract). *Note that, at the time of writing, this is only being used by a QA analysis script currently under development.* :jirab:`DM-4373` .. _release-v12-0-xytransform: Warp images based on an XYTransform ------------------------------------ Added the ability to warp images using a transformation defined by an :lclass:`lsst::afw::geom::XYTransform`. :jirab:`DM-4162` .. _release-v12-0-getcoordsystem: Add getCoordSystem to Coord and add UNKNOWN CoordSystem enum ------------------------------------------------------------ Added method :lmeth:`getCoordSystem` to :lclass:`lsst::afw::coord::Coord`. Also added ``UNKNOWN=-1`` as a new :lclass:`CoordSystem` enum (the existing enums retain their existing value). :lclass:`DM-4606` .. _release-v12-0-jointmatchlistwithcatalog: Adapt joinMatchListWithCatalog to facilitate and simplify denormalizing a match list ------------------------------------------------------------------------------------ The match lists created when performing image calibration (astrometry and photometry) are normalized (i.e. stripped down to a list of the matched reference and source ids and their distance) prior to being persisted. The ability to denormalize a match catalog is very useful (for post QA analysis, for example). This can now be done using the :lfunc:`joinMatchListWithCatalog` function in ``meas_algorithms``\ 's :lclass:`LoadReferenceObjectsTask`. It has been moved from ``meas_astrom``\ 's :lclass:`ANetBasicAstrometryTask` so that it can be easily accessed (requiring only that a reference object loader be initiated) and to allow it to work with any kind of reference catalog (i.e. other than ``a_net``). :jirab:`DM-3633` .. _release-v12-0-visualize-skymaps: Add a script for visualizing skymaps and CCDs --------------------------------------------- The ``skymap`` package now contains the script :command:`showVisitSkyMap.py` which provides a convenient way of visualizing the tracts, patches and CCDs contained in a set of visits. :jirab:`DM-4095` .. _release-v12-0-unpacked-matches: Add functions to generate "unpacked matches" to and from a catalog ------------------------------------------------------------------ Functions have been added to :lmod:`lsst::afw::catalogMatches` to provide the ability to convert a match list into a catalog and vice versa (this can be useful for post-processing analyses; QA analysis, for example). :jirab:`DM-4729` .. _release-v12-0-focal-plane-coords: Add a measurement algorithm which records the focal plane coordinates of sources -------------------------------------------------------------------------------- The :lclass:`SingleFrameFPPositionPlugin` measurement plugin, available in ``meas_base``, records the positions of source centroids in focal plane coordinates (which may be convenient for plotting). This plugin is not enabled by default, but may be switched on by requesting ``base_FPPosition`` in measurement configuration. :jirab:`DM-4234` .. _release-v12-0-jacobian-position-src: Add a measurement algorithm which records the Jacobian at the positions of sources ---------------------------------------------------------------------------------- The :lclass:`SingleFrameJacobianPlugin` calculates the ratio between the nominal Jacobian determinant at the source centroid (as determined by a user-specified pixel scale) and the actual Jacobian determinant as derived from the astrometric solution. This plugin is not enabled by default, but may be switched on by requesting ``base_Jacobian`` in measurement configuration. :jirab:`DM-4234` .. _release-v12-0-record-images-contributing-to-coadds: Add a measurement algorithm which records the number of input images contributing to a coadd sources ---------------------------------------------------------------------------------------------------- When run on a source detected on a coadd, :lclass:`SingleFrameInputCountPlugin` records the number of input images which were stacked to create the coadd at the position corresponding to the source centroid. The plugin is referred to as ``base_CountInputs``, and is enabled by default when performing measurement on coadded images. It is not appropriate to enable this plugin when processing single visit (i.e., not coadded) images. :jirab:`DM-4235` .. _release-v12-0-variance-at-source: Add a measurement algorithm which records the variance at the positions of sources ---------------------------------------------------------------------------------- The :lclass:`SingleFrameVariancePlugin` records the median variance in the background around the position of the source being measured. The plugin is referred to as ``base_Variance`` and is enabled by default when performing single frame measurement. :jirab:`DM-4235,DM-5427` .. _release-v12-0-source-flux-in-ap: Add a measurement algorithm which records source flux in an aperture scaled to the PSF -------------------------------------------------------------------------------------- The :lclass:`ScaledApertureFluxAlgorithm` measures the flux in a circular aperture with radius scaled to some user-specified multiple of the PSF FWHM. This plugin is not enabled by default, but may be switched on by requesting the ``base_ScaledApertureFlux`` in measurement configuration. :jirab:`DM-3257` .. _release-v12-0-blendedness: Add a measurement algorithm which quantifies the amount of "blendedness" of an object ------------------------------------------------------------------------------------- The :lclass:`BlendednessAlgorithm` measures the amount to which an object is blended. Both the flux and shape of each child object are compared to measurements at the same point on the full image. The size of the weight function used on both images is determined from the child object. The blendedness metric implemented is defined as ``1-childFlux/parentFlux``. The plugin is referred to as ``base_Blendedness`` and is not enabled by default. :jirab:`DM-4847` .. _release-v12-0-simple-shape-meas: Add a "simple" shape measurement algorithm ------------------------------------------ The :lclass:`SimpleShape` algorithm, provided in the ``meas_extensions_simpleShape`` package, computes the non-adaptive elliptical Gaussian-weighted moments of an image. The plugin is referred to as ``ext_simpleShape_SimpleShape`` and is not enabled by default. :jirab:`DM-5284` .. _release-v12-0-mirata-seljak-mandelbaum: Add Hirata-Seljak-Mandelbaum shape measurement algorithms --------------------------------------------------------- The ``meas_extensions_shapeHSM`` package has been added to the distribution. This provides a series of measurement algorithms based on the work by `Hirata and Seljak (2003) `__ and `Mandelbaum et al (2005) `__. Please cite those works if publishing results based on this code. These algorithms are disabled by default; they can be enabled by requesting ``ext_shapeHSM_HsmShapeBj``, ``ext_shapeHSM_HsmShapeLinear``, ``ext_shapeHSM_HsmShapeKsb``, ``ext_shapeHSM_HsmShapeRegauss``, ``ext_shapeHSM_HsmSourceMoments`` and/or ``ext_shapeHSM_HsmPsfMoments`` in the measurement configuration. :jirab:`DM-2141,DM-3384,DM-4780` .. _release-v12-0-interp-background: Add option to temporarily remove an interpolated background prior to detection ------------------------------------------------------------------------------ This has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background. The extra subtracted interpolated background is added back in after detection. Currently, the default setting for the config parameter ``doTempLocalBackround`` is set to ``False``. :jirab:`DM-4821` .. _release-v12-0-averagecoord: Add function to average coordinates ----------------------------------- Added function :lfunc:`lsst.afw.coord.averageCoord`, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinates. :jirab:`DM-4933` .. _release-v12-0-hsc-support: Integrate support for Hyper Suprime-Cam --------------------------------------- The ``obs_subaru`` camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST's continuous integration system. It will now be included as part of the lsst\_distrib release. *Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present.* :jirab:`DM-3518,DONE DM-4358,DM-5007` .. _release-v12-0-psf-shapelet: Reimplement PSF Shapelet approximations for CModel -------------------------------------------------- A new algorithm for computing multi-shapelet approximations (:lclass:`DoubleShapeletPsfApprox`) has been added to ``meas_modelfit``. This is simpler and more robust than the old algorithm, which has been renamed to :lclass:`GeneralShapeletPsfApprox`. The new algorithm is recommended for production use, and is now the default. :jirab:`DM-5197` .. _release-v12-0-propagate-flags-to-coadds: Propagate flags from individual visits to coadds ------------------------------------------------ A task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits. :jirab:`DM-4878,DM-5084` .. _release-v12-0-prototype-bfc: Prototype Brighter-Fatter correction ------------------------------------ Code for correcting for the Brighter-Fatter effect on CCDs is now available in the stack. It is enabled using the ``doBrighterFatter`` configuration option to :lclass:`IsrTask`. It requires a pre-generated correction kernel. Calculation of this kernel is not currently performed within the stack: a prototype exists, and will be merged to the Calibration Products Pipeline in a future cycle. :jirab:`DM-4837,DM-5082,DM-5130` .. _release-v12-0-apcorr-coadd-meas: Aperture correction on coadd measurements ----------------------------------------- Aperture corrections are now enabled for measurements performed on coadds. :jirab:`DM-5086` .. _release-v12-0-grown-footprints: Return grown Footprints from detection -------------------------------------- By default, :lclass:`Footprints` returned by :lclass:`SourceDetectionTask` are now grown by a multiple of the PSF size. :jirab:`DM-4410` .. _release-v12-0-meas-sky-objs: Enable measurement of "sky objects" in coadd processing ------------------------------------------------------- Sky objects correspond to source properties measured at positions when no objects have been detected. This enables us to better characterize the depth of the survey. This functionality is enabled by default; it can be disabled by setting ``nSkySourcesPerPatch`` to zero in the configuration of :lclass:`MergeDetectionsTask`. :jirab:`DM-4840,DM-5288` .. _release-v12-0-specify-output-dir: Specification of output directory is now mandatory -------------------------------------------------- When running a command line task which produces output it is now mandatory to specify an output directory (previously, if no output location was specified, data products were written back to the input repository). Output locations may be specified with the ``--rerun`` or ``--output`` command line options. More information is available on `community.lsst.org `__. :jirab:`DM-4236` .. _release-v12-0-bright-object-masks: Bright object masks ------------------- Given an input catalog listing the known positions and sizes of bright objects, a bit is set in the mask plane for every pixel lying within the object's footprint. :jirab:`DM-4831` .. _release-v12-0-cmodel-improvements: CModel fitting improvements --------------------------- CModel is a model fitting approach in which a pure exponential and pure de Vaucouleur are each fit separately, and then their linear combination is fit while the ellipse parameters are held fixed. Improvements in this release make CModel fitting faster and improves results on objects which are detected with an unphysically large likelihood radius. This has been achieved in three ways: - The initial approximate exponential fit that is used to determine the starting parameters and pixel region to use for the exp and dev fit now uses per-pixel variances; - The method used to determine the pixel region used in fitting has been adjusted to make smarter choices, using fewer pixels on average for all objects and many fewer pixels for unphysically large objects; - A new semi-empirical Bayesian prior on radius and ellipticity based on COSMOS distributions has been introduced. :jirab:`DM-4768` .. _release-v12-0-astropy-table-views: Astropy Table views into LSST Catalog objects --------------------------------------------- `Astropy Table `__ views into LSST catalog objects can now be created. These views share underlying data buffers (aside from flag fields), making them read-write, but rows and columns added on either side will not be visible in the other. Two equivalent interfaces are available: .. code-block:: python astropy_table = lsst_catalog.asAstropy() and (in Astropy >= v1.2): .. code-block:: python astropy_table = astropy.table.Table(lsst_catalog) `QTable `__ objects can also be used, resulting in columns that use Astropy's `units `__ package to enforce unit correctness. These interfaces have multiple options to control the details of the view, including how to handle columns that require copies; see the Python on-line help for :lmeth:`asAstropy` for more information. While LSST's catalog objects have features that make them particularly useful in building pipelines, Astropy's are much more appropriate for most analysis tasks, and we strongly recommend using them for any analysis tasks that need to add columns to tables or combine columns from multiple tables. :jirab:`DM-5641,DM-5642,DM-5643` .. _release-v12-0-afterburner-measurements: Add an "afterburner" measurement facility ----------------------------------------- This new functionality makes it possible to register plugins to calculate quantities based on the results of pixel measurement algorithms. This might include, for example, star-galaxy separation or applying aperture corrections. Afterburners of this type are run after measurement plugins, and do not have access to pixel data. :jirab:`DM-4887` .. _release-v12-0-task-registry: Tasks can now be kept in registries ----------------------------------- .. https://community.lsst.org/t/tasks-can-now-be-kept-in-registries/839 Related sets of tasks should now be kept in a registry as per :jira:`RFC-183`, with a common abstract base class. Tasks can now use an :lclass:`lsst.pex.config.RegistryField` config field to specify a subtask if that subtask is in a registry :jirap:`DM-6074`. The task is built and used the same way as if it was specified in an :lclass:`lsst.pex.config.ConfigurableField`, but retargeting and overriding config parameters is different. See `task documentation `__ for more information. See also `How to Write a Task `__ for guidelines for choosing between using :lclass:`lsst.pex.config.RegistryField` and :lclass:`lsst.pex.config.ConfigurableField` to hold a subtask. PSF determiners are now tasks that inherit from an abstract base class :lclass:`lsst.meas.algorithms.PsfDeterminerTask` :jirap:`DM-6077`. However, the effect on existing code was negligible because they were already configurables used from a registry. The way you retarget PSF determiners and override their config parameters remains unchanged. Reimplemented the registry for star selectors that was lost in :jira:`DM-5532`: :lclass:`lsst.meas.algorithms.starSelectorRegistry` :jirap:`DM-6474` One backwards incompatible change: in :jirab:`DM-6474` :lclass:`MeasurePsfTask` and :lclass:`MeasureApCorrTask` both now specify their star selectors using an :lclass:`lsst.pex.config.RegistryField`. This means the format for retargeting star selectors and overriding their config parameters has changed. The config override files in the various ``obs_`` packages are updated accordingly. .. _release-v12-0-afw-test-utilities: New test utilities in afw: BoxGrid and makeRampImage ---------------------------------------------------- .. https://community.lsst.org/t/new-test-utilities-in-afw-boxgrid-and-makerampimage/837 :lclass:`lsst.afw.geom.testUtils.BoxGrid` divides a bounding box into ``nCol x nRow`` equal sized sub-boxes (as equal sized as possible for integer boxes that do not divide evenly) that tile the larger box and have the same type. :lfunc:`lsst.afw.image.testUtils.makeRampImage` makes an image (``ImageX`` where ``X`` is any available type) with values that increase linearly between specified limits (linearly to the extent possible, for integer images). :jirab:`DM-5462` .. _release-v12-0-non-linearity-corrections: Correcting non-linearity ------------------------ .. https://community.lsst.org/t/correcting-non-linearity/816 Introduced a standard way to correct non-linearity (linearize data) as part of Instrument Signature Removal (ISR). Linearization is performed by new functors in ``ip_isr``: - :lclass:`LinearizeBase` is an abstract base class. It is called with an image and the detector information and the correction is performed in place (like all other ISR corrections in :lclass:`IsrTask`). - :lclass:`LinearizeSquared` performs a simple square correction: ``corrImage = uncorrImage + c0*uncorrImage^2`` where ``c0`` is the first coefficient in in the linearity coefficients of the amp into catalog. This is the model used by ``obs_subaru`` for SuprimeCam and HSC. - :lclass:`LinearizeLookupTable` uses a lookup table to determine an offset (read the code doc string for details). The lookup table is saved with the linearizer, but the linearizer also performs a sanity check against the provided detector when called. - You can easily add other linearizers as desired. - Each linearizer has a class variable ``LinearizationType``, a string whose value should be used as the linearization type in the amplifier info catalog. The linearizer checks this value when performing linearization. All detector in a camera must use the same type of linearizer. However linearization can easily be disabled on a detector-by-detector basis by setting linearity type to :lclass:`lsst.afw.cameraGeom.NullLinearityType`. For a camera that does not need linearization, do this for all detectors. Linearizers are obtained from the butler, like any other calibration product. - For :lclass:`LinearizeSquared` and other linearizers that get coefficients from the amplifier info catalog, only one instance is needed for all detectors. In that case the simplest technique is to define :lmeth:`map_linearize` and :lmeth:`bypass_linearize` methods on the camera mapper to return an instance. See the ``obs_subaru`` package for an example. - For :lclass:`LinearizeLookupTable` and other linearizers that store detector-specific data, the ``obs_`` package developer must pickle one linearizer for each detector and make them available as dataset type "linearizer". - If the camera does not want linearization then no "linearizer" dataset type is required because :lclass:`IsrTask` realizes linearization is not wanted before it tries to unpersist the linearizer. You may leave ``IsrConfig.doLinearize`` set to its default value of ``True`` without significant performance penalty. :jirab:`DM-5462,RFC-164` .. _release-v12-0-amplifier-catalogs: Amplifier information catalogs have changed ------------------------------------------- .. https://community.lsst.org/t/amplifier-information-catalogs-have-changed/801 The format of amplifier information catalogs has changed. **Your versions of afw and associated obs_ packages must be compatible** or else you will get errors when building a camera mapper (thus when running any nearly any command-line task). Amplifier information catalogs have a new field as of :jira:`DM-6147`: suspect level. If the value is not ``nan`` then pixels whose values are above this level are masked as ``SUSPECT``. The only cameras that specify a suspect level, so far, are HSC and SuprimeCam. However, a value can be set for any camera, if desired. ``SUSPECT`` is intended to indicate pixels with doubtful values due to errors that are difficult to correct accurately, e.g. a regime where linearity correction is not very stable. In addition, saturation level in the amplifier information catalog is now a floating point value (instead of an integer) and a value of ``nan`` means 'do not mask pixels as SAT.' :jirab:`DM-6147` .. _release-v12-0-background-subtraction: Changes in how background subtraction is done --------------------------------------------- .. https://community.lsst.org/t/changes-in-how-background-subtraction-is-done/756 Background estimation in Python is now done using different routines in ``meas_algorithms``. There is a new task :lclass:`SubtractBackgroundTask`, with full documentation and a working example. The existing function :lfunc:`getBackground` (which fits a background) is replaced by :lmeth:`SubtractBackgroundTask.fitBackground`. Changes from :lfunc:`getBackground` are: - :lmeth:`getBackground` could return ``None`` if the fit failed; in that situation :lmeth:`fitBackround` will raise :exc:`RuntimeError` instead of returning ``None``. - The argument ``image`` was renamed to ``maskedImage``, for clarity. - The config is not passed as an argument. - The debug display code uses different keys and is updated to use :lmod:`afw.display`. The existing function :lfunc:`estimateBackground` (which subtract a background from an exposure) is replaced by :lmeth:`SubtractBackgroundTask.run`. Changes from :lfunc:`estimateBackground` are: - You may pass in a background model (an :lclass:`lsst.afw.math.BackgroundList`). - It returns a struct containing the updated background model. - The config is not passed as an argument. - The debug display code displays the unsubtracted image and uses different keys and is updated to use :lmeth:`afw.display`. The task's config :lclass:`SubtractBackgroundConfig` replaces the old :lclass:`BackgroundConfig`. The field ``algorithm`` may no longer be ``None``; you must use the string ``"NONE"``, instead. See `discussion on Community `__ for details. :jirab:`DM-5323,RFC-155` .. _release-v12-0-star-selectors: Star selectors have changed --------------------------- .. https://community.lsst.org/t/star-selectors-have-changed/639 Star selectors are now tasks. They were already configurable and many added logs; now they are standard tasks. The star selector registry ``starSelectorRegistry`` was gone for awhile. Now that it is back, using a registry field from that registry is the preferred way to specify a star selector as a subtask of another task. Added :lclass:`BaseStarSelectorTask` (but for awhile it was called :lclass:`StarSelectorTask`) an abstract base class for star selectors with the following methods: - :lmeth:`selectStars` an abstract method that takes a catalog of sources and returns a catalog of stars. - :lmeth:`makePsfCandidates` a concrete method that takes a catalog of stars (as returned by `selectStars` and produces PSF candidates; it also returns a sub-catalog of those stars that were successfully turned into PSF candidates (which is usually all of them). - :lmeth:`run` a concrete method that selects stars, makes them into PSF candidates and optionally flags the stars. :jirab:`RFC-154,DM-5532` .. _release-v12-0-processccdtask: Backward-incompatible changes to ProcessCcdTask and subtasks ------------------------------------------------------------ Code changes ^^^^^^^^^^^^ - :lclass:`ProcessCoaddTask` is gone, along with all bin scripts that run it. Use the new `Multi-Band `__ code, instead. - :lclass:`ProcessCcdTask` has been rewritten, so its config has changed. Config override files will need to be updated. This will be done for the ``obs_`` packages as part of the merge, but if you have personal config override files then you will probably need to update them. - Camera-specific variants of :lclass:`ProcessCcdTask` are gone. You will run :command:`processCcdTask.py` to process images for all cameras. - For DECam :command:`processCcdTask.py` will use the LSST Stack's ISR by default. To read ``instcal`` files from the DECam Community Pipeline, replace the ISR task with ``DecamNullIsrTask`` by using a config override file containing the following: .. code-block:: python from lsst.obs.decam.decamNullIsr import DecamNullIsrTask config.isr.retarget(DecamNullIsrTask) - A new dynamic dataset type is available for adding data ID arguments to the argument parser for command-line tasks: :lclass:`ConfigDatasetType` obtains the dataset type from a config parameter. - Various subtasks have changed, including: - New camera-specific ISR task variants for SDSS and DECam: :lclass:`SdssNullIsrTask` and :lclass:`DecamNullIsrTask`. - New task :lclass:`DetectAndMeasureTask` detects and deblends sources and performs single-frame measurement. - New task :lclass:`CharacterizeImageTask` measures PSF and aperture correction, among other things. - :lclass:`CalibrateTask` has been rewritten. It now performs deep detection and measurement, astrometry and photometry. - Camera-specific variants of :lclass:`CalibrateTask` are gone. - :lclass:`ProcessImageTask` (formerly a base class for :lclass:`ProcessCcdTask` and :lclass:`ProcessCoaddTask`) is gone. Data product changes ^^^^^^^^^^^^^^^^^^^^ - ``icSrc`` no longer includes RA/Dec coordinate data, because the fit WCS is not available when the catalog is constructed. You will have to set the coord field yourself if you need it. - ``icExp`` and ``icExpBackground`` can optionally be written by :lclass:`CharacterizeImageTask`. They are so close to ``calexp`` and ``calexpBackground`` that they are not written by default. - ``icMatch`` is no longer being written. Algorithm changes ^^^^^^^^^^^^^^^^^ - PSF is fit somewhat differently. The new task fits the PSF in using a configurable number of iterations. By default each iteration starts with a simple Gaussian PSF whose sigma matches the PSF of the previous fit, but you can use the actual PSF each time. Using a Gaussian causes convergence in 2 iterations. Using the fit PSF slows convergence. - Sources in the ``icSrc`` catalog should have a more consistent minimum SNR for varying seeing. The old code detected once, using a Gaussian PSF with FWHM set by a config parameter. The new code performs detection using the PSF in the final PSF iteration. - The default star selector for ``MeasurePsfTask`` is ``objectSize`` rather than ``sizeMagnitude``. The ``objectSize`` star selector is preferred and was already being specified as an override by HSC. - The icSrc catalog is not matched to an astrometric reference catalog unless the star selector used to measure PSF can use the matches (which is unusual). - The astrometric and photometric solution now use the deeper ``src`` catalog instead of the shallower ``icSrc`` catalog, though with a new SNR cutoff whose default provides a depth similar to the ``icSrc`` catalog. - Fake source handling is temporarily missing; it will be re-added in :jira:`DM-5310`. :jirab:`DM-4692,DM-5348` .. _release-v12-0-bug-fixes: Bug Fixes ========= - :ref:`release-v12-0-persist-ltvn-header` - :ref:`release-v12-0-identifying-peaks-in-merge` - :ref:`release-v12-0-getchildren` - :ref:`release-v12-0-warping-wcs-diff-systems` - :ref:`release-v12-0-mininitialradius` - :ref:`release-v12-0-fix-cmodel-math` - :ref:`release-v12-0-dipole-centroid-slot` - :ref:`release-v12-0-example-updates` - :ref:`release-v12-0-log-task-failures` - :ref:`release-v12-0-skymap` - :ref:`release-v12-0-coadd-variance` - :ref:`release-v12-0-deblended-variance` - :ref:`release-v12-0-apcorr-logic` - :ref:`release-v12-0-catalog-columns` - :ref:`release-v12-0-wcslib` - :ref:`release-v12-0-obs-subaru-rotation` - :ref:`release-v12-0-conf-overrides-failure` - :ref:`release-v12-0-sdssshape-flags` - :ref:`release-v12-0-contant-background-interpolation` - :ref:`release-v12-0-filter-fallback-message` - :ref:`release-v12-0-pixel-padding` .. _release-v12-0-persist-ltvn-header: Persist LTVn headers as floating point numbers ---------------------------------------------- When persisting to a FITS file, these header cards were previously, incorrectly, stored as integers. :jirab:`DM-4133` .. _release-v12-0-identifying-peaks-in-merge: Fix bug when identifying existing peaks in a merge -------------------------------------------------- If two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixed. :jirab:`DM-2978` .. _release-v12-0-getchildren: Fix situation in which the getChildren method of SourceCatalog may return the wrong information ----------------------------------------------------------------------------------------------- The :lmeth:`getChildren` method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The :lmeth:`getChildren` method was updated to raise an exception if the precondition of sorting is not met. :jirab:`DM-2976` .. _release-v12-0-warping-wcs-diff-systems: Fix warping when the WCS have different coordinate systems ---------------------------------------------------------- Warping assumed that the sky representation of both WCS was identical. :jirab:`DM-4162` .. _release-v12-0-mininitialradius: Correct bad default minInitialRadius for CModel ----------------------------------------------- The ``minInitialRadius`` configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications. :jirab:`DM-3821` .. _release-v12-0-fix-cmodel-math: Correct algebraic error in CModel uncertainty calculation --------------------------------------------------------- There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude. :jirab:`DM-3821` .. _release-v12-0-dipole-centroid-slot: NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot ------------------------------------------------------------------------------------ Previously, initializing these algorithms was only possible if a centroid was already defined. That was not only unnecessary, but also made them more complicated to use, particularly in testing. :jirab:`DM-3940` .. _release-v12-0-example-updates: Update (some) example code to run with recent stack versions ------------------------------------------------------------ Changes in :lmod:`afw::table` had broken :file:`examples/calibrateTask.py` in ``pipe_tasks``. It has now been updated to comply with the latest :lmod:`afw::table` API. :jirab:`DM-4125` .. _release-v12-0-log-task-failures: Fix a failure to appropriately log failed task execution -------------------------------------------------------- When task execution fails, we add a message to the log (with level ``FATAL``). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved. :jirab:`DM-4218` .. _release-v12-0-skymap: Updates to Skymap packages -------------------------- Add functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and ``tract,patch`` indices. Functions include :lfunc:`findClosestTractPatchList`, :lfunc:`findAllTract`, and :lfunc:`findTractPatchList` which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively. :jirab:`DM-3775` .. _release-v12-0-coadd-variance: Fix variance in coadded images ------------------------------ Warping images in order to coadd them loses variance into covariance. This is mitigated by scaling the variance plane of the coadd. The scaling was being applied incorrectly in some cases. This has now been fixed. :jirab:`DM-4798` .. _release-v12-0-deblended-variance: Fix variance in deblended sources --------------------------------- The deblender incorrectly scaled the variance plane in deblended sources with the fraction of the total flux assigned to the source. This has been corrected. :jirab:`DM-4845` .. _release-v12-0-apcorr-logic: Fix logic for applying aperture corrections ------------------------------------------- This fixes a bug whereby the aperture corrections were being applied only after all the measurement plugins had run through, independent of their execution order. This resulted in plugins whose measurements rely on aperture corrected fluxes (i.e. with execution order > ``APCORR_ORDER``) being applied prior to the aperture correction, leading to erroneous results. The only plugin that was affected by this at this time was ``base_ClassificationExtendedness``. :jirab:`DM-4836` .. _release-v12-0-catalog-columns: More uniform support for assigning to catalog columns ----------------------------------------------------- Assignment of scalars or NumPy arrays to columns of afw.table.Catalog objects (e.g. ``catalog["column"] = value``) is now more uniformly supported across types (support was inconsistent before, and never allowed scalar or augmented assignment). Flag columns still do not support column assignment, and Flag column access still returns a copy, not a view, because Flag values are stored internally as individual bits within a larger integer. :jirab:`DM-4856` .. _release-v12-0-wcslib: Upgraded WCSLIB to version 5.13 ------------------------------- Version 5.13 of WCSLIB resolves memory corruption errors that could crash the stack in some circumstances. :jirab:`DM-4904,RFC-89,DM-4946,DM-3793` .. _release-v12-0-obs-subaru-rotation: Fix rotation for instrument signature removal in obs_subaru ----------------------------------------------------------- Approximately half of the HSC CCDs are rotated 180 deg with respect to the others. Two others have 90 deg rotations and another two have 270 deg rotations (see HSC CCD layout). The raw images for the rotated CCDs thus need to be rotated to match the rotation of their associated calibration frames (in the context of how they have currently been ingested) prior to applying the corrections. This is accomplished by rotating the exposure using the rotated context manager function in ``obs_subaru``\ 's :command:`isr.py` and the ``nQuarter`` specification in the policy file for each CCD. Currently, rotated uses ``afw``\ 's ``rotateImageBy90`` (which apparently rotates in a counter-clockwise direction) to rotated the exposure by ``4 - nQuarter`` turns. This turns out to be the wrong rotation for the odd ``nQuarter`` CCDs. This issue fixes this bug, leading to much improved processing of HSC CCD's 100, 101, 102, and 103. Note that, in the future, the ingestion of the calibration data will be updated such that no rotations are necessary (so they will then be removed from ``obs_subaru`` accordingly). :jirab:`DM-4998` .. _release-v12-0-conf-overrides-failure: Fix a silent failure to apply config overrides ---------------------------------------------- When applying a config override, using a variable which hadn't been defined should throw a ``NameError``, which ultimately propagates to the end user to notify them that something has gone awry. This warning was being incorrectly suppressed. :jirab:`DM-5729` .. _release-v12-0-sdssshape-flags: Correctly set flags for bad SdssShape measurements. --------------------------------------------------- The :lclass:`SdssShape` algorithm provides both shape and flux measurements. In some cases, a failed shape measurement could go un-noticed, resulting in an incorrect and unflagged flux measurement being associated with that source. This is now checked for, and bad fluxes are appropriately flagged. :jirab:`DM-3935` .. _release-v12-0-contant-background-interpolation: Fix CONSTANT background interpolation of bad data ------------------------------------------------- When performing interpolation over bad data (e.g. every pixel masked), all interpolation types other than ``CONSTANT`` would return ``NaN``\ s; ``CONSTANT`` would throw. This has now been changed so that ``CONSTANT`` also returns ``NaN``\ s. :jirab:`DM-5797` .. _release-v12-0-pixel-padding: Accommodate pixel padding when unpersisting reference catalog matches --------------------------------------------------------------------- The reference object loader in ``meas_algorithm``\ 's :command:`loadReferenceObjects.py` grows the ``bbox`` by the config parameter ``pixelMargin`` (padding to add to 4 all edges of the bounding box [pixels]) when setting the radius of the sky circle to be searched in the reference catalog. This is set to 50 by default but was not reflected by the radius parameter set in the metadata, which left open the possibility that some matches could reside outside the circle searched within the unpersisted radius. Additionally, the match metadata was being set after the exposure's WCS had been updated, also leading to an inconsistency with the sky circle that was actually searched. We now ensure that the actual sky circle that was searched for reference objects is the one set and persisted in the match metadata. :jirab:`DM-5686` .. _release-v12-0-filter-fallback-message: Correct misleading filter fallback error message ------------------------------------------------ When failing to load a ``calib``, if ``fallbackFilterName`` was not set, a confusing and apparently unrelated error message would be generated (``Unknown value type for filter: ``). This has been corrected to properly inform the user about the issue. :jirab:`DM-6151` .. _release-v12-0-internal-improvements: Build and Code Improvements =========================== - :ref:`release-v12-0-numpy-110` - :ref:`release-v12-0-boost-warning` - :ref:`release-v12-0-remove-task-display` - :ref:`release-v12-0-mask-to-defectlists` - :ref:`release-v12-0-ctrl-pool` - :ref:`release-v12-0-pipe-drivers` - :ref:`release-v12-0-test-tolerances` - :ref:`release-v12-0-filter-canonical-name` - :ref:`release-v12-0-clang-issues` - :ref:`release-v12-0-cmake-anaconda` - :ref:`release-v12-0-afwdata-tests` - :ref:`release-v12-0-disable-implicit-threading` - :ref:`release-v12-0-migrate-smart-pointers` .. _release-v12-0-numpy-110: Work-around incompatibilities with NumPy 1.10 --------------------------------------------- NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match. :jirab:`DM-4063,DM-4071,DM-4238`. .. _release-v12-0-boost-warning: When building boost warn user if user-config.jam or site-config.jam exists -------------------------------------------------------------------------- Building boost can fail if a :file:`user-config.jam` or :file:`site-config.jam` exist and have options which conflict with the LSST build configuration process. Introduce a warning message if either of these files are found to notify the user. :jirab:`DM-4198` .. _release-v12-0-remove-task-display: Remove deprecated Task.display() method --------------------------------------- This method has been deprecated since release 9.2 (S14). It has been removed from the codebase, and all stack code updated to directly interface with :lmod:`afw.display` or to use helper functions defined in ``meas_astrom``. :jirab:`DM-4428` .. _release-v12-0-mask-to-defectlists: Efficiency improvement in converting Masks to DefectLists --------------------------------------------------------- The previous version of routine was extremely memory intensive when large numbers of pixels were masked. :jirab:`DM-4800` .. _release-v12-0-ctrl-pool: Add a new task parallelization framework ---------------------------------------- The ``ctrl_pool`` package has been added to the LSST stack. This is a high-level parallelization framework used for distributing Task execution across a cluster, based on an MPI process pool. It is based on work carried out on Hyper Suprime-Cam. It is not intended to be the long-term solution to parallelized processing in the LSST stack, but meets our data processing needs until the fully-fledged parallelization middleware is available. :jirab:`DM-2983,DM-4835,DM-5409` .. _release-v12-0-pipe-drivers: Add parallel-processing top level tasks --------------------------------------- The new ``pipe_drivers`` package builds upon ``ctrl_pool``, above, to provide command-line scripts which coordinate distributed execution of the single-frame, coaddition and multiband processing steps using either the Python multiprocessing module or with a SLURM batch scheduler on a cluster. :jirab:`DM-3368,DM-3369,DM-3370` .. _release-v12-0-test-tolerances: Adjust test tolerances to be compatible with MKL-based NumPy ------------------------------------------------------------ Anaconda 2.5 ships, by default, with a version of NumPy built against Intel MKL rather than OpenBLAS. This can change some numerical results slightly, necessitating a change to test tolerances. :jirab:`DM-5108` .. _release-v12-0-filter-canonical-name: Now possible to directly get a Filter's canonical name and aliases ------------------------------------------------------------------ Added the convenience methods :lmeth:`getCanonicalName` and :lmeth:`getAliases` to :lmod:`lsst.afw.image.Filter`, accessible from both C++ and Python. These return the canonical name and the aliases, respectively, of the filter. This information was previously only available through an awkward sequence of method calls. :jirab:`DM-4816` .. _release-v12-0-clang-issues: Fix build issues with recent clang ---------------------------------- Recent releases of the clang C/C++ compiler, as shipped with Apple XCode, caused build failures in the stack. Although we believe this may be a problem with clang, we have worked around it within the stack code. We hope to track down the source of the error and, if appropriate, report it to the clang developers in future. :jirab:`DM-5590,DM-5609` .. _release-v12-0-cmake-anaconda: Fix incorrect linking against Anaconda-provided libraries when using CMake -------------------------------------------------------------------------- Some external packages---mariadb and mariadbclient---use a CMake based build system. This can incorrectly link against some libraries bundled with the Anaconda Python distribution, rather than the system-provided equivalents, resulting in a build failure. We have adjusted the build process of the affected packages to work around this error. :jirab:`DM-5595` .. _release-v12-0-afwdata-tests: Execute afw test suite when afwdata is not available ---------------------------------------------------- Some tests in the ``afw`` package rely on data from the ``afwdata`` package. The test suite would search for ``afwdata``, and skip all of the afw tests if ``afwdata`` is not available. This check has been made smarter, so that only tests which actually require ``afwdata`` are now skipped. :jirab:`DM-609` .. _release-v12-0-disable-implicit-threading: Disable implicit threading -------------------------- Low-level threading packages (such as OpenBLAS or MKL) can implicitly use many threads. Since the LSST stack also parallelizes at a higher level (e.g. using Python's multiprocessing module), this can cause undesirable contention. We now disable implicit threading when explicitly parallelizing at a higher level to protect the user from this. Implicit threading can be re-enabled by setting the ``LSST_ALLOW_IMPLICIT_THREADS`` environment variable. For more details, see this `Community post `__. :jirab:`DM-4719` .. _release-v12-0-migrate-smart-pointers: Migrate to standard smart pointers ---------------------------------- C++11 introduced new smart pointer types (``std::unique_ptr``, ``std::shared_ptr`` and ``std::weak_ptr``). We have migrated from the previously used Boost smart pointers to their standard equivalents. :jirab:`DM-5879,DM-4008,RFC-100,DM-5966` .. _release-v12-0-sui: Science User Interface ====================== During cycles winter and extra 2016, SUIT group has embarked on a major rewrite of the Firefly client side code, converting the Java/GWT based code to pure JavaScript (ES6) code, adopting React/Redux framework. The goal is to finish more than 90% of the client code conversion by the end of X16. We will make a release version of the code by end of September 2016. .. _release-v12-0-doc: Documentation improvements ========================== A number of documentation improvements were made, three of which may be of general interest: DM Developer Guide ------------------ New life has been breathed into the `DM Developer Guide`_ both on the surface and behind the scenes. While the primary audience for this are LSST developers, it can be a useful guide to anyone who wishes to participate in our open source development---all the fun without the paperwork! You can find it at `developer.lsst.io`_ .. _DM developer guide: https://developer.lsst.io .. _developer.lsst.io: https://developer.lsst.io LSST Technotes -------------- We have a lightweight bootstrap that allows contributors to produce ReStructuredText documents that are then published via our web services using Sphinx and some of our won sauce. You can see a rich example in Colin Slater's technote, "DMTN-006: False Positive Rates in the LSST Image Differencing Pipeline," at `dmtn-006.lsst.io`_. A `list of available technotes `_ is currently curated on the Community forum. .. _dmtn-006.lsst.io: http://dmtn-006.lsst.io/en/latest/ If you have been thinking hard about an LSST-related technical or scientific issue that you feel is below your "worth a paper" threshold, why not contribute it as a technote? We have a `lsst-technote-bootstrap`_ project to get you get started. .. _lsst-technote-bootstrap: https://github.com/lsst-sqre/lsst-technote-bootstrap Release Notes and Installation ------------------------------ You're reading them now on `pipelines.lsst.io`_\ ! We have moved our release note and installation instructions to our Spinx-based platform as well. This means that if you find a deficiency with our notes that needs clarification, or discover a new issue, you are quite welcome to fork-and-PR them; the GitHub repository is `pipelines_docs`_. .. _pipelines.lsst.io: https://pipelines.lsst.io/ .. _pipelines_docs: https://github.com/lsst/pipelines_docs