Overview of the AP pipeline#

lsst.ap.pipe is a data processing pipeline for Prompt Data Products. It operates on ingested raw data in a Butler repository. It also requires appropriate calibration products and templates. As it runs, the pipeline generates calibrated images, difference images, difference image source catalogs, and a source association database.

The initial motivation for lsst.ap.pipe, information about one of the original test datasets, and an outdated tutorial are available in DMTN-039.

The AP Pipeline calls several main tasks and their associated subtasks:

  1. IsrTask, which performs image reduction;

  2. CharacterizeImageTask, which estimates the background and point-spread function of an image;

  3. CalibrateTask, which performs photometric and astrometric calibration;

  4. AlardLuptonSubtractTask, which subtracts a warped template from an image;

  5. DetectAndMeasureTask, which detects and measures diaSources on an image difference and

  6. DiaPipelineTask, which makes a catalog of Difference Image Analysis (DIA) Objects from the DIASources created during image differencing.

In practice, lsst.ap.pipe is often discussed in the context of lsst.ap.verify. The former is responsible for running the AP Pipeline. The latter uses lsst.ap.pipe to verify the output.

ap_pipe is entirely written in Python. Key contents include:

  • ApPipe.yaml: a Pipeline configuration for running the entire AP Pipeline.

  • LSSTCam/ApPipeDaytime.yaml: the variant used for daytime (non-real-time) LSSTCam processing. It drops loadDiaCatalogs and has DiaPipelineTask read the DIAObject and DIASource history from the APDB during association, so that the duplicate DIASource check sees rows written by any earlier pass over the same image. Prompt Processing must not use it; the preload is what keeps the APDB out of its latency-critical path.

  • ApPipe-legacy.yaml and ApPipeWithPreconvolution-legacy.yaml: variants that write legacy afw images. See Image types.

By default the pipeline is limited to running on data taken in filter bands whose names match those used by the Rubin Observatory LSST Camera (that is ugrizy). In order to run on bands outside of these filters, one must add the associated columns to the Apdb schema and add the band names to the config of DiaPipelineTask.

Image types#

The image_type pipeline parameter, declared in _ingredients/ApPipe.yaml, selects the storage classes of the image datasets. The default, future, writes lsst.images types: VisitImage for preliminary_visit_image, and DifferenceImage for the template and difference images. The value legacy writes afw ExposureF datasets. The parameter also applies to the imported analysis_tools pipeline, and overrides the value that pipeline declares.

Two connections name a component of an image dataset. The component names differ between the two types, so the parameter cannot switch them:

  • buildTemplate:connections.wcs: preliminary_visit_image.sky_projection (future) or preliminary_visit_image.wcs (legacy).

  • filterDiaSource:connections.diffImVisitInfo: difference_image.obs_info (future) or difference_image.visitInfo (legacy).

The instrument directories provide ApPipe-legacy.yaml and, where the base pipeline exists, ApPipeWithPreconvolution-legacy.yaml. These set image_type: legacy and override both connections. LSSTComCamSim/ApPipe.yaml and LSSTCam/ApPipeDaytime.yaml have no legacy variant; Writing legacy afw images shows how to run them in legacy mode. The SingleFrame.yaml pipelines contain neither connection, so -c parameters:image_type=legacy alone switches them. The ApPipeWithFakes pipelines always write legacy images; see ApPipeWithFakes pipeline.