Getting started with the AP pipeline

Installation

lsst.ap.pipe is available from the LSST Science Pipelines. It is installed as part of the lsst_apps and lsst_distrib metapackages.

Ingesting data files

LSST-style image processing typically operates on Butler repositories and does not directly interface with data files. lsst.ap.pipe is no exception. The process of turning a set of raw data files and corresponding calibration products into a format the Butler understands is called ingestion. Ingestion can be somewhat camera-specific, and is outside the scope of the AP Pipeline.

A utility to ingest data before running lsst.ap.pipe is available in ap_verify. However, this works only on datasets which adhere to the ap_verify dataset format. Alternately, you may use a pre- ingested dataset or manually ingest files yourself following the directions for a given obs_ package, e.g., step 4 of the obs_decam README.

A standard ingestion workflow for DECam looks something like

ingestImagesDecam.py input_loc --filetype raw path/to/raw/files
ingestCalibs.py input_loc --calib calib_loc path/to/flats/and/biases --validity 999
ingestCalibs.py input_loc --calib calib_loc --calibType defect --mode=skip path/to/defects --validity 0

Note

Defect ingestion is a step unique to DECam. It presently requires --mode=skip, this mode interprets paths as relative to calib_loc, and the validity value is not used (but must be included). This interface may change when DM-5467 is completed.

Required data products

For the AP Pipeline to successfully process data, the following is required:

  • Raw science images and reference catalogs ingested into a main Butler repository
    • The reference catalogs must be in a directory called ref_cats with subdirectories for each catalog containing the appropriate catalog shards. We recommend using Pan-STARRS for photometry and gaia for astrometry. An example config file for using these two catalogs can be found in the ap_verify_hits2015 repository.
  • Calibration products (biases, flats, and defects, if applicable) ingested into a Butler repository you must specify with the --calib flag on the command line at runtime
    • To check if this requirement has been satisfied, you can inspect the calibRegistry.sqlite3 created in this repository and ensure the information in the flat, bias, and defect tables is accurate
  • Template images (of type deepCoadd by default) for difference imaging must be either in the main Butler repository or in another location you may specify with the --template flag on the command line at runtime

A sample dataset from the DECam HiTS survey that works with ap_pipe in the The dataset framework format is available as ap_verify_hits2015. However, this dataset must be ingested as described in Ingesting data files, and the reference catalog and defect files must be decompressed and extracted.

Please continue to Pipeline Tutorial for more details about running the AP Pipeline and interpreting the results.