Running the AP pipeline (Gen 2)¶

Setup¶

Pick up where you left off in Getting Started. This means you already have a repository of ingested DECam data and have setup the LSST Science Pipelines stack as well as ap_pipe and ap_association.

Your directory structure should look something like

repo/   <-- where you will be running ap_pipe
   registry.sqlite3   <--- created during image ingestion
   2000-01-31/
       g/
           image1.fits   <--- linked here by default during image ingestion
           image2.fits
           ...
   ...
   ref_cats/   <--- copied or linked here manually by you
      gaia/
         shard1
         shard2
         ...
      pan-starrs/
         shard1
         shard2
         ...
   calibs/
      calibRegistry.sqlite3   <--- created/updated during calib ingestion
      cpBIAS/
         bias1.fits   <--- linked here by default during calib ingestion
         bias2.fits
         ...
      cpFLAT/
         flat1.fits   <--- linked here by default during calib ingestion
         flat2.fits
         ...
      defects/   <--- put here during curated calib ingestion
      crosstalk/   <--- put here during curated calib ingestion
   templates/   <--- does not need to be here, but needs to exist somewhere
      repositoryCfg.yaml
      deepCoadd/
         g/
            0/
               0,0.fits
               0,1.fits
               ...

AP pipeline on the command line¶

The executable to run for the AP Pipeline (ApPipeTask) is in ap_pipe/bin/ap_pipe.py.

To process your ingested data, run

mkdir apdb/
make_apdb.py -c isolation_level=READ_UNCOMMITTED -c db_url="sqlite:///apdb/association.db"
ap_pipe.py repo --calib repo/calibs --rerun processed -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db" --id visit=123456^123457 ccdnum=42 filter=g --template templates

In this case, a processed directory will be created within repo/rerun and the results will be written there. See Setting up the Alert Production Database for ap_pipe for more information on make_apdb.py.

This example command only processes observations that have a dataId corresponding to visits 123456 and 123457, with ccdnum 42 and the g filter.

lsst.ap.pipe supports dataId parsing, e.g., ccdnum=3^6..12 will process ccdnums 3, 6, 7, 8, 9, 10, 11, and 12.

Note

Until a resolution for DM-12672 is found, you should include a filter in the dataId string for ap_pipe to run successfully.

If you prefer to have a standalone output repository, you may instead run

ap_pipe.py repo --calib repo/calibs --output path/to/put/processed/data/in -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db" --id visit=123456^123457 ccdnum=42 filter=g --template path/to/templates

In this case, the output directory will be created if it does not already exist. If you omit the --template flag, ap_pipe will assume the templates are somewhere in repo.

Note

If you are using the default (SQLite) association database, you must configure the database location, or ap_pipe will not run. The location is a path to a new or existing database file to be used for source associations (including associations with previously known objects, if the database already exists). In the examples above, it is configured with the -c option, but a personal config file may be more convenient if you intend to run ap_pipe many times.

Expected outputs¶

If you used the rerun option above, most of the output from ap_pipe should be written out in the repo/rerun/processed directory,. The exception is the source association database, which will be written to the location you configure. The result from running ap_pipe on DECam data should look something like

apdb/
   association.db   <--- the Alert Production Database with DIAObjects
repo/
   rerun/
      processed/
         repositoryCfg.yaml
         deepDiff/
            v123456/   <--- difference images and DIASource tables are in here
            v123457/
         123456/   <--- all other processed data products are in here (calexps etc.)
         123457/

This is one example, and your rerun or output directory structure may differ. Of course, to inspect this data with the Butler, you don’t need to know where it lives on disk. You should instead instantiate a Butler within python in the processed directory and access the data products that way.

For example, in python

import lsst.daf.persistence as dafPersist
butler = dafPersist.Butler('repo/rerun/processed')
dataId = {'visit': 123456, 'ccdnum': 42, 'filter': 'g'}
calexp = butler.get('calexp', dataId=dataId)
diffim = butler.get('deepDiff_differenceExp', dataId=dataId)
diaSourceTable = butler.get('deepDiff_diaSrc', dataId=dataId)

Calexp template mode¶

By default, ap_pipe assumes you would like to use PSF-matched coadd images as templates for difference imaging. However, the pipeline also supports using calibrated exposures (calexps) as templates instead. A configuration file config/calexpTemplates.py is included witha ap_pipe to enable this.

Note

This functionality is available in the Gen 2 alert production pipeline, but is not tested as thoroughly as coadd templates. For technical reasons, use of calexp templates will not be supported in the Gen 3 pipeline.

To use ap_pipe in calexp template mode, point to the config file with the --configfile (-C) flag and additionally specify the dataId of the template with the --templateId flag, e.g.,

-C $AP_PIPE_DIR/config/calexpTemplates.py --templateId visit=234567

Be sure to also specify the location of the repo containing the calexp templates with the --template flag if they are not in the main repo. A full command looks like

ap_pipe.py repo --calib repo/calibs --rerun processed -C $AP_PIPE_DIR/config/calexpTemplates.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template /path/to/calexp/templates --templateId visit=234567

Supplemental information¶

Previewing dataIds¶

So far, we have implicitly assumed that you know reasonable values to choose for the dataId values (i.e., visit, ccdnum, and filter for DECam). While it is your responsibility to ensure the data you want to process and your templates do indeed overlap with each other, ap_pipe supports the --show data flag.

To get a list of all the g-band dataIds available in repo in lieu of actually running ap_pipe, try

ap_pipe.py repo --calib repo/calibs --rerun processed --id filter=g --show data

Running on other cameras¶

Running ap_pipe on cameras other than DECam works much the same way: you need to provide a raw repo and either a rerun or an output repo, and you may need to provide calib or template repos. The calexp configuration file will work with any camera.

You will need to use a dataId formatted appropriately for the camera; check the camera’s obs package documentation or consult the –show data flag.

Common errors¶

‘No locations for get’: This means you are trying to access a data product which the Butler cannot find. It is common to encounter this if you do not have all of the calibration products in the right spot or a template image cannot be accessed.

Navigation

Running the AP pipeline (Gen 2)¶

Setup¶

AP pipeline on the command line¶

Expected outputs¶

Calexp template mode¶

Supplemental information¶

Previewing dataIds¶

Running on other cameras¶

Common errors¶