The Gen3 workflow

These directions are for users who wish to create their own data repositories independent of any Rubin/LSST-supported shared repositories. They assume the user has a recent weekly version of lsst_distrib setup.

Local repository setup

Create a Gen3 data repository and write curated calibration products to it (these include crosstalk, linearity, and defects). The latter step creates the DECam/calib collection. In the example here, REPO refers to the location you choose for the Butler repository.

Check that this worked successfully by querying all the collections currently in the repo, e.g.,

This should print a short list of collections, including several beginning with DECam/calib/curated and one called DECam/raw/all.

Ingest raw images

Ingest raw science and calibration (bias and flat) frames, which creates the DECam/raw/all collection. Next, run overscan correction on all the raws via the RunIsrForCrosstalkSources.yaml pipeline so they may be used as crosstalk sources during ISR (this is a DECam-specific step).

Check that this worked successfully, e.g.,

This should print tables of familiar-looking calibration exposures, with reasonable exposure and detector numbers, filter information, times, sky location, and other metadata.

Use cp_pipe to build nightly calibs

Next, build nightly (or similar) bias and flat frames using lsst.cp.pipe and certify them into a calib collection.

This example assumes the user has a single night of observations with bias frames numbered 1-6 and flat frames numbered 7-12, and wants to create nightly calib products valid for 24 hours. Note the bias building pipeline is camera-agnostic, but the flat-building pipeline has a prerequisite DECam-specific step which correctly handles inter-chip crosstalk.

Science time!

Now you can proceed with running ISR and other “processCcd” tasks via a Gen3 pipeline.

Some useful pipelines can be found in the pipelines/DECam directory of the lsst.ap.pipe and lsst.drp.pipe packages.