.. _lsst.obs.decam.gen3: ################# The Gen3 workflow ################# These directions are for users who wish to create their own data repositories independent of any Rubin/LSST-supported shared repositories. They assume the user has a recent weekly version of ``lsst_distrib`` setup. Local repository setup ---------------------- Create a Gen3 data repository and write curated calibration products to it (these include crosstalk, linearity, and defects). The latter step creates the ``DECam/calib`` collection. In the example here, ``REPO`` refers to the location you choose for the Butler repository. .. prompt:: bash cd REPO butler create REPO butler register-instrument REPO lsst.obs.decam.DarkEnergyCamera butler write-curated-calibrations REPO lsst.obs.decam.DarkEnergyCamera Check that this worked successfully by querying all the collections currently in the repo, e.g., .. prompt:: bash butler query-collections REPO --chains=tree This should print a short list of collections, including several beginning with ``DECam/calib/curated`` and one called ``DECam/raw/all``. Ingest raw images ----------------- Ingest raw science and calibration (bias and flat) frames, which creates the ``DECam/raw/all`` collection. Next, run overscan correction on all the raws via the ``RunIsrForCrosstalkSources.yaml`` pipeline so they may be used as crosstalk sources during ISR (this is a DECam-specific step). .. note:: Until `DM-30651 `__ is resolved, consider including the following configurations during the overscan crosstalk prep step. When running the ``RunIsrForCrosstalkSources.yaml`` pipeline, append: ``-c overscan:overscan.fitType='MEDIAN_PER_ROW'``. .. prompt:: bash butler ingest-raws REPO /path/to/raw/science/*.fits.fz --transfer link butler ingest-raws REPO /path/to/raw/calibs/*.fits.fz --transfer link pipetask run -b REPO -i DECam/raw/all -o DECam/raw/crosstalk-sources -p $CP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types Check that this worked successfully, e.g., .. prompt:: bash butler query-dimension-records REPO exposure --where "instrument='DECam' AND exposure.observation_type='zero'" butler query-dimension-records REPO exposure --where "instrument='DECam' AND exposure.observation_type='dome flat'" This should print tables of familiar-looking calibration exposures, with reasonable exposure and detector numbers, filter information, times, sky location, and other metadata. Use ``cp_pipe`` to build nightly calibs --------------------------------------- Next, build nightly (or similar) bias and flat frames using :py:mod:`lsst.cp.pipe` and certify them into a calib collection. This example assumes the user has a single night of observations with bias frames numbered 1-6 and flat frames numbered 7-12, and wants to create nightly calib products valid for 24 hours. Note the bias building pipeline is camera-agnostic, but the flat-building pipeline has a prerequisite DECam-specific step which correctly handles inter-chip crosstalk. .. note:: Until `DM-30651 `__ is resolved, consider including the following configurations during bias and flat building. When running the ``cpBias.yaml`` pipeline, append: ``-c isr:overscan.fitType='MEDIAN_PER_ROW'``. When running the ``cpFlat.yaml`` pipeline, append: ``-c isr:overscan.fitType='MEDIAN_PER_ROW' -c cpFlatNorm:level='AMP'``. .. prompt:: bash pipetask run -d "exposure IN (1, 2, 3, 4, 5, 6)" -b REPO -i DECam/raw/all,DECam/calib -o u/username/bias-construction-night1 -p $CP_PIPE_DIR/pipelines/cpBias.yaml --register-dataset-types butler certify-calibrations REPO u/username/bias-construction-night1 DECam/calib/run1 bias --begin-date 2021-01-01T00:00:00 --end-date 2021-01-01T23:59:59 pipetask run -d "exposure IN (7, 8, 9, 10, 11, 12)" -b REPO -i DECam/raw/all,DECam/raw/crosstalk-sources,DECam/calib -o u/username/flat-construction-night1 -p $CP_PIPE_DIR/pipelines/DarkEnergyCamera/cpFlat.yaml --register-dataset-types butler certify-calibrations REPO u/username/flat-construction-night1 DECam/calib/run1 flat --begin-date 2021-01-01T00:00:00 --end-date 2021-01-01T23:59:59 Science time! ------------- Now you can proceed with running ISR and other "processCcd" tasks via a Gen3 pipeline. .. note:: Until `DM-30651 `__ is resolved, consider configuring ``-c isr:overscan.fitType='MEDIAN_PER_ROW'`` as above when running ISR. Some useful pipelines can be found in the ``pipelines/DarkEnergyCamera`` directory of the :py:mod:`lsst.ap.pipe` package. These and other pipelines may move to ``recipes/DarkEnergyCamera`` as `RFC-775 `__ is implemented. .. _lsst.obs.decam.gen2: ############################ The deprecated Gen2 workflow ############################ Create a Gen2 data repository directory: .. prompt:: bash mkdir /path/to/repo echo lsst.obs.decam.DecamMapper > /path/to/repo/_mapper Ensure you have obs_decam setup and built as well as obs_decam_data. Import raw and calibration data into the data repository, for example: .. prompt:: bash ingestCuratedCalibs.py /path/to/repo --calib /path/to/calib/repo $OBS_DECAM_DATA_DIR/decam ingestImages.py /path/to/repo --filetype raw /path/to/raw/*.fits.fz ingestCalibs.py /path/to/repo --calib /path/to/calib/repo /path/to/ias-and-flat-files/*fits --validity 999 By default, ingesting calibration data only creates a repository database. When ingesting biases and flats, if you would like to also link these files (in the same way as images are ingested), use ``--mode=link``. Process data, noting you will want to turn astrometry and photometric calibration on if you have appropriate reference catalogs available: .. prompt:: bash processCcd.py /path/to/repo --id visit=283453 ccdnum=10 --output /path/to/your/output/repo/ -C /path/to/your/config/override/file --config calibrate.doAstrometry=False calibrate.doPhotoCal=False To use instcal files from the community pipeline, replace the ISR task with :ref:`DecamNullIsrTask` by using a config override file containing the following: .. code-block:: python from lsst.obs.decam.decamNullIsr import DecamNullIsrTask config.isr.retarget(DecamNullIsrTask)