The dataset framework

The ap_verify system is designed to allow integration testing of the Alert Production pipeline on a variety of LSST precursor data. The dataset framework provides a common format and delivery system for the test data. In effect, datasets serve as an adapter between raw observatory output and the LSST observatory interface (obs) framework.

Overview

Datasets are implemented as Git LFS repositories with a specific format. They provide the raw and calibration data files needed for an ap_verify run, and identify the observatory used to take the data. The observatory’s obs package can then be used by ap_verify to ingest the data into the LSST system and run the pipeline. Datasets are deliberately simple to allow them to be created and maintained without much knowledge of the LSST stack, particularly of the Butler I/O framework.

In depth

• Packaging data as a dataset
• Installing datasets
• Datasets vs. Butler repositories

Existing datasets

• HiTS2015
• HiTS2015 CI Subset