Setting up the Alert Production Database for ap_pipe

In its default configuration, the Alert Production Pipeline, as represented by lsst.ap.pipe.ApPipeTask (Gen 2) or pipelines/ApPipe.yaml (Gen 3), relies on a database to save and load DIASources and DIAObjects. When running as part of the operational system, this database will be provided externally. However, during testing and development, developers can run make_apdb.py to set up their own database. This page provides an overview of how to use make_apdb.py.

Configuring the database

The database is configured using ApdbConfig. ap_pipe.py command line users can pass configuration information to the script through the --config and --configfile command-line options. make_apdb.py also uses ApPipeConfig and the --config and --configfile options, so users can pass exactly the same arguments to make_apdb.py and ap_pipe.py. Supporting identical command line arguments for both scripts makes it easy to keep the database settings in sync.

For pipetask users the process is almost the same, except that pipetask’s config syntax is not exactly the same. The --config and --configfile options for pipetask use colons as separators between each task and its config; replace these with periods for make_apdb.py.

Note that apdb.db_url has no default; a value must be provided by the user.

Examples

Databases can be configured using direct config overrides (see Running the AP pipeline (Gen 2) for an explanation of the ap_pipe.py command line):

make_apdb.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED diaPipe.apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr
ap_pipe.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED diaPipe.apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr repo --calib repo/calibs --rerun myrun --id [optional IDs to process]

make_apdb.py ignores any ApPipeConfig fields not related to the APDB (in the example, differencer.coaddName), so there is no need to filter them out.

In Gen 3, this becomes (see Running the AP pipeline (Gen 3) for an explanation of pipetask):

make_apdb.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED diaPipe.apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr
pipetask run -p ApPipe.yaml -c diaPipe:apdb.isolation_level=READ_UNCOMMITTED diaPipe:apdb.db_url="sqlite:///databases/apdb.db" differencer:coaddName=dcr -b repo -o myrun

Databases can also be set up using config files:

make_apdb.py -C myApPipeConfig.py
ap_pipe.py repo --calib repo/calibs --rerun myrun -C myApPipeConfig.py --id [optional ID to process]

Further reading

• Running the AP pipeline (Gen 2)
• Running the AP pipeline (Gen 3)