Building a package with the installed Science Pipelines stack

You can build packages alongside an LSST Science Pipelines stack that you have installed with newinstall.sh, lsstsw, or Docker. This page describes how to build and set up packages cloned directly from GitHub.

Note

If you are developing with the LSST Docker images, refer to How to develop packages inside Docker containers. That page describes Docker-specific patterns and complements the general information on this page.

0. Prerequisites

Before developing an individual package, you need an installed Science Pipelines stack. You can install the Pipelines through any of these methods: newinstall.sh, lsstsw, or Docker.

Then set up the Pipelines software in a shell. See Setting up installed LSST Science Pipelines for more information.

Warning

newinstall.sh users: read EUPS tarball packages and compiler compatibility if the installed Pipelines stack uses prebuilt binary packages.

1. Clone a package

Use Git to clone the package you want to work on. Most LSST packages are available in the LSST’s GitHub organization (https://github.com/lsst). The repos.yaml file in the ‘repos’ repository also maps package names to repository URLs.

For example:

git clone https://github.com/lsst/pipe_tasks
cd pipe_tasks

You can also create a new package repository, though this is beyond this document’s scope.

You will work from a Git branch when developing a package. The DM Developer Guide describes the branching workflow that LSST staff use.

Note

Docker users: Clone the package onto your host filesystem rather than directly into the container by mounting a host directory in the container. See How to develop packages inside Docker containers.

Warning

lsstsw users: The lsstsw/build directory already includes clones of Git repositories. These repositories are reset when you run rebuild, though, so you can potentially lose local changes. It’s usually better to clone and work with Git repositories outside of the lsstsw directory.

2. Set up the package

From the package’s directory, set up the package itself in the EUPS stack:

setup -r . -t $USER

3. Build the package with Scons

scons -Q -j 6 opt=3

These flags configure Scons:

  • -Q: reduce logging to the terminal.

  • -j 6: build in parallel (for example, with ‘6’ CPUs).

  • opt=3: build with level 3 optimization. Use opt=0 (or opt=g with gcc compilers) for debugging.

This scons command will run several targets by default, in sequence:

  1. lib: build the C++ code and Pybind11 interface layer.

  2. python: install the Python code.

  3. tests: run the unit tests.

  4. example: compile the examples.

  5. doc: compile Doxygen-based documentation.

  6. shebang: convert the #!/usr/bin/env line in scripts for OS X compatibility (see DMTN-001).

You can build a subset of these targets by specifying one explicitly. For example, to compile C++, build the Python package and run tests:

scons -Q -j 6 opt=3 tests

Next steps

By following these steps, you have built a package from source alongside an installed Science Pipelines software stack. Now when you run the Science Pipelines, your new package will be used instead of the equivalent package provided by the Science Pipelines installation. Here are some tasks related to maintaining this development software stack:

Reviewing set up packages

Packages that are set up are part of the active Science Pipelines software stack. You can see what packages are currently set up by running:

eups list -s

You can also review what version of a single package is set up by running:

eups list <package name>

Setting up in a new shell

Whenever you open a new shell you need to set up both the LSST software environment and the LSST software stack. See Setting up installed LSST Science Pipelines for the basic procedure.

In addition to setting up the installed Science Pipelines software, you separately need to set up the development package itself. You can do this following the instruction in step 2. Set up the package.

Un-set up the development package

You can un-set up a development package to revert to the installed LSST Science Pipelines distribution.

To switch from a development package to the released package:

setup -j <package name> -t current

current is the default tag normally used for the installed LSST Science Pipelines software stack.

To un-set up a development package without replacing it:

unsetup -j <package name> -t $USER

This is useful if you are developing a new package that is not part of the installed LSST Science Pipelines software stack.