Building a package with the installed Science Pipelines stack¶
You can build packages alongside an LSST Science Pipelines stack that you have installed with newinstall.sh, lsstsw, or Docker. This pages describes how to build and set up packages cloned directly from GitHub.
If you are developing with the LSST Docker images, refer to How to develop packages inside Docker containers. That page describes Docker-specific patterns and complements the general information on this page.
Then set up the Pipelines software in a shell. See Setting up installed LSST Science Pipelines for more information.
newinstall.sh users: read EUPS tarball packages and compiler compatibility if the installed Pipelines stack uses prebuilt binary packages.
1. Clone a package¶
Use Git to clone the package you want to work on. Most LSST packages are available in the LSST’s GitHub organization (https://github.com/lsst). The repos.yaml file in the ‘repos’ repository also maps package names to repository URLs.
git clone https://github.com/lsst/pipe_tasks cd pipe_tasks
You can also create a new package repository, though this is beyond this document’s scope.
You will work from a Git branch when developing a package. The DM Developer Guide describes the branching workflow that LSST staff use.
Docker users: Clone the package onto your host filesystem rather then directly into the container by mounting a host directory in the container. See How to develop packages inside Docker containers.
lsstsw users: The
lsstsw/build directory already includes clones of Git repositories.
These repositories are reset when you run rebuild, though, so you can potentially lose local changes.
It’s usually better to clone and work with Git repositories outside of the
2. Set up the package¶
From the package’s directory, set up the package itself in the EUPS stack:
setup -r . -t $USER
3. Build the package with Scons¶
scons -Q -j 6 opt=3
These flags configure Scons:
-Q: reduce logging to the terminal.
-j 6: build in parallel (for example, with ‘6’ CPUs).
opt=3: build with level 3 optimization. Use
opt=gwith gcc compilers) for debugging.
scons command will run several targets by default, in sequence:
lib: build the C++ code and Pybind11 interface layer.
python: install the Python code.
tests: run the unit tests.
example: compile the examples.
doc: compile Doxygen-based documentation.
shebang: convert the
#!/usr/bin/envline in scripts for OS X compatibility (see DMTN-001).
You can build a subset of these targets by specifying one explicitly. For example, to compile C++, build the Python package and run tests:
scons -Q -j 6 opt=3 tests
By following these steps, you have built a package from source alongside an installed Science Pipelines software stack. Now when you run the Science Pipelines, your new package will be used instead of the equivalent package provided by the Science Pipelines installation. Here are some tasks related to maintaining this development software stack:
Reviewing set up packages¶
Packages that are set up are part of the active Science Pipelines software stack. You can see what packages are currently set up by running:
eups list -s
You can also review what version of a single package is set up by running:
eups list <package name>
Setting up in a new shell¶
Whenever you open a new shell you need to set up both the LSST software environment and the LSST software stack. See Setting up installed LSST Science Pipelines for the basic procedure.
In addition to setting up the installed Science Pipelines software, you separately need to set up the development package itself. You can do this following the instruction in step 2. Set up the package.
Un-set up the development package¶
You can un-set up a development package to revert to the installed LSST Science Pipelines distribution.
To switch from a development package to the released package:
setup -j <package name> -t current
current is the default tag normally used for the installed LSST Science Pipelines software stack.
To un-set up a development package without replacing it:
unsetup -j <package name> -t $USER
This is useful if you are developing a new package that is not part of the installed LSST Science Pipelines software stack.