Installation with lsstsw and lsst-build¶

1. Prerequisites¶

Before embarking on Science Pipelines development, ensure you have all software dependencies installed on your system:

• Pipelines build pre-requisites listed in the eups distrib installation page.
• A bash shell; lsstsw may not work with other shells like zsh or tcsh.
• Git LFS configured for LSST DM’s servers.

2. Obtaining a Development Stack with lsstsw¶

Code for the LSST Stack is distributed across many Git repositories (see github.com/lsst). lsstsw is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing.

Begin by choosing a working directory, then deploy lsstsw into it:

1
2
3
4

git clone https://github.com/lsst/lsstsw.git
cd lsstsw
./bin/deploy
. bin/setup.sh

The deploy script automates several things for you:

1. installs a miniconda Python environment specific to this lsstsw workspace,
2. installs EUPS in eups/current/,
3. clones lsst-build, which will run the build process for us,
4. clones versiondb, a robot-made Git repository of package dependency information, and
5. creates an empty Stack installation directory, stack/.

lsstsw clones repositories using HTTPS. Our guide to Setting up a Git credential helper will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials.

3. Build Science Pipelines¶

From the lsstsw directory, run:

rebuild lsst_apps

rebuild lsst_distrib

Once the rebuild step finishes, note the build number printed on screen. It is formatted as “bNNNN.” Tell EUPS this is the current build by making a clone of the build’s EUPS tag and calling it “current:”

eups tags --clone bNNNN current

The rebuild command accomplishes the following:

1. Clones all Science Pipelines packages from github.com/lsst. A high-bandwidth connection is helpful since the stack contains a non-trivial amount of code and test data.
2. Runs the Scons-based build process to compile C++, make Swig bindings, and ultimately create the lsst Python package. The Stack is built and installed into the stack/ directory inside your lsstsw/ work directory.

4. Sourcing the Pipelines in a New Shell¶

In every new shell session you will need to ‘setup’ the Science Pipelines. Do this by running the setup.sh from the lsstsw/ directory:

. bin/setup.sh

5. Testing Your Installation¶

Once the LSST Science Pipelines are installed, you can verify that it works by running a demo project. This demo processes a small amount of SDSS data.

6. Bonus: Developing a Package¶

An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. The Developer Guide describes Data Management’s workflow, but this section will get your started with the basics related to lsstsw and EUPS.

1. Stack packages are found in the lsstsw/build/ directory.

2. Create a new branch in a package’s Git repository,

   git checkout -b {{ticket-name}}
   

   Then declare this package for EUPS and set it up:

   eups declare -r . -t $USER {{package_name}} git
   setup -r . -t $USER
   

   Unpacking the eups declare arguments:

   • -r . is the path to the package’s repository, which is the current working directory. You don’t need to be in the repository’s directory if you provide the path appropriately.
   • -t $USER sets the EUPS tag. We use this because your username ($USER) is an allowed EUPS tag.
   • git is used as an EUPS version. Semantically we default to calling the version “git” to indicate this package’s version is the HEAD of a Git development branch.

   In the above eups declare command we associated the package version “git” with the tag “$USER.” In running setup, we told EUPS to setup the package and its dependencies with the version associated to the $USER tag. If the $USER tag isn’t found for dependencies, EUPS will revert to using versions of dependencies linked to the current tag. This is why we initially declared the entire lsstsw repository to have the version current.

3. Build the package with Scons:

   scons -Q -j 6 opt=3
   

   These flags tell SCons to build with flags:

   • -Q: reduce logging to the terminal,
   • -j 6: build in parallel (e.g., with ‘6’ CPUs),
   • opt=3: build with level 3 optimization.

   This scons command will run several targets by default, in sequence:

   1. lib: build the C++ code and SWIG interface layer
   2. python: install the Python code
   3. tests: run the test suite
   4. example: compile the examples,
   5. doc: compile Doxygen-based documentation, and
   6. shebang: convert the #!/usr/bin/env line in scripts for OS X compatibility (see DMTN-001).

   You can build a subset of these targets by specifying one explicitly. To simply compile C++, SWIG, build the Python package and run tests:

   scons -q -j 6 opt=3 tests