Installation with lsstsw and lsst-build

This page will guide you through installing the LSST Science Pipelines from source with lsstsw and lsst-build. These are the same tools LSST Data Management uses to build and test the Science Pipelines.

Since lsstsw presents the Science Pipelines as a directory of Git repositories clone from github.com/lsst, this installation method can be very convenient for developing Science Pipelines code.

Science Pipelines developers should also consult the DM Developer Guide, and the Workflow page in particular.

If you have difficulty installing LSST software:

• review the known installation issues for your platform.
• reach out on the Support forum at community.lsst.org.

1. Prerequisites

You need to install some prerequisites to build the LSST Stack from source:

• Install prerequisites for your platform: macOS, Debian / Ubuntu, or Centos / RedHat.
• If you opt not to use lsstsw ’s default Python environment you need to install these Python dependencies.
• If you intend to use a Git LFS repository, like ci_hsc or afwdata, install and configure Git LFS.

2. Obtaining a Development Stack with lsstsw

Code for the LSST Stack is distributed across many Git repositories (see github.com/lsst). lsstsw is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing.

Begin by choosing a working directory, then deploy lsstsw into it:

git clone https://github.com/lsst/lsstsw.git
cd lsstsw
./bin/deploy
. bin/setup.sh

The deploy script automates several things for you:

1. installs a miniconda Python environment specific to this lsstsw workspace,
2. installs EUPS in eups/current/,
3. clones lsst-build, which will run the build process for us,
4. clones versiondb, a robot-made Git repository of package dependency information, and
5. creates an empty Stack installation directory, stack/.

lsstsw clones repositories using HTTPS. Our guide to Setting up a Git credential helper will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials.

3. Build Science Pipelines

From the lsstsw directory, run:

rebuild lsst_distrib

Once the rebuild step finishes, note the build number printed on screen. It is formatted as “bNNNN.” Tell EUPS this is the current build by making a clone of the build’s EUPS tag and calling it “current:”

eups tags --clone bNNNN current

The rebuild command accomplishes the following:

1. Clones all Science Pipelines packages from github.com/lsst. A high-bandwidth connection is helpful since the stack contains a non-trivial amount of code and test data.
2. Runs the Scons-based build process to compile C++, make Swig bindings, and ultimately create the lsst Python package. The Stack is built and installed into the stack/ directory inside your lsstsw/ work directory.

Finally, set up the packages with EUPS:

setup lsst_distrib

4. Sourcing the Pipelines in a New Shell

In every new shell session you will need to ‘setup’ the Science Pipelines. Do this by running the setup.sh from the lsstsw/ directory:

. bin/setup.sh
setup lsst_distrib  # or an alternative top-level package

Note

If you are using a tcsh shell, run . bin/setup.csh instead (note csh extension).

5. Testing Your Installation

Once the LSST Science Pipelines are installed, you can verify that it works by running a demo project. This demo processes a small amount of SDSS data.

6. Bonus: Developing a Package

An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. The Developer Guide describes Data Management’s workflow, but this section will get your started with the basics related to lsstsw and EUPS.

1. Stack packages are found in the lsstsw/build/ directory.

2. Create a new branch in a package’s Git repository,

   git checkout -b {{ticket-name}}
   

   Then declare this package for EUPS and set it up:

   eups declare -r . -t $USER {{package_name}} git
   setup -r . -t $USER
   

   Unpacking the eups declare arguments:

   • -r . is the path to the package’s repository, which is the current working directory. You don’t need to be in the repository’s directory if you provide the path appropriately.
   • -t $USER sets the EUPS tag. We use this because your username ($USER) is an allowed EUPS tag.
   • git is used as an EUPS version. Semantically we default to calling the version “git” to indicate this package’s version is the HEAD of a Git development branch.

   In the above eups declare command we associated the package version “git” with the tag “$USER.” In running setup, we told EUPS to setup the package and its dependencies with the version associated to the $USER tag. If the $USER tag isn’t found for dependencies, EUPS will revert to using versions of dependencies linked to the current tag. This is why we initially declared the entire lsstsw repository to have the version current.

3. Build the package with Scons:

   scons -Q -j 6 opt=3
   

   These flags tell SCons to build with flags:

   • -Q: reduce logging to the terminal,
   • -j 6: build in parallel (e.g., with ‘6’ CPUs),
   • opt=3: build with level 3 optimization.

   This scons command will run several targets by default, in sequence:

   1. lib: build the C++ code and SWIG interface layer
   2. python: install the Python code
   3. tests: run the test suite
   4. example: compile the examples,
   5. doc: compile Doxygen-based documentation, and
   6. shebang: convert the #!/usr/bin/env line in scripts for OS X compatibility (see DMTN-001).

   You can build a subset of these targets by specifying one explicitly. To simply compile C++, SWIG, build the Python package and run tests:

   scons -q -j 6 opt=3 tests