Installation with lsstsw and lsst-build¶

This page guides you through installing the LSST Science Pipelines from source with lsstsw and lsst-build. These are the same tools LSST Data Management uses to build and test the Science Pipelines.

Since lsstsw presents the Science Pipelines as a directory of Git repositories cloned from github.com/lsst, this installation method can be very convenient for developing Science Pipelines code, particularly when modifying multiple packages at the same time. Other methods of installing LSST Science Pipelines software are newinstall.sh and Docker.

If you have issues using lsstsw, here are two ways to get help:

Review the known installation issues.
Ask a question on the LSST Community support forum.

1. Prerequisites¶

The LSST Science Pipelines are developed and tested primarily on CentOS, but can be compiled and run on macOS, Debian, Ubuntu, and other Linux distributions. See Platform compatibility for information about LSST’s official reference platform and build reports with other platforms, and follow the instructions under System prerequisites to ensure you have installed the prerequisite software for your platform.

2. Deploy lsstsw¶

Begin by choosing a working directory, then deploy lsstsw into it:

git clone https://github.com/lsst/lsstsw.git
cd lsstsw
./bin/deploy
source bin/envconfig

If you are running in a csh or tcsh, change the last line to:

source bin/envconfig.csh

For more information about the deploy command, see About the lsstsw deploy script.

If you intend to use a Git LFS repository, like testdata_ci_hsc or afwdata, you should configure Git LFS before you continue.

3. Build the Science Pipelines packages¶

From the lsstsw directory, run:

rebuild lsst_distrib

Once the rebuild step finishes, note the build number printed on screen. It is formatted as “bNNNN.” Tag this build as current so that EUPS can set it up by default:

eups tags --clone bNNNN current

Finally, set up the packages with EUPS:

setup lsst_distrib

See Setting up installed LSST Science Pipelines for more information.

Note

You can do more with the build command, including building from branches of GitHub repositories. For more information:

About the lsstsw rebuild command.
Building from branches.
lsstsw rebuild command reference.

4. Testing Your installation (optional)¶

Once the LSST Science Pipelines are installed, you can verify that it works by running a demo project.

5. Upgrading your installation¶

You can upgrade an lsstsw installation in-place by running the following commands from within your lsstsw/ directory.

Start a shell that has not sourced envconfig; you may have to comment out a line in e.g. your .bashrc. The bin/deploy script needs to run without an active environment to be able to install a new one.
git pull to download the latest environment definition.
bin/deploy to install that new conda environment.
Start a new shell for the final command, to ensure your shell environment is properly configured for the new lsstsw env, and source lsstsw/bin/envconfig if it is not automatically sourced during your shell startup.
rebuild -u lsst_distrib to download the latest repos definition file and rebuild the entire Science Pipelines codebase.

If you do not intend to use your older builds in the future, you can remove all of the sub-directories in your stack/VERSION/ (where VERSION is the old environment version) path before the upgrade, to save space and reduce the number of eups package versions.

Sourcing the Pipelines in a new shell¶

In every new shell session you will need to set up the Science Pipelines environment and EUPS package stack.

Run these two steps:

Activate the lsstsw software environment by sourcing the envconfig script in lsstsw’s bin directory:
```
source bin/envconfig
```
If you are running in a csh or tcsh, run this set up script instead:
```
source bin/envconfig.csh
```
Set up a top-level package:
```
setup lsst_distrib
```
Instead of lsst_distrib, you can set up a different top-level package like lsst_apps or any individual EUPS package you previously installed. See Top-level packages to install.

About the lsstsw deploy script¶

The deploy script automates several things to prepare an LSST development environment:

Installs Miniconda and a Python 3 environment specific to this lsstsw workspace, including (another) Git and Git LFS.
Installs EUPS into eups/current/.
Clones lsst-build, the tool that runs the build process.
Clones versiondb, a robot-managed Git repository of package dependency information.
Creates an empty stack installation directory, stack/.

This environment, including the EUPS, Miniconda, Git, and Git LFS software, is only activated when you source the bin/envconfig or bin/envconfig.csh scripts in a shell. Otherwise, lsstsw does not affect the software installed on your computer.

About the lsstsw rebuild command¶

The rebuild command accomplishes the following:

Clones all Science Pipelines packages from github.com/lsst. The repos.yaml file in the https://github.com/lsst/repos repository maps package names to GitHub repositories.
Runs the Scons-based build process to compile C++, make Pybind11 bindings, and ultimately create the lsst Python package. The stack is built and installed into the stack/ directory inside your lsstsw/ work directory.

lsstsw clones repositories using HTTPS (see repos.yaml). Our guide to Setting up a Git credential helper will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials.

Building from branches¶

lsstsw’s rebuild command enables you to clone and build development branches.

To build lsst_distrb, but use the Git branch my-feature when it’s available in a package’s repository:

rebuild -r my-feature lsst_distrib

Multiple ticket branches across multiple products can be built in order of priority:

rebuild -r feature-1 -r feature-2 lsst_distrib

In this example, a feature-1 branch will be used in any package’s Git repository. A feature-2 branch will be used secondarily in repositories where feature-1 doesn’t exist. Finally, lsstsw falls back to using the master branch for repositories that lack both feature-1 and feature-2.

lsstsw deploy command reference¶

usage: deploy.sh [-2|-3] [-b] [-h]

-b¶: Use bleeding-edge conda packages.

-h¶: Print the help message.

-r REF¶: Use a particular git ref of the conda packages in scipipe_conda_env.

lsstsw rebuild command reference¶

rebuild [-p] [-n] [-u] [-r <ref> [-r <ref2> [...]]] [-t <eupstag>] [product1 [product2 [...]]]

-p¶: Prep only.

-n¶: Do not run git fetch in already-downloaded repositories.

-u¶: Update the repos.yaml package index to the master branch on GitHub of https://github.com/lsst/repos.

-r <git ref>¶: Rebuild using the Git ref. A Git ref can be a branch name, tag, or commit SHA. Multiple -r arguments can be given, in order or priority.

-t¶: EUPS tag.

Navigation