Install with newinstall.sh and eups distrib

This page guides you through installing the LSST Science Pipelines software. This installation method is recommended for anyone who uses or develops the Pipelines software.

If you have installation issues, here are some ways to get help:

1. Prerequisites

You can install the LSST Science Pipelines on CentOS 7 (LSST’s reference platform) as well as other Linux distributions and macOS (see the LSST Stack Testing Status for information on the platforms we have tested with).

Before you begin, install prerequisite software for your platform:

2. Make an installation directory

First, choose where you want to install the LSST Science Pipelines. Create and change into that directory:

mkdir -p lsst_stack
cd lsst_stack

If you are installing the software for multiple users (a shared stack), see Setting unix permissions for shared installations.

3. Run newinstall.sh

Open a new shell session for the installation (or ensure you haven’t used LSST software in that shell before). If you need to reuse a shell, see Running newinstall.sh in an already set-up shell.

Next, run newinstall.sh to set up the environment you’ll install the LSST Science Pipelines into. For most use cases we recommend downloading and running newinstall.sh like this:

curl -OL https://raw.githubusercontent.com/lsst/lsst/14.0/scripts/newinstall.sh
bash newinstall.sh -ct

Always execute newinstall.sh with bash, as shown, regardless of what shell you are in.

We recommend that you opt into the provided Miniconda Python environment (see the links below for more information).

Then load this environment into your shell:

source loadLSST.bash # for bash
source loadLSST.csh  # for csh
source loadLSST.ksh  # for ksh
source loadLSST.zsh  # for zsh

Note

Here are ways to customize the newinstall.sh installation for specific needs:

For background information about newinstall.sh, see:

4. Install Science Pipelines packages

Install the LSST Science Pipelines packages by running eups distrib install for a top-level package and a tagged version.

This example installs the v14_0_rc1 tagged version (current release) of the lsst_distrib top-level package:

eups distrib install lsst_distrib -t v14_0_rc1 lsst_distrib
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
setup lsst_distrib

If pre-built binaries are available for your platform (and you ran newinstall.sh with the -t argument) the installation should take roughly 10 minutes. Otherwise, installation falls back to a source build which can take two hours, depending on the top-level package and your machine’s performance.

The last command, setup, activates the installed packages in your shell environment. You’ll need to run setup in each shell session you’ll use the LSST Science Pipelines in. See Setting up installed LSST Science Pipelines for more information.

Note

5. Test your installation (optional)

Once the LSST Science Pipelines are installed, you can verify that it works by running a demo pipeline. This demo processes a small amount of SDSS data and verifies that measurements match expected values.

See Testing the Science Pipelines installation with a demo for instructions.

Next steps

Add links to topics the reader should go to next. Examples:

  • A topic reminding a user to setup the stack in a new shell.
  • Topic explaining top-level packages.
  • Topic on compiling a package alongside an existing stack.

Advanced installation topics

The above steps guided you through LSST’s recommended installation. These topics provide additional information about the installation and ways to customize it:

Setting unix permissions for shared installations

You can make the LSST Science Pipelines installation accessible to multiple users on the same machine.

First, create a separate unix group (called lsst, for example) with a umask of 002 (all access permissions for the group and allow other users to read/execute).

Then set the ownership of the installation directory to the lsst group, have the SGID (2000) bit set, and allow group read/write/execute (mode 2775).

Running newinstall.sh in an already set-up shell

If you’ve run the LSST Science Pipelines previously, you may have conflicting environment variables in your shell. To be safe, run:

unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH

Then return to the instructions step 3. Run newinstall.sh.

What newinstall.sh does

newinstall.sh creates a self-contained environment on your machine where you can install, run, and develop the LSST Science Pipelines. You activate this environment in a shell by sourcing the loadLSST script in the installation directory (see Setting up).

Here is how newinstall.sh prepares the environment:

  • Identifies your operating system and compilers to determine what EUPS binary packages should be installed (the EUPS package root, see About EUPS tarball packages).
  • Installs a specific version of Python, through Miniconda, that is compatible with EUPS binary packages (see About the Miniconda Python installed by newinstall.sh).
  • Installs Conda packages that the LSST Science Pipelines depends on (see Python dependencies).
  • Checks for git on your systems and offers to install it if necessary.
  • Installs EUPS, the package manager used by the LSST software stack.

For information about newinstall.sh’s arguments, see newinstall.sh argument reference.

About the Miniconda Python installed by newinstall.sh

newinstall.sh can install a dedicated Python environment for your LSST Science Pipelines installation. This Python installation isn’t required, but we recommend it. See How to use your own Python with newinstall.sh if required.

The Python environment installed by newinstall.sh is Miniconda, a minimal version of Anaconda. By default, newinstall.sh installs Python 3.5.2. If you need to work with your own Python 2.7-only packages, see How to install a Python 2.7 environment with newinstall.sh.

In this Miniconda environment, newinstall.sh installs the Science Pipeline’s Python prerequisites. See Python dependencies for more information.

This Miniconda installation won’t affect your other Python installations (like the system’s Python, your own Anaconda or Miniconda, or virtual environments). The LSST Miniconda environment is only active when you source the loadLSST script installed by newinstall.sh (see Setting up installed LSST Science Pipelines).

If you install other Python packages in a shell where the LSST Miniconda is activated (with pip install or conda install) those packages are installed into the LSST Miniconda’s site-packages, not your system’s. The Python installed by newinstall.sh works like an isolated Python environment dedicated to LSST Science Pipelines code and your own related modules—effectively like a Conda environment or Python venv. This pattern is useful because it reduces the risk of having Python package version incompatibilities.

How to install a Python 2.7 environment with newinstall.sh

LSST Science Pipelines is backwards compatible with Python 2.7. If you need to run your own Python 2.7-only Python packages in conjunction with the Pipelines, you can have newinstall.sh install a Python 2.7 environment for you instead of the default Python 3.5 environment.

To select Python 2.7, run newinstall.sh with the -2 flag (in addition to other flags, like -t):

bash newinstall.sh -2

Then follow the remaining instructions at 3. Run newinstall.sh.

See also: About the Miniconda Python installed by newinstall.sh.

How to use your own Python with newinstall.sh

newinstall.sh creates a new Python environment by default (pre-configured with Python dependencies). If necessary, you can use your own pre-existing Python environment.

To do so, run newinstall.sh (see newinstall-run for details and command arguments). When newinstall-run prompts you to install Miniconda, type no.

Be aware of these caveats when using your own Python installation:

  • You are responsible for installing Python package dependencies. See Python dependencies.
  • Prebuilt binaries will not be available. eups distrib install will always install from source.

About EUPS tarball packages

EUPS distrib binary (tarball) packages significantly speed up your installation. Rather than compiling the LSST Science Pipelines from source, EUPS tarballs are prebuilt packages made specifically for supported platforms.

Platforms are defined by four factors:

  1. Operating system.
  2. Compiler.
  3. Miniconda (Python) version.
  4. lsstsw version (Git ref).

EUPS distrib binary packages are currently being built for these platform combinations:

EUPS distrib binary flavors
OS Compiler Python
macOS osx/10.9 clang-800.0.42.1 miniconda3-4.2.12 (Python 3)
macOS osx/10.9 clang-800.0.42.1 miniconda2-4.2.12 (Python 2)
Redhat redhat/el7 gcc-system miniconda3-4.2.12 (Python 3)
Redhat redhat/el7 gcc-system miniconda2-4.2.12 (Python 2)
Redhat redhat/el6 devtoolset-3 miniconda2-4.2.12 (Python 2)

When you run newinstall.sh, it looks at your system to identify your operating system and compiler. The version of newinstall.sh you run also determines the Miniconda version and the lsstsw build system versions.

Together, these four factors define the URL prefix (called an EUPS package root) that eups distrib install looks for binary packages from. If binary tarballs are unavailable for that EUPS package root, eups distrib install automatically falls back to compiling LSST Science Pipelines packages from source.

You can see the active EUPS package roots on your system by running:

eups distrib path

Here is an example of the output:

https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67
https://eups.lsst.codes/stack/src

Based on this example, eups distrib install will preferentially install EUPS distrib binary packages for the macOS 10.9 system, clang-800.0.42.1 compiler, and miniconda3-4.2.12-7c8e67 Python and lsstsw combination. If eups distrib install cannot find packages at that EUPS package root it will look in the second EUPS package root (https://eups.lsst.codes/stack/src), which provides source packages.

EUPS tarball packages and compiler compatibility

EUPS binary tarball packages are prebuilt on LSST’s continuous integration servers for a specific combination of operating system, compilers, Python, and Python dependencies. If you are developing packages alongside this installation, you might encounter application binary interface (ABI) incompatibilities if are using a different compiler version or a different Python environment.

In this case, the more reliable solution is to revert to a source installation. To do this, repeat the installation but run newinstall.sh without the -t argument:

bash newinstall.sh -c

Without the -t argument to newinstall.sh, eups distrib install will always build and install packages from source, ensuring ABI compatibility.

Installing other releases (including daily and weekly tags)

The instructions on this page guide you through installing the current release of the LSST Science Pipelines corresponding to this documentation. You can, however, install other releases by running eups distrib install with a different tag.

The common types of tags are:

  • Major releases, tagged as v<MAJOR>_<MINOR> (for example, v14_0).
  • Weekly builds, tagged as w_<YEAR>_<N> (for example, w_2017_33 is the 33rd weekly build in 2017).
  • Daily builds, tagged as d_<YEAR>_<MONTH>_<DAY> (for example, d_2017_09_01 is the daily build for September 1, 2017).

There are also tags pointing to the most recent releases:

  • Current major release, tagged as current.
  • Current weekly build, tagged as w_latest.
  • Current daily build, tagged as d_latest.

You can see all available tags at https://eups.lsst.codes/stack/src/tags (each tag has a .list file).

Note

Binary installations may not be available for all tags. From https://eups.lsst.codes/stack, browse subdirectories corresponding to your platform and look for .list files of available tags. eups distrib install automatically falls back to a source build if binaries are not available.

Warning

You need to ensure that the Python environment created by newinstall.sh (see step 3. Run newinstall.sh) is compatible with the tagged software.

For example, if you are installing a recent weekly you may need to download and run newinstall.sh from master:

curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh
bash newinstall.sh -ct

See https://github.com/lsst/lsst/tags for available tagged versions of newinstall.sh.

newinstall.sh argument reference

usage: newinstall.sh [-b] [-f] [-h] [-n] [-3|-2] [-t|-T] [-s|-S] [-P <path-to-python>]
-b

Run in batch mode. Don’t ask any questions and install all extra packages.

-c

Attempt to continue a previously failed install.

-n

No-op. Go through the motions but echo commands instead of running them.

-P <PATH_TO_PYTHON>

Use a specific python interpreter for EUPS.

-2

Use Python 2 if the script is installing its own Python.

-3

Use Python 3 if the script is installing its own Python. (default)

-t

Use pre-compiled EUPS “tarball” packages, if available.

-T

Do not use pre-compiled EUPS “tarball” packages. (default)

-s

Use EUPS source “eupspkg” packages, if available (compile from source).

-S

Do not use EUPS source “eupspkg” packages (do not compile from source).

-h

Display this help message.