Running with Docker

LSST provides versioned Docker images containing the Science Pipelines software. With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. Docker is an effective and reliable alternative to the newinstall.sh and lsstsw-based methods that install LSST software directly on your system.

If you have issues using the LSST Docker images, reach out on the LSST Community support forum.

Prerequisites

To download Docker images and run containers, you need Docker’s software. The Docker Community Edition is freely available for most platforms, including macOS, Linux, and Windows.

If you haven’t used Docker before, you might want to learn more about Docker, images, and containers. Docker’s Getting Started documentation is a good resource.

Quick start

This command downloads a current version of the LSST Science Pipelines Docker image (see also Finding images for different LSST Science Pipelines releases), starts a container, and opens a prompt:

docker run -ti lsstsqre/centos:7-stack-lsst_distrib-d_2020_01_13

Then in the container’s shell, load the LSST environment and activate the lsst_distrib top-level package:

source /opt/lsst/software/stack/loadLSST.bash
setup lsst_distrib

This step is equivalent to the set up instructions for a newinstall.sh-based installation. In fact, the images are internally based on newinstall.sh.

When you’re done with the container, exit from the container’s shell:

exit

This returns you to the original shell on your host system.

Next, learn more with these topics:

• How to mount a host directory into a container
• How to run a container in the background and attach to it
• How to develop packages inside Docker containers
• Finding images for different LSST Science Pipelines releases

How to mount a host directory into a container

When you run a Docker container, you’re working inside a system that is isolated from your host machine. The container’s filesystem is distinct from your host machine’s.

You can mount a host directory into the container, however. When you mount a host directory to a container, the data and code that resides on your host filesystem is accessible to the container’s filesystem. This is useful for processing data with the LSST Science Pipelines and even developing packages for the Science Pipelines.

To mount a local directory, add a -v <host directory>/<mount directory> argument to the docker run command. For example:

docker run -it -v `pwd`:/home/lsst/mnt lsstsqre/centos:7-stack-lsst_distrib-d_2020_01_13

The example mounts the current working directory (`pwd`) to the /home/lsst/mnt directory in the container.

If you run ls from the container’s prompt you should see all files in the current working directory of the host filesystem:

ls mnt

As usual with interactive mode (docker run -it), you can exit from the container’s shell to stop the container and return to the host shell:

exit

How to run a container in the background and attach to it

The Quick start showed you how to run a container in interactive mode. In this mode, Docker immediately opens a shell in the new container. When you exit from the shell, the container stops.

An alternative is to run a container in a detached state. With a detached container, the container won’t stop until you specify it.

To get started, run the container with the -d flag (detached):

docker run -itd --name lsst lsstsqre/centos:7-stack-lsst_distrib-d_2020_01_13

You still use the -it arguments to put the container in interactive mode, even though Docker doesn’t immediately open a container prompt for you.

The --name lsst argument gives the new container a name. You can choose whatever name makes sense for your work. This example uses the name “lsst.”

Next, from a shell on your host system (the same shell as before, or even a new shell) open a shell in the container with the docker exec command:

docker exec -it lsst /bin/bash

Your prompt is now a prompt in the container.

You can repeat this process, attaching to the container multiple times, to open multiple container shells.

To close a container shell, type exit.

Finally, to stop the container entirely, run this command from your host’s shell:

docker stop lsst

And delete the container:

docker rm lsst

How to develop packages inside Docker containers

You can develop code, including LSST Science Pipelines packages, with the LSST Science Pipelines Docker images. This section summarizes the containerized development workflow. Refer to Building a package with the installed Science Pipelines stack for general information.

Basic set up

These steps show how to run a container and build a LSST Science Pipelines package in it:

1. From the host shell, clone packages into the current working directory. For example:

   git clone https://github.com/lsst/pipe_tasks
   

   Any datasets you’re working with should be in the current working directory as well.

2. From the host shell, start the container with the current working directory mounted:

   docker run -itd -v `pwd`:/home/lsst/mnt --name lsst lsstsqre/centos:7-stack-lsst_distrib-d_2020_01_13
   

   This starts the container in a detached mode so you can open and exit multiple container shells. Follow the steps in How to run a container in the background and attach to it to open a shell in the container.

3. From the container’s shell, activate the LSST environment and setup the top-level package:

   source /opt/lsst/software/stack/loadLSST.bash
   setup lsst_distrib
   

4. From the container’s shell, change into the directory of the package you cloned and set it up. For example:

   cd mnt/pipe_tasks
   setup -r .
   

   Note

   Compared to the typical development work, the setup command shown here does not include the -t $USER argument to tag the development package. This is because the Docker container doesn’t have a $USER environment variable set by default. You can still set up and develop the package this way, it just won’t be tagged by EUPS.

5. From the container’s shell, build the package. For example:

   scons -Q -j 6 opt=3
   

The containerized development workflow

To develop packages with Docker containers you will use a combination of shells and applications on both the host system and inside the Docker container.

On the host system you will run your own code editors and git to develop the package. This way you don’t have to configure an editor of git inside the container. This is why we mount a local directory with the code and data in it.

In container shells you run commands to set up packages (setup), compile code (scons), test code (pytest), and run the Pipelines on data (processCcd.py, for example). Use docker exec to open multiple shells in the container (see How to run a container in the background and attach to it).

Cleaning up the development container

You can stop and delete the container at any time:

docker stop <container name>
docker rm <container name>

In this example, the container is named lsst.

Stopping and deleting a container doesn’t affect the data in the local directory you mounted into that container.

Finding images for different LSST Science Pipelines releases

LSST Science Pipelines Docker images are published as lsstsqre/centos on Docker Hub. These images are based on a CentOS base image.

Docker images are versioned with tags, allowing you to run any release of the LSST Science Pipelines software. The schema of these tags is:

<centos major version>-stack-<EUPS product>-<EUPS distrib tag>

For example:

7-stack-lsst_distrib-d_2020_01_13

This tag corresponds to:

• CentOS 7 operating system.
• lsst_distrib top-level package.
• d_2020_01_13 EUPS tag. See Installing other releases (including daily and weekly tags) for an overview of LSST’s EUPS tag schema.

You can see what tags are available by browsing lsstsqre/centos on Docker Hub.

See also

See Installing other releases (including daily and weekly tags) for information on the different types of EUPS tags.