Running with Docker¶
LSST provides versioned Docker images containing the Science Pipelines software. With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. Docker is an effective and reliable alternative to the newinstall.sh and lsstsw-based methods that install LSST software directly on your system.
If you have issues using the LSST Docker images, reach out on the LSST Community support forum.
To download Docker images and run containers, you need Docker’s software. The Docker Community Edition is freely available for most platforms, including macOS, Linux, and Windows.
If you haven’t used Docker before, you might want to learn more about Docker, images, and containers. Docker’s Getting Started documentation is a good resource.
This command downloads a weekly build of the LSST Science Pipelines, starts a container, and opens a prompt:
docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2017_35
Then in the container’s shell, load the LSST environment and set up a top-level package (
lsst_distrib in this case):
source /opt/lsst/software/stack/loadLSST.bash setup lsst_distrib
When you’re done with the container, exit from the container’s shell:
This returns you to the original shell on your host system.
Next, learn more with these topics:
How to mount a host directory into a container¶
When you run a Docker container, you’re working inside a system that is isolated from your host machine. The container’s filesystem is distinct from your host machine’s.
You can mount a host directory into the container, however. When you mount a host directory to a container, the data and code that resides on your host filesystem is accessible to the container’s filesystem. This is useful for processing data with the LSST Science Pipelines and even developing packages for the Science Pipelines.
To mount a local directory, add a
-v <host directory>/<mount directory> argument to the docker run command.
docker run -it -v `pwd`:/home/vagrant/mnt lsstsqre/centos:7-stack-lsst_distrib-w_2017_35
The example mounts the current working directory (
`pwd`) to the
/home/vagrant/mnt directory in the container.
If you run ls from the container’s prompt you should see all files in the current working directory of the host filesystem:
/home/vagrant is the default home directory for LSST Science Pipelines Docker images.
As usual with interactive mode (
docker run -it), you can
exit from the container’s shell to stop the container and return to the host shell:
How to run a container in the background and attach to it¶
The Quick start showed you how to run a container in interactive mode.
In this mode, Docker immediately opens a shell in the new container.
exit from the shell, the container stops.
An alternative is to run a container in a detached state. With a detached container, the container won’t stop until you specify it.
To get started, run the container with the
-d flag (detached):
docker run -itd --name lsst lsstsqre/centos:7-stack-lsst_distrib-w_2017_35
You still use the
-it arguments to put the container in interactive mode, even though Docker doesn’t immediately open a container prompt for you.
--name lsst argument gives the new container a name.
You can choose whatever name makes sense for your work.
This example uses the name “
Next, from a shell on your host system (the same shell as before, or even a new shell) open a shell in the container with the docker exec command:
docker exec -it lsst /bin/bash
Your prompt is now a prompt in the container.
You can repeat this process, attaching to the container multiple times, to open multiple container shells.
To close a container shell, type
Finally, to stop the container entirely, run this command from your host’s shell:
docker stop lsst
And delete the container:
docker rm lsst
How to develop packages inside Docker containers¶
You can develop code, including LSST Science Pipelines packages, with the LSST Science Pipelines Docker images. This section summarizes the containerized development workflow. Refer to Building a package with the installed Science Pipelines stack for general information.
Basic set up¶
These steps show how to run a container and build a LSST Science Pipelines package in it:
From the host shell, clone packages into the current working directory. For example:
git clone https://github.com/lsst/pipe_tasks
Any datasets you’re working with should be in the current working directory as well.
From the host shell, start the container with the current working directory mounted:
docker run -itd -v `pwd`:/home/vagrant/mnt --name lsst lsstsqre/centos:7-stack-lsst_distrib-w_2017_35
This starts the container in a detached mode so you can open and exit multiple container shells. Follow the steps in How to run a container in the background and attach to it to open a shell in the container.
From the container’s shell, activate the LSST environment and setup the top-level package:
source /opt/lsst/software/stack/loadLSST.bash setup lsst_distrib
From the container’s shell, change into the directory of the package you cloned and set it up. For example:
cd mnt/pipe_tasks setup -r .
Compared to the typical development work, the setup command shown here does not include the
-t $USERargument to tag the development package. This is because the Docker container doesn’t have a
$USERenvironment variable set by default. You can still set up and develop the package this way, it just won’t be tagged by EUPS.
From the container’s shell, build the package. For example:
scons -Q -j 6 opt=3
The containerized development workflow¶
To develop packages with Docker containers you will use a combination of shells and applications on both the host system and inside the Docker container.
On the host system you will run your own code editors and git to develop the package. This way you don’t have to configure an editor of git inside the container. This is why we mount a local directory with the code and data in it.
In container shells you run commands to set up packages (setup), compile code (scons), test code (pytest), and run the Pipelines on data (processCcd.py, for example). Use docker exec to open multiple shells in the container (see How to run a container in the background and attach to it).
Cleaning up the development container¶
You can stop and delete the container at any time:
docker stop <container name> docker rm <container name>
In this example, the container is named
Stopping and deleting a container doesn’t affect the data in the local directory you mounted into that container.
Finding images for different LSST Science Pipelines releases¶
Docker images are versioned with tags, allowing you to run any release of the LSST Science Pipelines software. The schema of these tags is:
<centos major version>-stack-<EUPS product>-<EUPS distrib tag>
This tag corresponds to:
- CentOS 7 operating system.
w_2017_35EUPS tag. See Installing other releases (including daily and weekly tags) for an overview of LSST’s EUPS tag schema.
You can see what tags are available by browsing lsstsqre/centos on Docker Hub.