Development Tutorial with lsstsw and lsst-build

The page Installing the LSST Science Pipelines describes how to obtain a released version of the Science Pipelines. However, developers will want to work and build directly against the Pipelines’s Git repositories. This page describes how to develop the Science Pipelines using the lsstsw build tool.

  1. Obtaining a Development Stack with lsstsw
  2. Working on a Ticket
  3. Continuous Integration with Jenkins
  4. Making a Pull Request and Merging
  5. Resetting your lsstsw development stack
  6. Extending the lsstsw Workflow

Developers should consult the LSST DM Developer Guide, which comprehensively covers DM coding standards and development workflows.

Prerequisites

Before embarking on Science Pipelines development, ensure you have all software dependencies installed on your system. These dependencies are listed in the guide to installing the stack from source.

For lsstsw-based development you will need to work in a bash shell.

Additionally, consult the Development Workflow page of the Developer Guide for guidance setting up Git and Git LFS for DM development and our policies on JIRA ticketing, Git branching, testing with Jenkins, code review and merging. We’ll link to relevant sections of the Development Workflow throughout this tutorial.

Obtaining a Development Stack with lsstsw

Code for the LSST Stack is distributed across many Git repositories (see github.com/lsst). lsstsw is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing.

Step 1. Get lsstsw

Begin by choosing a work directory, then clone lsstsw into it:

git clone https://github.com/lsst/lsstsw.git
cd lsstsw

Step 2. Deploy lsstsw

Prepare the development environment by running two commands in the lsstsw/ directory:

./bin/deploy
. bin/setup.sh

The deploy script automates several things for you:

  1. installs a miniconda Python environment specific to this lsstsw workspace,
  2. installs EUPS in eups/current/,
  3. clones lsst-build, which will run the build process for us,
  4. clones versiondb, a robot-made Git repository of package dependency information, and
  5. creates an empty Stack installation directory, stack/.

By default, lsstsw clones repositories using HTTPS. Our guide to Setting up a Git credential helper will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials.

The setup.sh step enables EUPS, the package manager used by LSST. Whenever you open a new terminal session, you need to run ‘. bin/setup.sh’ to activate your lsstsw environment.

Step 3. Download and build the stack

Run

rebuild lsst_apps

Initially this will git clone all of the Stack repositories. A high-bandwidth connection is helpful since the stack contains a non-trivial amount of code and test data.

Next, rebuild will run our Scons-based build process to compile C++, make Swig bindings, and ultimately create the lsst Python package. The Stack is built and installed into the stack/ directory inside your lsstsw/ work directory.

Note that we ran rebuild lsst_apps since lsst_apps is a meta package that depends on the entire Stack, thus ensuring you have a complete Stack to develop on.

Step 4. Tag the current build

Once the rebuild step finishes, take note of the build number printed on screen. It is formatted as “bNNNN.” Tell EUPS this is the current build by making a clone of the build’s EUPS tag and calling it “current:”

eups tags --clone bNNNN current

Note: this command will print eups tags: local variable 'tagNames' referenced before assignment; this is a known EUPS bug that doesn’t affect functionality.

You now have a working Stack, ready for development.

Working on a Ticket

At LSST Data Management, we use tickets on JIRA to track work. You might be assigned an existing ticket, or you might create a new ticket to work on. These tickets are named “DM-MMMMM.”

When beginning any Stack development work, ensure lsstsw is setup in your terminal sessions. From the lsstsw/ directory:

. bin/setup.sh

Step 1. Create ticket branches for repositories in development

Make a ticket branch for each repository involved in your ticket work. From a package’s repository in lsstsw/build:

git checkout -b tickets/DM-MMMM

(repeat for other packages in development)

Note that you can do local work on arbitrarily-named branches, but all commits that you intend to make a pull request for must be in tickets/DM-MMMM branches. If you want to push non-ticket work up an LSST repository on GitHub you can prefix your branch’s name with u/{{username}}/ (as in, your GitHub username). Our developer workflow page explains DM’s Git branch policy.

Next, create this branch on the GitHub remote. From a package’s repository in lsstsw/build:

git push -u

(repeat for other packages in development)

This initial push will create a remote branch origin/tickets/DM-MMMM and track it so that you can simply git push and git pull without arguments between the ticket branch on the origin remote and your local clone.

Step 2. Declare these repositories to EUPS

We need to tell EUPS about these development repositories (with eups declare) and set them up for building (with setup). From a package’s repository in lsstsw/build:

eups declare -r . -t $USER {{package_name}} git
setup -r . -t $USER

(repeat for other packages in development)

Unpacking the eups declare arguments:

  • -r . is the path to the package’s repository, which is the current working directory. You don’t need to be in the repository’s directory if you provide the path appropriately.
  • -t $USER sets the EUPS tag. We use this because your username ($USER) is an allowed EUPS tag.
  • git is used as an EUPS version. Semantically we default to calling the version “git” to indicate this package’s version is the HEAD of a Git development branch.

In the above eups declare command we associated the package version “git” with the tag “$USER.” In running setup, we told EUPS to setup the package and its dependencies with the version associated to the $USER tag. If the $USER tag isn’t found for dependencies, EUPS will revert to using versions of dependencies linked to the current tag. This is why we initially declared the entire lsstsw repository to have the version current.

Step 3. Compile and test with SCons

Develop the package(s) as you normally would. To build the Stack with the newly-developed package, run SCons from the repository of a package being developed:

scons -Q -j 6 opt=3

These flags tell SCons to build with flags:

  • -Q: reduce logging to the terminal,
  • -j 6: build in parallel (e.g., with ‘6’ CPUs),
  • opt=3: build with level 3 optimization.

This scons command will run several targets by default, in sequence:

  1. lib: build the C++ code and SWIG interface layer
  2. python: install the Python code
  3. tests: run the test suite
  4. example: compile the examples,
  5. doc: compile Doxygen-based documentation, and
  6. shebang: convert the #!/usr/bin/env line in scripts for OS X compatibility (see DMTN-001).

You can build a subset of these targets by specifying one explicitly. To simply compile C++, SWIG, build the Python package and run tests:

scons -q -j 6 opt=3 tests

If you are developing multiple packages simultaneously on the same ticket branch, you can compile and test all of them with the rebuild command from lsstsw/:

rebuild -r tickets/DM-MMMM lsst_apps

This will build all Stack repositories within the lsst_apps umbrella using the tickets/DM-MMMM ticket branch if available (falling back to the master branch).

Continuous Integration with Jenkins

We use a Jenkins instance to run continuous integration tests on the LSST Stack. Jenkins tests the Stack against multiple environments, ensuring that your code is robust.

Step 1. Ensure the code is pushed

git push all commits in development branches of packages to the remote development branches on GitHub.

Step 2. Log into ci.lsst.codes

Open https://ci.lsst.codes/job/stack-os-matrix/build?delay=0sec in a browser and setup an account if you have not already done so. Once logged in you will see the Jenkins job submission page. On that page:

  1. Enter the name(s) of development branches to include in the build in the BRANCHES field.
  2. Click the Submit button and wait.

You can monitor builds in the “Bot: Jenkins” HipChat room. If your build is shown to have failed, you can click on the ‘Console’ link in the HipChat message to see a build log.

Making a Pull Request and Merging

Once your code is passing tests, it’s ready to be packaged, sent for review, and ultimately merged.

Our DM Development Guide has a comprehensive section on DM’s code review process. Please review that document thoroughly to learn about the nuances of DM’s development workflow; here we outline the basic steps:

  1. Rebase your commit history against the latest master branch (or other integration branch) and update your development branch on GitHub.
  2. Create a GitHub pull request.
  3. Assign a reviewer.
  4. Discuss the code on the GitHub pull request page.
  5. Merge the ticket branch and mark the ticket as done.

Resetting your lsstsw development stack

Removing Eups username tags

Once your work is merged into master, Eups no longer needs to track the git development version; instead we can use the default current tag to refer to the latest build.

To remove your Eups username tag, run this command from each package repository involved in your previous development:

eups undeclare -t $USER {{package_name}} git

Replace the version name as needed if you didn’t use the default EUPS version ‘git’ (from Step 2 of Working on a Ticket).

Rebuilding your lsstsw development stack

Finally, you can also update your entire development stack. This involves pulling master branches for all Stack repositories and recompiling the Stack from source. lsstsw automates this with the rebuild command. Before rebuilding, ensure that any work in any Git repository has been pushed to GitHub. rebuild wipes the existing repositories. Unpushed work will be deleted.

From the lsstsw/ directory:

rebuild lsst_apps

Then re-tag the build as current (see above).

Extending the lsstsw Workflow

The above workflow described an idealized case of working on a single ticket. This section describes how to extend the basic workflow for more complex cases.

Refreshing the master for the entire stack

If the ticket is taking an extended time to develop, you may need to update the master branches of the entire Stack to reliably test and merge your ticket branch. The most robust way to do this is by rebuilding the lsstsw environment completely (see above).

Before doing, ensure that all work is pushed to branches on GitHub.

After the rebuild, you will need to EUPS tag the current Stack, following the instructions above.

Finally, checkout your work branches from the GitHub remote and declare these work repositories to EUPS following.