The Butler Command¶

Note

This page describes how to extend the butler subcommand; see Butler Command-Line Reference for documentation on the subcommands defined in lsst.daf.butler. Additional subcommands are defined in other packages, especially lsst.obs.base; use butler --help to get a complete list.

daf_butler provides a command line interface command called butler. It supports subcommands, some of which are implemented in daf_butler.

The butler command and subcommands are implemented in click, which is well documented and has a good quickstart guide. There is more to know about click’s commands, options, and arguments than is described here, and the click documentation is a good resource for that.

This guide includes an overview of how subcommands, options, and arguments are used with the butler command.

It then describes how other packages can add subcommands to the butler command by way of a plugin system, in Adding Butler Subcommands.

Finally, the butler command line framework can be used to write other commands that load subcommands. This is described in Writing Other Commands

Subcommands¶

Subcommands are like the “pull” in git pull. The subcommand is implemented as a python function and decorated with @click.command. The name of the function will become the name of the subcommand on the command line.

Example of a command or subcommand definition:

 import click


 @click.group()
 def git():
     """An example git-style interface."""
     pass


 # Notice this uses "@git" instead of "@click", this adds the command to the
 # git group.
 @git.command()
 def pull():
     """An example 'pull' subcommand."""
     print("pull!")


 if __name__ == "__main__":
     git()

This creates a command that is called by main, which has a subcommand pull. It automatically has a --help option:

$ python git.py pull --help
Usage: git.py pull [OPTIONS]

  An example 'pull' subcommand.

Options:
  --help  Show this message and exit.

And pull can be called as a subcommand:

$ python git.py pull
pull!

Naming¶

For two-word commands, the words in the function name should be separated by an underscore. When creating the command name click will convert the underscore to a dash.

For example, the function

 @click.command
 def register_instrument():
     ...

Becomes a subcommand of butler called register-instrument.

Butler Subcommands¶

When creating a subcommand to be loaded by the butler command, use the decorator @click.command(). The subcommand loader will find the command and add it to the butler command. Don’t worry about this too much for now, we will discuss it more later.

An example butler subcommand implementation:

@click.command()
def my_subcommand():
    """An example subcommand that can be loaded by the butler subcommand loader."""
    pass

Options¶

Options are like the --all and the --message <msg> in git commit --all --message <msg>. They are added to a command by decorating the command function with @click.option().

In addition to the long flag (like --message) it can have a short flag, like -m for message. If it has a long flag, the argument name to the command function is the long flag with the dashes removed. If there is only a short flag, that will be used to create the argument name (again, without the dash). If a string with no dashes is passed, that will be used as the argument name, overriding the argument name that would have been generated using either of the option flags. This is shown below in the section describing MWOptionDecorator.

An example of a subcommand that uses options:

import click


@click.group()
def git():
    """An example git-style interface."""
    pass


@git.command()
@click.option("-m", "--message", help="commit message")
@click.option("-a", "--all", help="commit all changed files", is_flag=True)
def commit(all, message):
    """An example 'commit' subcommand."""
    print(f"commit. all: {all}, message: {message}")


if __name__ == "__main__":
    git()

The help is automatically generated:

$ python git.py commit --help
Usage: git.py commit [OPTIONS]

  An example 'commit' subcommand.

Options:
  -m, --message TEXT  commit message
  -a, --all           commit all changed files
  --help              Show this message and exit.

And an example of calling the subcommand:

$ python git.py commit -a -m "example commit message"
commit. all: True, message: example commit message

Arguments¶

Arguments are parameters without flags like my_branch in git checkout my_branch. They are added to a command by decorating the command function with @click.argument().

An example of a subcommand that uses arguments:

import click


@click.group()
def git():
    """An example git-style interface."""
    pass


@git.command()
@click.argument("branch")
def checkout(branch):
    """An example 'checkout' subcommand.

    BRANCH In click, arguments are documented in the command function help.
          But you can use MWArgumentDecorator, described later, to
          automatically add argument help to your command function.
    """
    print(f"checkout branch {branch}")


if __name__ == "__main__":
    git()

The help is automatically generated:

$ python git.py checkout --help
Usage: git.py checkout [OPTIONS] BRANCH

  An example 'checkout' subcommand.

  BRANCH In click, arguments are documented in the command function help.
  But you can use MWArgumentDecorator, described later, to
  automatically add argument help to your command function.

Options:
  --help  Show this message and exit.

And an example of calling the subcommand:

$ python git.py checkout mybranch
checkout branch mybranch

Butler Command Line Interface Utilities¶

daf_butler provides utilities that can be used with Click for various purposes:

Shared Options and Arguments¶

It can be good to define an option or argument one time and use it with more than one command. This reduces code duplication and improves consistency in the command line interface. daf_butler provides MWOptionDecorator and MWArgumentDecorator to define reusable option and argument decorators.

Mostly they take the same arguments as @click.option and @click.argument.
MWArgumentDecorator accepts a help argument, and inserts that help text in the correct place in the command’s help output. (The standard @click.argument decorator does not take a help argument and instead requires the the argument docstring to be added to the command function.)

An example implementation of git checkout that uses MWArgumentDecorator and MWOptionDecorator:

import click

from lsst.daf.butler.cli.opt import MWOptionDecorator, MWArgumentDecorator

branch_argument = MWArgumentDecorator("branch", help="Checkout a branch")

# Notice a string with no dashes is passed ("make_new_branch"), it is used
# as the argument name in the command function where it is used. (This is
# available for any click.option)
new_branch_option = MWOptionDecorator(
    "-b",
    "make_new_branch",
    help="create and checkout a new branch",
    # is_flag makes the option take no values, uses a bool
    # which is true if the option is passed and false by default.
    is_flag=True,
)


@click.group()
def git():
    """An example git-style interface."""
    pass


@git.command()
@branch_argument()
@new_branch_option()
def checkout(branch, make_new_branch):
    """An example 'checkout' subcommand."""
    print(f"checkout branch {branch}, make new:{make_new_branch}")


if __name__ == "__main__":
    git()

By convention:

Shared options and arguments should be placed in a package that is as high in the dependency tree as is reasonable for that option.
Shared option definitions go in the file .../cli/opt/options.py in the package’s python directory tree. Shared arguments go in .../cli/opt/arguments.py
The shared option name should:
- Match or nearly match the long name of the option or argument.
- Be all lowercase.
- Have multiple words separated by underscores.
- Shared options should end with _option. Shared arguments should end with _argument.

Shared Option Groups¶

An option group decorator may be created for shared options that will frequently be used together. The option group decorator can then add all its options to a command with a single decorator call.

By convention:

Option group decorators should go in the file .../cli/opt/optionDecorators.py.
Option group decorators should inherit from lsst.daf.butler.cli.utils.OptionGroup. This makes it easy to define the option group:
1. Create a subclass of OptionGroup
2. In the subclass __init__ function, define a member parameter called decorators that is a list or tuple of the options that go in that group.

Defines an Option Group decorator:

 class pipeline_build_options(OptionGroup):  # noqa: N801
     """Decorator to add options to a command function for building a pipeline."""

     def __init__(self):
         self.decorators = [
             ctrlMpExecOpts.pipeline_option(),
             ctrlMpExecOpts.task_option(),
         ]

Uses an Option Group decorator:

@click.command()
@pipeline_build_options()
def build(pipeline, task):
    ...

Callbacks¶

Options and arguments take a callback argument whose value is a function to be executed before passing the value to the command function. This allows the value(s) to be manipulated or acted upon before the command function is executed. lsst.daf.butler.cli.utils provides several helpful callback functions:

split_commas: Accepts a list of strings that may contain comma separated values and splits them at the commas, returning a single list of values.
split_kv: Accepts a list of strings with key-value pairs that may be comma separated. It is very configurable, for example the user can specify the key-value separator token, output container type and ordering, and more. The docstring is a good resource to learn more.
to_upper: Converts the value to upper case.
options_file_option: Allows option values to be loaded from a yaml file.

Adding Butler Subcommands¶

Packages can add subcommands to the butler command using a plugin system. This section describes how to do that. To use the plugin system you should also read and understand the sections above about the butler command. Then, write your subcommands and arrange them as described below in Package Layout. Finally, declare them as butler command plugins as described in Manifest.

Package Layout¶

The following conventions are recommended but not required:

All command line interface code should go in a folder called cli under the package’s python hierarchy e.g. python/lsst/daf/butler/cli.
Commands go in a file .../cli/cmd/commands.py
Options go in a file .../cli/opt/options.py
Shared options go in a file .../cli/opt/sharedOptions.py
Arguments go in a file .../cli/opt/arguments.py
There may be a utils.py file, also usually in the cli folder.

cli
├── cmd
│    └── commands.py
├── opt
│    ├── arguments.py
│    ├── options.py
│    └── sharedOptions.py
├── resources.yaml
└── utils.yaml

Manifest¶

The butler command finds plugin commands by way of a resource manifest published in an environment variable. By convention it is usually in the cli folder and named resources.yaml.

The resources.yaml must have a heading cmd, this is the section for importable commands. It must contain two key-value pairs:

A key called import whose value names the package that the commands can be imported from.

A key called commands that contains a list of importable command names. Use the dash-separated command name, not the underscore-separated function name.

For example, the manifest file for butler plugin subcommands in obs_base is like this:

cmd:
  import: lsst.obs.base.cli.cmd
  commands:
    - register-instrument
    - write-curated-calibrations

Publish the resource manifest in an environment variable: in the package’s ups/<pkg>.table file, add a command to prepend DAF_BUTLER_PLUGINS with the location of the resource manifest. Make sure to use the environment variable for the location of the package.

The settings for obs_base are like this:

envPrepend(DAF_BUTLER_PLUGINS, $OBS_BASE_DIR/python/lsst/obs/base/cli/resources.yaml)

Writing Other Commands¶

Other commands that load their commands from plugins as describe above can be implemented using the butler command framework, described below.

The butler command uses a click.MultiCommand subclass called LoaderCLI. It dynamically loads subcommands from the local package and from plugin packages. LoaderCLI can be used to implement other commands that dynamically load subcommands.

It’s easy to create a new kind of command by copying the template below and making a few small changes:

Change the value of localCmdPkg so refers to importable commands in the local package.
If you will support plugin commands, decide on a new environment variable to refer to the plugin manifests and change the value of pluginEnvVar to that.
If you will not support plugin commands, simply delete pluginEnvVar.
Change the class name ButlerCLI to something more descriptive for your command, and change the argument @click.command(cls=ButlerCLI,... that refers to it.

 import click

 from lsst.daf.butler.cli.butler import LoaderCLI


 # Change the class name to better describe your command.
 class ButlerCLI(LoaderCLI):
     # Replace this value with the import path to your `cmd` module.
     localCmdPkg = "lsst.daf.butler.cli.cmd"

     # Replace this value with the manifest environment variable described
     # above.
     pluginEnvVar = "DAF_BUTLER_PLUGINS"


 # Change ``cls=ButlerCLI`` to be the same as your new class name above.
 @click.command(cls=ButlerCLI, context_settings=dict(help_option_names=["-h", "--help"]))
 # You can remove log_level_option if you do not support it. You can add
 # other command options here. (Subcommand options are declared elsewhere).
 @log_level_option()
 def cli(log_level):
     # Normally you would handle the function arguments here, if there are
     # any, and/or pass them to other functions. `log_level` is unique; it
     # is handled by `LoaderCLI.get_command` and `LoaderCLI.list_commands`,
     # and is called in one of those functions before this function is
     # called.
     pass


 def main():
     return cli()

Navigation