The Butler Command¶
daf_butler
provides a command line interface command called butler
. It supports subcommands, some of
which are implemented in daf_butler
.
The butler
command and subcommands are implemented in click, which is well
documented and has a good
quickstart guide. There is more to know about
click’s commands, options, and arguments than is described here, and the click documentation is a good
resource for that.
This guide includes an overview of how subcommands, options, and arguments are used with the
butler
command.
It then describes how other packages can add subcommands to the butler command by way of a plugin system, in Adding Butler Subcommands.
Finally, the butler
command line framework can be used to write other commands that load subcommands. This
is described in Writing Other Commands
Subcommands¶
Subcommands are like the “pull” in git pull
. The subcommand is implemented as a python function and
decorated with @click.command
. The name of the function will become the name of the subcommand on the
command line.
Example of a command or subcommand definition:
import click
@click.group()
def git():
"""An example git-style interface."""
pass
# Notice this uses "@git" instead of "@click", this adds the command to the
# git group.
@git.command()
def pull():
"""An example 'pull' subcommand."""
print("pull!")
if __name__ == "__main__":
git()
This creates a command that is called by main
, which has a subcommand pull
.
It automatically has a --help
option:
$ python git.py pull --help
Usage: git.py pull [OPTIONS]
An example 'pull' subcommand.
Options:
--help Show this message and exit.
And pull
can be called as a subcommand:
$ python git.py pull
pull!
Naming¶
For two-word commands, the words in the function name should be separated by an underscore. When creating the command name click will convert the underscore to a dash.
For example, the function
@click.command
def register_instrument():
...
Becomes a subcommand of butler
called register-instrument
.
Butler Subcommands¶
When creating a subcommand to be loaded by the butler
command, use the decorator @click.command()
.
The subcommand loader will find the command and add it to the butler
command. Don’t worry about this too
much for now, we will discuss it more later.
An example butler
subcommand implementation:
@click.command()
def my_subcommand():
"""An example subcommand that can be loaded by the butler subcommand loader."""
pass
Options¶
Options are like the --all
and the --message <msg>
in git commit --all --message <msg>
.
They are added to a command by decorating the command function with @click.option()
.
In addition to the long flag (like --message
) it can have a short flag, like -m
for message.
If it has a long flag, the argument name to the command function is the long flag with the dashes removed.
If there is only a short flag, that will be used to create the argument name (again, without the dash).
If a string with no dashes is passed, that will be used as the argument name, overriding the argument name
that would have been generated using either of the option flags. This is shown below in the section describing
MWOptionDecorator
.
An example of a subcommand that uses options:
import click
@click.group()
def git():
"""An example git-style interface."""
pass
@git.command()
@click.option("-m", "--message", help="commit message")
@click.option("-a", "--all", help="commit all changed files", is_flag=True)
def commit(all, message):
"""An example 'commit' subcommand."""
print(f"commit. all: {all}, message: {message}")
if __name__ == "__main__":
git()
The help is automatically generated:
$ python git.py commit --help
Usage: git.py commit [OPTIONS]
An example 'commit' subcommand.
Options:
-m, --message TEXT commit message
-a, --all commit all changed files
--help Show this message and exit.
And an example of calling the subcommand:
$ python git.py commit -a -m "example commit message"
commit. all: True, message: example commit message
Arguments¶
Arguments are parameters without flags like my_branch
in git checkout my_branch
.
They are added to a command by decorating the command function with @click.argument()
.
An example of a subcommand that uses arguments:
import click
@click.group()
def git():
"""An example git-style interface."""
pass
@git.command()
@click.argument("branch")
def checkout(branch):
"""An example 'checkout' subcommand.
BRANCH In click, arguments are documented in the command function help.
But you can use MWArgumentDecorator, described later, to
automatically add argument help to your command function.
"""
print(f"checkout branch {branch}")
if __name__ == "__main__":
git()
The help is automatically generated:
$ python git.py checkout --help
Usage: git.py checkout [OPTIONS] BRANCH
An example 'checkout' subcommand.
BRANCH In click, arguments are documented in the command function help.
But you can use MWArgumentDecorator, described later, to
automatically add argument help to your command function.
Options:
--help Show this message and exit.
And an example of calling the subcommand:
$ python git.py checkout mybranch
checkout branch mybranch
Butler Command Line Interface Utilities¶
daf_butler
provides utilities that can be used with Click for various
purposes:
Callbacks¶
Options and arguments take a callback
argument whose value is a function to be executed before passing the value to the command function.
This allows the value(s) to be manipulated or acted upon before the command function is executed.
lsst.daf.butler.cli.utils
provides several helpful callback functions:
split_commas
- Accepts a list of strings that may contain comma separated values and splits them at the commas, returning a single list of values.
split_kv
- Accepts a list of strings with key-value pairs that may be comma separated. It is very configurable, for example the user can specify the key-value separator token, output container type and ordering, and more. The docstring is a good resource to learn more.
to_upper
- Converts the value to upper case.
options_file_option
- Allows option values to be loaded from a
yaml
file.
Adding Butler Subcommands¶
Packages can add subcommands to the butler
command using a plugin system. This section describes how to do that.
To use the plugin system you should also read and understand the sections above about the butler command.
Then, write your subcommands and arrange them as described below in Package Layout.
Finally, declare them as butler
command plugins as described in Manifest.
Package Layout¶
The following conventions are recommended but not required:
- All command line interface code should go in a folder called
cli
under the package’s python hierarchy e.g.python/lsst/daf/butler/cli
. - Commands go in a file
.../cli/cmd/commands.py
- Options go in a file
.../cli/opt/options.py
- Shared options go in a file
.../cli/opt/sharedOptions.py
- Arguments go in a file
.../cli/opt/arguments.py
- There may be a
utils.py
file, also usually in thecli
folder.
cli
├── cmd
│ └── commands.py
├── opt
│ ├── arguments.py
│ ├── options.py
│ └── sharedOptions.py
├── resources.yaml
└── utils.yaml
Manifest¶
The butler
command finds plugin commands by way of a resource manifest published in an environment variable.
By convention it is usually in the cli
folder and named resources.yaml
.
The resources.yaml
must have a heading cmd
, this is the section for importable commands.
It must contain two key-value pairs:
- A key called
import
whose value names the package that the commands can be imported from.- A key called
commands
that contains a list of importable command names. Use the dash-separated command name, not the underscore-separated function name.
For example, the manifest file for butler
plugin subcommands in obs_base
is like this:
cmd:
import: lsst.obs.base.cli.cmd
commands:
- register-instrument
- write-curated-calibrations
Publish the resource manifest in an environment variable: in the package’s ups/<pkg>.table
file, add a
command to prepend DAF_BUTLER_PLUGINS
with the location of the resource manifest. Make sure to use the
environment variable for the location of the package.
The settings for obs_base
are like this:
envPrepend(DAF_BUTLER_PLUGINS, $OBS_BASE_DIR/python/lsst/obs/base/cli/resources.yaml)
Writing Other Commands¶
Other commands that load their commands from plugins as describe above can be implemented using the butler command framework, described below.
The butler
command uses a click.MultiCommand
subclass called LoaderCLI
.
It dynamically loads subcommands from the local package and from plugin packages.
LoaderCLI
can be used to implement other commands that dynamically load subcommands.
It’s easy to create a new kind of command by copying the template below and making a few small changes:
- Change the value of
localCmdPkg
so refers to importable commands in the local package. - If you will support plugin commands, decide on a new environment variable to refer to the plugin manifests
and change the value of
pluginEnvVar
to that. - If you will not support plugin commands, simply delete
pluginEnvVar
. - Change the class name
ButlerCLI
to something more descriptive for your command, and change the argument@click.command(cls=ButlerCLI,...
that refers to it.
import click
from lsst.daf.butler.cli.butler import LoaderCLI
# Change the class name to better describe your command.
class ButlerCLI(LoaderCLI):
# Replace this value with the import path to your `cmd` module.
localCmdPkg = "lsst.daf.butler.cli.cmd"
# Replace this value with the manifest environment variable described
# above.
pluginEnvVar = "DAF_BUTLER_PLUGINS"
# Change ``cls=ButlerCLI`` to be the same as your new class name above.
@click.command(cls=ButlerCLI,
context_settings=dict(help_option_names=["-h", "--help"]))
# You can remove log_level_option if you do not support it. You can add
# other command options here. (Subcommand options are declared elsewhere).
@log_level_option()
def cli(log_level):
# Normally you would handle the function arguments here, if there are
# any, and/or pass them to other functions. `log_level` is unique; it
# is handled by `LoaderCLI.get_command` and `LoaderCLI.list_commands`,
# and is called in one of those functions before this function is
# called.
pass
def main():
return cli()