Configuring command-line tasks

Command-line tasks are highly configurable. This page describes how to review configurations with --show config, and change configurations on the command line with --config, --config-file, or --configfile.

One special category of configuration is subtask retargeting. Retargeting uses concepts discussed on this page, but with additional considerations described in Retargeting subtasks of command-line tasks.

How to show a command-line task’s current configuration

The --show config argument allows you to review the current configuration details for command-line tasks.

Tip

--show config puts the command-line task in a dry-run mode where no data processed. Once you’re done reviewing configurations, remove the --show argument from the command-line task invocation to actually process data.

Alternatively, add run to the --show argument to show configurations and process data: --show config run.

Viewing all configurations

Use --show config to see all current configurations for a command-line task. For example:

processCcd.py REPOPATH --output output --show config

Note that both the input and output Butler data repositories repository must be set to show camera-specific task configuration defaults. Instead of an --output argument you can specify a --rerun, see Using Butler data repositories and reruns with command-line tasks.

Viewing a subset of configurations

To filter the --show config output, include a search term with wildcard matching (*) characters. For example, this will show any configuration that includes the string calibration:

processCcd.py REPOPATH --output output --show config="*calibration*"

How to set configurations with command-line arguments

Command-line tasks can be configured through a combination of two mechanisms: arguments on the command line (--config) or through configuration files (--config-file or --configfile). In general, simple configurations can be made through the command line, while complex configurations and subtask retargeting must done through configuration files (see How to use configuration files).

To change a configuration value on the command line, pass that configuration name and value to the --config argument. For example, to set a configuration named skyMap.projection to a value "TAN":

task.py REPOPATH --output output --config skyMap.projection="TAN"

You can provide multiple --config arguments on the same command line or set multiple configurations with a single --config argument:

task.py REPOPATH --output output --config config1="value1" config2="value2"

Only simple configuration values can be set through --config arguments, such as:

  • String values. For example: --config configName="value".
  • Scalar numbers. For example: --config configName=2.5.
  • Lists of integers. For example: --config intList=2,4,-87.
  • List of floating point numbers. For example: --config floatList=3.14,-5.6e7.
  • Boolean values. For example: --config configName=True configName2=False.

Specific types of configurations you cannot perform with the --config argument are:

For these more complex configuration types you must use configuration files, which are evaluated as Python code.

How to use configuration files

You can also provide configurations to a command-line task through a configuration file. In fact, configuration files are Python modules; anything you can do in Python you can do in a configuration file.

Configuration files give you full access to the configuration API, allowing you to import and retarget subtasks, and set configurations with complex types. These configurations can only be done through configuration files, not through command-line arguments.

Use a configuration file by providing its file path through a -C/--config-file/--configfile argument:

task.py REPOPATH --output output --config-file taskConfig.py

Multiple configuration files can be provided through the same --config-file argument and the --config-file argument itself can be repeated.

In a configuration file, configurations are attributes of a config object. If on the command line you set a configuration with a --config skyMap.projection="TAN" argument, in a configuration file the equivalent statement is:

config.skyMap.projection = "TAN"

config is the root configuration object for the command-line task. Settings for the command-line task itself are attributes of config. In that example, config.skyMap is a subtask and projection is a configuration of that skyMap subtask.

About configuration defaults and camera configuration override files

Command-line task configurations are a combination of configurations you provide and defaults from the observatory package and the task itself.

When a command-line task is run, it loads two camera-specific configuration files, if found: one for the observatory package, and one for a specific camera defined in that observatory package. For an example observatory package named obs_package, these configuration override files are, in order:

  • obs_package/config/taskName.py (overrides for an observatory package in general).
  • obs_package/config/cameraName/taskName.py (overrides for a specific camera, named “cameraName”).

The taskName is the command-line task, such as processCcd for :command:processCcd.py (this is the lsst.pipe.base.CmdLineTask._DefaultName class variable).

Here are two examples:

  • obs_lsstSim/config/makeCoaddTempExp.py: specifies which version of the image selector task to use for co-adding LSST simulated images using the obs_lsstSim observatory package.
  • obs_subaru/config/hsc/isr.py`: provides overrides for the instrument signature removal (aka detrending) task for the hsc camera (Hyper Suprime-Cam) in the obs_subaru observatory package.

Overall, the priority order for setting task configurations is configurations is (highest priority first):

  1. User-provided --config and --config-file arguments (computed left-to-right).
  2. Camera specific configuration override file in an observatory package.
  3. General configuration override file in an observatory package.
  4. Task defaults.