lsst.ctrl.mpexec

Command Line Scripts

The pipetask command is being ported from an argparse framework to a Click framework. During development the command implemented using Click is called pipetask2. At some point the current pipetask command will be removed and pipetask2 will be renamed to pipetask.

pipetask2

pipetask2 [OPTIONS] COMMAND [ARGS]...

Options

--log-level <log_level>

The Python log level to use.

--long-log

Make log messages appear in long format.

build

Build and optionally save pipeline definition.

This does not require input data to be specified.

pipetask2 build [OPTIONS]

Options

-p, --pipeline <pipeline>

Location of a pipeline definition file in YAML format.

-t, --task <TASK[:LABEL>

Task name to add to pipeline, must be a fully qualified task name. Task name can be followed by colon and label name, if label is not given then task base name (class name) is used as label.

--delete <LABEL>

Delete task with given label from pipeline.

-c, --config <LABEL:NAME=VALUE>

Config override, as a key-value pair.

-C, --config-file <LABEL:FILE>

Configuration override file(s), applies to a task with a given label.

--order-pipeline

Order tasks in pipeline based on their data dependencies, ordering is performed as last step before saving or executing pipeline.

-s, --save-pipeline <save_pipeline>

Location for storing resulting pipeline definition in YAML format.

--pipeline-dot <pipeline_dot>

Location for storing GraphViz DOT representation of a pipeline.

-i, --instrument <instrument>

Add an instrument which will be used to load config overrides when defining a pipeline. This must be the fully qualified class name.

--show <ITEM|ITEM=VALUE>

Dump various info to standard output. Possible items are: config, config=[Task::] or config=[Task::]<PATTERN>:NOIGNORECASE to dump configuration fields possibly matching given pattern and/or task label; history= to dump configuration history for a field, field name is specified as [Task::][SubTask.]Field; dump-config, dump-config=Task to dump complete configuration for a task given its label or all tasks; pipeline to show pipeline composition; graph to show information about quanta; workflow to show information about quanta and their dependency; tasks to show task composition.

--log-level <log_level>

The Python log level to use.

qgraph

Not implemented.

Build and optionally save pipeline and quantum graph.

pipetask2 qgraph [OPTIONS]

run

Not implemented.

Build and execute pipeline and quantum graph.

pipetask2 run [OPTIONS]

Contributing

lsst.ctrl.mpexec is developed at https://github.com/lsst/ctrl_mpexec. You can find Jira issues for this module under the ctrl_mpexec component.

Python API reference

lsst.ctrl.mpexec Package

Functions

graph2dot(qgraph, file)	Convert QuantumGraph into GraphViz digraph.
makeParser([fromfile_prefix_chars, parser_class])	Make instance of command line parser for CmdLineFwk.
pipeline2dot(pipeline, file)	Convert Pipeline into GraphViz digraph.

Classes

CmdLineFwk()	PipelineTask framework which executes tasks from command line.
ExecutionGraphFixup	Interface for classes which update quantum graphs before execution.
MPGraphExecutor(numProc, timeout, …[, …])	Implementation of QuantumGraphExecutor using same-host multiprocess execution of Quanta.
PreExecInit(butler, taskFactory[, skipExisting])	Initialization of registry for QuantumGraph execution.
QuantumExecutor	Class which abstracts execution of a single Quantum.
QuantumGraphExecutor	Class which abstracts QuantumGraph execution.
SingleQuantumExecutor(taskFactory[, …])	Executor class which runs one Quantum at a time.
TaskFactory	Class instantiating PipelineTasks.