ButlerInitializedTaskRunner

class lsst.pipe.base.ButlerInitializedTaskRunner(TaskClass, parsedCmd, doReturnResults=False)

Bases: lsst.pipe.base.TaskRunner

A TaskRunner for CmdLineTasks that require a butler keyword argument to be passed to their constructor.

Attributes Summary

TIMEOUT

Methods Summary

__call__(args)	Run the Task on a single target.
getTargetList(parsedCmd, **kwargs)	Get a list of (dataRef, kwargs) for TaskRunner.__call__.
makeTask([parsedCmd, args])	A variant of the base version that passes a butler argument to the task’s constructor.
precall(parsedCmd)	Hook for code that should run exactly once, before multiprocessing.
prepareForMultiProcessing()	Prepare this instance for multiprocessing
run(parsedCmd)	Run the task on all targets.

Attributes Documentation

TIMEOUT = 2592000

Methods Documentation

__call__(args)

Run the Task on a single target.

Parameters:

args

Arguments for Task.run()

Returns:

struct : lsst.pipe.base.Struct

Contains these fields if doReturnResults is True:

• dataRef: the provided data reference.
• metadata: task metadata after execution of run.
• result: result returned by task run, or None if the task fails.
• ``exitStatus`: 0 if the task completed successfully, 1 otherwise.

If doReturnResults is False the struct contains:

• ``exitStatus`: 0 if the task completed successfully, 1 otherwise.

Notes

This default implementation assumes that the args is a tuple containing a data reference and a dict of keyword arguments.

Warning

If you override this method and wish to return something when doReturnResults is False, then it must be picklable to support multiprocessing and it should be small enough that pickling and unpickling do not add excessive overhead.

getTargetList(parsedCmd, **kwargs)

Get a list of (dataRef, kwargs) for TaskRunner.__call__.

Parameters:

parsedCmd : argparse.Namespace

The parsed command object returned by lsst.pipe.base.ArgumentParser.parse_args.

kwargs

Any additional keyword arguments. In the default TaskRunner this is an empty dict, but having it simplifies overriding TaskRunner for tasks whose run method takes additional arguments (see case (1) below).

Notes

The default implementation of TaskRunner.getTargetList and TaskRunner.__call__ works for any command-line task whose run method takes exactly one argument: a data reference. Otherwise you must provide a variant of TaskRunner that overrides TaskRunner.getTargetList and possibly TaskRunner.__call__. There are two cases.

Case 1

If your command-line task has a run method that takes one data reference followed by additional arguments, then you need only override TaskRunner.getTargetList to return the additional arguments as an argument dict. To make this easier, your overridden version of getTargetList may call TaskRunner.getTargetList with the extra arguments as keyword arguments. For example, the following adds an argument dict containing a single key: “calExpList”, whose value is the list of data IDs for the calexp ID argument:

def getTargetList(parsedCmd):
    return TaskRunner.getTargetList(
        parsedCmd,
        calExpList=parsedCmd.calexp.idList
    )

It is equivalent to this slightly longer version:

@staticmethod
def getTargetList(parsedCmd):
    argDict = dict(calExpList=parsedCmd.calexp.idList)
    return [(dataId, argDict) for dataId in parsedCmd.id.idList]

Case 2

If your task does not meet condition (1) then you must override both TaskRunner.getTargetList and TaskRunner.__call__. You may do this however you see fit, so long as TaskRunner.getTargetList returns a list, each of whose elements is sent to TaskRunner.__call__, which runs your task.

makeTask(parsedCmd=None, args=None)

A variant of the base version that passes a butler argument to the task’s constructor.

Parameters:

parsedCmd : argparse.Namespace

Parsed command-line options, as returned by the ArgumentParser; if specified then args is ignored.

args

Other arguments; if parsedCmd is None then this must be specified.

Raises:

RuntimeError

Raised if parsedCmd and args are both None.

precall(parsedCmd)

Hook for code that should run exactly once, before multiprocessing.

Notes

Must return True if TaskRunner.__call__ should subsequently be called.

Warning

Implementations must take care to ensure that no unpicklable attributes are added to the TaskRunner itself, for compatibility with multiprocessing.

The default implementation writes package versions, schemas and configs, or compares them to existing files on disk if present.

prepareForMultiProcessing()

Prepare this instance for multiprocessing

Optional non-picklable elements are removed.

This is only called if the task is run under multiprocessing.

run(parsedCmd)

Run the task on all targets.

Parameters:

parsedCmd : argparse.Namespace

Parsed command argparse.Namespace.

Returns:

resultList : list

A list of results returned by TaskRunner.__call__, or an empty list if TaskRunner.__call__ is not called (e.g. if TaskRunner.precall returns False). See TaskRunner.__call__ for details.

Notes

The task is run under multiprocessing if TaskRunner.numProcesses is more than 1; otherwise processing is serial.