MemoryMetricTask¶

MemoryMetricTask creates a resident set size Measurement based on data collected by @timeMethod. It reads the raw timing data from the top-level CmdLineTask’s metadata, which is identified by the task configuration.

In general, it’s only useful to measure this metric for the top-level task being run. @timeMethod measures the peak memory usage from process start, so the results for any subtask will be contaminated by previous subtasks run on the same data ID.

Because @timeMethod gives platform-dependent results, this task may give incorrect results (e.g., units) when run in a distributed system with heterogeneous nodes.

Processing summary¶

MemoryMetricTask searches the metadata for @timeMethod-generated keys corresponding to the method of interest. If it finds matching keys, it stores the maximum memory usage as a Measurement.

Python API summary¶

from lsst.verify.tasks.commonMetrics import MemoryMetricTask

classMemoryMetricTask(**kwargs): A Task that computes the maximum resident set size using metadata produced by the `lsst.pipe.base.timeMethod` decorator...

attributeconfig: Access configuration fields and retargetable subtasks.

methodrun(metadata): Compute a measurement from science task metadata...

Butler datasets¶

Input datasets¶

metadata: The metadata of the top-level command-line task (e.g., ProcessCcdTask, ApPipeTask) being instrumented. Because the metadata produced by each top-level task is a different Butler dataset type, this dataset must be explicitly configured when running MemoryMetricTask or a MetricsControllerTask that contains it.

Retargetable subtasks¶

No subtasks.

Configuration fields¶

metadata¶

Data type: lsst.pipe.base.config.InputDatasetConfig
Field type: ConfigField

The target top-level task’s metadata. The name must be set to the metadata’s butler type, such as ‘processCcd_metadata’.

metric¶

Default: None
Field type: str Field

The fully qualified name of the metric to store the profiling information.

target¶

Default: None
Field type: str Field

The method to profile, optionally prefixed by one or more tasks in the format of lsst.pipe.base.Task.getFullMetadata().

Examples¶

from lsst.verify.tasks import MemoryMetricTask

config = MemoryMetricTask.ConfigClass()
config.metadata.name = "apPipe_metadata"
config.target = "apPipe:ccdProcessor.runDataRef"
config.metric = "pipe_tasks.ProcessCcdMemory"
task = MemoryMetricTask(config=config)

# config.metadata provided for benefit of MetricsControllerTask/Pipeline
# but since we've defined it we might as well use it
metadata = butler.get(config.metadata.name)
processCcdTime = task.run(metadata).measurement

Navigation