lsst.verify

lsst.verify is a framework for packages that measure software and data quality metrics. A metric can be any measurable scalar quantity; some examples are in the LSST Science Requirements Document (LPM-17), though packages can also define ad hoc metrics. Measurements made through lsst.verify can be uploaded to LSST’s SQUASH monitoring dashboard to help you see how code development affects performance.

See also

SQR-019: LSST Verification Framework API Demonstration.

Using lsst.verify

Contributing

lsst.verify is developed at https://github.com/lsst/verify. You can find Jira issues for this module under the verify component.

Task reference

Pipeline tasks

ApdbMetricTask

A base class for tasks that compute metrics from an alert production database.

CpuTimingMetricTask

A Task that computes a CPU time using metadata produced by the lsst.utils.timer.timeMethod decorator.

MemoryMetricTask

A Task that computes the maximum resident set size using metadata produced by the lsst.utils.timer.timeMethod decorator.

MetadataMetricTask

A base class for tasks that compute metrics from single metadata objects.

MetricTask

A base class for tasks that compute one metric from input datasets.

TimingMetricTask

A Task that computes a wall-clock time using metadata produced by the lsst.utils.timer.timeMethod decorator.

Tasks

ConfigApdbLoader

A Task that takes a science task config and returns the corresponding Apdb object.

DirectApdbLoader

A Task that takes a Apdb config and returns the corresponding Apdb object.

Script reference

Python API reference

lsst.verify Package

Functions

output_quantities(package_name, quantities)

Output measurements, as astropy.units.Quantity objects, from a pipeline task execution to a lsst.verify-formatted JSON file.

Classes

Blob(name, **datums)

Blob is a flexible container of data, as lsst.verify.Datum s, that are serializable to JSON.

BlobSet([blobs])

A collection of Blobs.

Datum([quantity, unit, label, description])

A value annotated with units, a plot label and description.

Job([measurements, metrics, specs, meta])

Container for Measurements, Blob s, and Metadata associated with a pipeline run.

Measurement(metric[, quantity, blobs, ...])

A measurement of a single Metric.

MeasurementNotes(metric_name)

Container for annotations (notes) associated with a single lsst.verify.Measurement.

MeasurementSet([measurements])

A collection of Measurements of Metrics.

Metadata(measurement_set[, data])

Container for verification framework job metadata.

MetadataQuery([terms])

Query of lsst.verify.Job.meta metadata.

Metric(name, description, unit[, tags, ...])

Container for the definition of a metric.

MetricSet([metrics])

A collection of Metrics.

Name([package, metric, spec])

Semantic name of a package, Metric or Specification in the lsst.verify framework.

Specification(name, **kwargs)

Specification base class.

SpecificationResolutionError

Error resolving a specification document's stated inheritance.

SpecificationSet([specifications, partials])

A collection of Specifications.

ThresholdSpecification(name, threshold, ...)

A threshold-type specification, associated with a Metric, that defines a binary comparison against a measurement.

VerifyError

Base error for verify.

VerifySpecificationError

Error accessing or using requirement specifications.

Class Inheritance Diagram

digraph inheritanceb603617720 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Blob" [URL="../../../py-api/lsst.verify.Blob.html#lsst.verify.Blob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Blob is a flexible container of data, as `lsst.verify.Datum` \s, that"]; "JsonSerializationMixin" -> "Blob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BlobSet" [URL="../../../py-api/lsst.verify.BlobSet.html#lsst.verify.BlobSet",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A collection of `~lsst.verify.Blob`\ s."]; "JsonSerializationMixin" -> "BlobSet" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Datum" [URL="../../../py-api/lsst.verify.Datum.html#lsst.verify.Datum",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A value annotated with units, a plot label and description."]; "QuantityAttributeMixin" -> "Datum" [arrowsize=0.5,style="setlinewidth(0.5)"]; "JsonSerializationMixin" -> "Datum" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Job" [URL="../../../py-api/lsst.verify.Job.html#lsst.verify.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Container for `~lsst.verify.Measurement`\ s, `~lsst.verify.Blob` \s,"]; "JsonSerializationMixin" -> "Job" [arrowsize=0.5,style="setlinewidth(0.5)"]; "JsonSerializationMixin" [URL="../../../py-api/lsst.verify.jsonmixin.JsonSerializationMixin.html#lsst.verify.jsonmixin.JsonSerializationMixin",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Mixin that provides JSON serialization support to subclasses."]; "Measurement" [URL="../../../py-api/lsst.verify.Measurement.html#lsst.verify.Measurement",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A measurement of a single `~lsst.verify.Metric`."]; "JsonSerializationMixin" -> "Measurement" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MeasurementNotes" [URL="../../../py-api/lsst.verify.MeasurementNotes.html#lsst.verify.MeasurementNotes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Container for annotations (notes) associated with a single"]; "MeasurementSet" [URL="../../../py-api/lsst.verify.MeasurementSet.html#lsst.verify.MeasurementSet",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A collection of `~lsst.verify.Measurement`\ s of"]; "JsonSerializationMixin" -> "MeasurementSet" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Metadata" [URL="../../../py-api/lsst.verify.Metadata.html#lsst.verify.Metadata",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Container for verification framework job metadata."]; "JsonSerializationMixin" -> "Metadata" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MetadataQuery" [URL="../../../py-api/lsst.verify.MetadataQuery.html#lsst.verify.MetadataQuery",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Query of `lsst.verify.Job.meta` metadata."]; "JsonSerializationMixin" -> "MetadataQuery" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Metric" [URL="../../../py-api/lsst.verify.Metric.html#lsst.verify.Metric",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Container for the definition of a metric."]; "JsonSerializationMixin" -> "Metric" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MetricSet" [URL="../../../py-api/lsst.verify.MetricSet.html#lsst.verify.MetricSet",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A collection of `Metric`\ s."]; "JsonSerializationMixin" -> "MetricSet" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Name" [URL="../../../py-api/lsst.verify.Name.html#lsst.verify.Name",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Semantic name of a package, `~lsst.verify.Metric` or"]; "QuantityAttributeMixin" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Mixin with common attributes for classes that wrap an"]; "Specification" [URL="../../../py-api/lsst.verify.Specification.html#lsst.verify.Specification",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Specification base class."]; "JsonSerializationMixin" -> "Specification" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SpecificationResolutionError" [URL="../../../py-api/lsst.verify.SpecificationResolutionError.html#lsst.verify.SpecificationResolutionError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Error resolving a specification document's stated inheritance."]; "SpecificationSet" [URL="../../../py-api/lsst.verify.SpecificationSet.html#lsst.verify.SpecificationSet",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A collection of `Specification`\ s."]; "JsonSerializationMixin" -> "SpecificationSet" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ThresholdSpecification" [URL="../../../py-api/lsst.verify.ThresholdSpecification.html#lsst.verify.ThresholdSpecification",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A threshold-type specification, associated with a `Metric`, that"]; "Specification" -> "ThresholdSpecification" [arrowsize=0.5,style="setlinewidth(0.5)"]; "VerifyError" [URL="../../../py-api/lsst.verify.VerifyError.html#lsst.verify.VerifyError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base error for verify."]; "VerifySpecificationError" [URL="../../../py-api/lsst.verify.VerifySpecificationError.html#lsst.verify.VerifySpecificationError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Error accessing or using requirement specifications."]; "VerifyError" -> "VerifySpecificationError" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

lsst.verify.bin.dispatchverify Module

Functions

build_argparser()

main()

Entrypoint for the dispatch_verify.py command line executable.

insert_lsstsw_metadata(job, config)

Insert metadata for lsstsw-based packages into Job.meta['packages'].

insert_extra_package_metadata(job, config)

Insert metadata for extra packages ('--package-repos') into Job.meta['packages'].

insert_env_metadata(job, env_name, metadata, ...)

Insert environment metadata into the Job.

validate_date_created(date_created)

Ensure date_created is a valid datetime string in UTC.

Classes

Configuration(args)

Configuration for dispatch_verify.py that reconciles command line and environment variable arguments.

Class Inheritance Diagram

digraph inheritance50d60cd382 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Configuration" [URL="../../../py-api/lsst.verify.bin.dispatchverify.Configuration.html#lsst.verify.bin.dispatchverify.Configuration",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Configuration for dispatch_verify.py that reconciles command line and"]; }

lsst.verify.extract_metricvalues Module

Tools for loading metric values from a butler and printing them, or from two butlers and differencing them.

These functions are used by the print_metricvalues script.

Functions

load_value(butler[, verbose])

Load all measured non-time/non-memory metrics in the given butler repo.

load_timing(butler[, verbose])

Load all measured timing metrics in the given butler repo.

load_memory(butler[, verbose])

Load all measured memory usage metrics in the given butler repo.

print_metrics(butler, kind, *[, ...])

Print all metrics with measured values in the given repo.

print_diff_metrics(butler1, butler2[, ...])

Load metric values from two repos and print their differences.

load_from_butler(butler, query[, ...])

Parameters:

lsst.verify.jsonmixin Module

Classes

JsonSerializationMixin()

Mixin that provides JSON serialization support to subclasses.

lsst.verify.metadata.eupsmanifest Module

Classes

Manifest(manifest_stream)

Iterator over packages in lsstsw's manifest.txt dataset.

lsst.verify.metadata.jenkinsci Module

Functions

get_jenkins_env()

Gather metadata entries from LSST DM Jenkins CI environment.

lsst.verify.metadata.lsstsw Module

Classes

LsstswRepos(dirname)

lsstsw package version information based on repos.yaml and checked out Git repositories.

lsst.verify.squash Module

Functions

get(api_url[, api_endpoint, api_user, ...])

GET request to the SQUASH API.

post(api_url, api_endpoint[, json_doc, ...])

POST a JSON document to SQUASH.

get_endpoint_url(api_url, api_endpoint, **kwargs)

Lookup SQUASH endpoint URL.

reset_endpoint_cache()

Reset the cache used by get_endpoint_url.

get_default_timeout()

Get the default HTTP client timeout setting.

get_default_api_version()

Get the default SQUASH API versioned used by the lsst.verify.squash client functions.

make_accept_header([version])

Make the Accept HTTP header for versioned SQUASH API requests.

lsst.verify.report Module

Classes

Report(measurements, specs)

Report tabulating specification pass/fail status for a set of lsst.verify.Measurements.

lsst.verify.tasks Package

Classes

AbstractMetadataMetricTask(**kwargs)

A base class for tasks that compute metrics from metadata values.

ApdbMetricConfig(*args, **kw)

A base class for APDB metric task configs.

ApdbMetricConnections(*[, config])

An abstract connections class defining a database input.

ApdbMetricTask(**kwargs)

A base class for tasks that compute metrics from an alert production database.

ConfigApdbLoader(**kwargs)

A Task that takes a science task config and returns the corresponding Apdb object.

CpuTimingMetricConfig

alias of TimeMethodMetricConfig

CpuTimingMetricTask(**kwargs)

A Task that computes a CPU time using metadata produced by the lsst.utils.timer.timeMethod decorator.

DirectApdbLoader(**kwargs)

A Task that takes a Apdb config and returns the corresponding Apdb object.

MemoryMetricConfig

alias of TimeMethodMetricConfig

MemoryMetricTask(**kwargs)

A Task that computes the maximum resident set size using metadata produced by the lsst.utils.timer.timeMethod decorator.

MetadataMetricConfig(*args, **kw)

A base class for metadata metric task configs.

MetadataMetricTask(**kwargs)

A base class for tasks that compute metrics from single metadata objects.

MetricComputationError

This class represents unresolvable errors in computing a metric.

MetricConfig(*args, **kw)

MetricConnections(*[, config])

An abstract connections class defining a metric output.

MetricTask(**kwargs)

A base class for tasks that compute one metric from input datasets.

SingleMetadataMetricConnections(*[, config])

An abstract connections class defining a metadata input.

TimingMetricConfig

alias of TimeMethodMetricConfig

TimingMetricTask(**kwargs)

A Task that computes a wall-clock time using metadata produced by the lsst.utils.timer.timeMethod decorator.