lsst.verify

lsst.verify is a framework for packages that measure software and data quality metrics. A metric can be any measurable scalar quantity; some examples are in the LSST Science Requirements Document (LPM-17), though packages can also define ad hoc metrics. Measurements made through lsst.verify can be uploaded to LSST’s SQUASH monitoring dashboard to help you see how code development affects performance.

See also

SQR-019: LSST Verification Framework API Demonstration.

Contributing

lsst.verify is developed at https://github.com/lsst/verify. You can find Jira issues for this module under the verify component.

Task reference

Tasks

ConfigPpdbLoader
A Task that takes a science task config and returns the corresponding Ppdb object.
MemoryMetricTask
A Task that computes the maximum resident set size using metadata produced by the lsst.pipe.base.timeMethod decorator.
MetadataMetricTask
A base class for tasks that compute metrics from single metadata objects.
MetricTask
A base class for tasks that compute one metric from input datasets.
MetricsControllerTask
A Task for executing a collection of lsst.verify.gen2tasks.MetricTask objects.
MultiMetadataMetricTask
A base class for tasks that compute metrics from multiple metadata objects.
PpdbMetricTask
A base class for tasks that compute metrics from a prompt products database.
SquashMetadataTask
A Task for adding SQuaSH-required metadata to a Job.
TimingMetricTask
A Task that computes a wall-clock time using metadata produced by the lsst.pipe.base.timeMethod decorator.

Python API reference

lsst.verify Package

Functions

output_quantities(package_name, quantities) Output measurements, as astropy.units.Quantity objects, from a pipeline task execution to a lsst.verify-formatted JSON file.

Classes

Blob(name, **datums) Blob is a flexible container of data, as lsst.verify.Datum s, that are serializable to JSON.
BlobSet([blobs]) A collection of Blobs.
Datum([quantity, unit, label, description]) A value annotated with units, a plot label and description.
Job([measurements, metrics, specs, meta]) Container for Measurements, Blob s, and Metadata associated with a pipeline run.
Measurement(metric[, quantity, blobs, …]) A measurement of a single Metric.
MeasurementNotes(metric_name) Container for annotations (notes) associated with a single lsst.verify.Measurement.
MeasurementSet([measurements]) A collection of Measurements of Metrics.
Metadata(measurement_set[, data]) Container for verification framework job metadata.
MetadataQuery([terms]) Query of lsst.verify.Job.meta metadata.
Metric(name, description, unit[, tags, …]) Container for the definition of a metric.
MetricSet([metrics]) A collection of Metrics.
Name([package, metric, spec]) Semantic name of a package, Metric or Specification in the lsst.verify framework.
Specification(name, **kwargs) Specification base class.
SpecificationResolutionError Error resolving a specification document’s stated inheritance.
SpecificationSet([specifications, partials]) A collection of Specifications.
ThresholdSpecification(name, threshold, …) A threshold-type specification, associated with a Metric, that defines a binary comparison against a measurement.
VerifyError Base error for verify.
VerifySpecificationError Error accessing or using requirement specifications.

Class Inheritance Diagram

Inheritance diagram of lsst.verify.blob.Blob, lsst.verify.blobset.BlobSet, lsst.verify.datum.Datum, lsst.verify.job.Job, lsst.verify.measurement.Measurement, lsst.verify.measurement.MeasurementNotes, lsst.verify.measurementset.MeasurementSet, lsst.verify.jobmetadata.Metadata, lsst.verify.metaquery.MetadataQuery, lsst.verify.metric.Metric, lsst.verify.metricset.MetricSet, lsst.verify.naming.Name, lsst.verify.spec.base.Specification, lsst.verify.errors.SpecificationResolutionError, lsst.verify.specset.SpecificationSet, lsst.verify.spec.threshold.ThresholdSpecification, lsst.verify.errors.VerifyError, lsst.verify.errors.VerifySpecificationError

lsst.verify.bin.dispatchverify Module

Functions

build_argparser()
main() Entrypoint for the dispatch_verify.py command line executable.
insert_lsstsw_metadata(job, config) Insert metadata for lsstsw-based packages into Job.meta['packages'].
insert_extra_package_metadata(job, config) Insert metadata for extra packages (‘–package-repos’) into Job.meta['packages'].
insert_env_metadata(job, env_name, metadata) Insert environment metadata into the Job.

Classes

Configuration(args) Configuration for dispatch_verify.py that reconciles command line and environment variable arguments.

Class Inheritance Diagram

Inheritance diagram of lsst.verify.bin.dispatchverify.Configuration

lsst.verify.jsonmixin Module

Classes

JsonSerializationMixin Mixin that provides JSON serialization support to subclasses.

lsst.verify.metadata.eupsmanifest Module

Classes

Manifest(manifest_stream) Iterator over packages in lsstsw’s manifest.txt dataset.

lsst.verify.metadata.jenkinsci Module

Functions

get_jenkins_env() Gather metadata entries from LSST DM Jenkins CI environment.

lsst.verify.metadata.lsstsw Module

Classes

LsstswRepos(dirname) lsstsw package version information based on repos.yaml and checked out Git repositories.

lsst.verify.squash Module

Functions

get(api_url[, api_endpoint, api_user, …]) GET request to the SQUASH API.
post(api_url, api_endpoint[, json_doc, …]) POST a JSON document to SQUASH.
get_endpoint_url(api_url, api_endpoint, **kwargs) Lookup SQUASH endpoint URL.
reset_endpoint_cache() Reset the cache used by get_endpoint_url.
get_default_timeout() Get the default HTTP client timeout setting.
get_default_api_version() Get the default SQUASH API versioned used by the lsst.verify.squash client functions.
make_accept_header([version]) Make the Accept HTTP header for versioned SQUASH API requests.

lsst.verify.report Module

Classes

Report(measurements, specs) Report tabulating specification pass/fail status for a set of lsst.verify.Measurements.

lsst.verify.gen2tasks Package

Functions

register(name) A class decorator that registers a lsst.verify.gen2tasks.MetricTask with a central repository.
registerMultiple(name) A class decorator that registers a lsst.verify.gen2tasks.MetricTask with a central repository.

Classes

MetricTask(**kwargs) A base class for tasks that compute one metric from input datasets.
MetricsControllerConfig Configuration options for MetricsControllerTask.
MetricsControllerTask([config]) A Task for executing a collection of lsst.verify.gen2tasks.MetricTask objects.
SquashMetadataTask([config, name, …]) A Task for adding SQuaSH-required metadata to a Job.

lsst.verify.tasks Package

Classes

ConfigPpdbLoader(**kwargs) A Task that takes a science task config and returns the corresponding Ppdb object.
MemoryMetricConfig alias of lsst.verify.tasks.commonMetrics.TimeMethodMetricConfig
MemoryMetricTask(**kwargs) A Task that computes the maximum resident set size using metadata produced by the lsst.pipe.base.timeMethod decorator.
MetadataMetricConfig A base class for metadata metric task configs.
MetadataMetricTask(**kwargs) A base class for tasks that compute metrics from single metadata objects.
MetricComputationError This class represents unresolvable errors in computing a metric.
MultiMetadataMetricConfig A base class for metadata metric task configs.
MultiMetadataMetricTask(**kwargs) A base class for tasks that compute metrics from multiple metadata objects.
PpdbMetricConfig A base class for PPDB metric task configs.
PpdbMetricTask(**kwargs) A base class for tasks that compute metrics from a prompt products database.
TimingMetricConfig alias of lsst.verify.tasks.commonMetrics.TimeMethodMetricConfig
TimingMetricTask(**kwargs) A Task that computes a wall-clock time using metadata produced by the lsst.pipe.base.timeMethod decorator.