lsst.verify

lsst.verify is a framework for packages that measure software and data quality metrics. A metric can be any measurable scalar quantity; some examples are in the LSST Science Requirements Document (LPM-17), though packages can also define ad hoc metrics. Measurements made through lsst.verify can be uploaded to LSST’s SQUASH monitoring dashboard to help you see how code development affects performance.

See also

SQR-019: LSST Verification Framework API Demonstration.

Python API reference

lsst.verify Package

LSST Science Pipelines Verification Framework.

Functions

output_quantities(package_name, quantities) Output measurements, as astropy.units.Quantity objects, from a pipeline task execution to a lsst.verify-formatted JSON file.

Classes

Blob(name, **datums) Blob is a flexible container of data, as lsst.verify.Datum s, that are serializable to JSON.
BlobSet([blobs]) A collection of Blobs.
Datum([quantity, unit, label, description]) A value annotated with units, a plot label and description.
Job([measurements, metrics, specs, meta]) Container for Measurements, Blob s, and Metadata associated with a pipeline run.
Measurement(metric[, quantity, blobs, …]) A measurement of a single Metric.
MeasurementNotes(metric_name) Container for annotations (notes) associated with a single lsst.verify.Measurement.
MeasurementSet([measurements]) A collection of Measurements of Metrics.
Metadata(measurement_set[, data]) Container for verification framework job metadata.
MetadataQuery([terms]) Query of lsst.verify.Job.meta metadata.
Metric(name, description, unit[, tags, …]) Container for the definition of a metric.
MetricComputationError This class represents unresolvable errors in computing a metric.
MetricSet([metrics]) A collection of Metrics.
Name([package, metric, spec]) Semantic name of a package, Metric or Specification in the lsst.verify framework.
Specification(name, **kwargs) Specification base class.
SpecificationResolutionError Error resolving a specification document’s stated inheritance.
SpecificationSet([specifications, partials]) A collection of Specifications.
ThresholdSpecification(name, threshold, …) A threshold-type specification, associated with a Metric, that defines a binary comparison against a measurement.
VerifyError Base error for verify.
VerifySpecificationError Error accessing or using requirement specifications.

Class Inheritance Diagram

Inheritance diagram of lsst.verify.blob.Blob, lsst.verify.blobset.BlobSet, lsst.verify.datum.Datum, lsst.verify.job.Job, lsst.verify.measurement.Measurement, lsst.verify.measurement.MeasurementNotes, lsst.verify.measurementset.MeasurementSet, lsst.verify.jobmetadata.Metadata, lsst.verify.metaquery.MetadataQuery, lsst.verify.metric.Metric, lsst.verify.metricTask.MetricComputationError, lsst.verify.metricset.MetricSet, lsst.verify.naming.Name, lsst.verify.spec.base.Specification, lsst.verify.errors.SpecificationResolutionError, lsst.verify.specset.SpecificationSet, lsst.verify.spec.threshold.ThresholdSpecification, lsst.verify.errors.VerifyError, lsst.verify.errors.VerifySpecificationError

lsst.verify.bin.dispatchverify Module

Upload LSST Science Pipelines Verification Job datasets to the SQUASH dashboard.

Job JSON files can be created by lsst.verify.Job.write or lsst.verify.output_quantities. A Job dataset consists of metric measurements, associated blobs, and pipeline execution metadata. Individual LSST Science Pipelines tasks typically write separate JSON datasets. This command can collect and combine multiple Job JSON datasets into a single Job upload.

Configuration

dispatch_verify.py is configurable from both the command line and environment variables. See the argument documenation for environment variable equivalents. Command line settings override environment variable configuration.

Metadata and environment

dispatch_verify.py can enrich Verification Job metadata with information from the environment. Currently dispatch_verify.py supports the Jenkins CI and the LSST Data Facility (LDF) execution environments.

In the Jenkins CI execution environment (--env=jenkins) the following environment variables are consumed:

  • BUILD_ID: ID in the CI system
  • BUILD_URL: CI page with information about the build
  • PRODUCT: the name of the product built, e.g. ‘validate_drp’
  • dataset: the name of the dataset processed, e.g. ‘validation_data_cfht’
  • label: the name of the platform where it runs

If --lsstsw is used, additional Git branch information is included with Science Pipelines package metadata.

In the LSST Data Facility execution environment (--env=ldf) the following environment variables are consumed:

  • DATASET: the name of the dataset processed, e.g ‘HSC RC2’
  • DATASET_REPO_URL: a reference URL with information about the dataset
  • RUN_ID: ID of the run in the LDF environment
  • RUN_ID_URL: a reference URL with information about the run
  • VERSION_TAG: the version tag of the LSST software used, e.g. ‘w_2018_18’

Note: currently it is not possible to gather Science Pipelines package metadata in the LDF environment, thus if --env=ldf is used --ignore-lsstsw is aslo used by default in this environment.

Functions

parse_args()
main() Entrypoint for the dispatch_verify.py command line executable.
insert_lsstsw_metadata(job, config) Insert metadata for lsstsw-based packages into Job.meta['packages'].
insert_extra_package_metadata(job, config) Insert metadata for extra packages (‘–package-repos’) into Job.meta['packages'].
insert_env_metadata(job, env_name, metadata) Insert environment metadata into the Job.

Classes

Configuration(args) Configuration for dispatch_verify.py that reconciles command line and environment variable arguments.

Class Inheritance Diagram

Inheritance diagram of lsst.verify.bin.dispatchverify.Configuration

lsst.verify.jsonmixin Module

Classes

JsonSerializationMixin Mixin that provides JSON serialization support to subclasses.

lsst.verify.metadata.eupsmanifest Module

API for parsing the manifest.txt file for EUPS packages found in lsstsw.

Classes

Manifest(manifest_stream) Iterator over packages in lsstsw’s manifest.txt dataset.

lsst.verify.metadata.jenkinsci Module

Functions

get_jenkins_env() Gather metadata entries from LSST DM Jenkins CI environment.

lsst.verify.metadata.lsstsw Module

APIs for building software provenance from lsstsw.

Classes

LsstswRepos(dirname) lsstsw package version information based on repos.yaml and checked out Git repositories.

lsst.verify.squash Module

SQUASH (https://squash.lsst.codes) client.

Data objects, particularly Job, Metric, and Specification, use this client to upload and retrieve data from the SQUASH verification database.

SQUASH will likely be replaced by a database served behind Data Management’s webserv API. This client is considered a shim during construction.

Functions

get(api_url[, api_endpoint, api_user, …]) GET request to the SQUASH API.
post(api_url, api_endpoint[, json_doc, …]) POST a JSON document to SQUASH.
get_endpoint_url(api_url, api_endpoint, **kwargs) Lookup SQUASH endpoint URL.
reset_endpoint_cache() Reset the cache used by get_endpoint_url.
get_default_timeout() Get the default HTTP client timeout setting.
get_default_api_version() Get the default SQUASH API versioned used by the lsst.verify.squash client functions.
make_accept_header([version]) Make the Accept HTTP header for versioned SQUASH API requests.

lsst.verify.report Module

Classes

Report(measurements, specs) Report tabulating specification pass/fail status for a set of lsst.verify.Measurements.

lsst.verify.compatibility Package

Classes

MetricTask([config, name, parentTask, log]) A base class for tasks that compute exactly one metric from arbitrary input datasets.
MetricsControllerConfig Global, metric-independent options for MetricsControllerTask.
MetricsControllerTask([config]) A Task for executing a collection of lsst.verify.compatibility.MetricTask objects.