lsst.analysis.tools

The Analysis Tools package is designed to assist in the creation of quality assurance (QA) plots and metrics from the outputs of a data reduction pipeline. The intention is that users have the flexibility to construct complex data analysis tasks from a set of simple building blocks. In this sense, a series of consistent, repeatable and high-quality plots and metrics can be generated for any given dataset.

Using Analysis Tools

For a tutorial on working with analysis_tools please see the getting started guide.

• Getting Started Guide
• Package Layout
• A Simple Plotting And Metric Example
• Adding an Action
• Adding a Plot Type
• numpy and other warnings
• Action Types
• Plot Types
• FAQs
• Detailed Package Layout

Need Help?

The first place to check for more information is the FAQs page which contains some helpful hints. More will be added to this page over time; if you find something that you think should be added here then please do!

Project members and in-kind contributors may ask for further help via the #rubinobs-analysis-tools channel on the LSSTC Slack workspace. All users of the analysis tools package, including community members, are also welcome to post any questions you might have on the Community Forum.

Contributing

The lsst.analysis.tools package is developed at github.com/lsst/analysis_tools.

Jira issues relating to this package can be found using the analysis_tools component.

Script reference

• verify_to_sasquatch.py
• build-gather-resource-usage-qg

Python API Reference

lsst.analysis.tools Package

This package contains various analysis tools and the PipelineTasks which run them.

The primary tool in this package is the AnalysisTool class. An AnalysisTool class has three stages: prep, process, and produce. Creating a new AnalysisTool involves choosing (through configuration) one or more AnalysisAction classes to run for each of those stages.

Both AnalysisTool and AnalysisAction are subclasses of the ConfigurableAction class. These objects are special types that are configured prior to any code execution, and behave as functions at runtime. The configuration state of a ConfigurableAction is saved separately from the object itself.

Functions

cos(values)	Return the sqrt of values.
divide(dividend, divisor)	Return dividend/divisor.
fluxToMag(flux[, flux_unit, return_millimags])	Convert fluxes to magnitudes.
getPatchCorners(tractInfo, patchId)	Calculate the corners of a patch, given tractInfo.
getTractCorners(skymap, tractId)	Calculate the corners of a tract, given a skymap.
isPercent(value)	Return true if the value is between 0-100
nanMax(vector)	Return the max of a vector.
nanMean(vector)	Return the mean of a vector.
nanMedian(vector)	Return the median of a vector.
nanMin(vector)	Return the max of a vector.
nanSigmaMad(vector)	Return the sigma_MAD of a vector.
nanStd(vector)
sigmaMad(vector)
sin(values)	Return the sin of values.
sqrt(values)	Return the sqrt of values.

Variables

filterwarnings_action	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_all_nan	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_all_nan_slice	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_divide_zero_divide	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_divide_zero_log	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_divide_zero_log10	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_dof_zero	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_cos	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_divide	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_log	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_log10	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_scalar_divide	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_sin	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_sqrt	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_invalid_value_subtract	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
numpy_mean_empty	str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

lsst.analysis.tools.actions.plot Package

Classes

BarPanel(*args, **kw)	A configurable class describing a panel in a bar plot.
BarPlot(*args, **kw)	A plotting tool which can take multiple keyed data inputs and can create one or more bar graphs.
ColorColorFitPlot(*args, **kw)	Make a color-color plot and overplot a prefited line to the fit region.
CompletenessHist(*args, **kw)	Makes plots of completeness and purity.
DiaSkyPanel(*args, **kw)	Configuration options for DiaSkyPlot panels.
DiaSkyPlot(*args, **kw)	Generic pseudo base class for plotting DiaSources (or DiaObjects) on the sky.
FocalPlaneGeometryPlot(*args, **kw)	Plots the focal plane distribution of a parameter in afw camera geometry units: amplifiers and detectors.
FocalPlanePlot(*args, **kw)	Plots the focal plane distribution of a parameter.
GridPanelConfig(*args, **kw)
GridPlot(*args, **kw)	Plot a series of plot elements onto a regularly spaced grid.
HistPanel(*args, **kw)	A Config class that holds parameters to configure a single panel of a histogram plot.
HistPlot(*args, **kw)	Make an N-panel plot with a configurable number of histograms displayed in each panel.
HistStatsPanel(*args, **kw)	A Config class that holds parameters to configure a the stats panel shown for histPlot.
InterpolateDetectorMetricPlot(*args, **kw)	Interpolate metrics evaluated at locations across a detector.
MatrixPlot(*args, **kw)	Make the plot of a matrix (2D array).
Med2Mad(*args, **kw)	Return the median +/- 2*nansigmamad values of an input vector to use as the minimum and maximum values of a colorbar range.
MinMax(*args, **kw)	Return the maximum and minimum values of an input vector to use as the minimum and maximum values of a colorbar range.
MultiVisitCoveragePlot(*args, **kw)	Plot the coverage for a set of visits.
PerTractPropertyMapPlot(*args, **kw)
RhoStatisticsPlot(*args, **kw)	Make multiple plots of rho statistics.
ScatterPlotStatsAction(*args, **kw)	Calculates the statistics needed for the scatter plot with two hists.
ScatterPlotWithTwoHists(*args, **kw)	Makes a scatter plot of the data with a marginal histogram for each axis.
SkyPlot(*args, **kw)	Plots the on sky distribution of a parameter.
SurveyWidePropertyMapPlot(*args, **kw)
WholeSkyPlot(*args, **kw)	Plots the on sky distribution of a parameter.
XYPlot(*args, **kw)	Make a plot (with errorbars) of one quantity (X) vs another (Y).

Class Inheritance Diagram

lsst.analysis.tools.actions.scalar Package

Classes

ApproxFloor(*args, **kw)	Returns the median of the lowest ten values of the sorted input.
CountAction(*args, **kw)	Performs count actions, with threshold-based filtering.
CountUniqueAction(*args, **kw)	Counts the number of unique rows in a given column.
DivideScalar(*args, **kw)	Calculate (A/B) for scalars.
FracInRange(*args, **kw)	Compute the fraction of a distribution that is between specified minimum and maximum values, and is not NaN.
FracNan(*args, **kw)	Compute the fraction of vector entries that are NaN.
FracThreshold(*args, **kw)	Compute the fraction of a distribution above or below a threshold.
IqrHistAction(*args, **kw)	Calculates the interquartile range of the given histogram data.
MaxAction(*args, **kw)	Returns the maximum of the given data.
MeanAction(*args, **kw)	Calculates the mean of the given data.
MedianAction(*args, **kw)	Calculates the median of the given data.
MedianHistAction(*args, **kw)	Calculates the median of the given histogram data.
MinAction(*args, **kw)	Returns the minimum of the given data.
RmsAction(*args, **kw)	Calculates the root mean square of the given data (without subtracting the mean as in StdevAction).
SigmaMadAction(*args, **kw)	Calculates the sigma mad of the given data.
StdevAction(*args, **kw)	Calculates the standard deviation of the given data.
SumAction(*args, **kw)	Returns the sum of all values in the column.
ValueAction(*args, **kw)	Extracts the first value from a vector.

Class Inheritance Diagram

lsst.analysis.tools.actions.vector Package

Classes

AddVector(*args, **kw)	Calculate (A+B).
AngularSeparation(*args, **kw)	Calculate the angular separation between two coordinate positions.
BandSelector(*args, **kw)	Makes a mask for sources observed in a specified set of bands.
CalcBinnedStatsAction(*args, **kw)
CalcE(*args, **kw)	Calculate a complex value representation of the ellipticity.
CalcE1(*args, **kw)	Calculate $e_1$ (distortion-type) or $g_1$ (shear-type).
CalcE2(*args, **kw)	Calculate $e_2$ (distortion-type) or $g_2$ (shear-type).
CalcEDiff(*args, **kw)	Calculate the difference of two ellipticities as a complex quantity.
CalcMomentSize(*args, **kw)	Calculate a size based on 2D moments.
CalcRhoStatistics(*args, **kw)	Calculate rho statistics.
CalcSn(*args, **kw)	Calculate the signal-to-noise ratio from a single flux vector.
CoaddPlotFlagSelector(*args, **kw)	This default setting makes it take the band from the kwargs.
ColorDiff(*args, **kw)	Calculate the difference between two colors from flux actions.
ColorError(*args, **kw)	Calculate the error in a color from two different flux error columns.
ConstantValue(*args, **kw)	Return a constant scalar value.
ConvertFluxToMag(*args, **kw)	Turn nano janskies into magnitudes.
ConvertUnits(*args, **kw)	Convert the units of a vector.
CosVector(*args, **kw)	Calculate cos(A)
DivideVector(*args, **kw)	Calculate (A/B)
DownselectVector(*args, **kw)	Get a vector from KeyedData, apply specified selector, return the shorter Vector.
ExtendednessSelector(*args, **kw)	A selector that picks between extended and point sources.
ExtinctionCorrectedMagDiff(*args, **kw)	Compute the difference between two magnitudes and correct for extinction By default bands are derived from the <band>_ prefix on flux columns, per the naming convention in the Object Table: e.g.
FiniteSelector(*args, **kw)	Return a mask of finite values for a vector key
FlagSelector(*args, **kw)	The base flag selector to use to select valid sources for QA.
FractionalDifference(*args, **kw)	Calculate (A-B)/B.
GalaxySelector(*args, **kw)	A selector that picks out galaxies based off of their extendedness values.
GoodDiaSourceSelector(*args, **kw)	Selects good DIA sources from diaSourceTables.
InjectedClassSelector(*args, **kw)	A selector for injected objects of a given class.
InjectedGalaxySelector(*args, **kw)	A selector for injected stars of a given class.
InjectedObjectSelector(*args, **kw)	A selector for injected objects.
InjectedStarSelector(*args, **kw)	A selector for injected stars of a given class.
IsMatchedObjectSameClass(*args, **kw)	Action to return whether matched objects are the same class.
LoadVector(*args, **kw)	Load and return a Vector from KeyedData.
Log10Vector(*args, **kw)	Calculate log10(A)
MagDiff(*args, **kw)	Calculate the difference between two magnitudes; each magnitude is derived from a flux column.
MagSelector(*args, **kw)	Selects points that have minMag < mag (AB) < maxMag.
MatchedObjectSelector(*args, **kw)	A selector that selects matched objects with finite distances.
MatchingFlagSelector(*args, **kw)	The default flag selector to apply pre matching.
MultiCriteriaDownselectVector(*args, **kw)	Get a vector from KeyedData, apply specified set of selectors with AND logic, and return the shorter Vector.
MultiplyVector(*args, **kw)	Calculate (A*B)
PerGroupStatistic(*args, **kw)	Compute per-group statistic values and return result as a vector with one element per group.
RAcosDec(*args, **kw)	Construct a vector of RA*cos(Dec) in order to have commensurate values between RA and Dec.
RaiseFromBaseVector(*args, **kw)	Calculate n**A
RaiseToPowerVector(*args, **kw)	Calculate A**n
RangeSelector(*args, **kw)	Selects rows within a range, inclusive of min/exclusive of max.
ReferenceGalaxySelector(*args, **kw)	A selector that selects galaxies from a catalog with a boolean column identifying unresolved sources.
ReferenceObjectSelector(*args, **kw)	A selector that selects all objects from a catalog with a boolean column identifying unresolved sources.
ReferenceStarSelector(*args, **kw)	A selector that selects stars from a catalog with a boolean column identifying unresolved sources.
ResidualWithPerGroupStatistic(*args, **kw)	Compute residual between individual elements of group and the per-group statistic.
SelectorBase(*args, **kw)
SinVector(*args, **kw)	Calculate sin(A)
SkyObjectSelector(*args, **kw)	Selects sky objects in the given band(s).
SkySourceSelector(*args, **kw)	Selects sky sources from sourceTables.
SnSelector(*args, **kw)	Selects points that have S/N > threshold in the given flux type.
SqrtVector(*args, **kw)	Calculate sqrt(A)
SquareVector(*args, **kw)	Calculate A**2
StarSelector(*args, **kw)	A selector that picks out stars based off of their extendedness values.
SubtractVector(*args, **kw)	Calculate (A-B).
ThresholdSelector(*args, **kw)	Return a mask corresponding to an applied threshold.
TreecorrConfig(*args, **kwargs)	Deprecated since version v28.0.
UnknownSelector(*args, **kw)	A selector that picks out unclassified objects based off of their extendedness values.
VectorSelector(*args, **kw)	Load a boolean vector from KeyedData and return it for use as a selector.
VisitPlotFlagSelector(*args, **kw)	Select on a set of flags appropriate for making visit-level plots (i.e., using sourceTable_visit catalogs).

Class Inheritance Diagram

lsst.analysis.tools.actions.keyedData Package

Classes

AddComputedVector(*args, **kw)	Compute a Vector from the specified VectorAction and add it to a copy of the KeyedData, returning the result.
CalcBinnedCompletenessAction(*args, **kw)	Calculate completeness and purity in a single magnitude bin.
CalcCompletenessHistogramAction(*args, **kw)	Action to calculate a histogram of completeness vs magnitude.
CalcRelativeDistances(*args, **kw)	Calculate relative distances in a matched catalog.
ChainedKeyedDataActions(*args, **kw)	Run a series of KeyedDataActions and accumulated their output into one KeyedData result.
KeyedDataKeyAccessAction(*args, **kw)	Fetches the value for a given top-level key in a KeyedData.
KeyedDataSelectorAction(*args, **kw)	Extract Vector specified by vectorKeys from input KeyedData and optionally apply selectors to down select extracted vectors.
KeyedScalars(*args, **kw)	Creates an output of type KeyedData, where the keys are given by the identifiers in scalarActions and the values are the results of the corresponding ScalarAction.
MagnitudeCompletenessConfig(*args, **kw)	Configuration for measuring magnitudes at given completeness thresholds.
StellarLocusFitAction(*args, **kw)	Determine Stellar Locus fit parameters from given input Vectors.

Class Inheritance Diagram

lsst.analysis.tools.tasks Package

Classes

AmpToAmpCorrelationAnalysisConfig(*args, **kw)
AmpToAmpCorrelationAnalysisTask(*[, config, ...])	Task to analyze the correlation between amplifier biases.
AssocDiaSrcDetectorVisitAnalysisConfig(...)
AssocDiaSrcDetectorVisitAnalysisTask(*[, ...])
AssociatedSourcesTractAnalysisConfig(*args, **kw)
AssociatedSourcesTractAnalysisTask(*[, ...])
AstrometricCatalogMatchConfig(*args, **kw)
AstrometricCatalogMatchTask(*[, config, ...])	Match a tract-level catalog to a reference catalog
AstrometricCatalogMatchVisitConfig(*args, **kw)
AstrometricCatalogMatchVisitTask(*[, ...])	Match a visit-level catalog to a reference catalog
CalexpSummaryAnalysisConfig(*args, **kw)
CalexpSummaryAnalysisTask(*[, config, log, ...])
CatalogMatchConfig(*args, **kw)
CatalogMatchTask(*[, config, log, initInputs])	The base task for matching catalogs.
CcdVisitTableAnalysisConfig(*args, **kw)
CcdVisitTableAnalysisTask(*[, config, log, ...])
ConsolidateResourceUsageConfig(*args, **kw)	Configuration definitions for ConsolidateResourceUsageTask.
ConsolidateResourceUsageConnections(*[, config])	Connection definitions for ConsolidateResourceUsageTask.
ConsolidateResourceUsageTask(*[, config, ...])	A PipelineTask that summarizes task resource usage into a single table with per-task rows.
DatasetMetadataAnalysisTask(*[, config, ...])
DiaFakesDetectorVisitAnalysisConfig(*args, **kw)
DiaFakesDetectorVisitAnalysisTask(*[, ...])
DiaFakesVisitAnalysisConfig(*args, **kw)
DiaFakesVisitAnalysisTask(*[, config, log, ...])
DiaObjectDetectorVisitAnalysisConfig(*args, **kw)
DiaObjectDetectorVisitAnalysisConnections(*)
DiaObjectDetectorVisitAnalysisTask(*[, ...])
DiaSourceDetectorVisitAnalysisConfig(*args, **kw)
DiaSourceDetectorVisitAnalysisConnections(*)
DiaSourceDetectorVisitAnalysisTask(*[, ...])
DiaSourceTableTractAnalysisConfig(*args, **kw)
DiaSourceTableTractAnalysisTask(*[, config, ...])
DiffMatchedAnalysisConfig(*args, **kw)
DiffMatchedAnalysisTask(*[, config, log, ...])
DiffimDetectorVisitMetricsAnalysisConfig(...)
DiffimDetectorVisitMetricsAnalysisTask(...)	Deprecated since version v29.0.
DiffimDetectorVisitSpatiallySampledPlotsConfig(...)
DiffimDetectorVisitSpatiallySampledPlotsTask(*)
GatherResourceUsageConfig(*args, **kw)	Configuration definitions for GatherResourceUsageTask.
GatherResourceUsageConnections(*[, config])	Connection definitions for GatherResourceUsageTask.
GatherResourceUsageTask(*[, config, log, ...])	A PipelineTask that gathers resource usage statistics from task metadata.
InjectedObjectAnalysisConfig(*args, **kw)
InjectedObjectAnalysisTask(*[, config, log, ...])	Make plots and metrics using a table of measured objects matched to injected inputs.
MakeMetricTableConfig(*args, **kw)
MakeMetricTableTask(*[, config, log, initInputs])	Turn metric bundles and combine them into a metric table.
MakeTractPostageStampConfig(*args, **kw)
MakeTractPostageStampTask(*[, config, log, ...])
MetadataAnalysisConfig(*args, **kw)
MetadataExposureDetectorAnalysisConfig(...)
MetadataExposureDetectorAnalysisTask(*[, ...])
MetricAnalysisConfig(*args, **kw)
MetricAnalysisTask(*[, config, log, initInputs])	Take a metric table and run an analysis tool on the data it contains.
ObjectEpochTableConfig(*args, **kw)
ObjectEpochTableTask(*[, config, log, ...])	Collect mean epochs for the observations that went into each object.
ObjectTableSurveyAnalysisTask(*[, config, ...])	A specialized AnalysisPipelineTask for multiple tracts.
ObjectTableTractAnalysisConfig(*args, **kw)
ObjectTableTractAnalysisConnections(*[, config])
ObjectTableTractAnalysisTask(*[, config, ...])
PerTractPropertyMapAnalysisConfig(*args, **kw)
PerTractPropertyMapAnalysisTask(*[, config, ...])
PhotometricCatalogMatchConfig(*args, **kw)
PhotometricCatalogMatchTask(*[, config, ...])	A wrapper task to provide the information that is specific to the photometric reference catalog.
RefCatObjectAnalysisConfig(*args, **kw)
RefCatObjectAnalysisTask(*[, config, log, ...])	Make plots and metrics using a table of objects matched to reference catalog sources.
RefCatObjectPhotometricAnalysisConfig(*args, ...)
RefCatObjectPhotometricAnalysisTask(*[, ...])	Make plots and metrics using a table of objects matched to photometric reference catalog sources.
RefCatSourceAnalysisConfig(*args, **kw)
RefCatSourceAnalysisTask(*[, config, log, ...])	Make plots and metrics using a table of objects matched to reference catalog sources.
RefCatSourcePhotometricAnalysisConfig(*args, ...)
RefCatSourcePhotometricAnalysisTask(*[, ...])	Make plots and metrics using a table of objects matched to photometric reference catalog sources.
ResourceUsageQuantumGraphBuilder(butler, *)	Custom quantum graph generator and pipeline builder for resource usage summary tasks.
SourceObjectTableAnalysisConfig(*args, **kw)
SourceObjectTableAnalysisTask(*[, config, ...])
SourceTableVisitAnalysisConfig(*args, **kw)
SourceTableVisitAnalysisTask(*[, config, ...])
SurveyWidePropertyMapAnalysisConfig(*args, **kw)
SurveyWidePropertyMapAnalysisTask(*[, ...])
TaskMetadataAnalysisTask(*[, config, log, ...])
TrailedDiaSrcDetectorVisitAnalysisConfig(...)
TrailedDiaSrcDetectorVisitAnalysisTask(*[, ...])
VerifyBiasResultsAnalysisConfig(*args, **kw)
VerifyBiasResultsAnalysisConnections(*[, config])
VerifyBiasResultsAnalysisTask(*[, config, ...])
VerifyCalibAnalysisConfig(*args, **kw)
VerifyCalibAnalysisConfigByFilter(*args, **kw)
VerifyCalibAnalysisTask(*[, config, log, ...])
VerifyCalibAnalysisTaskByFilter(*[, config, ...])
VerifyCalibDetectorConfig(*args, **kw)
VerifyCalibDetectorConfigByFilter(*args, **kw)
VerifyCalibDetectorTask(*[, config, log, ...])
VerifyCalibDetectorTaskByFilter(*[, config, ...])
WholeSkyAnalysisConfig(*args, **kw)
WholeSkyAnalysisTask(*[, config, log, ...])

Class Inheritance Diagram