PhotonTransferCurveDataset

class lsst.ip.isr.PhotonTransferCurveDataset(ampNames=[], ptcFitType=None, covMatrixSide=1, covMatrixSideFullCovFit=None, **kwargs)

Bases: IsrCalib

A simple class to hold the output data from the PTC task.

The dataset is made up of a dictionary for each item, keyed by the amplifiers’ names, which much be supplied at construction time. New items cannot be added to the class to save accidentally saving to the wrong property, and the class can be frozen if desired. inputExpIdPairs records the exposures used to produce the data. When fitPtc() or fitCovariancesAstier() is run, a mask is built up, which is by definition always the same length as inputExpIdPairs, rawExpTimes, rawMeans and rawVars, and is a list of bools, which are incrementally set to False as points are discarded from the fits.

Parameters:

ampNameslist

List with the names of the amplifiers of the detector at hand.

ptcFitTypestr, optional

Type of model fitted to the PTC: “EXPAPPROXIMATION”, or “FULLCOVARIANCE” or “FULLCOVARIANCE_NO_B”.

covMatrixSideint, optional

Maximum lag of measured covariances (size of square covariance matrices).

covMatrixSideFullCovFitint, optional

Maximum covariances lag for FULLCOVARIANCE fit. It should be less or equal than covMatrixSide.

kwargsdict, optional

Other keyword arguments to pass to the parent init.

Notes

The stored attributes are:

badAmpslist [str]

List with bad amplifiers names.

inputExpIdPairsdict, [str, list]

Dictionary keyed by amp names containing the input exposures IDs.

inputExpPairMjdStartListdict, [str, np.ndarray]

Dictionary keyed by amp names containing the start mjd from the first exposure in each flat pair.

expIdMaskdict, [str, np.ndarray]

Dictionary keyed by amp names containing the mask produced after outlier rejection. The mask produced by the “FULLCOVARIANCE” option may differ from the one produced in the other two PTC fit types.

expIdRolloffMaskdict, [str, np.ndarray]

Dictionary keyed by amp names containing the mask produced after outlier rejection and extended to fit the PTC rolloff.

rawExpTimesdict, [str, np.ndarray]

Dictionary keyed by amp names containing the unmasked exposure times.

rawMeansdict, [str, np.ndarray]

Dictionary keyed by amp names containing the unmasked average of the means of the exposures in each flat pair (units: adu).

rawVarsdict, [str, np.ndarray]

Dictionary keyed by amp names containing the variance of the difference image of the exposures in each flat pair (units: adu^2).

rawDeltasdict, [str, np.ndarray]

Dictionary keyed by amp names containing the scaled unmasked delta of the means of the exposures in each flat pair (mean2 - mean1) (units: adu).

rowMeanVariancedict, [str, np.ndarray]

Dictionary keyed by amp names containing the variance of the means of the rows of the difference image of the exposures in each flat pair (units: adu^2).

histVarsdict, [str, np.ndarray]

Dictionary keyed by amp names containing the variance of the difference image of the exposures in each flat pair estimated by fitting a Gaussian model.

histChi2Dofsdict, [str, np.ndarray]

Dictionary keyed by amp names containing the chi-squared per degree of freedom fitting the difference image to a Gaussian model.

kspValuesdict, [str, np.ndarray]

Dictionary keyed by amp names containing the KS test p-value from fitting the difference image to a Gaussian model.

gaindict, [str, float]

Dictionary keyed by amp names containing the fitted gains. May be adjusted by amp-offset gain ratios if configured in PTC solver.

gainUnadjusteddict, [str, float]

Dictionary keyed by amp names containing unadjusted (raw) fit gain values. May be the same as gain values if amp-offset adjustment is not turned on.

gainErrdict, [str, float]

Dictionary keyed by amp names containing the errors on the fitted gains.

gainListdict, [str, np.ndarray]

Dictionary keyed by amp names containing the gain estimated from each flat pair.

overscanMedianLevelListdict, [str, np.ndarray]

Dictionary keyed by amp names containing the median overscan level from each input flat pair (units: adu).

overscanMediandict `, [`str, float]

Dictionary keyed by amp names containing the median of overscanMedianLevelList[expIdMask] (units: adu).

overscanMedianSigmadict `, [`str, float]

Dictionary keyed by amp names containing the median absolute deviation of overscanMedianLevelList[expIdMask] (units: adu).

noiseListdict, [str, np.ndarray]

Dictionary keyed by amp names containing the mean overscan standard deviation from each flat pair (units: adu).

noisedict, [str, float]

Dictionary keyed by amp names containing the fitted noise (units: electron).

noiseErrdict, [str, float]

Dictionary keyed by amp names containing the errors on the fitted noise (units: electron).

ampOffsetsdict, [str, float]

Dictionary keyed by amp names containing amp-to-amp offsets (units: adu).

ptcFitParsdict, [str, np.ndarray]

Dictionary keyed by amp names containing the raveled array fitted parameters of the PTC model.

ptcFitParsErrordict, [str, np.ndarray]

Dictionary keyed by amp names containing the raveled array errors on the fitted parameters of the PTC model for ptcFitType.

ptcFitChiSqdict, [str, float]

Dictionary keyed by amp names containing the reduced chi squared of the fit.

ptcTurnoffdict [str, `float]

Flux value (in adu) where the variance of the PTC curve starts decreasing consistently.

ptcTurnoffSamplingErrordict [str, float]

Sampling error on the ptcTurnoff, based on the flux sampling of the input PTC (units: adu).

ptcRolloffdict [str, `float]

Flux value (in adu) where the variance of the PTC curve begins starts to move away from the PTC model by some threshold.

ptcRolloffErrordict [str, float]

Covariance error from the fit to the PTC rolloff (units: adu).

ptcRolloffdict [str, `float]

Curvature parameter of the PTC rolloff.

ptcRolloffErrordict [str, float]

Fitting error of the curvature parameter of the PTC rolloff.

nPixelCovariancesdict, [str, int]

Dictionary keyed by amp names containing the number of pixels that were used to measure the covariances.

covariancesdict, [str, np.ndarray]

Dictionary keyed by amp names containing a list of measured covariances per mean flux (units: adu^2).

covariancesModeldict, [str, np.ndarray]

Dictionary keyed by amp names containinging covariances model (Eq. 20 of Astier+19) per mean flux (units: adu^2).

covariancesSqrtWeightsdict, [str, np.ndarray]

Dictionary keyed by amp names containinging sqrt. of covariances weights (units: 1/adu).

aMatrixdict, [str, np.ndarray]

Dictionary keyed by amp names containing the “a” parameters from the model in Eq. 20 of Astier+19 (units: 1/electron).

bMatrixdict, [str, np.ndarray]

Dictionary keyed by amp names containing the “b” parameters from the model in Eq. 20 of Astier+19 (units: 1/electron).

noiseMatrixdict, [str, np.ndarray]

Dictionary keyed by amp names containing the “noise” parameters from the model in Eq. 20 of Astier+19 (units: electron^2).

covariancesModelNoBdict, [str, np.ndarray]

Dictionary keyed by amp names containing covariances model (with ‘b’=0 in Eq. 20 of Astier+19) per mean flux (units: adu^2). Will be removed after v29.

aMatrixNoBdict, [str, np.ndarray]

Dictionary keyed by amp names containing the “a” parameters from the model in Eq. 20 of Astier+19 (and ‘b’ = 0) (units: 1/electron). Will be removed after v29.

noiseMatrixNoBdict, [str, np.ndarray]

Dictionary keyed by amp names containing the “noise” parameters from the model in Eq. 20 of Astier+19, with ‘b’ = 0 (units: electron^2). Will be removed after v29.

finalVarsdict, [str, np.ndarray]

Dictionary keyed by amp names containing the masked variance of the difference image of each flat pair. If needed, each array will be right-padded with np.nan to match the length of rawExpTimes.

finalModelVarsdict, [str, np.ndarray]

Dictionary keyed by amp names containing the masked modeled variance of the difference image of each flat pair. If needed, each array will be right-padded with np.nan to match the length of rawExpTimes.

finalMeansdict, [str, np.ndarray]

Dictionary keyed by amp names containing the masked average of the means of the exposures in each flat pair. If needed, each array will be right-padded with np.nan to match the length of rawExpTimes.

photoChargesdict, [str, np.ndarray]

Dictionary keyed by amp names containing the integrated photocharge for linearity calibration.

photoChargeDeltasdict, [str, np.ndarray]

Dictionary keyed by amp names containing the delta for the integrated photocharge (photocharge2 - photocharge1).

auxValuesdict, [str, np.ndarray]

Dictionary of per-detector auxiliary header values that can be used for PTC, linearity computation.

Version 1.1 adds the ptcTurnoff attribute. Version 1.2 adds the histVars, histChi2Dofs, and kspValues

attributes.

Version 1.3 adds the noiseMatrix and noiseMatrixNoB attributes. Version 1.4 adds the auxValues attribute. Version 1.5 adds the covMatrixSideFullCovFit attribute. Version 1.6 adds the rowMeanVariance attribute. Version 1.7 adds the noiseList attribute. Version 1.8 adds the ptcTurnoffSamplingError attribute. Version 1.9 standardizes PTC noise units to electron. Version 2.0 adds the ampOffsets, gainUnadjusted, and

gainList attributes.

Version 2.1 deprecates the covariancesModelNoB, aMatrixNoB, and

noiseMatrixNoB attributes.

Version 2.2 adds the overscanMedianLevelList and

inputExpPairMjdStartList attributes.

Version 2.3 adds the overscanMedian and

overscanMedianSigma attrbutes.

Version 2.4 adds the nPixelCovariances attribute. Version 2.5 adds the rawDeltas and photoChargeDeltas attributes. Version 2.6 adds the expIdRolloffMask, ptcRolloff, ptcRolloffError,

ptcRolloffTau, and ptcRolloffTauError attributes. Also deprecates the POLYNOMIAL fit type.

Attributes Summary

metadata
requiredAttributes

Methods Summary

appendPartialPtc(partialPtc)	Append a partial PTC dataset to this dataset.
apply(target)	Method to apply the calibration to the target object.
calibInfoFromDict(dictionary)	Handle common keywords.
determineCalibClass(metadata, message)	Attempt to find calibration class in metadata.
evalPtcModel(mu)	Computes the covariance model at specific signal levels.
fromDetector(detector)	Read metadata parameters from a detector.
fromDict(dictionary)	Construct a calibration from a dictionary of properties.
fromTable(tableList)	Construct calibration from a list of tables.
getExpIdsUsed(ampName)	Get the exposures used, i.e. not discarded, for a given amp.
getGoodAmps()	Get the good amps from this PTC.
getGoodPoints(ampName)	Get the good points used for a given amp in the PTC.
getMetadata()	Retrieve metadata associated with this calibration.
readFits(filename, **kwargs)	Read calibration data from a FITS file.
readText(filename, **kwargs)	Read calibration representation from a yaml/ecsv file.
setAmpValuesPartialDataset(ampName[, ...])	Set the amp values for a partial PTC Dataset (from cpExtractPtcTask).
setAuxValuesPartialDataset(auxDict)	Set a dictionary of auxiliary values for a partial dataset.
setMetadata(metadata)	Store a copy of the supplied metadata with this calibration.
sort(sortIndex)	Sort the components of the PTC by a given sort index.
toDict()	Return a dictionary containing the calibration properties.
toTable()	Construct a list of tables containing the information in this calibration.
updateMetadata(**kwargs)	Update calibration metadata.
updateMetadataFromExposures(exposures)	Extract and unify metadata information.
validate([other])	Validate that this calibration is defined and can be used.
validateGainNoiseTurnoffValues(ampName[, doWarn])	Ensure the gain, read noise, and PTC turnoff have sensible values.
writeFits(filename)	Write calibration data to a FITS file.
writeText(filename[, format])	Write the calibration data to a text file.

Attributes Documentation

metadata

requiredAttributes

Methods Documentation

appendPartialPtc(partialPtc)

Append a partial PTC dataset to this dataset.

Parameters:

partialPtclsst.ip.isr.PhotonTransferCurveDataset

Partial PTC to append. Should only have one element.

apply(target)

Method to apply the calibration to the target object.

Parameters:

targetobject

Thing to validate against.

Returns:

validbool

Returns true if the calibration was applied correctly.

Raises:

NotImplementedError

Raised if not implemented.

calibInfoFromDict(dictionary)

Handle common keywords.

This isn’t an ideal solution, but until all calibrations expect to find everything in the metadata, they still need to search through dictionaries.

Parameters:

dictionarydict or lsst.daf.base.PropertyList

Source for the common keywords.

Raises:

RuntimeError

Raised if the dictionary does not match the expected OBSTYPE.

classmethod determineCalibClass(metadata, message)

Attempt to find calibration class in metadata.

Parameters:

metadatadict or lsst.daf.base.PropertyList

Metadata possibly containing a calibration class entry.

messagestr

Message to include in any errors.

Returns:

calibClassobject

The class to use to read the file contents. Should be an lsst.ip.isr.IsrCalib subclass.

Raises:

ValueError

Raised if the resulting calibClass is the base lsst.ip.isr.IsrClass (which does not implement the content methods).

evalPtcModel(mu)

Computes the covariance model at specific signal levels.

Parameters:

munumpy.array, (N,)

List of mean signals in ADU.

Returns:

covModelnumpy.array, (N, M, M)

Covariances model at mu (in ADU^2).

Raises:

RuntimeError

Raised if ptcFitType is invalid.

Notes

Computes the covModel for all mu, and it returns cov[N, M, M], where the variance model is cov[:,0,0]. Both mu and cov are in ADUs and ADUs squared. This routine evaluates the approximation in Eq. 16 of Astier+19 (1905.08677) if self.ptcFitType == EXPAPPROXIMATION, and Eq. 20 of Astier+19 if self.ptcFitType == FULLCOVARIANCE(_NO_B).

The EXPAPPROXIMATION model (Eq. 16 of Astier+19) is only an approximation for the variance (cov[0,0]), so the function returns covModel of shape (N,), representing an array of [C_{00}] if self.ptcFitType == EXPAPPROXIMATION.

fromDetector(detector)

Read metadata parameters from a detector.

Parameters:

detectorlsst.afw.cameraGeom.detector

Input detector with parameters to use.

Returns:

caliblsst.ip.isr.PhotonTransferCurveDataset

The calibration constructed from the detector.

classmethod fromDict(dictionary)

Construct a calibration from a dictionary of properties. Must be implemented by the specific calibration subclasses.

Parameters:

dictionarydict

Dictionary of properties.

Returns:

caliblsst.ip.isr.PhotonTransferCurveDataset

Constructed calibration.

Raises:

RuntimeError

Raised if the supplied dictionary is for a different calibration.

classmethod fromTable(tableList)

Construct calibration from a list of tables. This method uses the fromDict method to create the calibration, after constructing an appropriate dictionary from the input tables.

Parameters:

tableListlist [lsst.afw.table.Table]

List of tables to use to construct the datasetPtc.

Returns:

caliblsst.ip.isr.PhotonTransferCurveDataset

The calibration defined in the tables.

getExpIdsUsed(ampName)

Get the exposures used, i.e. not discarded, for a given amp. If no mask has been created yet, all exposures are returned.

Parameters:

ampNamestr

Returns:

expIdsUsedlist [tuple]

List of pairs of exposure ids used in PTC.

getGoodAmps()

Get the good amps from this PTC.

getGoodPoints(ampName)

Get the good points used for a given amp in the PTC.

Parameters:

ampNamestr

Amplifier’s name.

Returns:

goodPointsnp.ndarray

Boolean array of good points used in PTC.

getMetadata()

Retrieve metadata associated with this calibration.

Returns:

metalsst.daf.base.PropertyList

Metadata. The returned PropertyList can be modified by the caller and the changes will be written to external files.

classmethod readFits(filename, **kwargs)

Read calibration data from a FITS file.

Parameters:

filenamestr

Filename to read data from.

kwargsdict or collections.abc.Mapping`, optional

Set of key=value pairs to pass to the fromTable method.

Returns:

caliblsst.ip.isr.IsrCalib

Calibration contained within the file.

classmethod readText(filename, **kwargs)

Read calibration representation from a yaml/ecsv file.

Parameters:

filenamestr

Name of the file containing the calibration definition.

kwargsdict or collections.abc.Mapping`, optional

Set of key=value pairs to pass to the fromDict or fromTable methods.

Returns:

calibIsrCalibType

Calibration class.

Raises:

RuntimeError

Raised if the filename does not end in “.ecsv” or “.yaml”.

setAmpValuesPartialDataset(ampName, inputExpIdPair=(-1, -1), inputExpPairMjdStart=nan, rawExpTime=nan, rawMean=nan, rawVar=nan, rawDelta=nan, rowMeanVariance=nan, photoCharge=nan, photoChargeDelta=nan, ampOffset=nan, expIdMask=False, expIdRolloffMask=False, nPixelCovariance=-1, covariance=None, covSqrtWeights=None, gain=nan, noise=nan, overscanMedianLevel=nan, histVar=nan, histChi2Dof=nan, kspValue=0.0)

Set the amp values for a partial PTC Dataset (from cpExtractPtcTask).

Parameters:

ampNamestr

Name of the amp to set the values.

inputExpIdPairtuple [int]

Exposure IDs of input pair.

inputExpPairMjdStartfloat, optional

The start MJD of first exposure in the flat pair.

rawExpTimefloat, optional

Exposure time for this exposure pair (units: sec).

rawMeanfloat, optional

Average of the means of the exposures in this pair (units: adu).

rawVarfloat, optional

Variance of the difference of the exposures in this pair (units: adu^2).

rawDeltafloat, optional

Delta of the means of the exposure in this pair (units: adu).

rowMeanVariancefloat, optional

Variance of the means of the rows in the difference image of the exposures in this pair (units: adu^2).

photoChargefloat, optional

Integrated photocharge for flat pair for linearity calibration (arbitrary units).

photoChargeDeltafloat, optional

Delta between integrated photocharge for the flat pair (arbitrary units).

ampOffsetfloat, optional

Amp offset for this amplifier.

expIdMaskbool, optional

Flag setting if this exposure pair should be used (True) or not used (False).

expIdRolloffMaskbool, optional

Flag setting if this exposure pair should be used (True) or not used for rolloff search (False).

nPixelCovarianceint, optional

Number of pixels that went into the covariance measurement.

covariancenp.ndarray or None, optional

Measured covariance for this exposure pair (units: adu^2).

covSqrtWeightsnp.ndarray or None, optional

Measured sqrt of covariance weights in this exposure pair (units: 1/adu).

gainfloat, optional

Estimated gain for this exposure pair (units: electron/adu).

noisefloat, optional

Estimated read noise for this exposure pair (units: electron).

overscanMedianLevelfloat, optional

Average of the median overscan levels for this exposure pair. (units: adu)

histVarfloat, optional

Variance estimated from fitting a histogram with a Gaussian model (units: adu).

histChi2Doffloat, optional

Chi-squared per degree of freedom from Gaussian histogram fit.

kspValuefloat, optional

KS test p-value from the Gaussian histogram fit.

setAuxValuesPartialDataset(auxDict)

Set a dictionary of auxiliary values for a partial dataset.

Parameters:

auxDictdict [str, float]

Dictionary of float values.

setMetadata(metadata)

Store a copy of the supplied metadata with this calibration.

Parameters:

metadatalsst.daf.base.PropertyList

Metadata to associate with the calibration. Will be copied and overwrite existing metadata.

sort(sortIndex)

Sort the components of the PTC by a given sort index.

The PTC is sorted in-place.

Parameters:

sortIndexlist or np.ndarray

The sorting index, which must be the same length as the number of elements of the PTC.

toDict()

Return a dictionary containing the calibration properties. The dictionary should be able to be round-tripped through fromDict.

Returns:

dictionarydict

Dictionary of properties.

toTable()

Construct a list of tables containing the information in this calibration.

The list of tables should create an identical calibration after being passed to this class’s fromTable method.

Returns:

tableListlist [astropy.table.Table]

List of tables containing the linearity calibration information.

updateMetadata(**kwargs)

Update calibration metadata. This calls the base class’s method after ensuring the required calibration keywords will be saved.

Parameters:

setDatebool, optional

Update the CALIBDATE fields in the metadata to the current time. Defaults to False.

kwargs

Other keyword parameters to set in the metadata.

updateMetadataFromExposures(exposures)

Extract and unify metadata information.

Parameters:

exposureslist

Exposures or other calibrations to scan.

validate(other=None)

Validate that this calibration is defined and can be used.

Parameters:

otherobject, optional

Thing to validate against.

Returns:

validbool

Returns true if the calibration is valid and appropriate.

validateGainNoiseTurnoffValues(ampName, doWarn=False)

Ensure the gain, read noise, and PTC turnoff have sensible values.

Parameters:

ampNamestr

Amplifier’s name.

writeFits(filename)

Write calibration data to a FITS file.

Parameters:

filenamestr

Filename to write data to.

Returns:

usedstr

The name of the file used to write the data.

writeText(filename, format='auto')

Write the calibration data to a text file.

Parameters:

filenamestr

Name of the file to write.

formatstr

Format to write the file as. Supported values are:

"auto" : Determine filetype from filename. "yaml" : Write as yaml. "ecsv" : Write as ecsv.

Returns:

usedstr

The name of the file used to write the data. This may differ from the input if the format is explicitly chosen.

Raises:

RuntimeError

Raised if filename does not end in a known extension, or if all information cannot be written.

Notes

The file is written to YAML/ECSV format and will include any associated metadata.