PhotonTransferCurveExtractTask¶
- class lsst.cp.pipe.PhotonTransferCurveExtractTask(*, config: PipelineTaskConfig | None = None, log: logging.Logger | LsstLogAdapter | None = None, initInputs: Dict[str, Any] | None = None, **kwargs: Any)¶
Bases:
PipelineTask
,CmdLineTask
Task to measure covariances from flat fields.
This task receives as input a list of flat-field images (flats), and sorts these flats in pairs taken at the same time (the task will raise if there is one one flat at a given exposure time, and it will discard extra flats if there are more than two per exposure time). This task measures the mean, variance, and covariances from a region (e.g., an amplifier) of the difference image of the two flats with the same exposure time.
The variance is calculated via afwMath, and the covariance via the methods in Astier+19 (appendix A). In theory, var = covariance[0,0]. This should be validated, and in the future, we may decide to just keep one (covariance). At this moment, if the two values differ by more than the value of
thresholdDiffAfwVarVsCov00
(default: 1%), a warning will be issued.The measured covariances at a given exposure time (along with other quantities such as the mean) are stored in a PTC dataset object (
PhotonTransferCurveDataset
), which gets partially filled at this stage (the remainder of the attributes of the dataset will be filled after running the second task of the PTC-measurement pipeline,PhotonTransferCurveSolveTask
).The number of partially-filled
PhotonTransferCurveDataset
objects will be less than the number of input exposures because the task combines input flats in pairs. However, it is required at this moment that the number of input dimensions matches bijectively the number of output dimensions. Therefore, a number of “dummy” PTC datasets are inserted in the output list. This output list will then be used as input of the next task in the PTC-measurement pipeline,PhotonTransferCurveSolveTask
, which will assemble the multiplePhotonTransferCurveDataset
objects into a single one in order to fit the measured covariances as a function of flux to one of three models (seePhotonTransferCurveSolveTask
for details).Reference: Astier+19: “The Shape of the Photon Transfer Curve of CCD sensors”, arXiv:1905.08677.
Attributes Summary
Methods Summary
applyOverrides
(config)A hook to allow a task to change the values of its config after the camera-specific overrides are loaded but before any command-line overrides are applied.
Empty (clear) the metadata for this Task and all sub-Tasks.
Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
Get metadata for all tasks.
Get the task name as a hierarchical name including parent task names.
getGainFromFlatPair
(im1Area, im2Area, ...[, ...])Estimate the gain from a single pair of flats.
getImageAreasMasksStats
(exposure1, exposure2)Get image areas in a region as well as masks and statistic objects.
getName
()Get the name of the task.
getReadNoiseFromMetadata
(taskMetadata, ampName)Gets readout noise for an amp from ISR metadata.
Return resource configuration for this task.
Get the schemas generated by this task.
Get a dictionary of all tasks as a shallow copy.
makeCovArray
(inputTuple, maxRangeFromTuple)Make covariances array from tuple.
makeField
(doc)Make a
lsst.pex.config.ConfigurableField
for this task.makeSubtask
(name, **keyArgs)Create a subtask as a new instance as the
name
attribute of this task.measureMeanVarCov
(im1Area, im2Area, ...)Calculate the mean of each of two exposures and the variance and covariance of their difference.
parseAndRun
([args, config, log, doReturnResults])Parse an argument list and run the command.
run
(inputExp, inputDims, taskMetadata)Measure covariances from difference of flat pairs
runQuantum
(butlerQC, inputRefs, outputRefs)Ensure that the input and output dimensions are passed along.
timer
(name[, logLevel])Context manager to log performance data for an arbitrary block of code.
writeConfig
(butler[, clobber, doBackup])Write the configuration used for processing the data, or check that an existing one is equal to the new one if present.
writeMetadata
(dataRef)Write the metadata produced from processing the data.
writePackageVersions
(butler[, clobber, ...])Compare and write package versions.
writeSchemas
(butler[, clobber, doBackup])Write the schemas returned by
lsst.pipe.base.Task.getAllSchemaCatalogs
.Attributes Documentation
Methods Documentation
- classmethod applyOverrides(config)¶
A hook to allow a task to change the values of its config after the camera-specific overrides are loaded but before any command-line overrides are applied.
- Parameters:
- configinstance of task’s
ConfigClass
Task configuration.
- configinstance of task’s
Notes
This is necessary in some cases because the camera-specific overrides may retarget subtasks, wiping out changes made in ConfigClass.setDefaults. See LSST Trac ticket #2282 for more discussion.
Warning
This is called by CmdLineTask.parseAndRun; other ways of constructing a config will not apply these overrides.
- getAllSchemaCatalogs() Dict[str, Any] ¶
Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
- Returns:
- schemacatalogs
dict
Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
lsst.afw.table
Catalog type) for all tasks in the hierarchy, from the top-level task down through all subtasks.
- schemacatalogs
Notes
This method may be called on any task in the hierarchy; it will return the same answer, regardless.
The default implementation should always suffice. If your subtask uses schemas the override
Task.getSchemaCatalogs
, not this method.
- getFullMetadata() TaskMetadata ¶
Get metadata for all tasks.
- Returns:
- metadata
TaskMetadata
The keys are the full task name. Values are metadata for the top-level task and all subtasks, sub-subtasks, etc.
- metadata
Notes
The returned metadata includes timing information (if
@timer.timeMethod
is used) and any metadata set by the task. The name of each item consists of the full task name with.
replaced by:
, followed by.
and the name of the item, e.g.:topLevelTaskName:subtaskName:subsubtaskName.itemName
using
:
in the full task name disambiguates the rare situation that a task has a subtask and a metadata item with the same name.
- getFullName() str ¶
Get the task name as a hierarchical name including parent task names.
- Returns:
- fullName
str
The full name consists of the name of the parent task and each subtask separated by periods. For example:
The full name of top-level task “top” is simply “top”.
The full name of subtask “sub” of top-level task “top” is “top.sub”.
The full name of subtask “sub2” of subtask “sub” of top-level task “top” is “top.sub.sub2”.
- fullName
- getGainFromFlatPair(im1Area, im2Area, imStatsCtrl, mu1, mu2, correctionType='NONE', readNoise=None)¶
Estimate the gain from a single pair of flats.
The basic premise is 1/g = <(I1 - I2)^2/(I1 + I2)> = 1/const, where I1 and I2 correspond to flats 1 and 2, respectively. Corrections for the variable QE and the read-noise are then made following the derivation in Robert Lupton’s forthcoming book, which gets
1/g = <(I1 - I2)^2/(I1 + I2)> - 1/mu(sigma^2 - 1/2g^2).
This is a quadratic equation, whose solutions are given by:
- g = mu +/- sqrt(2*sigma^2 - 2*const*mu + mu^2)/(2*const*mu*2
2*sigma^2)
where ‘mu’ is the average signal level and ‘sigma’ is the amplifier’s readnoise. The positive solution will be used. The way the correction is applied depends on the value supplied for correctionType.
- correctionType is one of [‘NONE’, ‘SIMPLE’ or ‘FULL’]
‘NONE’ : uses the 1/g = <(I1 - I2)^2/(I1 + I2)> formula. ‘SIMPLE’ : uses the gain from the ‘NONE’ method for the
1/2g^2 term.
- ‘FULL’solves the full equation for g, discarding the
non-physical solution to the resulting quadratic.
- Parameters:
- im1Area
lsst.afw.image.maskedImage.MaskedImageF
Masked image from exposure 1.
- im2Area
lsst.afw.image.maskedImage.MaskedImageF
Masked image from exposure 2.
- imStatsCtrl
lsst.afw.math.StatisticsControl
Statistics control object.
- mu1: `float`
Clipped mean of im1Area (ADU).
- mu2: `float`
Clipped mean of im2Area (ADU).
- correctionType
str
, optional The correction applied, one of [‘NONE’, ‘SIMPLE’, ‘FULL’]
- readNoise
float
, optional Amplifier readout noise (ADU).
- im1Area
- Returns:
- gain
float
Gain, in e/ADU.
- gain
- Raises:
- RuntimeError: if
correctionType
is not one of ‘NONE’, ‘SIMPLE’, or ‘FULL’.
- RuntimeError: if
- getImageAreasMasksStats(exposure1, exposure2, region=None)¶
Get image areas in a region as well as masks and statistic objects.
- Parameters:
- exposure1
lsst.afw.image.exposure.ExposureF
First exposure of flat field pair.
- exposure2
lsst.afw.image.exposure.ExposureF
Second exposure of flat field pair.
- region
lsst.geom.Box2I
, optional Region of each exposure where to perform the calculations (e.g, an amplifier).
- exposure1
- Returns:
- im1Area
lsst.afw.image.maskedImage.MaskedImageF
Masked image from exposure 1.
- im2Area
lsst.afw.image.maskedImage.MaskedImageF
Masked image from exposure 2.
- imStatsCtrl
lsst.afw.math.StatisticsControl
Statistics control object.
- mu1:
float
Clipped mean of im1Area (ADU).
- mu2:
float
Clipped mean of im2Area (ADU).
- im1Area
- getReadNoiseFromMetadata(taskMetadata, ampName)¶
Gets readout noise for an amp from ISR metadata.
- Parameters:
- taskMetadata
list
[lsst.pipe.base.TaskMetadata
] List of exposures metadata from ISR.
- ampName
str
Amplifier name.
- taskMetadata
- Returns:
- readNoise
float
Median of the overscan readnoise in the post-ISR metadata of the input exposures (ADU). Returns ‘None’ if the median could not be calculated.
- readNoise
- getResourceConfig() ResourceConfig | None ¶
Return resource configuration for this task.
- Returns:
- Object of type
ResourceConfig
orNone
if resource - configuration is not defined for this task.
- Object of type
- getSchemaCatalogs() Dict[str, Any] ¶
Get the schemas generated by this task.
- Returns:
- schemaCatalogs
dict
Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
lsst.afw.table
Catalog type) for this task.
- schemaCatalogs
See also
Task.getAllSchemaCatalogs
Notes
Warning
Subclasses that use schemas must override this method. The default implementation returns an empty dict.
This method may be called at any time after the Task is constructed, which means that all task schemas should be computed at construction time, not when data is actually processed. This reflects the philosophy that the schema should not depend on the data.
Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.
- getTaskDict() Dict[str, ReferenceType[Task]] ¶
Get a dictionary of all tasks as a shallow copy.
- Returns:
- taskDict
dict
Dictionary containing full task name: task object for the top-level task and all subtasks, sub-subtasks, etc.
- taskDict
- makeCovArray(inputTuple, maxRangeFromTuple)¶
Make covariances array from tuple.
- Parameters:
- inputTuple
numpy.ndarray
Structured array with rows with at least (mu, afwVar, cov, var, i, j, npix), where: mu :
float
0.5*(m1 + m2), where mu1 is the mean value of flat1 and mu2 is the mean value of flat2.
- maxRangeFromTuple
int
Maximum range to select from tuple.
- inputTuple
- Returns:
- cov
numpy.array
Covariance arrays, indexed by mean signal mu.
- vCov
numpy.array
Variance arrays, indexed by mean signal mu.
- muVals
numpy.array
List of mean signal values.
- cov
- classmethod makeField(doc: str) ConfigurableField ¶
Make a
lsst.pex.config.ConfigurableField
for this task.- Parameters:
- doc
str
Help text for the field.
- doc
- Returns:
- configurableField
lsst.pex.config.ConfigurableField
A
ConfigurableField
for this task.
- configurableField
Examples
Provides a convenient way to specify this task is a subtask of another task.
Here is an example of use:
class OtherTaskConfig(lsst.pex.config.Config): aSubtask = ATaskClass.makeField("brief description of task")
- makeSubtask(name: str, **keyArgs: Any) None ¶
Create a subtask as a new instance as the
name
attribute of this task.- Parameters:
- name
str
Brief name of the subtask.
- keyArgs
Extra keyword arguments used to construct the task. The following arguments are automatically provided and cannot be overridden:
“config”.
“parentTask”.
- name
Notes
The subtask must be defined by
Task.config.name
, an instance ofConfigurableField
orRegistryField
.
- measureMeanVarCov(im1Area, im2Area, imStatsCtrl, mu1, mu2)¶
Calculate the mean of each of two exposures and the variance and covariance of their difference. The variance is calculated via afwMath, and the covariance via the methods in Astier+19 (appendix A). In theory, var = covariance[0,0]. This should be validated, and in the future, we may decide to just keep one (covariance).
- Parameters:
- im1Area
lsst.afw.image.maskedImage.MaskedImageF
Masked image from exposure 1.
- im2Area
lsst.afw.image.maskedImage.MaskedImageF
Masked image from exposure 2.
- imStatsCtrl
lsst.afw.math.StatisticsControl
Statistics control object.
- mu1: `float`
Clipped mean of im1Area (ADU).
- mu2: `float`
Clipped mean of im2Area (ADU).
- im1Area
- Returns:
- mu
float
orNaN
0.5*(mu1 + mu2), where mu1, and mu2 are the clipped means of the regions in both exposures. If either mu1 or m2 are NaN’s, the returned value is NaN.
- varDiff
float
orNaN
Half of the clipped variance of the difference of the regions inthe two input exposures. If either mu1 or m2 are NaN’s, the returned value is NaN.
- covDiffAstier
list
orNaN
- List with tuples of the form (dx, dy, var, cov, npix), where:
If either mu1 or m2 are NaN’s, the returned value is NaN.
- mu
- classmethod parseAndRun(args=None, config=None, log=None, doReturnResults=False)¶
Parse an argument list and run the command.
- Parameters:
- args
list
, optional - config
lsst.pex.config.Config
-type, optional Config for task. If
None
useTask.ConfigClass
.- log
logging.Logger
-type, optional Log. If
None
use the default log.- doReturnResults
bool
, optional If
True
, return the results of this task. Default isFalse
. This is only intended for unit tests and similar use. It can easily exhaust memory (if the task returns enough data and you call it enough times) and it will fail when using multiprocessing if the returned data cannot be pickled.
- args
- Returns:
- struct
lsst.pipe.base.Struct
Fields are:
argumentParser
the argument parser (
lsst.pipe.base.ArgumentParser
).parsedCmd
the parsed command returned by the argument parser’s
parse_args
method (argparse.Namespace
).taskRunner
the task runner used to run the task (an instance of
Task.RunnerClass
).resultList
results returned by the task runner’s
run
method, one entry per invocation (list
). This will typically be a list ofStruct
, each containing at least anexitStatus
integer (0 or 1); seeTask.RunnerClass
(TaskRunner
by default) for more details.
- struct
Notes
Calling this method with no arguments specified is the standard way to run a command-line task from the command-line. For an example see
pipe_tasks
bin/makeSkyMap.py
or almost any other file in that directory.If one or more of the dataIds fails then this routine will exit (with a status giving the number of failed dataIds) rather than returning this struct; this behaviour can be overridden by specifying the
--noExit
command-line option.
- run(inputExp, inputDims, taskMetadata)¶
Measure covariances from difference of flat pairs
- Parameters:
- inputExp
dict
[float
,list
Dictionary that groups references to flat-field exposures that have the same exposure time (seconds), or that groups them sequentially by their exposure id.
- inputDims
list
List of exposure IDs.
- taskMetadata
list
[lsst.pipe.base.TaskMetadata
] List of exposures metadata from ISR.
- inputExp
- Returns:
- results
lsst.pipe.base.Struct
The resulting Struct contains:
outputCovariances
A list containing the per-pair PTC measurements (
list
[lsst.ip.isr.PhotonTransferCurveDataset
])
- results
- runQuantum(butlerQC, inputRefs, outputRefs)¶
Ensure that the input and output dimensions are passed along.
- Parameters:
- butlerQC
ButlerQuantumContext
Butler to operate on.
- inputRefs
InputQuantizedConnection
Input data refs to load.
- ouptutRefs
OutputQuantizedConnection
Output data refs to persist.
- butlerQC
- timer(name: str, logLevel: int = 10) Iterator[None] ¶
Context manager to log performance data for an arbitrary block of code.
- Parameters:
See also
timer.logInfo
Examples
Creating a timer context:
with self.timer("someCodeToTime"): pass # code to time
- writeConfig(butler, clobber=False, doBackup=True)¶
Write the configuration used for processing the data, or check that an existing one is equal to the new one if present.
- Parameters:
- butler
lsst.daf.persistence.Butler
Data butler used to write the config. The config is written to dataset type
CmdLineTask._getConfigName
.- clobber
bool
, optional A boolean flag that controls what happens if a config already has been saved:
- doBackup
bool
, optional Set to
True
to backup the config files if clobbering.
- butler
- writeMetadata(dataRef)¶
Write the metadata produced from processing the data.
- Parameters:
- dataRef
Butler data reference used to write the metadata. The metadata is written to dataset type
CmdLineTask._getMetadataName
.
- writePackageVersions(butler, clobber=False, doBackup=True, dataset='packages')¶
Compare and write package versions.
- Parameters:
- butler
lsst.daf.persistence.Butler
Data butler used to read/write the package versions.
- clobber
bool
, optional A boolean flag that controls what happens if versions already have been saved:
- doBackup
bool
, optional If
True
and clobbering, old package version files are backed up.- dataset
str
, optional Name of dataset to read/write.
- butler
- Raises:
- TaskError
Raised if there is a version mismatch with current and persisted lists of package versions.
Notes
Note that this operation is subject to a race condition.
- writeSchemas(butler, clobber=False, doBackup=True)¶
Write the schemas returned by
lsst.pipe.base.Task.getAllSchemaCatalogs
.- Parameters:
- butler
lsst.daf.persistence.Butler
Data butler used to write the schema. Each schema is written to the dataset type specified as the key in the dict returned by
getAllSchemaCatalogs
.- clobber
bool
, optional A boolean flag that controls what happens if a schema already has been saved:
- doBackup
bool
, optional Set to
True
to backup the schema files if clobbering.
- butler
Notes
If
clobber
isFalse
and an existing schema does not match a current schema, then some schemas may have been saved successfully and others may not, and there is no easy way to tell which is which.