FgcmFitCycleTask

FgcmFitCycleTask runs a single fgcm fit on the star observations generated from FgcmBuildStarsTask using the look-up table generated from FgcmMakeLutTask. This code is meant to be run multiple times until convergence, and the results output from one “fit cycle” are used as an input to the subsequent fit cycle. One final cleanup run is performed to generate the tables required for input to FgcmOutputProductsTask.

This is the third task in a typical fgcmcal processing chain. The first is FgcmMakeLutTask, the second is FgcmBuildStarsTableTask or FgcmBuildStarsTask, and the fourth is FgcmOutputProductsTask.

FgcmFitCycleTask is available as a command-line task, fgcmFitCycle.py.

Processing summary

FgcmFitCycleTask reads in the star observations and the look-up table, performs an atmosphere and instrument fit, and outputs fit parameters as well as a comprehensive list of QA plots. If the config option isFinalCycle is set to True then additional datafiles are output that are used by FgcmOutputProductsTask. Please see the cookbook for information on deciding when and how to set the isFinalCycle parameter to True.

Aside from the butler outputs (listed below), the FgcmFitCycleTask will output a new config file in the current working directory with recommended settings for the subsequent fit cycle. Furthermore, there are a wide range of diagnostic QA plots that are output by the task. For details on the contents of these QA plots, please see the cookbook as well as Burke, Rykoff, et al. 2018.

fgcmFitCycle.py command-line interface

Note that no --id arguments are used by fgcmFitCycle.py.

fgcmFitCycle.py REPOPATH [@file [@file2 ...]] [--output OUTPUTREPO | --rerun RERUN] [--configfile configfile] [other options]

Key arguments:

REPOPATH
The input Butler repository’s URI or file path.

Key options:

--configfile:
The config file to use. For all runs except the initial, this should be the config file output from the previous cycle.

See also

See Command-line task argument reference for details and additional options.

Python API summary

from lsst.fgcmcal.fgcmFitCycle import FgcmFitCycleTask
classFgcmFitCycleTask(butler=None, initInputs=None, **kwargs)

Run Single fit cycle for FGCM global calibration...

attributeconfig

Access configuration fields and retargetable subtasks.

methodrun(**kwargs)

Run task algorithm on in-memory data...

methodrunDataRef(butler)

Run a single fit cycle for FGCM...

See also

See the FgcmFitCycleTask API reference for complete details.

Butler datasets

When run as the fgcmFitCycle.py command-line task, or directly through the runDataRef method, FgcmFitCycleTask obtains datasets from the input Butler data repository and persists outputs to the output Butler data repository. Note that configurations for FgcmFitCycleTask, and its subtasks, affect what datasets are persisted and what their content is.

Input datasets

camera
Camera geometry and detector object
fgcmLookupTable
FGCM look-up table produced by FgcmMakeLutTask
fgcmVisitCatalog
Catalog (lsst.afw.table) of visit metadata produced by FgcmBuildStarsTask
fgcmStarObservations
Catalog of star observations produced by FgcmBuildStarsTask
fgcmStarIds
Catalog of unique star ids, positions, and number of observations produced by FgcmBuildStarsTask
fgcmStarIndices
Catalog of indices linking unique star ids to star observations produced by FgcmBuildStarsTask
fgcmReferenceStars
Catalog of reference stars matched to unique star ids produced by FgcmBuildStarsTask
fgcmFitParameters
Catalog of fit parameters from previous fit cycle (if available)

Output datasets

fgcmFitParameters
Catalog of fit parameters. Not output if run as part of FgcmCalibrateTractTask
fgcmFlaggedStars
Catalog of flagged star ids, either bad or reserved from fit. Not output if run as part of FgcmCalibrateTractTask
fgcmZeropoints
Catalog of zero-point information. Only output if isFinalCycle is True. Not output if run as part of FgcmCalibrateTractTask
fgcmAtmosphereParameters
Catalog of atmosphere parameters. Only output if isFinalCycle is True. Not output if run as part of FgcmCalibrateTractTask
fgcmStandardStars
Catalog of standard stars from fit. Only output if isFinalCycle is True. Not output if run as part of FgcmCalibrateTractTask

Retargetable subtasks

No subtasks.

Configuration fields

aperCorrFitNBins

Default
10
Field type
int Field
Number of aperture bins used in aperture correction fit. When set to 0no fit will be performed, and the config.aperCorrInputSlopes will be used if available.

aperCorrInputSlopeDict

Default
{}
Field type
DictField
Key type
str
Value type
float
Per-band specification of aperture correction input slope parameters. These are used on the first fit iteration, and aperture correction parameters will be updated from the data if config.aperCorrFitNBins > 0. It is recommended to set this when there is insufficient data to fit the parameters (e.g. tract mode).

aperCorrInputSlopes

Default
[]
Field type
float ListField (optional)
Aperture correction input slope parameters. These are used on the first fit iteration, and aperture correction parameters will be updated from the data if config.aperCorrFitNBins > 0. It is recommended to set thiswhen there is insufficient data to fit the parameters (e.g. tract mode). If set, must be same length as config.bands, and matched band-by-band. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use aperCorrInputSlopeDict instead.

applyRefStarColorCuts

Default
True
Field type
bool Field
Apply color cuts to reference stars?

approxThroughput

Default
(1.0,)
Field type
float ListField (optional)
Approximate overall throughput at start of calibration observations. May be 1 element (same for all bands) or the same length as config.bands. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use approxThroughputDict instead.

approxThroughputDict

Default
{}
Field type
DictField
Key type
str
Value type
float
Per-band specification of the approximate overall throughput at the start of calibration observations. Must have one entry per band. Typically should be 1.0.

autoHighCutNSig

Default
4.0
Field type
float Field
Number of sigma for automatic computation of (high) outlier cut. Cut is based on exposure gray width (per band), unless useRepeatabilityForExpGrayCuts is set, in which case the star repeatability is used (also per band).

autoPhotometricCutNSig

Default
3.0
Field type
float Field
Number of sigma for automatic computation of (low) photometric cut. Cut is based on exposure gray width (per band), unless useRepeatabilityForExpGrayCuts is set, in which case the star repeatability is used (also per band).

bands

Default
[]
Field type
str ListField
Bands to run calibration

brightObsGrayMax

Default
0.15
Field type
float Field
Maximum gray extinction to be considered bright observation

ccdGrayFocalPlaneChebyshevOrder

Default
3
Field type
int Field
Order of the 2D chebyshev polynomials for focal plane fit.

ccdGrayFocalPlaneDict

Default
{}
Field type
DictField
Key type
str
Value type
bool
Per-band specification on whether to compute focal-plane residual (‘ccd gray’) corrections.

ccdGrayFocalPlaneFitMinCcd

Default
1
Field type
int Field
Minimum number of ‘good’ CCDs required to perform focal-plane gray corrections. If there are fewer good CCDs then the gray correction is computed per-ccd.

ccdGrayMaxStarErr

Default
0.1
Field type
float Field
Maximum error on a star observation to use in ccd gray (achromatic residual) computation

ccdGraySubCcd

Default
False
Field type
bool Field (optional)
Compute CCD gray terms on sub-ccd scale Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use ccdGraySubCcdDict instead.

ccdGraySubCcdChebyshevOrder

Default
1
Field type
int Field
Order of the 2D chebyshev polynomials for sub-ccd gray fit.

ccdGraySubCcdDict

Default
{}
Field type
DictField
Key type
str
Value type
bool
Per-band specification on whether to compute achromatic per-ccd residual (‘ccd gray’) on a sub-ccd scale.

ccdGraySubCcdTriangular

Default
True
Field type
bool Field
Should the sub-ccd gray chebyshev matrix be triangular to suppress high-order cross terms?

coatingMjds

Default
(0.0,)
Field type
float ListField
Mirror coating dates in MJD

colorSplitBands

Default
('g', 'i')
Field type
str ListField
Required length
2
Band names to use to split stars by color. Must have 2 entries.

colorSplitIndices

Default
None
Field type
int ListField (optional)
Band indices to use to split stars by color Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use colorSplitBands instead.

connections

Data type
lsst.pipe.base.config.Connections
Field type
ConfigField
Configurations describing the connections of the PipelineTask to datatypes

cycleNumber

Default
None
Field type
int Field
FGCM fit cycle number. This is automatically incremented after each run and stage of outlier rejection. See cookbook for details.

deltaMagBkgOffsetPercentile

Default
0.25
Field type
float Field
Percentile brightest stars on a visit/ccd to use to compute net offset from local background subtraction.

deltaMagBkgPerCcd

Default
False
Field type
bool Field
Compute net offset from local background subtraction per-ccd? Otherwise, use computation per visit.

doReferenceCalibration

Default
True
Field type
bool Field
Use reference catalog as additional constraint on calibration

epochMjds

Default
(0.0,)
Field type
float ListField
Epoch boundaries in MJD

expGrayErrRecoverCut

Default
0.05
Field type
float Field
Maximum exposure gray error to be able to recover bad ccds via interpolation. Visits with more gray variance will only get CCD zeropoints if there are sufficient star observations (minStarPerCcd) on that CCD.

expGrayHighCut

Default
(0.0,)
Field type
float ListField (optional)
Maximum (positive) exposure gray for a visit to be considered photometric. Must be same length as config.bands, and matched band-by-band. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use expGrayHighCutDict instead.

expGrayHighCutDict

Default
{}
Field type
DictField
Key type
str
Value type
float
Per-band specification on maximum (positive) achromatic exposure residual (‘gray term’) for a visit to be considered photometric. Must have one entry per band. Broad-band filters should be 0.2.

expGrayInitialCut

Default
-0.25
Field type
float Field
Maximum exposure/visit gray value for initial selection of possible photometric observations.

expGrayPhotometricCut

Default
(0.0,)
Field type
float ListField (optional)
Maximum (negative) exposure gray for a visit to be considered photometric. Must be same length as config.bands, and matched band-by-band. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use expGrayPhotometricCutDict instead.

expGrayPhotometricCutDict

Default
{}
Field type
DictField
Key type
str
Value type
float
Per-band specification on maximum (negative) achromatic exposure residual (‘gray term’) for a visit to be considered photometric. Must have one entry per band. Broad-band filters should be -0.05.

expGrayRecoverCut

Default
-1.0
Field type
float Field
Maximum (negative) exposure gray to be able to recover bad ccds via interpolation. Visits with more gray extinction will only get CCD zeropoints if there are sufficient star observations (minStarPerCcd) on that CCD.

expVarGrayPhotometricCut

Default
0.0005
Field type
float Field (optional)
Maximum exposure variance to be considered possibly photometric Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use expVarGrayPhotometricCutDict instead.

expVarGrayPhotometricCutDict

Default
{}
Field type
DictField
Key type
str
Value type
float
Per-band specification on maximum exposure variance to be considered possibly photometric. Must have one entry per band. Broad-band filters should be 0.0005.

filterMap

Default
{}
Field type
DictField
Key type
str
Value type
str
Mapping from ‘filterName’ to band.

fitBands

Default
[]
Field type
str ListField
Bands to use in atmospheric fit. The bands not listed here will have the atmosphere constrained from the ‘fitBands’ on the same night. Must be a subset of config.bands

fitFlag

Default
(0,)
Field type
int ListField (optional)
Flag for which bands are directly constrained in the FGCM fit. Bands set to 0 will have the atmosphere constrained from observations in other bands on the same night. Must be same length as config.bands, and matched band-by-band. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use fitBands instead.

fitMirrorChromaticity

Default
False
Field type
bool Field
Fit (intraband) mirror chromatic term?

focalPlaneSigmaClip

Default
4.0
Field type
float Field
Number of sigma to clip outliers per focal-plane.

freezeStdAtmosphere

Default
False
Field type
bool Field
Freeze atmosphere parameters to standard (for testing)

instrumentParsPerBand

Default
False
Field type
bool Field
Model instrumental parameters per band? Otherwise, instrumental parameters (QE changes with time) are shared among all bands.

instrumentSlopeMinDeltaT

Default
20.0
Field type
float Field
Minimum time change (in days) between observations to use in constraining instrument slope.

isFinalCycle

Default
False
Field type
bool Field
Is this the final cycle of the fitting? Will automatically compute final selection of stars and photometric exposures, and will output zeropoints and standard stars for use in fgcmOutputProducts

latitude

Default
None
Field type
float Field
Observatory latitude

mapLongitudeRef

Default
0.0
Field type
float Field
Reference longitude for plotting maps

mapNSide

Default
256
Field type
int Field
Healpix nside for plotting maps

maxCcdGrayErr

Default
0.05
Field type
float Field
Maximum error on CCD gray offset to be considered photometric

maxIterBeforeFinalCycle

Default
50
Field type
int Field
Maximum fit iterations, prior to final cycle. The number of iterations will always be 0 in the final cycle for cleanup and final selection.

minCcdPerExp

Default
5
Field type
int Field
Minimum number of good CCDs per exposure/visit to be used in calibration fit. Visits with fewer good CCDs will have CCD zeropoints estimated where possible.

minExpPerNight

Default
10
Field type
int Field
Minimum number of good exposures/visits to consider a partly photometric night

minObsPerBand

Default
2
Field type
int Field
Minimum good observations per band

minStarPerCcd

Default
5
Field type
int Field
Minimum number of good stars per CCD to be used in calibration fit. CCDs with fewer stars will have their calibration estimated from other CCDs in the same visit, with zeropoint error increased accordingly.

minStarPerExp

Default
600
Field type
int Field
Minimum number of good stars per exposure/visit to be used in calibration fit. Visits with fewer good stars will have CCD zeropoints estimated where possible.

modelMagErrors

Default
True
Field type
bool Field
Should FGCM model the magnitude errors from sky/fwhm? (False means trust inputs)

nCore

Default
4
Field type
int Field
Number of cores to use

nExpPerRun

Default
1000
Field type
int Field
Number of exposures to run in each chunk

nStarPerRun

Default
200000
Field type
int Field
Number of stars to run in each chunk

outfileBase

Default
None
Field type
str Field
Filename start for plot output files

outputStandardsBeforeFinalCycle

Default
False
Field type
bool Field
Output standard stars prior to final cycle? Used in debugging.

outputZeropointsBeforeFinalCycle

Default
False
Field type
bool Field
Output standard stars prior to final cycle? Used in debugging.

precomputeSuperStarInitialCycle

Default
False
Field type
bool Field
Precompute superstar flat for initial cycle

quietMode

Default
False
Field type
bool Field
Be less verbose with logging.

randomSeed

Default
None
Field type
int Field (optional)
Random seed for fgcm for consistency in tests.

refStarOutlierNSig

Default
4.0
Field type
float Field
Number of sigma compared to average mag for reference star to be considered an outlier. Computed per-band, and if it is an outlier in any band it is rejected from fits.

refStarSnMin

Default
50.0
Field type
float Field
Reference star signal-to-noise minimum to use in calibration. Set to <=0 for no cut.

requiredBands

Default
[]
Field type
str ListField
Bands that are required for a star to be considered a calibration star. Must be a subset of config.bands

requiredFlag

Default
(0,)
Field type
int ListField (optional)
Flag for which bands are required for a star to be considered a calibration star in the FGCM fit. Typically this should be the same as fitFlag. Must be same length as config.bands, and matched band-by-band. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use requiredBands instead.

reserveFraction

Default
0.1
Field type
float Field
Fraction of stars to reserve for testing

saveMetadata

Default
True
Field type
bool Field
Flag to enable/disable metadata saving for a task, enabled by default.

sedFudgeFactors

Default
(0,)
Field type
float ListField (optional)
Fudge factors for computing linear SED from colors. Must be same length as config.bands, and matched band-by-band. Deprecated: This field has been deprecated and will be removed after v20. Please use sedSlopeTermMap and sedSlopeMap.

sedboundaryterms

Data type
lsst.fgcmcal.sedterms.SedboundarytermDict
Field type
ConfigField
Mapping from bands to SED boundary term names used is sedterms.

sedterms

Data type
lsst.fgcmcal.sedterms.SedtermDict
Field type
ConfigField
Mapping from terms to bands for fgcm linear SED approximations.

sigFgcmMaxEGray

Default
(0.05,)
Field type
float ListField (optional)
Maximum (absolute) gray value for observation in sigma_FGCM. May be 1 element (same for all bands) or the same length as config.bands. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use sigFgcmMaxEGrayDict instead.

sigFgcmMaxEGrayDict

Default
{}
Field type
DictField
Key type
str
Value type
float
Per-band specification for maximum (absolute) achromatic residual (gray value) for observations in sigma_fgcm (raw repeatability). Broad-band filters should be 0.05.

sigFgcmMaxErr

Default
0.01
Field type
float Field
Maximum mag error for fitting sigma_FGCM

sigma0Phot

Default
0.003
Field type
float Field
Systematic error floor for all zeropoints

sigmaCalFitPercentile

Default
(0.05, 0.15)
Field type
float ListField
Magnitude percentile range to fit systematic error floor

sigmaCalPlotPercentile

Default
(0.05, 0.95)
Field type
float ListField
Magnitude percentile range to plot systematic error floor

sigmaCalRange

Default
(0.001, 0.003)
Field type
float ListField
Allowed range for systematic error floor estimation

starColorCuts

Default
('NO_DATA',)
Field type
str ListField
Encoded star-color cuts (to be cleaned up)

superStarSigmaClip

Default
5.0
Field type
float Field
Number of sigma to clip outliers when selecting for superstar flats

superStarSubCcd

Default
True
Field type
bool Field (optional)
Compute superstar flat on sub-ccd scale Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use superStarSubCcdDict instead.

superStarSubCcdChebyshevOrder

Default
1
Field type
int Field
Order of the 2D chebyshev polynomials for sub-ccd superstar fit. Global default is first-order polynomials, and should be overridden on a camera-by-camera basis depending on the ISR.

superStarSubCcdDict

Default
{}
Field type
DictField
Key type
str
Value type
bool
Per-band specification on whether to compute superstar flat on sub-ccd scale. Must have one entry per band.

superStarSubCcdTriangular

Default
False
Field type
bool Field
Should the sub-ccd superstar chebyshev matrix be triangular to suppress high-order cross terms?

useQuadraticPwv

Default
False
Field type
bool Field
Model PWV with a quadratic term for variation through the night?

useRepeatabilityForExpGrayCuts

Default
(False,)
Field type
bool ListField (optional)
Use star repeatability (instead of exposures) for computing photometric cuts? Recommended for tract mode or bands with few exposures. May be 1 element (same for all bands) or the same length as config.bands. Deprecated: This field is no longer used, and has been deprecated by DM-23699. It will be removed after v20. Use useRepeatabilityForExpGrayCutsDict instead.

useRepeatabilityForExpGrayCutsDict

Default
{}
Field type
DictField
Key type
str
Value type
bool
Per-band specification on whether to use star repeatability (instead of exposures) for computing photometric cuts. Recommended for tract mode or bands with few visits.

utBoundary

Default
None
Field type
float Field
Boundary (in UTC) from day-to-day

washMjds

Default
(0.0,)
Field type
float ListField
Mirror wash MJDs

Examples

See the cookbook for worked examples.