CameraMapper

class lsst.obs.base.CameraMapper(policy, repositoryDir, root=None, registry=None, calibRoot=None, calibRegistry=None, provided=None, parentRegistry=None, repositoryCfg=None)

Bases: lsst.daf.persistence.mapper.Mapper

CameraMapper is a base class for mappers that handle images from a camera and products derived from them. This provides an abstraction layer between the data on disk and the code.

Public methods: keys, queryMetadata, getDatasetTypes, map, canStandardize, standardize

Mappers for specific data sources (e.g., CFHT Megacam, LSST simulations, etc.) should inherit this class.

The CameraMapper manages datasets within a “root” directory. Note that writing to a dataset present in the input root will hide the existing dataset but not overwrite it. See #2160 for design discussion.

A camera is assumed to consist of one or more rafts, each composed of multiple CCDs. Each CCD is in turn composed of one or more amplifiers (amps). A camera is also assumed to have a camera geometry description (CameraGeom object) as a policy file, a filter description (Filter class static configuration) as another policy file, and an optional defects description directory.

Information from the camera geometry and defects are inserted into all Exposure objects returned.

The mapper uses one or two registries to retrieve metadata about the images. The first is a registry of all raw exposures. This must contain the time of the observation. One or more tables (or the equivalent) within the registry are used to look up data identifier components that are not specified by the user (e.g. filter) and to return results for metadata queries. The second is an optional registry of all calibration data. This should contain validity start and end entries for each calibration dataset in the same timescale as the observation time.

Subclasses will typically set MakeRawVisitInfoClass:

MakeRawVisitInfoClass: a class variable that points to a subclass of MakeRawVisitInfo, a functor that creates an lsst.afw.image.VisitInfo from the FITS metadata of a raw image.

Subclasses must provide the following methods:

_extractDetectorName(self, dataId): returns the detector name for a CCD (e.g., “CFHT 21”, “R:1,2 S:3,4”) as used in the AFW CameraGeom class given a dataset identifier referring to that CCD or a subcomponent of it.

_computeCcdExposureId(self, dataId): see below

_computeCoaddExposureId(self, dataId, singleFilter): see below

Subclasses may also need to override the following methods:

_transformId(self, dataId): transformation of a data identifier from colloquial usage (e.g., “ccdname”) to proper/actual usage (e.g., “ccd”), including making suitable for path expansion (e.g. removing commas). The default implementation does nothing. Note that this method should not modify its input parameter.

getShortCcdName(self, ccdName): a static method that returns a shortened name suitable for use as a filename. The default version converts spaces to underscores.

_getCcdKeyVal(self, dataId): return a CCD key and value by which to look up defects in the defects registry. The default value returns (“ccd”, detector name)

_mapActualToPath(self, template, actualId): convert a template path to an actual path, using the actual dataset identifier.

The mapper’s behaviors are largely specified by the policy file. See the MapperDictionary.paf for descriptions of the available items.

The ‘exposures’, ‘calibrations’, and ‘datasets’ subpolicies configure mappings (see Mappings class).

Common default mappings for all subclasses can be specified in the “policy/{images,exposures,calibrations,datasets}.yaml” files. This provides a simple way to add a product to all camera mappers.

Functions to map (provide a path to the data given a dataset identifier dictionary) and standardize (convert data into some standard format or type) may be provided in the subclass as “map_{dataset type}” and “std_{dataset type}”, respectively.

If non-Exposure datasets cannot be retrieved using standard daf_persistence methods alone, a “bypass_{dataset type}” function may be provided in the subclass to return the dataset instead of using the “datasets” subpolicy.

Implementations of map_camera and bypass_camera that should typically be sufficient are provided in this base class.

Notes

TODO:

• Handle defects the same was as all other calibration products, using the calibration registry
• Instead of auto-loading the camera at construction time, load it from the calibration registry
• Rewrite defects as AFW tables so we don’t need pyfits to unpersist them; then remove all mention of pyfits from this package.

Attributes Summary

packageName

Methods Summary

Mapper(cfg)	Instantiate a Mapper from a configuration.
backup(datasetType, dataId)	Rename any existing object with the given type and dataId.
bypass_camera(datasetType, pythonType, ...)	Return the (preloaded) camera object.
bypass_defects(datasetType, pythonType, ...)	Return a defect based on the butler location returned by map_defects
bypass_expIdInfo(datasetType, pythonType, ...)	Hook to retrieve an lsst.obs.base.ExposureIdInfo for an exposure
canStandardize(datasetType)	Return true if this mapper can standardize an object of the given dataset type.
getCameraName()	Return the name of the camera that this CameraMapper is for.
getDatasetTypes()	Return a list of the mappable dataset types.
getDefaultLevel()
getDefaultSubLevel(level)
getImageCompressionSettings(datasetType, dataId)	Stuff image compression settings into a daf.base.PropertySet
getKeys(datasetType, level)	Return a dict of supported keys and their value types for a given dataset type at a given level of the key hierarchy.
getPackageDir()	Return the base directory of this package
getPackageName()	Return the name of the package containing this CameraMapper.
getRegistry()	Get the registry used by this mapper.
getShortCcdName(ccdName)	Convert a CCD name to a form useful as a filename
keys()	Return supported keys.
map(datasetType, dataId[, write])	Map a data id using the mapping method for its dataset type.
map_camera(dataId[, write])	Map a camera dataset.
map_defects(dataId[, write])	Map defects dataset.
map_expIdInfo(dataId[, write])
map_skypolicy(dataId)	Map a sky policy.
queryMetadata(datasetType, format, dataId)	Get possible values for keys given a partial data id.
standardize(datasetType, item, dataId)	Standardize an object using the standardization method for its data set type, if it exists.
std_bfKernel(item, dataId)	Disable standardization for bfKernel
std_raw(item, dataId)	Standardize a raw dataset by converting it to an Exposure instead of an Image
std_skypolicy(item, dataId)	Standardize a sky policy by returning the one we use.
validate(dataId)	Validate a dataId’s contents.

Attributes Documentation

packageName = None

Methods Documentation

Mapper(cfg)

Instantiate a Mapper from a configuration. In come cases the cfg may have already been instantiated into a Mapper, this is allowed and the input var is simply returned.

Parameters:cfg – the cfg for this mapper. It is recommended this be created by calling Mapper.cfg() Returns:a Mapper instance

backup(datasetType, dataId)

Rename any existing object with the given type and dataId.

The CameraMapper implementation saves objects in a sequence of e.g.:

• foo.fits
• foo.fits~1
• foo.fits~2

All of the backups will be placed in the output repo, however, and will not be removed if they are found elsewhere in the _parent chain. This means that the same file will be stored twice if the previous version was found in an input repo.

bypass_camera(datasetType, pythonType, butlerLocation, dataId)

Return the (preloaded) camera object.

bypass_defects(datasetType, pythonType, butlerLocation, dataId)

Return a defect based on the butler location returned by map_defects

Parameters:

butlerLocation : lsst.daf.persistence.ButlerLocation

locationList = path to defects FITS file

dataId : dict

Butler data ID; “ccd” must be set.

Note: the name “bypass_XXX” means the butler makes no attempt to convert the ButlerLocation

into an object, which is what we want for now, since that conversion is a bit tricky.

bypass_expIdInfo(datasetType, pythonType, location, dataId)

Hook to retrieve an lsst.obs.base.ExposureIdInfo for an exposure

canStandardize(datasetType)

Return true if this mapper can standardize an object of the given dataset type.

classmethod getCameraName()

Return the name of the camera that this CameraMapper is for.

getDatasetTypes()

Return a list of the mappable dataset types.

getDefaultLevel()

getDefaultSubLevel(level)

getImageCompressionSettings(datasetType, dataId)

Stuff image compression settings into a daf.base.PropertySet

This goes into the ButlerLocation’s “additionalData”, which gets passed into the boost::persistence framework.

Parameters:

datasetType : str

Type of dataset for which to get the image compression settings.

dataId : dict

Dataset identifier.

Returns:

additionalData : lsst.daf.base.PropertySet

Image compression settings.

getKeys(datasetType, level)

Return a dict of supported keys and their value types for a given dataset type at a given level of the key hierarchy.

Parameters:

datasetType : str

Dataset type or None for all dataset types.

level : str or None

Level or None for all levels or ‘’ for the default level for the camera.

Returns:

dict

Keys are strings usable in a dataset identifier, values are their value types.

classmethod getPackageDir()

Return the base directory of this package

classmethod getPackageName()

Return the name of the package containing this CameraMapper.

getRegistry()

Get the registry used by this mapper.

Returns:

Registry or None

The registry used by this mapper for this mapper’s repository.

static getShortCcdName(ccdName)

Convert a CCD name to a form useful as a filename

The default implementation converts spaces to underscores.

keys()

Return supported keys.

Returns:

iterable

List of keys usable in a dataset identifier

map(datasetType, dataId, write=False)

Map a data id using the mapping method for its dataset type.

Parameters:

datasetType : string

The datasetType to map

dataId : DataId instance

The dataId to use when mapping

write : bool, optional

Indicates if the map is being performed for a read operation (False) or a write operation (True)

Returns:

ButlerLocation or a list of ButlerLocation

The location(s) found for the map operation. If write is True, a list is returned. If write is False a single ButlerLocation is returned.

Raises:

NoResults

If no locaiton was found for this map operation, the derived mapper class may raise a lsst.daf.persistence.NoResults exception. Butler catches this and will look in the next Repository if there is one.

map_camera(dataId, write=False)

Map a camera dataset.

map_defects(dataId, write=False)

Map defects dataset.

Returns:

lsst.daf.butler.ButlerLocation

Minimal ButlerLocation containing just the locationList field (just enough information that bypass_defects can use it).

map_expIdInfo(dataId, write=False)

map_skypolicy(dataId)

Map a sky policy.

queryMetadata(datasetType, format, dataId)

Get possible values for keys given a partial data id.

Parameters:

• datasetType – see documentation about the use of datasetType
• key – this is used as the ‘level’ parameter
• format –
• dataId – see documentation about the use of dataId

Returns:

standardize(datasetType, item, dataId)

Standardize an object using the standardization method for its data set type, if it exists.

std_bfKernel(item, dataId)

Disable standardization for bfKernel

bfKernel is a calibration product that is numpy array, unlike other calibration products that are all images; all calibration images are sent through _standardizeExposure due to CalibrationMapping, but we don’t want that to happen to bfKernel

std_raw(item, dataId)

Standardize a raw dataset by converting it to an Exposure instead of an Image

std_skypolicy(item, dataId)

Standardize a sky policy by returning the one we use.

validate(dataId)

Validate a dataId’s contents.

If the dataId is valid, return it. If an invalid component can be transformed into a valid one, copy the dataId, fix the component, and return the copy. Otherwise, raise an exception.