DarkEnergyCameraCPCalibFormatter

class lsst.obs.decam.DarkEnergyCameraCPCalibFormatter(*args, **kwargs)

Bases: DarkEnergyCameraRawFormatter

DECam Community Pipeline calibrations (bias, dark, flat, fringe) are multi-extension FITS files with detector=index+1.

Attributes Summary

can_read_from_local_file	Declare whether read_from_file is available to this formatter.
can_read_from_stream	Declare whether read_from_stream is available to this formatter.
can_read_from_uri	Declare whether read_from_uri is available to this formatter.
checked_parameters	The parameters passed by the butler user, after checking them against the storage class and transforming None into an empty dict (dict).
data_id	Return Data ID associated with this formatter (DataCoordinate).
dataset_ref	Return Dataset Ref associated with this formatter (DatasetRef).
default_extension	Default extension to use when writing a file.
file_descriptor	File descriptor associated with this formatter (FileDescriptor).
filterDefinitions
metadata	The metadata read from this file.
observationInfo	The ObservationInfo extracted from this file's metadata (ObservationInfo, read-only).
reader	The reader object that backs this formatter's read operations.
supported_extensions	Set of all extensions supported by this formatter.
supported_write_parameters	Parameters understood by this formatter that can be used to control how a dataset is serialized.
unsupported_parameters	Support all parameters.
wcsFlipX	Control whether the WCS is flipped in the X-direction (bool)
write_parameters	Parameters to use when writing out datasets.
write_recipes	Detailed write Recipes indexed by recipe name.

Methods Summary

attachComponentsFromMetadata(exposure)	Attach all lsst.afw.image.Exposure components derived from metadata (including the stripped metadata itself).
can_accept(in_memory_dataset)	Indicate whether this formatter can accept the specified storage class directly.
fromMetadata(metadata[, obsInfo, ...])	Construct a possibly-limited formatter from known metadata.
getDetector(id)	Return the detector that acquired this raw exposure.
get_write_extension()	Extension to use when writing a file.
isOnSky()	Boolean to determine if the exposure is thought to be on the sky.
makeFilterLabel()	Construct a FilterLabel from metadata.
makeRawSkyWcsFromBoresight(boresight, ...)	Class method to make a raw sky WCS from boresight and detector.
makeVisitInfo()	Construct a VisitInfo from metadata.
makeWcs(visitInfo, detector)	Create a SkyWcs from information about the exposure.
make_updated_location(location)	Return a new Location updated with this formatter's extension.
name()	Return the fully qualified name of the formatter.
predict_path()	Return the path that would be returned by write.
read([component, expected_size, cache_manager])	Read a Dataset.
readComponent(component)	Read a component dataset.
readFull()	Read the full dataset (while still accounting for parameters).
readImage()	Read just the image component of the Exposure.
readMetadata()	Read all header metadata directly into a PropertyList.
read_directly_from_possibly_cached_uri([...])	Read from arbitrary URI, checking for possible presence in local
read_from_local_file(path[, component, ...])	Read a dataset from a URI guaranteed to refer to the local file system.
read_from_possibly_cached_local_file([...])	Read a dataset ensuring that a local file is used, checking the cache for it.
read_from_possibly_cached_stream([...])	Read from a stream, checking for possible presence in local cache.
read_from_stream(stream[, component, ...])	Read from an open file descriptor.
read_from_uri(uri[, component, expected_size])	Read a dataset from a URI that can be local or remote.
segregate_parameters([parameters])	Segregate the supplied parameters.
stripMetadata()	Remove metadata entries that are parsed into components.
to_bytes(in_memory_dataset)	Serialize the in-memory dataset to bytes.
validate_extension(location)	Check the extension of the provided location for compatibility.
validate_write_recipes(recipes)	Validate supplied recipes for this formatter.
write(in_memory_dataset[, cache_manager])	Write a Dataset.
write_direct(in_memory_dataset, uri[, ...])	Serialize and write directly to final location.
write_local_file(in_memory_dataset, uri)	Serialize the in-memory dataset to a local file.
write_locally_then_move(in_memory_dataset, uri)	Write file to file system and then move to final location.

Attributes Documentation

can_read_from_local_file: ClassVar[bool] = True

Declare whether read_from_file is available to this formatter.

can_read_from_stream: ClassVar[bool] = False

Declare whether read_from_stream is available to this formatter.

can_read_from_uri: ClassVar[bool] = False

Declare whether read_from_uri is available to this formatter.

checked_parameters

data_id

Return Data ID associated with this formatter (DataCoordinate).

dataset_ref

Return Dataset Ref associated with this formatter (DatasetRef).

default_extension: ClassVar[str | None] = '.fits'

Default extension to use when writing a file.

Can be None if the extension is determined dynamically. Use the get_write_extension method to get the actual extension to use.

file_descriptor

File descriptor associated with this formatter (FileDescriptor).

filterDefinitions = <lsst.obs.base.filters.FilterDefinitionCollection object>

metadata

The metadata read from this file. It will be stripped as components are extracted from it (lsst.daf.base.PropertyList).

observationInfo

The ObservationInfo extracted from this file’s metadata (ObservationInfo, read-only).

reader

The reader object that backs this formatter’s read operations.

This is computed on first use and then cached. It should never be accessed when writing. Currently assumes a local file.

supported_extensions: ClassVar[Set[str]] = frozenset({'.fit', '.fits', '.fits.fz', '.fits.gz', '.fz'})

Set of all extensions supported by this formatter.

Any extension assigned to the default_extension property will be automatically included in the list of supported extensions.

supported_write_parameters: ClassVar[Set[str] | None] = None

Parameters understood by this formatter that can be used to control how a dataset is serialized. None indicates that no parameters are supported.

unsupported_parameters: ClassVar[Set[str]] = frozenset({})

Support all parameters.

wcsFlipX = True

Control whether the WCS is flipped in the X-direction (bool)

write_parameters

Parameters to use when writing out datasets.

write_recipes

Detailed write Recipes indexed by recipe name.

Methods Documentation

attachComponentsFromMetadata(exposure)

Attach all lsst.afw.image.Exposure components derived from metadata (including the stripped metadata itself).

Parameters:

exposurelsst.afw.image.Exposure

Exposure to attach components to (modified in place). Must already have a detector attached.

can_accept(in_memory_dataset: Any) → bool

Indicate whether this formatter can accept the specified storage class directly.

Parameters:

in_memory_datasetobject

The dataset that is to be written.

Returns:

acceptsbool

If True the formatter can write data of this type without requiring datastore to convert it. If False the datastore will attempt to convert before writing.

Notes

The base class always returns False even if the given type is an instance of the storage class type. This will result in a storage class conversion no-op but also allows mocks with mocked storage classes to work properly.

classmethod fromMetadata(metadata, obsInfo=None, storageClass=None, location=None)

Construct a possibly-limited formatter from known metadata.

Parameters:

metadatalsst.daf.base.PropertyList

Raw header metadata, with any fixes (see astro_metadata_translator.fix_header) applied but nothing stripped.

obsInfoastro_metadata_translator.ObservationInfo, optional

Structured information already extracted from metadata. If not provided, will be read from metadata on first use.

storageClasslsst.daf.butler.StorageClass, optional

StorageClass for this file. If not provided, the formatter will only support makeWcs, makeVisitInfo, makeFilter, and other operations that operate purely on metadata and not the actual file.

locationlsst.daf.butler.Location, optional.

Location of the file. If not provided, the formatter will only support makeWcs, makeVisitInfo, makeFilter, and other operations that operate purely on metadata and not the actual file.

Returns:

formatterFitsRawFormatterBase

An instance of cls.

getDetector(id)

Return the detector that acquired this raw exposure.

Parameters:

idint

The identifying number of the detector to get.

Returns:

detectorDetector

The detector associated with that id.

get_write_extension() → str

Extension to use when writing a file.

isOnSky()

Boolean to determine if the exposure is thought to be on the sky.

Returns:

onSkybool

Returns True if the observation looks like it was taken on the sky. Returns False if this observation looks like a calibration observation.

Notes

If there is tracking RA/Dec information associated with the observation it is assumed that the observation is on sky. Currently the observation type is not checked.

makeFilterLabel()

Construct a FilterLabel from metadata.

Returns:

filterFilterLabel

Object that identifies the filter for this image.

classmethod makeRawSkyWcsFromBoresight(boresight, orientation, detector)

Class method to make a raw sky WCS from boresight and detector.

Parameters:

boresightlsst.geom.SpherePoint

The ICRS boresight RA/Dec

orientationlsst.geom.Angle

The rotation angle of the focal plane on the sky.

detectorlsst.afw.cameraGeom.Detector

Where to get the camera geometry from.

Returns:

skyWcsSkyWcs

Reversible mapping from pixel coordinates to sky coordinates.

makeVisitInfo()

Construct a VisitInfo from metadata.

Returns:

visitInfoVisitInfo

Structured metadata about the observation.

makeWcs(visitInfo, detector)

Create a SkyWcs from information about the exposure.

If VisitInfo is not None, use it and the detector to create a SkyWcs, otherwise return the metadata-based SkyWcs (always created, so that the relevant metadata keywords are stripped).

Parameters:

visitInfoVisitInfo

The information about the telescope boresight and camera orientation angle for this exposure.

detectorDetector

The detector used to acquire this exposure.

Returns:

skyWcsSkyWcs

Reversible mapping from pixel coordinates to sky coordinates.

Raises:

InitialSkyWcsError

Raised if there is an error generating the SkyWcs, chained from the lower-level exception if available.

make_updated_location(location: Location) → Location

Return a new Location updated with this formatter’s extension.

Parameters:

locationLocation

The location to update.

Returns:

updatedLocation

A new Location with a new file extension applied.

classmethod name() → str

Return the fully qualified name of the formatter.

Returns:

namestr

Fully-qualified name of formatter class.

predict_path() → str

Return the path that would be returned by write.

Does not write any data file.

Uses the FileDescriptor associated with the instance.

Returns:

pathstr

Path within datastore that would be associated with the location stored in this Formatter.

read(component: str | None = None, expected_size: int = -1, cache_manager: AbstractDatastoreCacheManager | None = None) → Any

Read a Dataset.

Parameters:

componentstr, optional

Component to read from the file. Only used if the StorageClass for reading differed from the StorageClass used to write the file.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be used for verification or to decide whether to do a direct read or a file download. -1 indicates the file size is not known.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to cache a remote file locally or read a cached file that is already local.

Returns:

in_memory_datasetobject

The requested Dataset.

Raises:

FormatterNotImplementedError

Raised if no implementations were found that could read this resource.

Notes

This method should not be subclassed. Instead formatter subclasses should re-implement the specific read_from_* methods as appropriate. Each of these methods has a corresponding class property that must be True for the method to be called.

The priority for reading is:

• read_from_uri

• read_from_stream

• read_from_local_file

• read_from_uri (but with a local file)

Any of these methods can return NotImplemented if there is a desire to skip to the next one in the list. If a dataset is being requested with no component, no parameters, and it should also be added to the local cache, the first two calls will be skipped (unless read_from_stream is the only implemented read method) such that a local file will be used.

A Formatter can also read a file from within a Zip file if the URI associated with the FileDescriptor corresponds to a file with a zip extension and a URI fragment of the form zip-path={path_in_zip}. When reading a file from within a Zip file the priority for reading is:

• read_from_stream

• read_from_local_file

• read_from_uri

There are multiple cases that must be handled for reading:

For a single file:

• No component requested, read the whole file.

• Component requested, optionally read the component efficiently, else read the whole file and extract the component.

• Derived component requested, read whole file or read relevant component and derive.

Disassembled Composite:

• The file to read here is the component itself. Formatter only knows about this one component file. Should be no component specified in the read call but the FileDescriptor will know which component this is.

• A derived component. The file to read is a component but not the specified component. The caching needs the component from which it’s derived.

readComponent(component)

Read a component dataset.

Parameters:

componentstr, optional

Component to read from the file.

Returns:

objcomponent-dependent

In-memory component object.

Raises:

KeyError

Raised if the requested component cannot be handled.

readFull()

Read the full dataset (while still accounting for parameters).

Returns:

objcomponent-dependent

In-memory component object.

readImage()

Read just the image component of the Exposure.

Returns:

imageImage

In-memory image component.

readMetadata()

Read all header metadata directly into a PropertyList.

Returns:

metadataPropertyList

Header metadata.

read_directly_from_possibly_cached_uri(component: str | None = None, expected_size: int = -1, *, cache_manager: AbstractDatastoreCacheManager | None = None) → Any

Read from arbitrary URI, checking for possible presence in local

cache.

Parameters:

componentstr, optional

Component to read from the file. Only used if the StorageClass for reading differed from the StorageClass used to write the file.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be -1 indicates the file size is not known.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to check if there is a copy of the file in the local cache.

Returns:

in_memory_datasetobject or NotImplemented

The requested Dataset or an indication that the read mode was not implemented.

Notes

This method will first check the datastore cache in case the file is present locally. This method will not cache a remote dataset and will only do a size check for local files to avoid unnecessary round trips to a remote server.

The URI will be read by calling read_from_uri.

read_from_local_file(path: str, component: str | None = None, expected_size: int = -1) → Any

Read a dataset from a URI guaranteed to refer to the local file system.

Parameters:

pathstr

Path to a local file that should be read.

componentstr or None, optional

The component to be read from the dataset.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be -1 indicates the file size is not known.

Returns:

in_memory_datasetobject or NotImplemented

The Python object read from the resource or NotImplemented.

Raises:

FormatterNotImplementedError

Raised if there is no implementation written to read data from a local file.

Notes

This method will only be called if the class property can_read_from_local_file is True and other options were not used.

read_from_possibly_cached_local_file(component: str | None = None, expected_size: int = -1, *, cache_manager: AbstractDatastoreCacheManager | None = None) → Any

Read a dataset ensuring that a local file is used, checking the cache for it.

Parameters:

componentstr, optional

Component to read from the file. Only used if the StorageClass for reading differed from the StorageClass used to write the file.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be used for verification or to decide whether to do a direct read or a file download. -1 indicates the file size is not known.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to cache a remote file locally or read a cached file that is already local.

Returns:

in_memory_datasetobject or NotImplemented

The requested Dataset or an indication that the read mode was not implemented.

Notes

The file will be downloaded and cached if it is a remote resource. The file contents will be read using read_from_local_file or read_from_uri, with preference given to the former.

read_from_possibly_cached_stream(component: str | None = None, expected_size: int = -1, *, cache_manager: AbstractDatastoreCacheManager | None = None) → Any

Read from a stream, checking for possible presence in local cache.

Parameters:

componentstr, optional

Component to read from the file. Only used if the StorageClass for reading differed from the StorageClass used to write the file.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be used for verification or to decide whether to do a direct read or a file download. -1 indicates the file size is not known.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to check if there is a copy of the file in the local cache.

Returns:

in_memory_datasetobject or NotImplemented

The requested Dataset or an indication that the read mode was not implemented.

Notes

Calls read_from_stream but will first check the datastore cache in case the file is present locally. This method will not download a file to the local cache.

read_from_stream(stream: BinaryIO | ResourceHandleProtocol, component: str | None = None, expected_size: int = -1) → Any

Read from an open file descriptor.

Parameters:

streamlsst.resources.ResourceHandleProtocol or typing.BinaryIO

File stream to use to read the dataset.

componentstr or None, optional

The component to be read from the dataset.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be -1 indicates the file size is not known.

Returns:

in_memory_datasetobject or NotImplemented

The Python object read from the stream or NotImplemented.

Notes

Only called if the class property can_read_from_stream is True.

read_from_uri(uri: ResourcePath, component: str | None = None, expected_size: int = -1) → Any

Read a dataset from a URI that can be local or remote.

Parameters:

urilsst.resources.ResourcePath

URI to use to read the dataset. This URI can be local or remote and can refer to the actual resource or to a locally cached file.

componentstr or None, optional

The component to be read from the dataset.

expected_sizeint, optional

If known, the expected size of the resource to read. This can be -1 indicates the file size is not known.

Returns:

in_memory_datasetobject or NotImplemented

The Python object read from the resource or NotImplemented.

Raises:

FormatterNotImplementedError

Raised if there is no support for direct reads from a, possibly, remote URI.

Notes

This method is only called if the class property can_read_from_uri is set to True.

It is possible that a cached local file will be given to this method even if it was originally a remote URI. This can happen if the original write resulted in the file being added to the local cache.

If the full file is being read this file will not be added to the local cache. Consider returning NotImplemented in this situation, for example if there are no parameters or component specified, and allowing the system to fall back to calling read_from_local_file (which will populate the cache if configured to do so).

segregate_parameters(parameters: dict[str, Any] | None = None) → tuple[dict, dict]

Segregate the supplied parameters.

This splits the parameters into those understood by the formatter and those not understood by the formatter.

Any unsupported parameters are assumed to be usable by associated assemblers.

Parameters:

parametersdict, optional

Parameters with values that have been supplied by the caller and which might be relevant for the formatter. If None parameters will be read from the registered FileDescriptor.

Returns:

supporteddict

Those parameters supported by this formatter.

unsupporteddict

Those parameters not supported by this formatter.

stripMetadata()

Remove metadata entries that are parsed into components.

to_bytes(in_memory_dataset: Any) → bytes

Serialize the in-memory dataset to bytes.

Parameters:

in_memory_datasetobject

The Python object to serialize.

Returns:

serialized_datasetbytes

Bytes representing the serialized dataset.

Raises:

FormatterNotImplementedError

Raised if the formatter has not implemented the method. This will not cause a problem if write_local_file has been implemented.

classmethod validate_extension(location: Location) → None

Check the extension of the provided location for compatibility.

Parameters:

locationLocation

Location from which to extract a file extension.

Raises:

ValueError

Raised if the formatter does not understand this extension.

classmethod validate_write_recipes(recipes: Mapping[str, Any] | None) → Mapping[str, Any] | None

Validate supplied recipes for this formatter.

The recipes are supplemented with default values where appropriate.

Parameters:

recipesdict

Recipes to validate.

Returns:

validateddict

Validated recipes.

Raises:

RuntimeError

Raised if validation fails. The default implementation raises if any recipes are given.

write(in_memory_dataset: Any, cache_manager: AbstractDatastoreCacheManager | None = None) → None

Write a Dataset.

Parameters:

in_memory_datasetobject

The Dataset to serialize.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to cache the written file.

Raises:

FormatterNotImplementedError

Raised if the formatter subclass has not implemented write_local_file and to_bytes was not called.

Exception

Raised if there is an error serializing the dataset to disk.

Notes

The intent is for subclasses to implement either to_bytes or write_local_file or both and not to subclass this method.

write_direct(in_memory_dataset: Any, uri: ResourcePath, cache_manager: AbstractDatastoreCacheManager | None = None) → bool

Serialize and write directly to final location.

Parameters:

in_memory_datasetobject

The Dataset to serialize.

urilsst.resources.ResourcePath

URI to use when writing the serialized dataset.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to cache the written file.

Returns:

writtenbool

Flag to indicate whether the direct write did happen.

Raises:

Exception

Raised if there was a failure from serializing to bytes that was not FormatterNotImplementedError.

Notes

This method will call to_bytes to serialize the in-memory dataset and then will call the write method directly.

If the dataset should be cached or is local the file will not be written and the method will return False. This is because local URIs should be written to a temporary file name and then renamed to allow atomic writes. That path is handled by write_locally_then_move through write_local_file) and is preferred over this method being subclassed and the atomic write re-implemented.

write_local_file(in_memory_dataset: Any, uri: ResourcePath) → None

Serialize the in-memory dataset to a local file.

Parameters:

in_memory_datasetobject

The Python object to serialize.

uriResourcePath

The URI to use when writing the file.

Raises:

FormatterNotImplementedError

Raised if the formatter subclass has not implemented this method or has failed to implement the to_bytes method.

Notes

By default this method will attempt to call to_bytes and then write these bytes to the file.

write_locally_then_move(in_memory_dataset: Any, uri: ResourcePath, cache_manager: AbstractDatastoreCacheManager | None = None) → None

Write file to file system and then move to final location.

Parameters:

in_memory_datasetobject

The Dataset to serialize.

urilsst.resources.ResourcePath

URI to use when writing the serialized dataset.

cache_managerAbstractDatastoreCacheManager

A cache manager to use to allow a formatter to cache the written file.

Raises:

FormatterNotImplementedError

Raised if the formatter subclass has not implemented write_local_file.

Exception

Raised if there is an error serializing the dataset to disk.