Starlink NDF I/O

This is an experimental HDF5 serialization backend to demonstrate that the package can be used to write files in formats other than FITS. It does write files in HDF5 format that can be read by the Starlink software. We do not recommend you use it in a production environment at this time to create archival products.

lsst.images.ndf Package

Archive implementations that write HDS-on-HDF5 files compatible with the Starlink N-Dimensional Data Format (NDF) data model.

Files written by this archive are valid NDF files readable by applications and libraries from the Starlink Software Collection (KAPPA, hdstrace, CUPID, etc.) although the LSST data model adds extensions that are not known to the Starlink tooling.

The NDF data model is described in Jenness et al (2015).

For a lsst.images.MaskedImage the data and variance arrays map to the standard NDF equivalents. Any FITS headers are stored in the standard .MORE.FITS extension and the WCS is stored as a serialized AST FrameSet, modified to include a GRID frame in addition to a PIXEL frame, in the .WCS component. Starlink tools only allow 8-bit masks so the mask is stored directly as a QUALITY component when only 8 bits are needed but if more bits are required the full mask is stored in the .MORE.LSST.MASK extension as a 3-D unsigned byte NDF where the third dimension matches the internal representation of the 3-D mask. The QUALITY component includes a collapsed version of the full mask to enable Starlink software to apply a coarse mask. In the future the collapsing is intending to be more granular to allow mask planes to be combined based on their related concepts. Since the NDF data model has a subset that is compatible with the MaskedImage data model, an NDF can be read as a MaskedImage with full data, variance, mask, WCS, and FITS header support, although of course any extensions will be ignored.

For more complex image types such as lsst.images.VisitImage the LSST components are stored in the .MORE.LSST extension, including the JSON component containing the Pydantic models. Data arrays are represented as NDFs in the .MORE.LSST extension and referenced by path from the JSON. Tabular data is not supported by the NDF data model and is not currently representable. It is expected that they would become individual columns in an extension, all with the same length.

These files use HDF5 format and can be read by any HDF5 tooling such as h5dump or the Python h5py, although the data model includes group attributes to allow it to be read by the Starlink libraries that add additional semantic meaning to data structures. The HDS-on-HDF5 format is described in Jenness (2015).

HDF5 files are not generally usable for remote access of components through cloud storage. There is no facility to store byte offsets to components in any of the data structures.

Functions

archive_path_to_hdf5_path(archive_path)	Translate a serialization archive path to an NDF HDF5 path.
archive_path_to_hdf5_path_components(...)	Return HDS-compatible path components for an archive path.
read(cls, path, **kwargs)	Read an object from an NDF (HDS-on-HDF5) file.
write(obj[, filename, metadata, ...])	Write a serializable object to an NDF (HDS-on-HDF5) file.

Classes

NdfInputArchive(file)	Reads HDS-on-HDF5 NDF files written by NdfOutputArchive.
NdfOutputArchive(file[, ...])	An OutputArchive implementation that writes HDS-on-HDF5 files compatible with the Starlink NDF data model.
NdfPointerModel(*, path)	Reference to an NDF-archive sub-tree by HDF5 path.