MultiblockReader

class lsst.pipe.base.quantum_graph.MultiblockReader(stream: IO[bytes], int_size: int)

Bases: object

A helper object for reader multi-block files.

Methods Summary

open_in_zip(zf, name, *, int_size)

Open a reader for a file in a zip archive.

read_all_bytes_in_zip(zf, name, *, int_size, ...)

Iterate over all of the byte blocks in a file in a zip archive.

read_all_models_in_zip(zf, name, model_type, ...)

Iterate over all of the models in a file in a zip archive.

read_bytes(address)

Read raw bytes from the multi-block file.

read_model(address, model_type, decompressor)

Read a single compressed JSON block.

Methods Documentation

classmethod open_in_zip(zf: zipfile.ZipFile, name: str, *, int_size: int) Iterator[MultiblockReader]

Open a reader for a file in a zip archive.

Parameters:
zfzipfile.ZipFile

Zip archive to read the file from.

namestr

Base name for the multi-block file; an extension will be added.

int_sizeint

Number of bytes to use for all integers.

Returns:
readercontextlib.AbstractContextManager [ MultiblockReader ]

Context manager that returns a reader when entered.

classmethod read_all_bytes_in_zip(zf: zipfile.ZipFile, name: str, *, int_size: int, page_size: int) Iterator[bytes]

Iterate over all of the byte blocks in a file in a zip archive.

Parameters:
zfzipfile.ZipFile

Zip archive to read the file from.

namestr

Base name for the multi-block file; an extension will be added.

int_sizeint

Number of bytes to use for all integers.

page_sizeint

Approximate number of bytes to read at a time.

Returns:
byte_iterIterator [ bytes ]

Iterator over blocks.

classmethod read_all_models_in_zip(zf: zipfile.ZipFile, name: str, model_type: type[_T], decompressor: Decompressor, *, int_size: int, page_size: int) Iterator[_T]

Iterate over all of the models in a file in a zip archive.

Parameters:
zfzipfile.ZipFile

Zip archive to read the file from.

namestr

Base name for the multi-block file; an extension will be added.

model_typetype [ pydantic.BaseModel ]

Pydantic model to validate JSON with.

decompressorDecompressor

Object with a decompress method that takes and returns bytes.

int_sizeint

Number of bytes to use for all integers.

page_sizeint

Approximate number of bytes to read at a time.

Returns:
model_iterIterator [ pydantic.BaseModel ]

Iterator over model instances.

read_bytes(address: Address) bytes | None

Read raw bytes from the multi-block file.

Parameters:
addressAddress

Offset and size of the data to read.

Returns:
databytes or None

Data read directly, or None if the address has zero size.

read_model(address: Address, model_type: type[_T], decompressor: Decompressor) _T | None

Read a single compressed JSON block.

Parameters:
addressAddress

Size and offset of the block.

model_typetype [ pydantic.BaseModel ]

Pydantic model to validate JSON with.

decompressorDecompressor

Object with a decompress method that takes and returns bytes.

Returns:
modelpydantic.BaseModel

Validated model.