MultiblockReader

class lsst.pipe.base.quantum_graph.MultiblockReader(stream: IO[bytes], int_size: int)

Bases: object

A helper object for reader multi-block files.

Methods Summary

open_in_zip(zf, name, *, int_size)	Open a reader for a file in a zip archive.
read_all_bytes_in_zip(zf, name, *, int_size, ...)	Iterate over all of the byte blocks in a file in a zip archive.
read_all_models_in_zip(zf, name, model_type, ...)	Iterate over all of the models in a file in a zip archive.
read_bytes(address)	Read raw bytes from the multi-block file.
read_model(address, model_type, decompressor)	Read a single compressed JSON block.

Methods Documentation

classmethod open_in_zip(zf: zipfile.ZipFile, name: str, *, int_size: int) → Iterator[MultiblockReader]

Open a reader for a file in a zip archive.

Parameters:

zfzipfile.ZipFile

Zip archive to read the file from.

namestr

Base name for the multi-block file; an extension will be added.

int_sizeint

Number of bytes to use for all integers.

Returns:

readercontextlib.AbstractContextManager [ MultiblockReader ]

Context manager that returns a reader when entered.

classmethod read_all_bytes_in_zip(zf: zipfile.ZipFile, name: str, *, int_size: int, page_size: int) → Iterator[bytes]

Iterate over all of the byte blocks in a file in a zip archive.

Parameters:

zfzipfile.ZipFile

Zip archive to read the file from.

namestr

Base name for the multi-block file; an extension will be added.

int_sizeint

Number of bytes to use for all integers.

page_sizeint

Approximate number of bytes to read at a time.

Returns:

byte_iterIterator [ bytes ]

Iterator over blocks.

classmethod read_all_models_in_zip(zf: zipfile.ZipFile, name: str, model_type: type[_T], decompressor: Decompressor, *, int_size: int, page_size: int) → Iterator[_T]

Iterate over all of the models in a file in a zip archive.

Parameters:

zfzipfile.ZipFile

Zip archive to read the file from.

namestr

Base name for the multi-block file; an extension will be added.

model_typetype [ pydantic.BaseModel ]

Pydantic model to validate JSON with.

decompressorDecompressor

Object with a decompress method that takes and returns bytes.

int_sizeint

Number of bytes to use for all integers.

page_sizeint

Approximate number of bytes to read at a time.

Returns:

model_iterIterator [ pydantic.BaseModel ]

Iterator over model instances.

read_bytes(address: Address) → bytes | None

Read raw bytes from the multi-block file.

Parameters:

addressAddress

Offset and size of the data to read.

Returns:

databytes or None

Data read directly, or None if the address has zero size.

read_model(address: Address, model_type: type[_T], decompressor: Decompressor) → _T | None

Read a single compressed JSON block.

Parameters:

addressAddress

Size and offset of the block.

model_typetype [ pydantic.BaseModel ]

Pydantic model to validate JSON with.

decompressorDecompressor

Object with a decompress method that takes and returns bytes.

Returns:

modelpydantic.BaseModel

Validated model.