MultiblockWriter

class lsst.pipe.base.quantum_graph.MultiblockWriter(stream: ~typing.IO[bytes], int_size: int, file_size: int = 0, addresses: dict[~uuid.UUID, ~lsst.pipe.base.quantum_graph._multiblock.Address] = <factory>)

Bases: object

A helper object for writing multi-block files.

Attributes Summary

addresses	Running map of all addresses added to the file so far.
file_size	Running size of the full file.
int_size	Number of bytes to use for all integers.
stream	A binary file-like object to write to.

Methods Summary

open_in_zip(zf, name, int_size[, use_tempfile])	Open a writer for a file in a zip archive.
write_bytes(id, data)	Write raw bytes to the multi-block file.
write_model(id, model, compressor)	Write raw bytes to the multi-block file.

Attributes Documentation

addresses: dict[UUID, Address] = <dataclasses._MISSING_TYPE object>

Running map of all addresses added to the file so far.

When the multi-block file is fully written, this is appended to the AddressWriter.addresses to write the corresponding address file.

file_size: int = 0

Running size of the full file.

int_size: int = <dataclasses._MISSING_TYPE object>

Number of bytes to use for all integers.

stream: IO[bytes] = <dataclasses._MISSING_TYPE object>

A binary file-like object to write to.

Methods Documentation

classmethod open_in_zip(zf: zipfile.ZipFile, name: str, int_size: int, use_tempfile: bool = False) → Iterator[MultiblockWriter]

Open a writer for a file in a zip archive.

Parameters

zfzipfile.ZipFile

Zip archive to add the file to.

namestr

Base name for the multi-block file; an extension will be added.

int_sizeint

Number of bytes to use for all integers.

use_tempfilebool, optional

If True, send writes to a temporary file and only add the file to the zip archive when the context manager closes. This involves more overall I/O, but it permits multiple multi-block files to be open for writing in the same zip archive at once.

Returns

writercontextlib.AbstractContextManager [ MultiblockWriter ]

Context manager that returns a writer when entered.

write_bytes(id: UUID, data: bytes) → Address

Write raw bytes to the multi-block file.

Parameters

iduuid.UUID

Unique ID of the object described by this block.

databytes

Data to store directly.

Returns

addressAddress

Address of the bytes just written.

write_model(id: UUID, model: BaseModel, compressor: Compressor) → Address

Write raw bytes to the multi-block file.

Parameters

iduuid.UUID

Unique ID of the object described by this block.

modelpydantic.BaseModel

Model to convert to JSON and compress.

compressorCompressor

Object with a compress method that takes and returns bytes.

Returns

addressAddress

Address of the bytes just written.