Processor

class lsst.daf.relation.Processor

Bases: ABC

An inheritable framework for processing multi-engine relation trees.

Individual Engine classes have different definitions of what it means to process a relation tree, and no single engine can handle a tree with engines. This class provides a recursive algorithm that fills that role, with abstract method hooks for implementing Transfer and Materialize operations.

Notes

The Processor algorithm walks the tree recursively until it either finds:

• a Relation with a Relation.payload that is not None, which is returned as-is;

• a Materialization operation, for which a payload is computed via a call to the materialize hook, and then attached to both the original relation (modifying it in-place) and the returned one;

• a Transfer operation, for which a payload is computed via a call to the transfer hook, and then the attached to the returned relation only.

In addition, Processor never calls either hook on trivial methods - Engine.get_join_identity_payload and Engine.get_doomed_payload are called instead. This can (for example) avoid executing asking a database to execute a SQL query when the relation tree knows in advance the result will have no real content. It also special-cases Transfer operations that are followed immediately by a Materialization, allowing both operations to be handled by a single call.

Methods Summary

materialize(target, name)	Hook for implementing materialization operations.
process(relation)	Main entry point for processing a relation tree.
transfer(source, destination, materialize_as)	Hook for implementing transfers between engines.

Methods Documentation

abstract materialize(target: Relation, name: str) → Any

Hook for implementing materialization operations.

This method should be called only by the Processor base class.

Parameters:

targetRelation

Relation to be materialized. Any upstream Transfer operations in this tree are guaranteed to already have a payload already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.

namestr

The name of the Materialization operation, to be used as needed in the engine-specific payload.

Returns:

payload

Payload for this relation that should be cached.

process(relation: Relation) → Relation

Main entry point for processing a relation tree.

Parameters:

relationRelation

Root of the relation tree to process. On return, relations that hold a Materialization relation will have a new payload attached, if they did not have one already.

Returns:

processedRelation

A version of the relation tree in which any relation with a Transfer operation has a copy of the original Transfer that has a payload attached.

abstract transfer(source: Relation, destination: Engine, materialize_as: str | None) → Any

Hook for implementing transfers between engines.

This method should be called only by the Processor base class.

Parameters:

sourceRelation

Relation to be transferred. Any upstream Transfer operations in this tree are guaranteed to already have a payload already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.

destinationEngine

Engine the relation is being transferred to.

materialize_asstr or None

If not None, the name of a Materialization operation that immediately follows the transfer being implemented, in which case the returned payload should be appropriate for caching with the Materialization.

Returns:

payload

Payload for this relation in the destination engine.