Processor

class lsst.daf.relation.Processor

Bases: abc.ABC

An inheritable framework for processing multi-engine relation trees.

Individual Engine classes have different definitions of what it means to process a relation tree, and no single engine can handle a tree with engines. This class provides a recursive algorithm that fills that role, with abstract method hooks for implementing Transfer and Materialize operations.

Notes

The Processor algorithm walks the tree recursively until it either finds:

  • a Relation with a Relation.payload that is not None, which is returned as-is;
  • a Materialization operation, for which a payload is computed via a call to the materialize hook, and then attached to both the original relation (modifying it in-place) and the returned one;
  • a Transfer operation, for which a payload is computed via a call to the transfer hook, and then the attached to the returned relation only.

In addition, Processor never calls either hook on trivial methods - Engine.get_join_identity_payload and Engine.get_doomed_payload are called instead. This can (for example) avoid executing asking a database to execute a SQL query when the relation tree knows in advance the result will have no real content. It also special-cases Transfer operations that are followed immediately by a Materialization, allowing both operations to be handled by a single call.

Methods Summary

materialize(target, name) Hook for implementing materialization operations.
process(relation) Main entry point for processing a relation tree.
transfer(source, destination, materialize_as) Hook for implementing transfers between engines.

Methods Documentation

materialize(target: Relation, name: str) → Any

Hook for implementing materialization operations.

This method should be called only by the Processor base class.

Parameters:
target : Relation

Relation to be materialized. Any upstream Transfer operations in this tree are guaranteed to already have a payload already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.

name : str

The name of the Materialization operation, to be used as needed in the engine-specific payload.
Returns:
payload

Payload for this relation that should be cached.
process(relation: Relation) → Relation

Main entry point for processing a relation tree.

Parameters:
relation : Relation

Root of the relation tree to process. On return, relations that hold a Materialization relation will have a new payload attached, if they did not have one already.
Returns:
processed : Relation

A version of the relation tree in which any relation with a Transfer operation has a copy of the original Transfer that has a payload attached.
transfer(source: Relation, destination: Engine, materialize_as: str | None) → Any

Hook for implementing transfers between engines.

This method should be called only by the Processor base class.

Parameters:
source : Relation

Relation to be transferred. Any upstream Transfer operations in this tree are guaranteed to already have a payload already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.

destination : Engine

Engine the relation is being transferred to.

materialize_as : str or None

If not None, the name of a Materialization operation that immediately follows the transfer being implemented, in which case the returned payload should be appropriate for caching with the Materialization.
Returns:
payload

Payload for this relation in the destination engine.