Processor¶
- class lsst.daf.relation.Processor¶
Bases:
ABC
An inheritable framework for processing multi-engine relation trees.
Individual
Engine
classes have different definitions of what it means to process a relation tree, and no single engine can handle a tree with engines. This class provides a recursive algorithm that fills that role, with abstract method hooks for implementingTransfer
andMaterialize
operations.Notes
The
Processor
algorithm walks the tree recursively until it either finds:a
Relation
with aRelation.payload
that is notNone
, which is returned as-is;a
Materialization
operation, for which a payload is computed via a call to thematerialize
hook, and then attached to both the original relation (modifying it in-place) and the returned one;a
Transfer
operation, for which a payload is computed via a call to thetransfer
hook, and then the attached to the returned relation only.
In addition,
Processor
never calls either hook ontrivial
methods -Engine.get_join_identity_payload
andEngine.get_doomed_payload
are called instead. This can (for example) avoid executing asking a database to execute a SQL query when the relation tree knows in advance the result will have no real content. It also special-casesTransfer
operations that are followed immediately by aMaterialization
, allowing both operations to be handled by a single call.Methods Summary
materialize
(target, name)Hook for implementing materialization operations.
process
(relation)Main entry point for processing a relation tree.
transfer
(source, destination, materialize_as)Hook for implementing transfers between engines.
Methods Documentation
- abstract materialize(target: Relation, name: str) Any ¶
Hook for implementing materialization operations.
This method should be called only by the
Processor
base class.- Parameters:
- target
Relation
Relation to be materialized. Any upstream
Transfer
operations in this tree are guaranteed to already have apayload
already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.- name
str
The name of the
Materialization
operation, to be used as needed in the engine-specific payload.
- target
- Returns:
- payload
Payload for this relation that should be cached.
- process(relation: Relation) Relation ¶
Main entry point for processing a relation tree.
- Parameters:
- relation
Relation
Root of the relation tree to process. On return, relations that hold a
Materialization
relation will have a newpayload
attached, if they did not have one already.
- relation
- Returns:
- abstract transfer(source: Relation, destination: Engine, materialize_as: str | None) Any ¶
Hook for implementing transfers between engines.
This method should be called only by the
Processor
base class.- Parameters:
- source
Relation
Relation to be transferred. Any upstream
Transfer
operations in this tree are guaranteed to already have apayload
already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.- destination
Engine
Engine the relation is being transferred to.
- materialize_as
str
orNone
If not
None
, the name of aMaterialization
operation that immediately follows the transfer being implemented, in which case the returnedpayload
should be appropriate for caching with theMaterialization
.
- source
- Returns:
- payload
Payload for this relation in the
destination
engine.