Engine

class lsst.daf.relation.iteration.Engine(*, name: str = 'iteration', functions: dict[str, _F] = <factory>, relation_name_counter: int = 0)

Bases: lsst.daf.relation.GenericConcreteEngine

A concrete engine that treats relations as iterables with Mapping rows.

See the iteration module documentation for details.

Attributes Summary

name
relation_name_counter

Methods Summary

append_binary(operation, lhs, rhs) Hook for maintaining the engine’s conform invariants through BinaryOperation.apply.
append_unary(operation, target) Hook for maintaining the engine’s conform invariants through UnaryOperation.apply.
apply_custom_unary_operation(operation, target) Convert a custom UnaryOperation to a RowIterable.
backtrack_unary(operation, tree, preferred) Attempt to insert a unary operation in another engine upstream of this one by via operation commutators.
conform(relation) Ensure a relation tree satisfies this engine’s invariants.
convert_column_container(expression) Convert a ColumnContainer to a Python callable.
convert_column_expression(expression) Convert a ColumnExpression to a Python callable.
convert_predicate(predicate) Convert a Predicate to a Python callable.
execute(relation) Execute a native iteration relation, returning a Python iterable.
get_doomed_payload(columns) Return a payload for a leaf relation that has no rows.
get_function(name) Return the named column expression function.
get_join_identity_payload() Return a payload for a leaf relation that is the join identity.
get_relation_name(prefix) Return a name suitable for a new relation in this engine.
make_doomed_relation(columns, messages, name) Construct a leaf relation with no rows and one or more messages explaining why.
make_join_identity_relation(name) Construct a leaf relation with no columns and exactly one row.
make_leaf(columns, payload, *, name, …) Create a nontrivial leaf relation in this engine.
materialize(target, name, name_prefix) Mark that a target relation’s payload should be cached.
transfer(target, payload) Mark that a relation’s payload should be transferred from some other engine to this one.

Attributes Documentation

name = 'iteration'
relation_name_counter = 0

Methods Documentation

append_binary(operation: BinaryOperation, lhs: Relation, rhs: Relation) → Relation

Hook for maintaining the engine’s conform invariants through BinaryOperation.apply.

This method should only be called by BinaryOperation.apply and the engine’s own methods and helper classes. External code should call BinaryOperation.apply or a Relation factory method instead.

Parameters:
operation : BinaryOperation

Operation to apply; should already be filtered through BinaryOperation._begin_apply.

lhs : Relation

One relation to apply the operation to directly.

rhs : Relation

The other relation to apply the operation to directly.

Returns:
relation : Relation

Relation that includes the given operation acting on lhs and rhs, or a simplified equivalent.

Notes

Implementations should delegate back to UnaryOperation._finish_apply to actually create a UnaryOperationRelation and perform final simplification and checks. This is all the default implementation does.

append_unary(operation: UnaryOperation, target: Relation) → Relation

Hook for maintaining the engine’s conform invariants through UnaryOperation.apply.

This method should only be called by UnaryOperation.apply and the engine’s own methods and helper classes. External code should call UnaryOperation.apply or a Relation factory method instead.

Parameters:
operation : UnaryOperation

Operation to apply; should already be filtered through UnaryOperation._begin_apply.

target : Relation

Relation to apply the operation to directly.

Returns:
relation : Relation

Relation that includes the given operation acting on target, or a simplified equivalent.

Notes

Implementations should delegate back to UnaryOperation._finish_apply to actually create a UnaryOperationRelation and perform final simplification and checks. This is all the default implementation does.

apply_custom_unary_operation(operation: lsst.daf.relation._unary_operation.UnaryOperation, target: lsst.daf.relation._relation.Relation) → lsst.daf.relation.iteration._row_iterable.RowIterable

Convert a custom UnaryOperation to a RowIterable.

This method must be implemented in a subclass engine in order to support any custom UnaryOperation.

Parameters:
operation : UnaryOperation

Operation to apply. Guaranteed to be a Marker, Reordering, or RowFilter subclass.

target : Relation

Target of the unary operation. Typically this will be passed to execute and the result used to construct a new RowIterable.

Returns:
rows : RowIterable

Iterable over rows, with each row a mapping keyed by ColumnTag.

backtrack_unary(operation: lsst.daf.relation._unary_operation.UnaryOperation, tree: lsst.daf.relation._relation.Relation, preferred: lsst.daf.relation._engine.Engine) → tuple

Attempt to insert a unary operation in another engine upstream of this one by via operation commutators.

Parameters:
operation : UnaryOperation

Unary operation to apply.

tree : Relation

Relation tree the operation logically acts on; any upstream insertion of the given operation should be equivalent to applying it to the root of this tree. Caller guarantees that tree.engine == self.

preferred : Engine

Engine in which the operation or its commuted equivalent should be performed.

Returns:
new_tree : Relation

Possibly-updated relation tree.

done : bool

If True, the operation has been fully inserted upstream in the preferred engine. If False, either tree was returned unmodified or only a part of the operation (e.g. a projection whose columns are superset of the given projection’s) was inserted upstream.

messages : Sequence [ str ]

Messages explaining why backtracking insertion was unsuccessful or incomplete. Should be sentences with no trailing . and no capitalization; they will be joined with semicolons.

conform(relation: Relation) → Relation

Ensure a relation tree satisfies this engine’s invariants.

This can include reordering operations (in a way consistent with their commutators) and/or inserting MarkerRelation nodes.

Parameters:
relation : Relation

Original relation tree.

Returns:
conformed : Relation

Relation tree that satisfies this engine’s invariants.

Notes

The default implementation returns the given relation. Engines with a non-trivial conform implementation should always call it on any relations they are passed, as algorithms that process the relation tree are not guaranteed to maintain those invariants themselves. It is recommended to use a custom MarkerRelation to indicate trees that satisfy invariants, allowing the corresponding conform implementation to short-circuit quickly.

convert_column_container(expression: lsst.daf.relation._columns._container.ColumnContainer) → collections.abc.Callable[collections.abc.Mapping[lsst.daf.relation._columns._tag.ColumnTag, Any], collections.abc.Container]

Convert a ColumnContainer to a Python callable.

Parameters:
expression : ColumnContainer

Expression to convert.

Returns:
callable

Callable that takes a single Mapping argument (with ColumnTag keys and regular Python values, representing a row in a relation), returning the evaluated expression as collections.abc.Container instance.

convert_column_expression(expression: lsst.daf.relation._columns._expression.ColumnExpression) → collections.abc.Callable[collections.abc.Mapping[lsst.daf.relation._columns._tag.ColumnTag, Any], Any]

Convert a ColumnExpression to a Python callable.

Parameters:
expression : ColumnExpression

Expression to convert.

Returns:
callable

Callable that takes a single Mapping argument (with ColumnTag keys and regular Python values, representing a row in a relation), returning the evaluated expression as another regular Python value.

convert_predicate(predicate: lsst.daf.relation._columns._predicate.Predicate) → collections.abc.Callable[collections.abc.Mapping[lsst.daf.relation._columns._tag.ColumnTag, Any], bool]

Convert a Predicate to a Python callable.

Parameters:
predicate : Predicate

Expression to convert.

Returns:
callable

Callable that takes a single Mapping argument (with ColumnTag keys and regular Python values, representing a row in a relation), returning the evaluated expression as a bool.

execute(relation: lsst.daf.relation._relation.Relation) → lsst.daf.relation.iteration._row_iterable.RowIterable

Execute a native iteration relation, returning a Python iterable.

Parameters:
relation : Relation

Relation to execute.

Returns:
rows : RowIterable

Iterable over rows, with each row a mapping keyed by ColumnTag.

Notes

This method does not in general iterate over the relation’s rows; while some operations like Sort and Deduplication do require processing all rows up front (which will happen during a call to execute), most return lazy iterables that do little or nothing until actually iterated over.

This method requires all relations in the tree to have the same engine (self). Use the Processor class to handle trees with multiple engines.

get_doomed_payload(columns: collections.abc.Set[lsst.daf.relation._columns._tag.ColumnTag]) → lsst.daf.relation.iteration._row_iterable.RowIterable

Return a payload for a leaf relation that has no rows.

Parameters:
columns : Set [ ColumnTag ]

The columns the relation should have.

Returns:
payload

The engine-specific content for this relation.

get_function(name: str) → Optional[_F, None]

Return the named column expression function.

Parameters:
name : str

Name of the function, from ColumnFunction.name or PredicateFunction.name

Returns:
function

Engine-specific callable, or None if no match was found.

Notes

This implementation first looks for a symbol with this name in the built-in operator module, to handle the common case (shared by both the iteration and sql engines) where these functions are appropriate for the engine due to operator overloading. When this fails, the name is looked up in the functions attribute.

get_join_identity_payload() → lsst.daf.relation.iteration._row_iterable.RowIterable

Return a payload for a leaf relation that is the join identity.

Returns:
payload

The engine-specific content for this relation.

get_relation_name(prefix: str = 'leaf') → str

Return a name suitable for a new relation in this engine.

Parameters:
prefix : str, optional

Prefix to include in the returned name.

Returns:
name : str

Name for the relation; guaranteed to be unique over all of the relations in this engine.

Notes

This implementation combines the given prefix with both the current relation_name_counter value and a random hexadecimal suffix.

make_doomed_relation(columns: Set[ColumnTag], messages: Sequence[str], name: str = '0') → Relation

Construct a leaf relation with no rows and one or more messages explaining why.

Parameters:
columns : Set [ ColumnTag ]

The columns in this relation.

messages : Sequence [ `str ]

One or more messages explaining why the relation has no rows.

name : str, optional

Name used to identify and reconstruct this relation.

Returns:
relation : Relation

Doomed relation.

Notes

This is simplify a convenience method that delegates to LeafRelation.make_doomed. Derived engines with a nontrivial conform should override this method to conform the return value.

make_join_identity_relation(name: str = 'I') → Relation

Construct a leaf relation with no columns and exactly one row.

Parameters:
engine : Engine

The engine that is responsible for interpreting this relation.

name : str, optional

Name used to identify and reconstruct this relation.

Returns:
relation : Relation

Relation with no columns and one row.

make_leaf(columns: collections.abc.Set[lsst.daf.relation._columns._tag.ColumnTag], payload: lsst.daf.relation.iteration._row_iterable.MaterializedRowIterable, *, name: str = '', messages: collections.abc.Sequence[str] = (), name_prefix: str = 'leaf', parameters: Optional[Any, None] = None) → lsst.daf.relation._leaf_relation.LeafRelation

Create a nontrivial leaf relation in this engine.

This is a convenience method that simply forwards all arguments to the LeafRelation constructor; see that class for details.

materialize(target: Relation, name: str | None = None, name_prefix: str = 'materialization_') → Relation

Mark that a target relation’s payload should be cached.

Parameters:
target : Relation

Relation to mark.

name : str, optional

Name to use for the cached payload within the engine.

name_prefix : str, optional

Prefix to pass to get_relation_name; ignored if name is provided.

Returns:
relation : Relation

New relation that marks its upstream tree for caching, unless the materialization was simplified away.

See also

Processor.materialize

Notes

The base class implementation calls Materialization.simplify to avoid materializations of leaf relations or other materializations. Override implementations should generally do the same.

transfer(target: Relation, payload: Any | None = None) → Relation

Mark that a relation’s payload should be transferred from some other engine to this one.

Parameters:
target : Relation

Relation to transfer. If target.engine == self, this relation will be returned directly and no transfer will be performed. Back-to-back transfers from one engine to another and back again are also simplified away (via a call to Transfer.simplify). Sequences of transfers involving more than two engines are not simplified.

payload, optional

Destination-engine-specific content for the relation to attach to the transfer. Most Transfer relations do not have a payload; their ability to do so is mostly to support the special relation trees returned by the Processor class.

Returns:
relation : Relation

New relation that marks its upstream tree to be transferred to a new engine.

See also

Processor.transfer

Notes

The default implementation calls conform on the target relation using the target relation’s engine (i.e. not self). All override implementations should do this as well.