Materialization¶

class lsst.daf.relation.Materialization(target: Relation, payload: Any = None, *, name: str)¶

Bases: lsst.daf.relation.MarkerRelation

A marker operation that indicates that the upstream tree should be evaluated only once, with the results saved and reused for subsequent processing.

Materialization is the only provided operation for which UnaryOperationRelation.is_locked defaults to True.

Also unlike most operations, the payload value for a Materialization if frequently not None, as this is where engine-specific state is cached for future reuse.

Attributes Summary

`columns`	The columns in this relation (`Set` [ `ColumnTag` ] ).
`engine`	The engine that is responsible for interpreting this relation (`Engine`).
`is_join_identity`	Whether a `join` to this relation will result in the other relation being returned directly (`bool`).
`is_locked`	Whether this relation and those upstream of it should be considered fixed by tree-manipulation algorithms (`bool`).
`is_trivial`	Whether this relation has no real content (`bool`).
`max_rows`	The maximum number of rows this relation might have (`int` or `None`).
`min_rows`	The minimum number of rows this relation might have (`int`).
`payload`

Methods Summary

`attach_payload`(payload)	Attach an engine-specific `payload` to this relation.
`chain`(rhs)	Return a new relation with all rows from this relation and another.
`join`(rhs, predicate, *, backtrack, transfer)	Return a new relation that joins this one to the given one.
`materialized`(name, None] = None, *, name_prefix)	Return a new relation that indicates that this relation’s payload should be cached after it is first processed.
`reapply`(target, payload, None] = None)	Mark a new target relation, returning a new instance of the same type.
`simplify`(target)
`sorted`(terms, *, preferred_engine, …)	Return a new relation that sorts rows according to a sequence of column expressions.
`transferred_to`(destination)	Return a new relation that transfers this relation to a new engine.
`with_calculated_column`(tag, expression, *, …)	Return a new relation that adds a calculated column to this one.
`with_only_columns`(columns, *, …)	Return a new relation whose columns are a subset of this relation’s.
`with_rows_satisfying`(predicate, *, …)	Return a new relation that filters out rows via a boolean expression.
`without_duplicates`(*, preferred_engine, …)	Return a new relation that removes any duplicate rows from this one.

Attributes Documentation

columns¶: The columns in this relation (Set [ ColumnTag ] ).

engine¶: The engine that is responsible for interpreting this relation (Engine).

is_join_identity¶

Whether a join to this relation will result in the other relation being returned directly (bool).

Join identity relations have exactly one row and no columns.

See also

LeafRelation.make_join_identity

is_locked¶: Whether this relation and those upstream of it should be considered fixed by tree-manipulation algorithms (bool).

is_trivial¶

Whether this relation has no real content (bool).

A trivial relation is either a join identity with no columns and exactly one row, or a relation with an arbitrary number of columns and no rows (i.e. min_rows==max_rows==0).

max_rows¶

The maximum number of rows this relation might have (int or None).

This is None for relations whose size is not bounded from above.

min_rows¶: The minimum number of rows this relation might have (int).

payload = None¶

Methods Documentation

attach_payload(payload: Any) → None¶

Attach an engine-specific payload to this relation.

This method may be called exactly once on a MarkerRelation instance that was not initialized with a payload, despite the fact that Relation objects are otherwise considered immutable.

Parameters:	payload Engine-specific content to attach.
Raises:	TypeError Raised if this relation already has a payload, or if this marker subclass can never have a payload. `TypeError` is used here for consistency with other attempts to assign to an attribute of an immutable object.

chain(rhs: lsst.daf.relation._relation.Relation) → lsst.daf.relation._relation.Relation¶

Return a new relation with all rows from this relation and another.

This is a convenience method that constructs and applies a Chain operation.

Parameters:	rhs : `Relation` Other relation to chain to `self`. Must have the same columns and engine as `self`.
Returns:	relation : `Relation` New relation with all rows from both relations. If the engine `preserves order` for chains, all rows from `self` will appear before all rows from `rhs`, in their original order. This method never returns an operand directly, even if the other has `max_rows==0`, as it is assumed that even relations with no rows are useful to preserve in the tree for `diagnostics`.
Raises:	ColumnError Raised if the two relations do not have the same columns. EngineError Raised if the two relations do not have the same engine. RowOrderError Raised if `self` or `rhs` is unnecessarily ordered; see `expect_unordered`.

join(rhs: Relation, predicate: Predicate | None = None, *, backtrack: bool = True, transfer: bool = False) → Relation¶

Return a new relation that joins this one to the given one.

This is a convenience method that constructs and applies a Join operation, via PartialJoin.apply.

Parameters:	rhs : `Relation` Relation to join to `self`. predicate : `Predicate`, optional Boolean expression that must evaluate to true in order to join a a pair of rows, in addition to an implicit equality constraint on any columns in both relations. backtrack : `bool`, optional If `True` (default) and `self.engine != rhs.engine`, attempt to insert this join before a transfer upstream of `self`, as long as this can be done without breaking up any locked relations or changing the resulting relation content. transfer : `bool`, optional If `True` (`False` is default) and `self.engine != rhs.engine`, insert a new `Transfer` before the `Join`. If `backtrack` is also true, the transfer is added only if the backtrack attempt fails.
Returns:	relation : `Relation` New relation that joins `self` to `rhs`. May be `self` or `rhs` if the other is a `join identity`.
Raises:	ColumnError Raised if the given predicate requires columns not present in `self` or `rhs`. EngineError Raised if it was impossible to insert this operation in `rhs.engine` via backtracks or transfers on `self`, or if the predicate was not supported by the engine. RowOrderError Raised if `self` or `rhs` is unnecessarily ordered; see `expect_unordered`.

Notes

This method does not treat self and rhs symmetrically: it always considers rhs fixed, and only backtracks into or considers applying transfers to self.

materialized(name: Optional[str, None] = None, *, name_prefix: str = 'materialization') → lsst.daf.relation._relation.Relation¶

Return a new relation that indicates that this relation’s payload should be cached after it is first processed.

This is a convenience method that constructs and applies a Materialization operation.

Parameters:	name : `str`, optional Name to use for the cached payload within the engine (e.g. the name for a temporary table in SQL). If not provided, a name will be created via a call to `Engine.get_relation_name`. name_prefix : `str`, optional Prefix to pass to `Engine.get_relation_name`; ignored if `name` is provided. Unlike most operations, `Materialization` relations are locked by default, since they reflect user intent to mark a specific tree as cacheable.
Returns:	relation : `Relation` New relation that marks its upstream tree for caching. May be `self` if it is already a `LeafRelation` or another materialization (in which case the given name or name prefix will be ignored).
Raises:

See also

Processor.materialize

reapply(target: lsst.daf.relation._relation.Relation, payload: Optional[Any, None] = None) → lsst.daf.relation._marker_relation.MarkerRelation¶

Mark a new target relation, returning a new instance of the same type.

Parameters:	target : `Relation` New relation to mark. payload, optional Payload to attach to the new relation.
Returns:	relation : `MarkerRelation` A new relation with the given target.

Notes

This method is primarily intended for use by operations that “unroll” a relation tree to perform some modification upstream and then “replay” the operations and markers that were downstream. MarkerRelation implementations with state that depends on the target will need to override this method to update that state accordingly.

classmethod simplify(target: Relation) → bool¶

sorted(terms: Sequence[SortTerm], *, preferred_engine: Engine | None = None, backtrack: bool = True, transfer: bool = False, require_preferred_engine: bool = False) → Relation¶

Return a new relation that sorts rows according to a sequence of column expressions.

This is a convenience method that constructs and applies a Sort operation.

Parameters:	terms : `Sequence` [ `SortTerm` ] Ordered sequence of column expressions to sort on, with whether to apply them in ascending or descending order. preferred_engine : `Engine`, optional Engine that the operation would ideally be performed in. If this is not equal to `self.engine`, the `backtrack`, `transfer`, and `require_preferred_engine` arguments control the behavior. backtrack : `bool`, optional If `True` (default) and the current engine is not the preferred engine, attempt to insert this sort before a transfer upstream of the current relation, as long as this can be done without breaking up any locked relations or changing the resulting relation content. transfer : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, insert a new `Transfer` before the `Sort`. If `backtrack` is also true, the transfer is added only if the backtrack attempt fails. require_preferred_engine : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, raise `EngineError`. If `backtrack` is also true, the exception is only raised if the backtrack attempt fails. Ignored if `transfer` is true.
Returns:	relation : `Relation` New relation with sorted rows. Will be `self` if `terms` is empty. If `self` is already a sort operation relation, the operations will be merged by concatenating their terms, which may result in duplicate sort terms that have no effect.
Raises:	ColumnError Raised if any column required by a `SortTerm` is not present in `self.columns`. EngineError Raised if `require_preferred_engine=True` and it was impossible to insert this operation in the preferred engine, or if a `SortTerm` expression was not supported by the engine.

transferred_to(destination: Engine) → Relation¶

Return a new relation that transfers this relation to a new engine.

This is a convenience method that constructs and applies a Transfer operation.

Parameters:	destination : `Engine` Engine for the new relation.
Returns:	relation : `Relation` New relation in the given engine. Will be `self` if `self.engine == destination`.
Raises:

with_calculated_column(tag: ColumnTag, expression: ColumnExpression, *, preferred_engine: Engine | None = None, backtrack: bool = True, transfer: bool = False, require_preferred_engine: bool = False) → Relation¶

Return a new relation that adds a calculated column to this one.

This is a convenience method chat constructs and applies a Calculation operation.

Parameters:	tag : `ColumnTag` Identifier for the new column. expression : `ColumnExpression` Expression used to populate the new column. preferred_engine : `Engine`, optional Engine that the operation would ideally be performed in. If this is not equal to `self.engine`, the `backtrack`, `transfer`, and `require_preferred_engine` arguments control the behavior. backtrack : `bool`, optional If `True` (default) and the current engine is not the preferred engine, attempt to insert this calculation before a transfer upstream of the current relation, as long as this can be done without breaking up any locked relations or changing the resulting relation content. transfer : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, insert a new `Transfer` before the `Calculation`. If `backtrack` is also true, the transfer is added only if the backtrack attempt fails. require_preferred_engine : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, raise `EngineError`. If `backtrack` is also true, the exception is only raised if the backtrack attempt fails. Ignored if `transfer` is true.
Returns:	relation : `Relation` Relation that contains the calculated column.
Raises:	ColumnError Raised if the expression requires columns that are not present in `self.columns`, or if `tag` is already present in `self.columns`. EngineError Raised if `require_preferred_engine=True` and it was impossible to insert this operation in the preferred engine, or if the expression was not supported by the engine.

with_only_columns(columns: Set[ColumnTag], *, preferred_engine: Engine | None = None, backtrack: bool = True, transfer: bool = False, require_preferred_engine: bool = False) → Relation¶

Return a new relation whose columns are a subset of this relation’s.

This is a convenience method that constructs and applies a Projection operation.

Parameters:	columns : `Set` [ `ColumnTag` ] Columns to be propagated to the new relation; must be a subset of `self.columns`. preferred_engine : `Engine`, optional Engine that the operation would ideally be performed in. If this is not equal to `self.engine`, the `backtrack`, `transfer`, and `require_preferred_engine` arguments control the behavior. backtrack : `bool`, optional If `True` (default) and the current engine is not the preferred engine, attempt to insert this projection before a transfer upstream of the current relation, as long as this can be done without breaking up any locked relations or changing the resulting relation content. transfer : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, insert a new `Transfer` before the `Projection`. If `backtrack` is also true, the transfer is added only if the backtrack attempt fails. require_preferred_engine : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, raise `EngineError`. If `backtrack` is also true, the exception is only raised if the backtrack attempt fails. Ignored if `transfer` is true.
Returns:	relation : `Relation` New relation with only the given columns. Will be `self` if `columns == self.columns`.
Raises:	ColumnError Raised if `columns` is not a subset of `self.columns`. EngineError Raised if `require_preferred_engine=True` and it was impossible to insert this operation in the preferred engine.

with_rows_satisfying(predicate: Predicate, *, preferred_engine: Engine | None = None, backtrack: bool = True, transfer: bool = False, require_preferred_engine: bool = False) → Relation¶

Return a new relation that filters out rows via a boolean expression.

This is a convenience method that constructions and applies a Selection operation.

Parameters:	predicate : `Predicate` Boolean expression that evaluates to `False` for rows that should be included and `False` for rows that should be filtered out. preferred_engine : `Engine`, optional Engine that the operation would ideally be performed in. If this is not equal to `self.engine`, the `backtrack`, `transfer`, and `require_preferred_engine` arguments control the behavior. backtrack : `bool`, optional If `True` (default) and the current engine is not the preferred engine, attempt to insert this selection before a transfer upstream of the current relation, as long as this can be done without breaking up any locked relations or changing the resulting relation content. transfer : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, insert a new `Transfer` before the `Selection`. If `backtrack` is also true, the transfer is added only if the backtrack attempt fails. require_preferred_engine : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, raise `EngineError`. If `backtrack` is also true, the exception is only raised if the backtrack attempt fails. Ignored if `transfer` is true.
Returns:	relation : `Relation` New relation with only the rows that satisfy the given predicate. May be `self` if the predicate is `trivially True`.
Raises:	ColumnError Raised if `predicate.columns_required` is not a subset of `self.columns`. EngineError Raised if `require_preferred_engine=True` and it was impossible to insert this operation in the preferred engine, or if the expression was not supported by the engine.

without_duplicates(*, preferred_engine: Engine | None = None, backtrack: bool = True, transfer: bool = False, require_preferred_engine: bool = False) → Relation¶

Return a new relation that removes any duplicate rows from this one.

This is a convenience method that constructs and applies a Deduplication operation.

Parameters:	preferred_engine : `Engine`, optional Engine that the operation would ideally be performed in. If this is not equal to `self.engine`, the `backtrack`, `transfer`, and `require_preferred_engine` arguments control the behavior. backtrack : `bool`, optional If `True` (default) and the current engine is not the preferred engine, attempt to insert this deduplication before a transfer upstream of the current relation, as long as this can be done without breaking up any locked relations or changing the resulting relation content. transfer : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, insert a new `Transfer` before the `Deduplication`. If `backtrack` is also true, the transfer is added only if the backtrack attempt fails. require_preferred_engine : `bool`, optional If `True` (`False` is default) and the current engine is not the preferred engine, raise `EngineError`. If `backtrack` is also true, the exception is only raised if the backtrack attempt fails. Ignored if `transfer` is true.
Returns:	relation : `Relation` Relation with no duplicate rows. This may be `self` if it can be determined that there is no duplication already, but this is not guaranteed.
Raises:	EngineError Raised if `require_preferred_engine=True` and it was impossible to insert this operation in the preferred engine.

Navigation

Materialization¶