QueryContext

class lsst.daf.butler.registry.queries.QueryContext

Bases: lsst.daf.relation.Processor, contextlib.AbstractContextManager

A context manager interface for query operations that require some connection-like state.

Notes

QueryContext implementations are usually paired with a QueryBackend implementation, with the division of responsibilities as follows:

  • QueryContext implements the lsst.daf.relation.Processor interface, and is hence responsible for executing multi-engine relation trees.
  • QueryContext manages all state whose lifetime is a single query or set of related queries (e.g. temporary tables) via its context manager interface. Methods that do not involve this state should not require the context manager to have been entered.
  • QueryContext objects should be easily to construct by registry helper code that doesn’t have access to the full Registry data structure itself, while QueryBackend instances can generally only be constructed by code that does see essentially the full registry (for example, SqlQueryBackend holds a RegistryManagerInstances struct, while SqlQueryContext can be constructed with just a Database and ColumnTypeInfo).
  • QueryBackend.context is a factory for the associated QueryContext type.
  • QueryBackend methods that return relations accept the QueryContext returned by its context method in case those methods require state that should be cleaned up after the query is complete.

Attributes Summary

is_open Whether the context manager has been entered (bool).
preferred_engine Return the relation engine that this context prefers to execute operations in (lsst.daf.relation.Engine).

Methods Summary

any(relation, *, execute, exact) Check whether this relation has any result rows at all.
count(relation, *, exact, discard) Count the number of rows in the given relation.
drop_invalidated_postprocessing(relation, …) Return a modified relation tree without iteration-engine operations that require columns that are not in the given set.
fetch_iterable(relation) Execute the given relation and return its rows as an iterable of mappings.
make_data_coordinate_predicate(…) Return a Predicate that represents a data ID constraint.
make_data_id_relation(data_ids, dimension_names) Transform a set of data IDs into a relation.
make_initial_relation(relation, None] = None) Construct an initial relation suitable for this context.
make_spatial_region_overlap_predicate(lhs, rhs) Return a Predicate that tests whether two regions overlap
make_spatial_region_skypix_predicate(…) Return a Predicate that tests whether two region columns overlap
make_timespan_overlap_predicate(tag, timespan) Return a Predicate that tests whether a timespan column overlaps a timespan literal.
materialize(base, name) Hook for implementing materialization operations.
process(relation) Main entry point for processing a relation tree.
restore_columns(relation, columns_required) Return a modified relation tree that attempts to restore columns that were dropped by a projection operation.
strip_postprocessing(relation) Return a modified relation tree without any iteration-engine operations and any transfer to the iteration engine at the end.
transfer(source, destination, …) Hook for implementing transfers between engines.

Attributes Documentation

is_open

Whether the context manager has been entered (bool).

preferred_engine

Return the relation engine that this context prefers to execute operations in (lsst.daf.relation.Engine).

Methods Documentation

any(relation: lsst.daf.relation._relation.Relation, *, execute: bool = True, exact: bool = True) → bool

Check whether this relation has any result rows at all.

Parameters:
relation : Relation

Relation to be checked.

execute : bool, optional

If True, execute at least a LIMIT 1 query if it cannot be determined prior to execution that the query would return no rows.

exact : bool, optional

If True, run the full query and perform post-query filtering if needed, until at least one result row is found. If False, the returned result does not account for post-query filtering, and hence may be True even when all result rows would be filtered out.

Returns:
any_rows : bool

Whether the relation has any rows, or if it may have any rows if exact=False.

Raises:
RuntimeError

Raised if an exact check was requested and could not be obtained without executing the query.

count(relation: lsst.daf.relation._relation.Relation, *, exact: bool = True, discard: bool = False) → int

Count the number of rows in the given relation.

Parameters:
relation : Relation

Relation whose rows are to be counted.

exact : bool, optional

If True (default), return the exact number of rows. If False, returning an upper bound is permitted if it can be done much more efficiently, e.g. by running a SQL SELECT COUNT(*) query but ignoring client-side filtering that would otherwise take place.

discard : bool, optional

If True, compute the exact count even if it would require running the full query and then throwing away the result rows after counting them. If False, this is an error, as the user would usually be better off executing the query first to fetch its rows into a new query (or passing exact=False). Ignored if exact=False.

Returns:
n_rows : int

Number of rows in the relation, or an upper bound. This includes duplicates, if there are any.

Raises:
RuntimeError

Raised if an exact count was requested and could not be obtained without fetching and discarding rows.

drop_invalidated_postprocessing(relation: lsst.daf.relation._relation.Relation, new_columns: collections.abc.Set[lsst.daf.relation._columns._tag.ColumnTag]) → lsst.daf.relation._relation.Relation

Return a modified relation tree without iteration-engine operations that require columns that are not in the given set.

Parameters:
relation : Relation

Original relation tree.

new_columns : Set [ ColumnTag ]

The only columns that postprocessing operations may require if they are to be retained in the returned relation tree.

Returns:
modified : Relation

Modified relation tree with postprocessing operations incompatible with new_columns removed.

fetch_iterable(relation: lsst.daf.relation._relation.Relation) → lsst.daf.relation.iteration._row_iterable.RowIterable

Execute the given relation and return its rows as an iterable of mappings.

Parameters:
relation : Relation

Relation representing the query to execute.

Returns:
rows : RowIterable

An iterable over rows, with each row a mapping from ColumnTag to column value.

Notes

A transfer to iteration_engine will be added to the root (end) of the relation tree if the root is not already in the iteration engine.

Any transfers from other engines or persistent materializations will be handled by delegating to process before execution in the iteration engine.

To ensure the result is a multi-pass Python collection in memory, ensure the given tree ends with a materialization operation in the iteration engine.

make_data_coordinate_predicate(data_coordinate: lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, full: Optional[bool, None] = None) → lsst.daf.relation._columns._predicate.Predicate

Return a Predicate that represents a data ID constraint.

Parameters:
data_coordinate : DataCoordinate

Data ID whose keys and values should be transformed to predicate equality constraints.

full : bool, optional

Whether to include constraints on implied dimensions (default is to include implied dimensions if data_coordinate has them).

Returns:
predicate : lsst.daf.relation.column_expressions.Predicate

New predicate

make_data_id_relation(data_ids: collections.abc.Set[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate], dimension_names: collections.abc.Iterable[str]) → lsst.daf.relation._relation.Relation

Transform a set of data IDs into a relation.

data_ids : Set [ DataCoordinate ]
Data IDs to upload. All must have at least the dimensions given, but may have more.
dimension_names : Iterable [ str ]
Names of dimensions that will be the columns of the relation.
Returns:
relation : Relation

Relation in the iteration engine.

make_initial_relation(relation: Optional[lsst.daf.relation._relation.Relation, None] = None) → lsst.daf.relation._relation.Relation

Construct an initial relation suitable for this context.

Parameters:
relation : Relation, optional

A user-provided initial relation. Must be included by implementations when provided, but may be modified (e.g. by adding a transfer to a new engine) when need to satisfy the context’s invariants.

make_spatial_region_overlap_predicate(lhs: lsst.daf.relation._columns._expression.ColumnExpression, rhs: lsst.daf.relation._columns._expression.ColumnExpression) → lsst.daf.relation._columns._predicate.Predicate

Return a Predicate that tests whether two regions overlap

This operation only works with iteration engines; it is usually used to refine the result of a join or constraint on SkyPixDimension columns in SQL.

lhs : lsst.daf.relation.column_expressions.ColumnExpression
Expression for one spatial region.
rhs : lsst.daf.relation.column_expressions.ColumnExpression
Expression for the other spatial region.
Returns:
predicate : lsst.daf.relation.column_expressions.Predicate

New predicate with lhs and rhs as its required columns.

make_spatial_region_skypix_predicate(dimension: lsst.daf.butler.core.dimensions._skypix.SkyPixDimension, region: lsst.sphgeom._sphgeom.Region) → lsst.daf.relation._columns._predicate.Predicate

Return a Predicate that tests whether two region columns overlap

This operation only works with iteration engines; it is usually used to refine the result of a join on SkyPixDimension columns in SQL.

dimension : SkyPixDimension
Dimension whose key column is being constrained.
region : lsst.sphgeom.Region
Spatial region constraint to test against.
Returns:
predicate : lsst.daf.relation.column_expressions.Predicate

New predicate with the DimensionKeyColumn associated with dimension as its only required column.

make_timespan_overlap_predicate(tag: lsst.daf.relation._columns._tag.ColumnTag, timespan: lsst.daf.butler.core.timespan.Timespan) → lsst.daf.relation._columns._predicate.Predicate

Return a Predicate that tests whether a timespan column overlaps a timespan literal.

Parameters:
tag : ColumnTag

Identifier for a timespan column.

timespan : Timespan

Timespan literal selected rows must overlap.

Returns:
predicate : lsst.daf.relation.column_expressions.Predicate

New predicate.

materialize(base: lsst.daf.relation._relation.Relation, name: str) → Any

Hook for implementing materialization operations.

This method should be called only by the Processor base class.

Parameters:
target : Relation

Relation to be materialized. Any upstream Transfer operations in this tree are guaranteed to already have a payload already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.

name : str

The name of the Materialization operation, to be used as needed in the engine-specific payload.

Returns:
payload

Payload for this relation that should be cached.

process(relation: Relation) → Relation

Main entry point for processing a relation tree.

Parameters:
relation : Relation

Root of the relation tree to process. On return, relations that hold a Materialization relation will have a new payload attached, if they did not have one already.

Returns:
processed : Relation

A version of the relation tree in which any relation with a Transfer operation has a copy of the original Transfer that has a payload attached.

restore_columns(relation: lsst.daf.relation._relation.Relation, columns_required: collections.abc.Set[lsst.daf.relation._columns._tag.ColumnTag]) → tuple

Return a modified relation tree that attempts to restore columns that were dropped by a projection operation.

Parameters:
relation : Relation

Original relation tree.

columns_required : Set [ ColumnTag ]

Columns to attempt to restore. May include columns already present in the relation.

Returns:
modified : Relation

Possibly-modified tree with any projections that had dropped requested columns replaced by projections that do not drop these columns. Care is taken to ensure that join common columns and deduplication behavior is preserved, even if that means some columns are not restored.

columns_found : set [ ColumnTag ]

Columns from those requested that are present in modified.

strip_postprocessing(relation: lsst.daf.relation._relation.Relation) → tuple

Return a modified relation tree without any iteration-engine operations and any transfer to the iteration engine at the end.

Parameters:
relation : Relation

Original relation tree.

Returns:
modified : Relation

Stripped relation tree, with engine != iteration_engine.

stripped : UnaryOperation

Operations that were stripped, in the same order they should be reapplied (with transfer=True, preferred_engine=iteration_engine) to recover the original tree.

transfer(source: lsst.daf.relation._relation.Relation, destination: lsst.daf.relation._engine.Engine, materialize_as: str | None[str, None]) → Any

Hook for implementing transfers between engines.

This method should be called only by the Processor base class.

Parameters:
source : Relation

Relation to be transferred. Any upstream Transfer operations in this tree are guaranteed to already have a payload already attached (or some intervening relation does), so the relation’s own engine should be capable of processing it on its own.

destination : Engine

Engine the relation is being transferred to.

materialize_as : str or None

If not None, the name of a Materialization operation that immediately follows the transfer being implemented, in which case the returned payload should be appropriate for caching with the Materialization.

Returns:
payload

Payload for this relation in the destination engine.