Query

final class lsst.daf.butler.queries.Query(driver: QueryDriver, tree: QueryTree | None = None)

Bases: QueryBase

A method-chaining builder for butler queries.

Parameters:

driverQueryDriver

Implementation object that knows how to actually execute queries.

treeQueryTree, optional

Description of the query as a tree of joins and column expressions. Defaults to the result of a call to make_identity_query_tree.

Notes

Query objects should never be constructed directly by users; use Butler.query instead.

A Query object represents the first stage of query construction, in which constraints and joins are defined (roughly corresponding to the WHERE and FROM clauses in SQL). The various “results” objects represent the second (and final) stage, where the columns returned are specified and any sorting or integer slicing can be applied. Result objects are obtained from the data_ids, datasets, and dimension_records methods.

Query and query-result objects are always immutable (except for caching information fetched from the database or server), so modifier methods always return a new object without modifying the current one.

Attributes Summary

constraint_dataset_types	The names of all dataset types joined into the query.
constraint_dimensions	Dimensions currently present in the query, either directly or indirectly.
expression_factory	A factory for column expressions using overloaded operators.

Methods Summary

any(*[, execute, exact])	Test whether the query would return any rows.
data_ids([dimensions])	Return a result object that is a DataCoordinate iterable.
datasets(dataset_type[, collections, find_first])	Return a result object that is a DatasetRef iterable.
dimension_records(element)	Return a result object that is a DimensionRecord iterable.
explain_no_results([execute])	Return human-readable messages that may help explain why the query yields no results.
general(dimensions, *names[, ...])	Execute query returning general result.
join_data_coordinates(iterable)	Return a new query that joins in an explicit table of data IDs.
join_dataset_search(dataset_type[, collections])	Return a new query with a search for a dataset joined in.
join_dimensions(dimensions)	Return a new query that joins the logical tables for additional dimensions.
materialize(*[, dimensions, datasets])	Execute the query, save its results to a temporary location, and return a new query that represents fetching or joining against those saved results.
where(*args[, bind])	Return a query with a boolean-expression filter on its rows.

Attributes Documentation

constraint_dataset_types

The names of all dataset types joined into the query.

The existence of datasets of these types constrains the data IDs of any type of result. Fields for these dataset types are also usable in ‘where’ expressions.

constraint_dimensions

Dimensions currently present in the query, either directly or indirectly.

This includes dimensions that are present in any joined subquery (such as a dataset search, materialization, or data ID upload) or where argument, as well as any required or implied dependency of those dimensions.

expression_factory

A factory for column expressions using overloaded operators. (ExpressionFactory).

Notes

Typically this attribute will be assigned to a single-character local variable, and then its (dynamic) attributes can be used to obtain references to columns that can be included in a query:

with butler.query() as query:
    x = query.expression_factory
    query = query.where(
        x.instrument == "LSSTCam",
        x.visit.day_obs > 20240701,
        x.any(x.band == "u", x.band == "y"),
    )

As shown above, the returned object also has an any method to create combine expressions with logical OR (as well as not_ and all, though the latter is rarely necessary since where already combines its arguments with AND).

Proxies for fields associated with individual datasets but not dimension records (dataset_id, ingest_date, run, collection, as well as timespan for CALIBRATION collection searches) can be obtained with dict-like access instead:

with butler.query() as query:
    query = query.order_by(x["raw"].ingest_date)

Expression proxy objects that correspond to scalar columns overload the standard comparison operators (==, !=, <, >, <=, >=) and provide in_range, in_iterable, and in_query methods for membership tests. For order_by contexts, they also have a desc property to indicate that the sort order for that expression should be reversed.

Proxy objects for region and timespan fields have an overlaps method, and timespans also have begin and end properties to access scalar expression proxies for the bounds.

All proxy objects also have an is_null property.

Literal values can be created by calling ExpressionFactory.literal, but can almost always be created implicitly via overloaded operators instead.

Methods Documentation

any(*, execute: bool = True, exact: bool = True) → bool

Test whether the query would return any rows.

Parameters:

executebool, optional

If True, execute at least a LIMIT 1 query if it cannot be determined prior to execution that the query would return no rows.

exactbool, optional

If True, run the full query and perform post-query filtering if needed, until at least one result row is found. If False, the returned result does not account for post-query filtering, and hence may be True even when all result rows would be filtered out.

Returns:

anybool

True if the query would (or might, depending on arguments) yield result rows. False if it definitely would not.

data_ids(dimensions: DimensionGroup | Iterable[str] | str | None = None) → DataCoordinateQueryResults

Return a result object that is a DataCoordinate iterable.

Parameters:

dimensionsDimensionGroup, str, or Iterable [str], optional

The dimensions of the data IDs to yield, as either DimensionGroup instances or str names. Will be automatically expanded to a complete DimensionGroup. These dimensions do not need to match the query’s current dimensions. Default is constraint_dimensions.

Returns:

data_idsDataCoordinateQueryResults

Data IDs matching the given query parameters. These are guaranteed to identify all dimensions (DataCoordinate.hasFull returns True), but will not contain DimensionRecord objects (DataCoordinate.hasRecords returns False). Call with_dimension_records on the returned object to include dimension records as well.

datasets(dataset_type: str | DatasetType, collections: str | Iterable[str] | None = None, *, find_first: bool = True) → DatasetRefQueryResults

Return a result object that is a DatasetRef iterable.

Parameters:

dataset_typestr or DatasetType

The dataset type to search for.

collectionsstr or Iterable [ str ], optional

The collection or collections to search, in order. If not provided or None, and the dataset has not already been joined into the query, the default collection search path for this butler is used.

find_firstbool, optional

If True (default), for each result data ID, only yield one DatasetRef of each DatasetType, from the first collection in which a dataset of that dataset type appears (according to the order of collections passed in). If True, collections must not be ....

Returns:

refslsst.daf.butler.queries.DatasetRefQueryResults

Dataset references matching the given query criteria. Nested data IDs are guaranteed to include values for all implied dimensions (i.e. DataCoordinate.hasFull will return True), but will not include dimension records (DataCoordinate.hasRecords will be False) unless with_dimension_records is called on the result object (which returns a new one).

Raises:

lsst.daf.butler.registry.DatasetTypeExpressionError

Raised when the dataset_type expression is invalid.

lsst.daf.butler.registry.NoDefaultCollectionError

Raised when collections is None and default butler collections are not defined.

TypeError

Raised when the arguments are incompatible, such as when a collection wildcard is passed when find_first is True

dimension_records(element: str) → DimensionRecordQueryResults

Return a result object that is a DimensionRecord iterable.

Parameters:

elementstr

The name of a dimension element to obtain records for.

Returns:

recordslsst.daf.butler.queries.DimensionRecordQueryResults

Data IDs matching the given query parameters.

explain_no_results(execute: bool = True) → Iterable[str]

Return human-readable messages that may help explain why the query yields no results.

Parameters:

executebool, optional

If True (default) execute simplified versions (e.g. LIMIT 1) of aspects of the tree to more precisely determine where rows were filtered out.

Returns:

messagesIterable [ str ]

String messages that describe reasons the query might not yield any results.

general(dimensions: DimensionGroup | Iterable[str], *names: str, dimension_fields: Mapping[str, Set[str]] | None = None, dataset_fields: Mapping[str, Set[Literal['dataset_id', 'ingest_date', 'run', 'collection', 'timespan']] | ellipsis] | None = None, find_first: bool | None = None) → GeneralQueryResults

Execute query returning general result.

This is an experimental interface and may change at any time.

Parameters:

dimensionsDimensionGroup or Iterable [ str ]

The dimensions that span all fields returned by this query.

*namesstr

Names of dimensions fields (in “dimension.field” format), dataset fields (in “dataset_type.field” format) to include in this query.

dimension_fieldsMapping [str, Set [str]], optional

Dimension record fields included in this query, keyed by dimension element name.

dataset_fieldsMapping [str, Set | ... ], optional

Dataset fields included in this query, the key in the mapping is dataset type name. Ellipsis (...) can be used for value to include all dataset fields needed to extract DatasetRef instances later.

find_firstbool, optional

Whether this query requires find-first resolution for a dataset. This is ignored and can be omitted if the query has no dataset fields. It must be explicitly set to False if there are multiple dataset types with fields, or if any dataset’s collections or timespan fields are included in the results.

Returns:

resultGeneralQueryResults

Query result that can be iterated over.

Notes

The dimensions of the returned query are automatically expanded to include those associated with all dimension and dataset fields; the dimensions argument is just the minimal dimensions to return.

join_data_coordinates(iterable: Iterable[DataCoordinate]) → Query

Return a new query that joins in an explicit table of data IDs.

Parameters:

iterableIterable [DataCoordinate]

Iterable of DataCoordinate. All items must have the same dimensions. Must have at least one item.

Returns:

queryQuery

A new query object with the data IDs joined in.

join_dataset_search(dataset_type: str | DatasetType, collections: Iterable[str] | None = None) → Query

Return a new query with a search for a dataset joined in.

Parameters:

dataset_typestr or DatasetType

Dataset type or name. May not refer to a dataset component.

collectionsIterable [ str ], optional

Iterable of collections to search. Order is preserved, but will not matter if the dataset search is only used as a constraint on dimensions or if find_first=False when requesting results. If not present or None, the default collection search path will be used.

Returns:

queryQuery

A new query object with dataset columns available and rows restricted to those consistent with the found data IDs.

Raises:

DatasetTypeError

Raised if given dataset type is inconsistent with the registered dataset type.

MissingDatasetTypeError

Raised if the dataset type has not been registered and only a str dataset type name was given.

Notes

This method may require communication with the server unless the dataset type and collections have already been referenced by the same query context.

join_dimensions(dimensions: Iterable[str] | DimensionGroup) → Query

Return a new query that joins the logical tables for additional dimensions.

Parameters:

dimensionsIterable [ str ] or DimensionGroup

Names of dimensions to join in.

Returns:

queryQuery

A new query object with the dimensions joined in.

Notes

Dimensions are automatically joined in whenever needed, so this method should rarely need to be called directly.

materialize(*, dimensions: Iterable[str] | DimensionGroup | None = None, datasets: Iterable[str] | None = None) → Query

Execute the query, save its results to a temporary location, and return a new query that represents fetching or joining against those saved results.

Parameters:

dimensionsIterable [ str ] or DimensionGroup, optional

Dimensions to include in the temporary results. Default is to include all dimensions in the query.

datasetsIterable [ str ], optional

Names of dataset types that should be included in the new query; default is to include constraint_dataset_types.

Returns:

queryQuery

A new query object whose that represents the materialized rows.

Notes

Only dimension key columns and (at the discretion of the implementation) certain dataset columns are actually materialized, since at this stage we do not know which dataset or dimension record fields are actually needed in result rows, and these can be joined back in on the materialized dimension keys. But all constraints on those dimension keys (including dataset existence) are applied to the materialized rows.

where(*args: str | Predicate | DataCoordinate | Mapping[str, Any], bind: Mapping[str, Any] | None = None, **kwargs: Any) → Query

Return a query with a boolean-expression filter on its rows.

Parameters:

*args

Constraints to apply, combined with logical AND. Arguments may be str expressions to parse, Predicate objects (these are typically constructed via expression_factory) or data IDs.

bindMapping

Mapping from string identifier appearing in a string expression to a literal value that should be substituted for it. This is recommended instead of embedding literals directly into the expression, especially for strings, timespans, or other types where quoting or formatting is nontrivial.

**kwargs

Data ID key value pairs that extend and override any present in *args.

Returns:

queryQuery

A new query object with the given row filters (as well as any already present in self). All row filters are combined with logical AND.

Notes

Expressions referring to dimensions or dimension elements are resolved automatically. References to dataset fields (see expression_factory for the distinction) cannot be resolved by default; they must either be preceded by a call to join_dataset_search or must be passed to DatasetRefQueryResults.where instead.

Data ID values are not checked for consistency; they are extracted from args and then kwargs and combined, with later values overriding earlier ones.