Registry¶
-
class
lsst.daf.butler.
Registry
(database: Database, defaults: RegistryDefaults, managers: RegistryManagerInstances)¶ Bases:
object
Registry interface.
Parameters: - database :
Database
Database instance to store Registry.
- defaults :
RegistryDefaults
, optional Default collection search path and/or output
RUN
collection.- attributes :
type
Manager class implementing
ButlerAttributeManager
.- opaque :
type
Manager class implementing
OpaqueTableStorageManager
.- dimensions :
type
Manager class implementing
DimensionRecordStorageManager
.- collections :
type
Manager class implementing
CollectionManager
.- datasets :
type
Manager class implementing
DatasetRecordStorageManager
.- datastoreBridges :
type
Manager class implementing
DatastoreRegistryBridgeManager
.- dimensionConfig :
DimensionConfig
, optional Dimension universe configuration, only used when
create
is True.- writeable :
bool
, optional If True then Registry will support write operations.
- create :
bool
, optional If True then database schema will be initialized, it must be empty before instantiating Registry.
Attributes Summary
defaultConfigFile
Path to configuration defaults. defaults
Default collection search path and/or output RUN
collection (RegistryDefaults
).dimensions
All dimensions recognized by this Registry
(DimensionUniverse
).Methods Summary
associate
(collection, refs)Add existing datasets to a TAGGED
collection.certify
(collection, refs, timespan)Associate one or more datasets with a calibration collection and a validity range within it. copy
(defaults)Create a new Registry
backed by the same data repository and connection as this one, but independent defaults.createFromConfig
(config, str, None] = None, …)Create registry database and return Registry
instance.decertify
(collection, datasetType, …)Remove or adjust datasets to clear a validity range within a calibration collection. deleteOpaqueData
(tableName, **where)Remove records from an opaque table. disassociate
(collection, refs)Remove existing datasets from a TAGGED
collection.expandDataId
(dataId, Mapping[str, Any], …)Expand a dimension-based data ID to include additional information. fetchOpaqueData
(tableName, **where)Retrieve records from an opaque table. findDataset
(datasetType, str], dataId, …)Find a dataset given its DatasetType
and data ID.fromConfig
(config, RegistryConfig, Config, …)Create Registry
subclass instance fromconfig
.getCollectionChain
(parent)Return the child collections in a CHAINED
collection.getCollectionDocumentation
(collection)Retrieve the documentation string for a collection. getCollectionSummary
(collection)Return a summary for the given collection. getCollectionType
(name)Return an enumeration value indicating the type of the given collection. getDataset
(id)Retrieve a Dataset entry. getDatasetLocations
(ref)Retrieve datastore locations for a given dataset. getDatasetType
(name)Get the DatasetType
.getDatastoreBridgeManager
()Return an object that allows a new Datastore
instance to communicate with thisRegistry
.insertDatasets
(datasetType, str], dataIds, …)Insert one or more datasets into the Registry
insertDimensionData
(element, str], *data, …)Insert one or more dimension records into the database. insertOpaqueData
(tableName, *data)Insert records into an opaque table. isWriteable
()Return True
if this registry allows write operations, andFalse
otherwise.makeQueryBuilder
(summary)Return a QueryBuilder
instance capable of constructing and managing more complex queries than those obtainable viaRegistry
interfaces.queryCollections
(expression, datasetType, …)Iterate over the collections whose names match an expression. queryDataIds
(dimensions, str]], …)Query for data IDs matching user-provided criteria. queryDatasetAssociations
(datasetType, …)Iterate over dataset-collection combinations where the dataset is in the collection. queryDatasetTypes
(expression, *, components)Iterate over the dataset types whose names match an expression. queryDatasets
(datasetType, *, collections, …)Query for and iterate over dataset references matching user-provided criteria. queryDimensionRecords
(element, str], *, …)Query for dimension information matching user-provided criteria. refresh
()Refresh all in-memory state by querying the database. registerCollection
(name, type, doc)Add a new collection if one with the given name does not exist. registerDatasetType
(datasetType)Add a new DatasetType
to the Registry.registerOpaqueTable
(tableName, spec)Add an opaque (to the Registry
) table for use by aDatastore
or other data repository client.registerRun
(name, doc)Add a new run if one with the given name does not exist. removeCollection
(name)Completely remove the given collection. removeDatasetType
(name)Remove the named DatasetType
from the registry.removeDatasets
(refs)Remove datasets from the Registry. setCollectionChain
(parent, children)Define or redefine a CHAINED
collection.setCollectionDocumentation
(collection, doc)Set the documentation string for a collection. syncDimensionData
(element, str], row, Any], …)Synchronize the given dimension record with the database, inserting if it does not already exist and comparing values if it does. transaction
(*, savepoint)Return a context manager that represents a transaction. Attributes Documentation
-
defaultConfigFile
= None¶ Path to configuration defaults. Accessed within the
configs
resource or relative to a search path. Can be None if no defaults specified.
-
defaults
¶ Default collection search path and/or output
RUN
collection (RegistryDefaults
).This is an immutable struct whose components may not be set individually, but the entire struct can be set by assigning to this property.
-
dimensions
¶ All dimensions recognized by this
Registry
(DimensionUniverse
).
Methods Documentation
-
associate
(collection: str, refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]) → None¶ Add existing datasets to a
TAGGED
collection.If a DatasetRef with the same exact integer ID is already in a collection nothing is changed. If a
DatasetRef
with the sameDatasetType
and data ID but with different integer ID exists in the collection,ConflictingDefinitionError
is raised.Parameters: - collection :
str
Indicates the collection the datasets should be associated with.
- refs :
Iterable
[DatasetRef
] An iterable of resolved
DatasetRef
instances that already exist in thisRegistry
.
Raises: - ConflictingDefinitionError
If a Dataset with the given
DatasetRef
already exists in the given collection.- AmbiguousDatasetError
Raised if
any(ref.id is None for ref in refs)
.- MissingCollectionError
Raised if
collection
does not exist in the registry.- TypeError
Raise adding new datasets to the given
collection
is not allowed.
- collection :
-
certify
(collection: str, refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef], timespan: lsst.daf.butler.core.timespan.Timespan) → None¶ Associate one or more datasets with a calibration collection and a validity range within it.
Parameters: - collection :
str
The name of an already-registered
CALIBRATION
collection.- refs :
Iterable
[DatasetRef
] Datasets to be associated.
- timespan :
Timespan
The validity range for these datasets within the collection.
Raises: - AmbiguousDatasetError
Raised if any of the given
DatasetRef
instances is unresolved.- ConflictingDefinitionError
Raised if the collection already contains a different dataset with the same
DatasetType
and data ID and an overlapping validity range.- TypeError
Raised if
collection
is not aCALIBRATION
collection or if one or more datasets are of a dataset type for whichDatasetType.isCalibration
returnsFalse
.
- collection :
-
copy
(defaults: Optional[lsst.daf.butler.registry._defaults.RegistryDefaults] = None) → lsst.daf.butler.registry._registry.Registry¶ Create a new
Registry
backed by the same data repository and connection as this one, but independent defaults.Parameters: - defaults :
RegistryDefaults
, optional Default collections and data ID values for the new registry. If not provided,
self.defaults
will be used (but future changes to either registry’s defaults will not affect the other).
Returns: Notes
Because the new registry shares a connection with the original, they also share transaction state (despite the fact that their
transaction
context manager methods do not reflect this), and must be used with care.- defaults :
-
classmethod
createFromConfig
(config: Union[lsst.daf.butler.registry._config.RegistryConfig, str, None] = None, dimensionConfig: Union[lsst.daf.butler.core.dimensions._config.DimensionConfig, str, None] = None, butlerRoot: Optional[str] = None) → lsst.daf.butler.registry._registry.Registry¶ Create registry database and return
Registry
instance.This method initializes database contents, database must be empty prior to calling this method.
Parameters: - config :
RegistryConfig
orstr
, optional Registry configuration, if missing then default configuration will be loaded from registry.yaml.
- dimensionConfig :
DimensionConfig
orstr
, optional Dimensions configuration, if missing then default configuration will be loaded from dimensions.yaml.
- butlerRoot :
str
, optional Path to the repository root this
Registry
will manage.
Returns: - config :
-
decertify
(collection: str, datasetType: Union[str, lsst.daf.butler.core.datasets.type.DatasetType], timespan: lsst.daf.butler.core.timespan.Timespan, *, dataIds: Optional[Iterable[Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any]]]] = None) → None¶ Remove or adjust datasets to clear a validity range within a calibration collection.
Parameters: - collection :
str
The name of an already-registered
CALIBRATION
collection.- datasetType :
str
orDatasetType
Name or
DatasetType
instance for the datasets to be decertified.- timespan :
Timespan
, optional The validity range to remove datasets from within the collection. Datasets that overlap this range but are not contained by it will have their validity ranges adjusted to not overlap it, which may split a single dataset validity range into two.
- dataIds :
Iterable
[DataId
], optional Data IDs that should be decertified within the given validity range If
None
, all data IDs forself.datasetType
will be decertified.
Raises: - TypeError
Raised if
collection
is not aCALIBRATION
collection or ifdatasetType.isCalibration() is False
.
- collection :
-
deleteOpaqueData
(tableName: str, **where) → None¶ Remove records from an opaque table.
Parameters: - tableName :
str
Logical name of the opaque table. Must match the name used in a previous call to
registerOpaqueTable
.- where
Additional keyword arguments are interpreted as equality constraints that restrict the deleted rows (combined with AND); keyword arguments are column names and values are the values they must have.
- tableName :
-
disassociate
(collection: str, refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]) → None¶ Remove existing datasets from a
TAGGED
collection.collection
andref
combinations that are not currently associated are silently ignored.Parameters: - collection :
str
The collection the datasets should no longer be associated with.
- refs :
Iterable
[DatasetRef
] An iterable of resolved
DatasetRef
instances that already exist in thisRegistry
.
Raises: - AmbiguousDatasetError
Raised if any of the given dataset references is unresolved.
- MissingCollectionError
Raised if
collection
does not exist in the registry.- TypeError
Raise adding new datasets to the given
collection
is not allowed.
- collection :
-
expandDataId
(dataId: Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, graph: Optional[lsst.daf.butler.core.dimensions._graph.DimensionGraph] = None, records: Union[lsst.daf.butler.core.named.NamedKeyMapping[lsst.daf.butler.core.dimensions._elements.DimensionElement, typing.Union[lsst.daf.butler.core.dimensions._records.DimensionRecord, NoneType]][lsst.daf.butler.core.dimensions._elements.DimensionElement, Optional[lsst.daf.butler.core.dimensions._records.DimensionRecord]], Mapping[str, Optional[lsst.daf.butler.core.dimensions._records.DimensionRecord]], None] = None, withDefaults: bool = True, **kwargs) → lsst.daf.butler.core.dimensions._coordinate.DataCoordinate¶ Expand a dimension-based data ID to include additional information.
Parameters: - dataId :
DataCoordinate
ordict
, optional Data ID to be expanded; augmented and overridden by
kwds
.- graph :
DimensionGraph
, optional Set of dimensions for the expanded ID. If
None
, the dimensions will be inferred from the keys ofdataId
andkwds
. Dimensions that are indataId
orkwds
but not ingraph
are silently ignored, providing a way to extract and expand a subset of a data ID.- records :
Mapping
[str
,DimensionRecord
], optional Dimension record data to use before querying the database for that data, keyed by element name.
- withDefaults :
bool
, optional Utilize
self.defaults.dataId
to fill in missing governor dimension key-value pairs. Defaults toTrue
(i.e. defaults are used).- **kwargs
Additional keywords are treated like additional key-value pairs for
dataId
, extending and overriding
Returns: - expanded :
DataCoordinate
A data ID that includes full metadata for all of the dimensions it identifieds, i.e. guarantees that
expanded.hasRecords()
andexpanded.hasFull()
both returnTrue
.
- dataId :
-
fetchOpaqueData
(tableName: str, **where) → Iterator[dict]¶ Retrieve records from an opaque table.
Parameters: - tableName :
str
Logical name of the opaque table. Must match the name used in a previous call to
registerOpaqueTable
.- where
Additional keyword arguments are interpreted as equality constraints that restrict the returned rows (combined with AND); keyword arguments are column names and values are the values they must have.
Yields: - row :
dict
A dictionary representing a single result row.
- tableName :
-
findDataset
(datasetType: Union[lsst.daf.butler.core.datasets.type.DatasetType, str], dataId: Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any], None] = None, *, collections: Optional[Any] = None, timespan: Optional[lsst.daf.butler.core.timespan.Timespan] = None, **kwargs) → Optional[lsst.daf.butler.core.datasets.ref.DatasetRef]¶ Find a dataset given its
DatasetType
and data ID.This can be used to obtain a
DatasetRef
that permits the dataset to be read from aDatastore
. If the dataset is a component and can not be found using the provided dataset type, a dataset ref for the parent will be returned instead but with the correct dataset type.Parameters: - datasetType :
DatasetType
orstr
A
DatasetType
or the name of one.- dataId :
dict
orDataCoordinate
, optional A
dict
-like object containing theDimension
links that identify the dataset within a collection.- collections, optional.
An expression that fully or partially identifies the collections to search for the dataset; see Collection expressions for more information. Defaults to
self.defaults.collections
.- timespan :
Timespan
, optional A timespan that the validity range of the dataset must overlap. If not provided, any
CALIBRATION
collections matched by thecollections
argument will not be searched.- **kwargs
Additional keyword arguments passed to
DataCoordinate.standardize
to convertdataId
to a trueDataCoordinate
or augment an existing one.
Returns: - ref :
DatasetRef
A reference to the dataset, or
None
if no matching Dataset was found.
Raises: Notes
This method simply returns
None
and does not raise an exception even when the set of collections searched is intrinsically incompatible with the dataset type, e.g. ifdatasetType.isCalibration() is False
, but onlyCALIBRATION
collections are being searched. This may make it harder to debug some lookup failures, but the behavior is intentional; we consider it more important that failed searches are reported consistently, regardless of the reason, and that adding additional collections that do not contain a match to the search path never changes the behavior.- datasetType :
-
classmethod
fromConfig
(config: Union[ButlerConfig, RegistryConfig, Config, str], butlerRoot: Optional[Union[str, ButlerURI]] = None, writeable: bool = True, defaults: Optional[RegistryDefaults] = None) → Registry¶ Create
Registry
subclass instance fromconfig
.Registry database must be inbitialized prior to calling this method.
Parameters: - config :
ButlerConfig
,RegistryConfig
,Config
orstr
Registry configuration
- butlerRoot :
str
orButlerURI
, optional Path to the repository root this
Registry
will manage.- writeable :
bool
, optional If
True
(default) create a read-write connection to the database.- defaults :
RegistryDefaults
, optional Default collection search path and/or output
RUN
collection.
Returns: - config :
-
getCollectionChain
(parent: str) → lsst.daf.butler.registry.wildcards.CollectionSearch¶ Return the child collections in a
CHAINED
collection.Parameters: - parent :
str
Name of the chained collection. Must have already been added via a call to
Registry.registerCollection
.
Returns: - children :
CollectionSearch
An object that defines the search path of the collection. See Collection expressions for more information.
Raises: - parent :
-
getCollectionDocumentation
(collection: str) → Optional[str]¶ Retrieve the documentation string for a collection.
Parameters: - name :
str
Name of the collection.
Returns: - name :
-
getCollectionSummary
(collection: str) → lsst.daf.butler.registry.summaries.CollectionSummary¶ Return a summary for the given collection.
Parameters: - collection :
str
Name of the collection for which a summary is to be retrieved.
Returns: - summary :
CollectionSummary
Summary of the dataset types and governor dimension values in this collection.
- collection :
-
getCollectionType
(name: str) → lsst.daf.butler.registry._collectionType.CollectionType¶ Return an enumeration value indicating the type of the given collection.
Parameters: - name :
str
The name of the collection.
Returns: - type :
CollectionType
Enum value indicating the type of this collection.
Raises: - MissingCollectionError
Raised if no collection with the given name exists.
- name :
-
getDataset
(id: int) → Optional[lsst.daf.butler.core.datasets.ref.DatasetRef]¶ Retrieve a Dataset entry.
Parameters: - id :
int
The unique identifier for the dataset.
Returns: - ref :
DatasetRef
orNone
A ref to the Dataset, or
None
if no matching Dataset was found.
- id :
-
getDatasetLocations
(ref: lsst.daf.butler.core.datasets.ref.DatasetRef) → Iterable[str]¶ Retrieve datastore locations for a given dataset.
Parameters: - ref :
DatasetRef
A reference to the dataset for which to retrieve storage information.
Returns: - datastores :
Iterable
[str
] All the matching datastores holding this dataset.
Raises: - AmbiguousDatasetError
Raised if
ref.id
isNone
.
- ref :
-
getDatasetType
(name: str) → lsst.daf.butler.core.datasets.type.DatasetType¶ Get the
DatasetType
.Parameters: - name :
str
Name of the type.
Returns: - type :
DatasetType
The
DatasetType
associated with the given name.
Raises: - KeyError
Requested named DatasetType could not be found in registry.
- name :
-
getDatastoreBridgeManager
() → DatastoreRegistryBridgeManager¶ Return an object that allows a new
Datastore
instance to communicate with thisRegistry
.Returns: - manager :
DatastoreRegistryBridgeManager
Object that mediates communication between this
Registry
and its associated datastores.
- manager :
-
insertDatasets
(datasetType: Union[lsst.daf.butler.core.datasets.type.DatasetType, str], dataIds: Iterable[Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any]]], run: Optional[str] = None) → List[lsst.daf.butler.core.datasets.ref.DatasetRef]¶ Insert one or more datasets into the
Registry
This always adds new datasets; to associate existing datasets with a new collection, use
associate
.Parameters: - datasetType :
DatasetType
orstr
A
DatasetType
or the name of one.- dataIds :
Iterable
ofdict
orDataCoordinate
Dimension-based identifiers for the new datasets.
- run :
str
, optional The name of the run that produced the datasets. Defaults to
self.defaults.run
.
Returns: - refs :
list
ofDatasetRef
Resolved
DatasetRef
instances for all given data IDs (in the same order).
Raises: - datasetType :
-
insertDimensionData
(element: Union[lsst.daf.butler.core.dimensions._elements.DimensionElement, str], *data, conform: bool = True) → None¶ Insert one or more dimension records into the database.
Parameters: - element :
DimensionElement
orstr
The
DimensionElement
or name thereof that identifies the table records will be inserted into.- data :
dict
orDimensionRecord
(variadic) One or more records to insert.
- conform :
bool
, optional If
False
(True
is default) perform no checking or conversions, and assume thatelement
is aDimensionElement
instance anddata
is a one or moreDimensionRecord
instances of the appropriate subclass.
- element :
-
insertOpaqueData
(tableName: str, *data) → None¶ Insert records into an opaque table.
Parameters: - tableName :
str
Logical name of the opaque table. Must match the name used in a previous call to
registerOpaqueTable
.- data
Each additional positional argument is a dictionary that represents a single row to be added.
- tableName :
-
makeQueryBuilder
(summary: lsst.daf.butler.registry.queries._structs.QuerySummary) → lsst.daf.butler.registry.queries._builder.QueryBuilder¶ Return a
QueryBuilder
instance capable of constructing and managing more complex queries than those obtainable viaRegistry
interfaces.This is an advanced interface; downstream code should prefer
Registry.queryDataIds
andRegistry.queryDatasets
whenever those are sufficient.Parameters: - summary :
queries.QuerySummary
Object describing and categorizing the full set of dimensions that will be included in the query.
Returns: - builder :
queries.QueryBuilder
Object that can be used to construct and perform advanced queries.
- summary :
-
queryCollections
(expression: Any = Ellipsis, datasetType: Optional[lsst.daf.butler.core.datasets.type.DatasetType] = None, collectionTypes: Iterable[lsst.daf.butler.registry._collectionType.CollectionType] = frozenset({<CollectionType.RUN: 1>, <CollectionType.TAGGED: 2>, <CollectionType.CHAINED: 3>, <CollectionType.CALIBRATION: 4>}), flattenChains: bool = False, includeChains: Optional[bool] = None) → Iterator[str]¶ Iterate over the collections whose names match an expression.
Parameters: - expression :
Any
, optional An expression that fully or partially identifies the collections to return, such as a
str
,re.Pattern
, or iterable thereof.can be used to return all collections, and is the default. See Collection expressions for more information.
- datasetType :
DatasetType
, optional If provided, only yield collections that may contain datasets of this type. This is a conservative approximation in general; it may yield collections that do not have any such datasets.
- collectionTypes :
AbstractSet
[CollectionType
], optional If provided, only yield collections of these types.
- flattenChains :
bool
, optional If
True
(False
is default), recursively yield the child collections of matchingCHAINED
collections.- includeChains :
bool
, optional If
True
, yield records for matchingCHAINED
collections. Default is the opposite offlattenChains
: include either CHAINED collections or their children, but not both.
Yields: - collection :
str
The name of a collection that matches
expression
.
- expression :
-
queryDataIds
(dimensions: Union[Iterable[Union[lsst.daf.butler.core.dimensions._elements.Dimension, str]], lsst.daf.butler.core.dimensions._elements.Dimension, str], *, dataId: Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any], None] = None, datasets: Optional[Any] = None, collections: Optional[Any] = None, where: Optional[str] = None, components: Optional[bool] = None, bind: Optional[Mapping[str, Any]] = None, check: bool = True, **kwargs) → lsst.daf.butler.registry.queries._results.DataCoordinateQueryResults¶ Query for data IDs matching user-provided criteria.
Parameters: - dimensions :
Dimension
orstr
, or iterable thereof The dimensions of the data IDs to yield, as either
Dimension
instances orstr
. Will be automatically expanded to a completeDimensionGraph
.- dataId :
dict
orDataCoordinate
, optional A data ID whose key-value pairs are used as equality constraints in the query.
- datasets :
Any
, optional An expression that fully or partially identifies dataset types that should constrain the yielded data IDs. For example, including “raw” here would constrain the yielded
instrument
,exposure
,detector
, andphysical_filter
values to only those for which at least one “raw” dataset exists incollections
. Allowed types includeDatasetType
,str
,re.Pattern
, and iterables thereof. Unlike other dataset type expressions,...
is not permitted - it doesn’t make sense to constrain data IDs on the existence of all datasets. See DatasetType expressions for more information.- collections: `Any`, optional
An expression that fully or partially identifies the collections to search for datasets, such as a
str
,re.Pattern
, or iterable thereof.can be used to return all collections. Must be provided if
datasets
is, and is ignored if it is not. See Collection expressions for more information. If not provided,self.default.collections
is used.- where :
str
, optional A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or (as a shortcut for the primary key column of a dimension table) dimension name. See Dimension expressions for more information.
- components :
bool
, optional If
True
, apply all dataset expression patterns to component dataset type names as well. IfFalse
, never apply patterns to components. IfNone
(default), apply patterns to components only if their parent datasets were not matched by the expression. Fully-specified component datasets (str
orDatasetType
instances) are always included.- bind :
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- check :
bool
, optional If
True
(default) check the query for consistency before executing it. This may reject some valid queries that resemble common mistakes (e.g. queries for visits without specifying an instrument).- **kwargs
Additional keyword arguments are forwarded to
DataCoordinate.standardize
when processing thedataId
argument (and may be used to provide a constraining data ID even when thedataId
argument isNone
).
Returns: - dataIds :
DataCoordinateQueryResults
Data IDs matching the given query parameters. These are guaranteed to identify all dimensions (
DataCoordinate.hasFull
returnsTrue
), but will not containDimensionRecord
objects (DataCoordinate.hasRecords
returnsFalse
). CallDataCoordinateQueryResults.expanded
on the returned object to fetch those (and consider usingDataCoordinateQueryResults.materialize
on the returned object first if the expected number of rows is very large). See documentation for those methods for additional information.
Raises: - dimensions :
-
queryDatasetAssociations
(datasetType: Union[str, lsst.daf.butler.core.datasets.type.DatasetType], collections: Any = Ellipsis, *, collectionTypes: Iterable[lsst.daf.butler.registry._collectionType.CollectionType] = frozenset({<CollectionType.RUN: 1>, <CollectionType.TAGGED: 2>, <CollectionType.CHAINED: 3>, <CollectionType.CALIBRATION: 4>}), flattenChains: bool = False) → Iterator[lsst.daf.butler.core.datasets.association.DatasetAssociation]¶ Iterate over dataset-collection combinations where the dataset is in the collection.
This method is a temporary placeholder for better support for assocation results in
queryDatasets
. It will probably be removed in the future, and should be avoided in production code whenever possible.Parameters: - datasetType :
DatasetType
orstr
A dataset type object or the name of one.
- collections: `Any`, optional
An expression that fully or partially identifies the collections to search for datasets. See
queryCollections
and Collection expressions for more information. If not provided,self.default.collections
is used.- collectionTypes :
AbstractSet
[CollectionType
], optional If provided, only yield associations from collections of these types.
- flattenChains :
bool
, optional If
True
(default) search in the children ofCHAINED
collections. IfFalse
,CHAINED
collections are ignored.
Yields: - association :
DatasetAssociation
Object representing the relationship beween a single dataset and a single collection.
Raises: - datasetType :
-
queryDatasetTypes
(expression: Any = Ellipsis, *, components: Optional[bool] = None) → Iterator[lsst.daf.butler.core.datasets.type.DatasetType]¶ Iterate over the dataset types whose names match an expression.
Parameters: - expression :
Any
, optional An expression that fully or partially identifies the dataset types to return, such as a
str
,re.Pattern
, or iterable thereof.can be used to return all dataset types, and is the default. See DatasetType expressions for more information.
- components :
bool
, optional If
True
, apply all expression patterns to component dataset type names as well. IfFalse
, never apply patterns to components. IfNone
(default), apply patterns to components only if their parent datasets were not matched by the expression. Fully-specified component datasets (str
orDatasetType
instances) are always included.
Yields: - datasetType :
DatasetType
A
DatasetType
instance whose name matchesexpression
.
- expression :
-
queryDatasets
(datasetType: Any, *, collections: Optional[Any] = None, dimensions: Optional[Iterable[Union[lsst.daf.butler.core.dimensions._elements.Dimension, str]]] = None, dataId: Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any], None] = None, where: Optional[str] = None, findFirst: bool = False, components: Optional[bool] = None, bind: Optional[Mapping[str, Any]] = None, check: bool = True, **kwargs) → lsst.daf.butler.registry.queries._results.DatasetQueryResults¶ Query for and iterate over dataset references matching user-provided criteria.
Parameters: - datasetType
An expression that fully or partially identifies the dataset types to be queried. Allowed types include
DatasetType
,str
,re.Pattern
, and iterables thereof. The special valuecan be used to query all dataset types. See DatasetType expressions for more information.
- collections: optional
An expression that fully or partially identifies the collections to search for datasets, such as a
str
,re.Pattern
, or iterable thereof.can be used to find datasets from all
RUN
collections (no other collections are necessary, because all datasets are in aRUN
collection). See Collection expressions for more information. If not provided,self.default.collections
is used.- dimensions :
Iterable
ofDimension
orstr
Dimensions to include in the query (in addition to those used to identify the queried dataset type(s)), either to constrain the resulting datasets to those for which a matching dimension exists, or to relate the dataset type’s dimensions to dimensions referenced by the
dataId
orwhere
arguments.- dataId :
dict
orDataCoordinate
, optional A data ID whose key-value pairs are used as equality constraints in the query.
- where :
str
, optional A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or (as a shortcut for the primary key column of a dimension table) dimension name. See Dimension expressions for more information.
- findFirst :
bool
, optional If
True
(False
is default), for each result data ID, only yield oneDatasetRef
of eachDatasetType
, from the first collection in which a dataset of that dataset type appears (according to the order ofcollections
passed in). IfTrue
,collections
must not contain regular expressions and may not be.
- components :
bool
, optional If
True
, apply all dataset expression patterns to component dataset type names as well. IfFalse
, never apply patterns to components. IfNone
(default), apply patterns to components only if their parent datasets were not matched by the expression. Fully-specified component datasets (str
orDatasetType
instances) are always included.- bind :
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- check :
bool
, optional If
True
(default) check the query for consistency before executing it. This may reject some valid queries that resemble common mistakes (e.g. queries for visits without specifying an instrument).- **kwargs
Additional keyword arguments are forwarded to
DataCoordinate.standardize
when processing thedataId
argument (and may be used to provide a constraining data ID even when thedataId
argument isNone
).
Returns: - refs :
queries.DatasetQueryResults
Dataset references matching the given query criteria.
Raises: Notes
When multiple dataset types are queried in a single call, the results of this operation are equivalent to querying for each dataset type separately in turn, and no information about the relationships between datasets of different types is included. In contexts where that kind of information is important, the recommended pattern is to use
queryDataIds
to first obtain data IDs (possibly with the desired dataset types and collections passed as constraints to the query), and then use multiple (generally much simpler) calls toqueryDatasets
with the returned data IDs passed as constraints.
-
queryDimensionRecords
(element: Union[lsst.daf.butler.core.dimensions._elements.DimensionElement, str], *, dataId: Union[lsst.daf.butler.core.dimensions._coordinate.DataCoordinate, Mapping[str, Any], None] = None, datasets: Optional[Any] = None, collections: Optional[Any] = None, where: Optional[str] = None, components: Optional[bool] = None, bind: Optional[Mapping[str, Any]] = None, check: bool = True, **kwargs) → Iterator[lsst.daf.butler.core.dimensions._records.DimensionRecord]¶ Query for dimension information matching user-provided criteria.
Parameters: - element :
DimensionElement
orstr
The dimension element to obtain records for.
- dataId :
dict
orDataCoordinate
, optional A data ID whose key-value pairs are used as equality constraints in the query.
- datasets :
Any
, optional An expression that fully or partially identifies dataset types that should constrain the yielded records. See
queryDataIds
and DatasetType expressions for more information.- collections: `Any`, optional
An expression that fully or partially identifies the collections to search for datasets. See
queryDataIds
and Collection expressions for more information.- where :
str
, optional A string expression similar to a SQL WHERE clause. See
queryDataIds
and Dimension expressions for more information.- components :
bool
, optional Whether to apply dataset expressions to components as well. See
queryDataIds
for more information.- bind :
Mapping
, optional Mapping containing literal values that should be injected into the
where
expression, keyed by the identifiers they replace.- check :
bool
, optional If
True
(default) check the query for consistency before executing it. This may reject some valid queries that resemble common mistakes (e.g. queries for visits without specifying an instrument).- **kwargs
Additional keyword arguments are forwarded to
DataCoordinate.standardize
when processing thedataId
argument (and may be used to provide a constraining data ID even when thedataId
argument isNone
).
Returns: - dataIds :
DataCoordinateQueryResults
Data IDs matching the given query parameters.
- element :
-
refresh
() → None¶ Refresh all in-memory state by querying the database.
This may be necessary to enable querying for entities added by other
Registry
instances after this one was constructed.
-
registerCollection
(name: str, type: lsst.daf.butler.registry._collectionType.CollectionType = <CollectionType.TAGGED: 2>, doc: Optional[str] = None) → None¶ Add a new collection if one with the given name does not exist.
Parameters: - name :
str
The name of the collection to create.
- type :
CollectionType
Enum value indicating the type of collection to create.
- doc :
str
, optional Documentation string for the collection.
Notes
This method cannot be called within transactions, as it needs to be able to perform its own transaction to be concurrent.
- name :
-
registerDatasetType
(datasetType: lsst.daf.butler.core.datasets.type.DatasetType) → bool¶ Add a new
DatasetType
to the Registry.It is not an error to register the same
DatasetType
twice.Parameters: - datasetType :
DatasetType
The
DatasetType
to be added.
Returns: Raises: - ValueError
Raised if the dimensions or storage class are invalid.
- ConflictingDefinitionError
Raised if this DatasetType is already registered with a different definition.
Notes
This method cannot be called within transactions, as it needs to be able to perform its own transaction to be concurrent.
- datasetType :
-
registerOpaqueTable
(tableName: str, spec: lsst.daf.butler.core.ddl.TableSpec) → None¶ Add an opaque (to the
Registry
) table for use by aDatastore
or other data repository client.Opaque table records can be added via
insertOpaqueData
, retrieved viafetchOpaqueData
, and removed viadeleteOpaqueData
.Parameters: - tableName :
str
Logical name of the opaque table. This may differ from the actual name used in the database by a prefix and/or suffix.
- spec :
ddl.TableSpec
Specification for the table to be added.
- tableName :
-
registerRun
(name: str, doc: Optional[str] = None) → None¶ Add a new run if one with the given name does not exist.
Parameters: Notes
This method cannot be called within transactions, as it needs to be able to perform its own transaction to be concurrent.
-
removeCollection
(name: str) → None¶ Completely remove the given collection.
Parameters: - name :
str
The name of the collection to remove.
Raises: - MissingCollectionError
Raised if no collection with the given name exists.
Notes
If this is a
RUN
collection, all datasets and quanta in it are also fully removed. This requires that those datasets be removed (or at least trashed) from any datastores that hold them first.A collection may not be deleted as long as it is referenced by a
CHAINED
collection; theCHAINED
collection must be deleted or redefined first.- name :
-
removeDatasetType
(name: str) → None¶ Remove the named
DatasetType
from the registry.Warning
Registry caches the dataset type definitions. This means that deleting the dataset type definition may result in unexpected behavior from other butler processes that are active that have not seen the deletion.
Parameters: - name :
str
Name of the type to be removed.
Raises: - lsst.daf.butler.registry.OrphanedRecordError
Raised if an attempt is made to remove the dataset type definition when there are already datasets associated with it.
Notes
If the dataset type is not registered the method will return without action.
- name :
-
removeDatasets
(refs: Iterable[lsst.daf.butler.core.datasets.ref.DatasetRef]) → None¶ Remove datasets from the Registry.
The datasets will be removed unconditionally from all collections, and any
Quantum
that consumed this dataset will instead be marked with having a NULL input.Datastore
records will not be deleted; the caller is responsible for ensuring that the dataset has already been removed from all Datastores.Parameters: - refs :
Iterable
ofDatasetRef
References to the datasets to be removed. Must include a valid
id
attribute, and should be considered invalidated upon return.
Raises: - refs :
-
setCollectionChain
(parent: str, children: Any) → None¶ Define or redefine a
CHAINED
collection.Parameters: - parent :
str
Name of the chained collection. Must have already been added via a call to
Registry.registerCollection
.- children :
Any
An expression defining an ordered search of child collections, generally an iterable of
str
; see Collection expressions for more information.
Raises: - parent :
-
setCollectionDocumentation
(collection: str, doc: Optional[str]) → None¶ Set the documentation string for a collection.
Parameters:
-
syncDimensionData
(element: Union[lsst.daf.butler.core.dimensions._elements.DimensionElement, str], row: Union[Mapping[str, Any], lsst.daf.butler.core.dimensions._records.DimensionRecord], conform: bool = True) → bool¶ Synchronize the given dimension record with the database, inserting if it does not already exist and comparing values if it does.
Parameters: - element :
DimensionElement
orstr
The
DimensionElement
or name thereof that identifies the table records will be inserted into.- row :
dict
orDimensionRecord
The record to insert.
- conform :
bool
, optional If
False
(True
is default) perform no checking or conversions, and assume thatelement
is aDimensionElement
instance anddata
is a one or moreDimensionRecord
instances of the appropriate subclass.
Returns: Raises: - ConflictingDefinitionError
Raised if the record exists in the database (according to primary key lookup) but is inconsistent with the given one.
- element :
-
transaction
(*, savepoint: bool = False) → Iterator[None]¶ Return a context manager that represents a transaction.
- database :