CollectionManager¶
- class lsst.daf.butler.registry.interfaces.CollectionManager(*, registry_schema_version: VersionTuple | None = None)¶
Bases:
Generic
[_Key
],VersionedExtension
An interface for managing the collections (including runs) in a
Registry
.- Parameters:
- registry_schema_version
VersionTuple
orNone
, optional Version of registry schema.
- registry_schema_version
Notes
Each layer in a multi-layer
Registry
has its own record for any collection for which it has datasets (or quanta). Different layers may use different IDs for the same collection, so any usage of the IDs obtained through theCollectionManager
APIs are strictly for internal (toRegistry
) use.Methods Summary
addCollectionForeignKey
(tableSpec, *[, ...])Add a foreign key (field and constraint) referencing the collection table.
addRunForeignKey
(tableSpec, *[, prefix, ...])Add a foreign key (field and constraint) referencing the run table.
clone
(db, caching_context)Make an independent copy of this manager instance bound to a new
Database
instance.Add children to the end of a CHAINED collection.
find
(name)Return the collection record associated with the given name.
getCollectionForeignKeyName
([prefix])Return the name of the field added by
addCollectionForeignKey
if called with the same prefix.getDocumentation
(key)Retrieve the documentation string for a collection.
getParentChains
(key)Find all CHAINED collection names that directly contain the given collection.
getRunForeignKeyName
([prefix])Return the name of the field added by
addRunForeignKey
if called with the same prefix.initialize
(db, context, *, caching_context)Construct an instance of the manager.
lookup_name_sql
(sql_key, sql_from_clause)Return a SQLAlchemy column and FROM clause that enable a query to look up a collection name from the key.
Add children to the beginning of a CHAINED collection.
refresh
()Ensure all other operations on this manager are aware of any collections that may have been registered by other clients since it was initialized or last refreshed.
register
(name, type[, doc])Ensure that a collection of the given name and type are present in the layer this manager is associated with.
remove
(name)Completely remove a collection.
Remove children from a CHAINED collection.
resolve_wildcard
(wildcard, *[, ...])Iterate over collection records that match a wildcard.
setDocumentation
(key, doc)Set the documentation string for a collection.
update_chain
(parent_collection_name, ...[, ...])Replace all of the children in a chained collection with a new list.
Methods Documentation
- abstract classmethod addCollectionForeignKey(tableSpec: TableSpec, *, prefix: str = 'collection', onDelete: str | None = None, constraint: bool = True, **kwargs: Any) FieldSpec ¶
Add a foreign key (field and constraint) referencing the collection table.
- Parameters:
- tableSpec
ddl.TableSpec
Specification for the table that should reference the collection table. Will be modified in place.
- prefix
str
, optional A name to use for the prefix of the new field; the full name may have a suffix (and is given in the returned
ddl.FieldSpec
).- onDelete
str
, optional One of “CASCADE” or “SET NULL”, indicating what should happen to the referencing row if the collection row is deleted.
None
indicates that this should be an integrity error.- constraint
bool
, optional If
False
(True
is default), add a field that can be joined to the collection primary key, but do not add a foreign key constraint.- **kwargs
Additional keyword arguments are forwarded to the
ddl.FieldSpec
constructor (only thename
anddtype
arguments are otherwise provided).
- tableSpec
- Returns:
- fieldSpec
ddl.FieldSpec
Specification for the field being added.
- fieldSpec
- abstract classmethod addRunForeignKey(tableSpec: TableSpec, *, prefix: str = 'run', onDelete: str | None = None, constraint: bool = True, **kwargs: Any) FieldSpec ¶
Add a foreign key (field and constraint) referencing the run table.
- Parameters:
- tableSpec
ddl.TableSpec
Specification for the table that should reference the run table. Will be modified in place.
- prefix
str
, optional A name to use for the prefix of the new field; the full name may have a suffix (and is given in the returned
ddl.FieldSpec
).- onDelete
str
, optional One of “CASCADE” or “SET NULL”, indicating what should happen to the referencing row if the collection row is deleted.
None
indicates that this should be an integrity error.- constraint
bool
, optional If
False
(True
is default), add a field that can be joined to the run primary key, but do not add a foreign key constraint.- **kwargs
Additional keyword arguments are forwarded to the
ddl.FieldSpec
constructor (only thename
anddtype
arguments are otherwise provided).
- tableSpec
- Returns:
- fieldSpec
ddl.FieldSpec
Specification for the field being added.
- fieldSpec
- abstract clone(db: Database, caching_context: CachingContext) Self ¶
Make an independent copy of this manager instance bound to a new
Database
instance.- Parameters:
- Returns:
- instance
CollectionManager
New manager instance with the same configuration as this instance, but bound to a new Database object.
- instance
- abstract extend_collection_chain(parent_collection_name: str, child_collection_names: list[str]) None ¶
Add children to the end of a CHAINED collection.
If any of the children already existed in the chain, they will be moved to the new position at the end of the chain.
- Parameters:
- Raises:
- MissingCollectionError
If any of the specified collections do not exist.
- CollectionTypeError
If the parent collection is not a CHAINED collection.
- CollectionCycleError
If this operation would create a collection cycle.
Notes
If this function is called within a call to
Butler.transaction
, it will hold a lock that prevents other processes from modifying the parent collection until the end of the transaction. Keep these transactions short.
- abstract find(name: str) CollectionRecord[_Key] ¶
Return the collection record associated with the given name.
- Parameters:
- name
str
Name of the collection.
- name
- Returns:
- record
CollectionRecord
Object representing the collection, including its type and ID. If
record.type is CollectionType.RUN
, this will be aRunRecord
instance. Ifrecord.type is CollectionType.CHAIN
, this will be aChainedCollectionRecord
instance.
- record
- Raises:
- MissingCollectionError
Raised if the given collection does not exist.
Notes
Collections registered by another client of the same layer since the last call to
initialize
orrefresh
may not be found.
- abstract classmethod getCollectionForeignKeyName(prefix: str = 'collection') str ¶
Return the name of the field added by
addCollectionForeignKey
if called with the same prefix.
- abstract getDocumentation(key: _Key) str | None ¶
Retrieve the documentation string for a collection.
- abstract getParentChains(key: _Key) set[str] ¶
Find all CHAINED collection names that directly contain the given collection.
- abstract classmethod getRunForeignKeyName(prefix: str = 'run') str ¶
Return the name of the field added by
addRunForeignKey
if called with the same prefix.
- abstract classmethod initialize(db: Database, context: StaticTablesContext, *, caching_context: CachingContext, registry_schema_version: VersionTuple | None = None) CollectionManager ¶
Construct an instance of the manager.
- Parameters:
- db
Database
Interface to the underlying database engine and namespace.
- context
StaticTablesContext
Context object obtained from
Database.declareStaticTables
; used to declare any tables that should always be present in a layer implemented with this manager.- caching_context
CachingContext
Object controlling caching of information returned by managers.
- registry_schema_version
VersionTuple
orNone
Schema version of this extension as defined in registry.
- db
- Returns:
- manager
CollectionManager
An instance of a concrete
CollectionManager
subclass.
- manager
- lookup_name_sql(sql_key: ColumnElement[_Key], sql_from_clause: FromClause) tuple[sqlalchemy.sql.elements.ColumnElement[str], sqlalchemy.sql.selectable.FromClause] ¶
Return a SQLAlchemy column and FROM clause that enable a query to look up a collection name from the key.
- Parameters:
- sql_key
sqlalchemy.ColumnElement
SQL column expression that evaluates to the collection key.
- sql_from_clause
sqlalchemy.FromClause
SQL FROM clause from which
sql_key
was obtained.
- sql_key
- Returns:
- sql_name
sqlalchemy.ColumnElement
[str
] SQL column expression that evalutes to the collection name.
- sql_from_clause
sqlalchemy.FromClause
SQL FROM clause that includes the given
sql_from_clause
and any table needed to providedsql_name
.
- sql_name
- abstract prepend_collection_chain(parent_collection_name: str, child_collection_names: list[str]) None ¶
Add children to the beginning of a CHAINED collection.
If any of the children already existed in the chain, they will be moved to the new position at the beginning of the chain.
- Parameters:
- Raises:
- MissingCollectionError
If any of the specified collections do not exist.
- CollectionTypeError
If the parent collection is not a CHAINED collection.
- CollectionCycleError
If this operation would create a collection cycle.
Notes
If this function is called within a call to
Butler.transaction
, it will hold a lock that prevents other processes from modifying the parent collection until the end of the transaction. Keep these transactions short.
- abstract refresh() None ¶
Ensure all other operations on this manager are aware of any collections that may have been registered by other clients since it was initialized or last refreshed.
- abstract register(name: str, type: CollectionType, doc: str | None = None) tuple[lsst.daf.butler.registry.interfaces._collections.CollectionRecord[_Key], bool] ¶
Ensure that a collection of the given name and type are present in the layer this manager is associated with.
- Parameters:
- Returns:
- record
CollectionRecord
Object representing the collection, including its type and ID. If
type is CollectionType.RUN
, this will be aRunRecord
instance. Iftype is CollectionType.CHAIN
, this will be aChainedCollectionRecord
instance.- registered
bool
True if the collection was registered,
False
if it already existed.
- record
- Raises:
- TransactionInterruption
Raised if this operation is invoked within a
Database.transaction
context.- DatabaseConflictError
Raised if a collection with this name but a different type already exists.
Notes
Concurrent registrations of the same collection should be safe; nothing should happen if the types are consistent, and integrity errors due to inconsistent types should happen before any database changes are made.
- abstract remove(name: str) None ¶
Completely remove a collection.
Any existing
CollectionRecord
objects that correspond to the removed collection are considered invalidated.- Parameters:
- name
str
Name of the collection to remove.
- name
Notes
If this collection is referenced by foreign keys in tables managed by other objects, the ON DELETE clauses of those tables will be invoked. That will frequently delete many dependent rows automatically (via “CASCADE”, but it may also cause this operation to fail (with rollback) unless dependent rows that do not have an ON DELETE clause are removed first.
- abstract remove_from_collection_chain(parent_collection_name: str, child_collection_names: list[str]) None ¶
Remove children from a CHAINED collection.
- Parameters:
- Raises:
- MissingCollectionError
If any of the specified collections do not exist.
- CollectionTypeError
If the parent collection is not a CHAINED collection.
Notes
If this function is called within a call to
Butler.transaction
, it will hold a lock that prevents other processes from modifying the parent collection until the end of the transaction. Keep these transactions short.
- abstract resolve_wildcard(wildcard: CollectionWildcard, *, collection_types: Set[CollectionType] = frozenset({CollectionType.RUN, CollectionType.TAGGED, CollectionType.CHAINED, CollectionType.CALIBRATION}), done: set[str] | None = None, flatten_chains: bool = True, include_chains: bool | None = None) list[lsst.daf.butler.registry.interfaces._collections.CollectionRecord[_Key]] ¶
Iterate over collection records that match a wildcard.
- Parameters:
- wildcard
CollectionWildcard
Names and/or patterns for collections.
- collection_types
collections.abc.Set
[CollectionType
], optional If provided, only yield collections of these types.
- done
set
[str
], optional A
set
of collection names that will not be returned (presumably because they have already been returned in some higher-level logic) that will also be updated with the names of the collections returned.- flatten_chains
bool
, optional If
True
(default) recursively yield the child collections ofCHAINED
collections.- include_chains
bool
, optional If
True
, return records forCHAINED
collections themselves. The default is the opposite offlatten_chains
: either return records for CHAINED collections or their children, but not both.
- wildcard
- Returns:
- records
list
[CollectionRecord
] Matching collection records.
- records
- abstract setDocumentation(key: _Key, doc: str | None) None ¶
Set the documentation string for a collection.
- Parameters:
- key_Key
Internal primary key value for the collection.
- doc
str
, optional Docstring for the collection with the given key.
- abstract update_chain(parent_collection_name: str, child_collection_names: list[str], allow_use_in_caching_context: bool = False) None ¶
Replace all of the children in a chained collection with a new list.
- Parameters:
- parent_collection_name
str
The name of a CHAINED collection to be modified.
- child_collection_names
list
[ `str ` ] A child collection name or list of child collection names to be assigned to the parent.
- allow_use_in_caching_context
bool
, optional If
True
, skip a check that would otherwise disallow this function from being called inside an active caching context. (Only exists for legacy use, will eventually be removed).
- parent_collection_name
- Raises:
- MissingCollectionError
If any of the specified collections do not exist.
- CollectionTypeError
If the parent collection is not a CHAINED collection.
- CollectionCycleError
If this operation would create a collection cycle.
Notes
If this function is called within a call to
Butler.transaction
, it will hold a lock that prevents other processes from modifying the parent collection until the end of the transaction. Keep these transactions short.