ApdbSql

class lsst.dax.apdb.ApdbSql(config: lsst.dax.apdb.apdbSql.ApdbSqlConfig)

Bases: lsst.dax.apdb.Apdb

Implementation of APDB interface based on SQL database.

The implementation is configured via standard pex_config mechanism using ApdbSqlConfig configuration class. For an example of different configurations check config/ folder.

Parameters:
config : ApdbSqlConfig

Configuration object.

Methods Summary

countUnassociatedObjects() Return the number of DiaObjects that have only one DiaSource associated with them.
dailyJob() Implement daily activities like cleanup/vacuum.
getDiaForcedSources(region, object_ids, …) Return catalog of DiaForcedSource instances from a given region.
getDiaForcedSourcesHistory(start_time, …) Returns catalog of DiaForcedSource instances from a given time period.
getDiaObjects(region) Returns catalog of DiaObject instances from a given region.
getDiaObjectsHistory(start_time, end_time, …) Returns catalog of DiaObject instances from a given time period including the history of each DiaObject.
getDiaSources(region, object_ids, None], …) Return catalog of DiaSource instances from a given region.
getDiaSourcesHistory(start_time, end_time, …) Returns catalog of DiaSource instances from a given time period.
getSSObjects() Returns catalog of SSObject instances.
makeField(doc) Make a ConfigurableField for Apdb.
makeSchema(drop) Create or re-create whole database schema.
reassignDiaSources(idMap, int]) Associate DiaSources with SSObjects, dis-associating them from DiaObjects.
store(visit_time, objects, sources, …) Store all three types of catalogs in the database.
storeSSObjects(objects) Store or update SSObject catalog.
tableDef(table) Return table schema definition for a given table.
tableRowCount() Returns dictionary with the table names and row counts.

Methods Documentation

countUnassociatedObjects() → int

Return the number of DiaObjects that have only one DiaSource associated with them.

Used as part of ap_verify metrics.

Returns:
count : int

Number of DiaObjects with exactly one associated DiaSource.

Notes

This method can be very inefficient or slow in some implementations.

dailyJob() → None

Implement daily activities like cleanup/vacuum.

What should be done during daily activities is determined by specific implementation.

getDiaForcedSources(region: lsst.sphgeom._sphgeom.Region, object_ids: Optional[Iterable[int], None], visit_time: lsst.daf.base.dateTime.dateTime.DateTime) → Optional[pandas.core.frame.DataFrame, None]

Return catalog of DiaForcedSource instances from a given region.

Parameters:
region : lsst.sphgeom.Region

Region to search for DIASources.

object_ids : iterable [ int ], optional

List of DiaObject IDs to further constrain the set of returned sources. If list is empty then empty catalog is returned with a correct schema.

visit_time : lsst.daf.base.DateTime

Time of the current visit.
Returns:
catalog : pandas.DataFrame, or None

Catalog containing DiaSource records. None is returned if read_sources_months configuration parameter is set to 0.
Raises:
NotImplementedError

Raised if object_ids is None.

Notes

Even though base class allows None to be passed for object_ids, this class requires object_ids to be not-None. NotImplementedError is raised if None is passed.

This method returns DiaForcedSource catalog for a region with additional filtering based on DiaObject IDs. Only a subset of DiaSource history is returned limited by read_forced_sources_months config parameter, w.r.t. visit_time. If object_ids is empty then an empty catalog is always returned with a correct schema (columns/types).

getDiaForcedSourcesHistory(start_time: lsst.daf.base.dateTime.dateTime.DateTime, end_time: lsst.daf.base.dateTime.dateTime.DateTime, region: Optional[lsst.sphgeom._sphgeom.Region, None] = None) → pandas.core.frame.DataFrame

Returns catalog of DiaForcedSource instances from a given time period.

Parameters:
start_time : dafBase.DateTime

Starting time for DiaForcedSource history search. DiaForcedSource record is selected when its midPointTai falls into an interval between start_time (inclusive) and end_time (exclusive).

end_time : dafBase.DateTime

Upper limit on time for DiaForcedSource history search.

region : lsst.sphgeom.Region, optional

Region to search for DiaForcedSources, if not specified then whole sky is searched. If region is specified then some returned records may fall outside of this region.
Returns:
catalog : pandas.DataFrame

Catalog containing DiaObject records.

Notes

This part of API may not be very stable and can change before the implementation finalizes. Some implementations may not support region filtering, they will return records from the whole sky.

getDiaObjects(region: lsst.sphgeom._sphgeom.Region) → pandas.core.frame.DataFrame

Returns catalog of DiaObject instances from a given region.

This method returns only the last version of each DiaObject. Some records in a returned catalog may be outside the specified region, it is up to a client to ignore those records or cleanup the catalog before futher use.

Parameters:
region : lsst.sphgeom.Region

Region to search for DIAObjects.
Returns:
catalog : pandas.DataFrame

Catalog containing DiaObject records for a region that may be a superset of the specified region.
getDiaObjectsHistory(start_time: lsst.daf.base.dateTime.dateTime.DateTime, end_time: lsst.daf.base.dateTime.dateTime.DateTime, region: Optional[lsst.sphgeom._sphgeom.Region, None] = None) → pandas.core.frame.DataFrame

Returns catalog of DiaObject instances from a given time period including the history of each DiaObject.

Parameters:
start_time : dafBase.DateTime

Starting time for DiaObject history search. DiaObject record is selected when its validityStart falls into an interval between start_time (inclusive) and end_time (exclusive).

end_time : dafBase.DateTime

Upper limit on time for DiaObject history search.

region : lsst.sphgeom.Region, optional

Region to search for DiaObjects, if not specified then whole sky is searched. If region is specified then some returned records may fall outside of this region.
Returns:
catalog : pandas.DataFrame

Catalog containing DiaObject records.

Notes

This part of API may not be very stable and can change before the implementation finalizes.

getDiaSources(region: lsst.sphgeom._sphgeom.Region, object_ids: Optional[Iterable[int], None], visit_time: lsst.daf.base.dateTime.dateTime.DateTime) → Optional[pandas.core.frame.DataFrame, None]

Return catalog of DiaSource instances from a given region.

Parameters:
region : lsst.sphgeom.Region

Region to search for DIASources.

object_ids : iterable [ int ], optional

List of DiaObject IDs to further constrain the set of returned sources. If None then returned sources are not constrained. If list is empty then empty catalog is returned with a correct schema.

visit_time : lsst.daf.base.DateTime

Time of the current visit.
Returns:
catalog : pandas.DataFrame, or None

Catalog containing DiaSource records. None is returned if read_sources_months configuration parameter is set to 0.

Notes

This method returns DiaSource catalog for a region with additional filtering based on DiaObject IDs. Only a subset of DiaSource history is returned limited by read_sources_months config parameter, w.r.t. visit_time. If object_ids is empty then an empty catalog is always returned with the correct schema (columns/types). If object_ids is None then no filtering is performed and some of the returned records may be outside the specified region.

getDiaSourcesHistory(start_time: lsst.daf.base.dateTime.dateTime.DateTime, end_time: lsst.daf.base.dateTime.dateTime.DateTime, region: Optional[lsst.sphgeom._sphgeom.Region, None] = None) → pandas.core.frame.DataFrame

Returns catalog of DiaSource instances from a given time period.

Parameters:
start_time : dafBase.DateTime

Starting time for DiaSource history search. DiaSource record is selected when its midPointTai falls into an interval between start_time (inclusive) and end_time (exclusive).

end_time : dafBase.DateTime

Upper limit on time for DiaSource history search.

region : lsst.sphgeom.Region, optional

Region to search for DiaSources, if not specified then whole sky is searched. If region is specified then some returned records may fall outside of this region.
Returns:
catalog : pandas.DataFrame

Catalog containing DiaObject records.

Notes

This part of API may not be very stable and can change before the implementation finalizes.

getSSObjects() → pandas.core.frame.DataFrame

Returns catalog of SSObject instances.

Returns:
catalog : pandas.DataFrame

Catalog containing SSObject records, all existing records are returned.
classmethod makeField(doc: str) → lsst.pex.config.configurableField.ConfigurableField

Make a ConfigurableField for Apdb.

Parameters:
doc : str

Help text for the field.
Returns:
configurableField : lsst.pex.config.ConfigurableField

A ConfigurableField for Apdb.
makeSchema(drop: bool = False) → None

Create or re-create whole database schema.

Parameters:
drop : bool

If True then drop all tables before creating new ones.
reassignDiaSources(idMap: Mapping[int, int]) → None

Associate DiaSources with SSObjects, dis-associating them from DiaObjects.

Parameters:
idMap : Mapping

Maps DiaSource IDs to their new SSObject IDs.
Raises:
ValueError

Raised if DiaSource ID does not exist in the database.
store(visit_time: lsst.daf.base.dateTime.dateTime.DateTime, objects: pandas.core.frame.DataFrame, sources: Optional[pandas.core.frame.DataFrame, None] = None, forced_sources: Optional[pandas.core.frame.DataFrame, None] = None) → None

Store all three types of catalogs in the database.

Parameters:
visit_time : lsst.daf.base.DateTime

Time of the visit.

objects : pandas.DataFrame

Catalog with DiaObject records.

sources : pandas.DataFrame, optional

Catalog with DiaSource records.

forced_sources : pandas.DataFrame, optional

Catalog with DiaForcedSource records.

Notes

This methods takes DataFrame catalogs, their schema must be compatible with the schema of APDB table:

  • column names must correspond to database table columns
  • types and units of the columns must match database definitions, no unit conversion is performed presently
  • columns that have default values in database schema can be omitted from catalog
  • this method knows how to fill interval-related columns of DiaObject (validityStart, validityEnd) they do not need to appear in a catalog
  • source catalogs have diaObjectId column associating sources with objects
storeSSObjects(objects: pandas.core.frame.DataFrame) → None

Store or update SSObject catalog.

Parameters:
objects : pandas.DataFrame

Catalog with SSObject records.

Notes

If SSObjects with matching IDs already exist in the database, their records will be updated with the information from provided records.

tableDef(table: lsst.dax.apdb.apdbSchema.ApdbTables) → Optional[lsst.dax.apdb.apdbSchema.TableDef, None]

Return table schema definition for a given table.

Parameters:
table : ApdbTables

One of the known APDB tables.
Returns:
tableSchema : TableDef or None

Table schema description, None is returned if table is not defined by this implementation.
tableRowCount() → Dict[str, int]

Returns dictionary with the table names and row counts.

Used by ap_proto to keep track of the size of the database tables. Depending on database technology this could be expensive operation.

Returns:
row_counts : dict

Dict where key is a table name and value is a row count.