Organizing and identifying datasets¶
Each dataset in a repository is associated with an opaque unique integer ID, which we currently call its dataset_id, and it’s usually seen in Python code as the value of DatasetRef.id.
This is the number used as the primary key in most Registry tables that refer to datasets, and it’s the only way the contents of a Datastore are matched to those in a Registry.
With that number, the dataset is fully identified, and anything else about it can be unambiguously looked up.
We call a DatasetRef whose id attribute is not None a resolved DatasetRef.
Most of the time, however, users identify a dataset using a combination of three other attributes:
- a dataset type;
- a data ID;
- a collection.
Most collections are constrained to contain only one dataset with a particular dataset type and data ID, so this combination is usually enough to resolve a dataset (see Collections for exceptions).
A dataset’s type and data ID are intrinsic to it — while there may be many datasets with a particular dataset type and/or data ID, the dataset type and data ID associated with a dataset are set and fixed when it is created.
A DatasetRef always has both a dataset type attribute and a data ID, though the latter may be empty.
Dataset types are discussed below in Dataset types, while data IDs are one aspect of the larger Dimensions system and are discussed in Data IDs.
In contrast, the relationship between dataset and collections is many-to-many: a collection typically contains many different datasets, and a particular dataset may belong to multiple collections. As a result, is is common to search for datasets in multiple collections (often in a well-defined order), and interfaces that provide that functionality can accept a collection search path in many different forms. Collections are discussed further below in Collections.
Dataset types¶
The names “dataset” and “dataset type” (which daf_butler inherits from its daf_persistence predecessor) are intended to evoke the relationship between an instance and its class in object-oriented programming, but this is a metaphor, not a relationship that maps to any particular Python objects: we don’t have any Python class that fully represents the dataset concept (DatasetRef is the closest), and the DatasetType class is a regular class, not a metaclass.
So a dataset type is represented in Python as a DatasetType instance.
A dataset type defines both the dimensions used in a dataset’s data ID (so all data IDs for a particular dataset type have the same keys, at least when put in standard form) and the storage class that corresponds to its in-memory Python type and maps to the file format (or generalization thereof) used by a Datastore to store it.
These are associated with an arbitrary string name.
Beyond that definition, what a dataset type means isn’t really specified by the butler itself, but we expect higher-level code that uses butler to make that clear, and one anticipated case is worth calling out here: a dataset type roughly corresponds to the role its datasets play in a processing pipeline. In other words, a particular pipeline will typically accept particular dataset types as inputs and produce particular dataset types as outputs (and may produce and consume other dataset types as intermediates). And while the exact dataset types used may be configurable, changing a dataset type will generally involve substituting one dataset type for a very similar one (most of the time with the same dimensions and storage class).
Collections¶
Collections are lightweight groups of datasets defined in the Registry.
Groups of self-consistent calibration datasets, the outputs of a processing run, and the set of all raw images for a particular instrument are all examples of collections.
Collections are referred to in code simply as str names; various Registry methods can be used to manage them and obtain additional information about them when relevant.
There are multiple types of collections, corresponding to the different values of the CollectionType enum.
All collection types are usable in the same way in any context where existing datasets are being queried or retrieved, though the actual searches may be implemented quite differently in terms of database queries.
Collection types behave completely differently in terms of how and when datasets can be added to or remove from them.
Run Collections¶
A dataset is always added to a CollectionType.RUN collection when it is inserted into the Registry, and can never be removed from it without fully removing the dataset from the Registry.
There is no other way to add a dataset to a RUN collection.
The run collection name must be used in any file path templates used by any Datastore in order to guarantee uniqueness (other collection types are too flexible to guarantee continued uniqueness over the life of the dataset).
The name “run” reflects the fact that we expect most RUN collections to be used to store the outputs of processing runs, but they should also be used in any other context in which their lack of flexibility is acceptable, as they are the most efficient type of collection to store and query.
RUN collections that do represent the outputs of processing runs can be associated with a host name string and a timespan, and are expected to be the way in which some provenance is associated with datasets (e.g. a dataset that contains a list of software versions would have the same RUN as the datasets produced by a processing run that used those versions).
Like most collections, a RUN can contain at most one dataset with a particular dataset type and data ID.
Tagged Collections¶
CollectionType.TAGGED collections are the most flexible type of collection; datasets can be associated with or disassociated from a TAGGED collection at any time, as long as the usual contraint on a collection having only one dataset with a particular dataset type and data ID is maintained.
Membership in a TAGGED collection is implemented in the Registry database as a single row in a many-to-many join table (a “tag”) and is completely decoupled from the actual storage of the dataset.
Tags are thus both extremely lightweight relative to copies or re-ingests of files or other Datastore content, and slightly more expensive to store and possibly query than the RUN or CHAINED collection representations (which have no per-dataset costs).
The latter is rarely important, but higher-level code should avoid automatically creating TAGGED collections that may not ever be used.
Calibration Collections¶
CollectionType.CALIBRATION collections associate each dataset they contain with a temporal validity range.
The usual constraint on dataset type and data ID uniqueness is enforced as a function of time, not collection-wide - so for any particular dataset type and data ID combination, the validity range timespans may not overlap (but may be - and usually are - adjacent).
In other respects, CALIBRATION collections closely resemble TAGGED collections: they are also backed by a many-to-many join table (where each row has a timespan as well as a collection identifier and a dataset identifier), and datasets can be associated or disassociated from them similarly freely.
We use slightly different nomenclature for these operations, reflecting the high-level actions they represent: certifying a dataset adds it to a CALIBRATION collection with a particular validity range, and decertifying a dataset removes some or all of that validity range.
The same dataset can be present in a CALIBRATION collection multiple times with different validity ranges.
Chained Collections¶
A CollectionType.CHAINED collection is essentially a multi-collection search path that has been saved in the Registry database and associated with a name of its own.
Querying a CHAINED collection simply queries its child collections in order, and a CHAINED collection is always (and only) updated when its child collections are.
CHAINED collections may contain other chained collections, as long as they do not contain cycles, and they can also include restrictions on the dataset types to search for within each child collection (see Collection expressions).
The usual constraint on dataset type and data ID uniqueness within a collection is only lazily enforced for chained collections: operations that query them either deduplicate results themselves or terminate single-dataset searches after the first match in a child collection is found.
In some methods, like Registry.queryDatasets, this behavior is optional: passing findFirst=True will enforce the constraint, while findFirst=False will not.