RawIngestTask¶
- class lsst.obs.base.RawIngestTask(config: ~lsst.obs.base.ingest.RawIngestConfig, *, butler: ~lsst.daf.butler._butler.Butler, on_success: ~collections.abc.Callable[[list[lsst.daf.butler._file_dataset.FileDataset]], ~typing.Any] = <function _do_nothing>, on_metadata_failure: ~collections.abc.Callable[[~lsst.resources._resourcePath.ResourcePath, Exception], ~typing.Any] = <function _do_nothing>, on_ingest_failure: ~collections.abc.Callable[[~lsst.obs.base.ingest.RawExposureData, Exception], ~typing.Any] = <function _do_nothing>, **kwargs: ~typing.Any)¶
Bases:
Task
Driver Task for ingesting raw data into Gen3 Butler repositories.
- Parameters:
- config
RawIngestConfig
Configuration for the task.
- butler
Butler
Writeable butler instance, with
butler.run
set to the appropriateRUN
collection for these raw datasets.- on_success
Callable
, optional A callback invoked when all of the raws associated with an exposure are ingested. Will be passed a list of
FileDataset
objects, each containing one or more resolvedDatasetRef
objects. If this callback raises it will interrupt the entire ingest process, even ifRawIngestConfig.failFast
isFalse
.- on_metadata_failure
Callable
, optional A callback invoked when a failure occurs trying to translate the metadata for a file. Will be passed the URI and the exception, in that order, as positional arguments. Guaranteed to be called in an
except
block, allowing the callback to re-raise or replace (withraise ... from
) to override the task’s usual error handling (beforeRawIngestConfig.failFast
logic occurs).- on_ingest_failure
Callable
, optional A callback invoked when dimension record or dataset insertion into the database fails for an exposure. Will be passed a
RawExposureData
instance and the exception, in that order, as positional arguments. Guaranteed to be called in anexcept
block, allowing the callback to re-raise or replace (withraise ... from
) to override the task’s usual error handling (beforeRawIngestConfig.failFast
logic occurs).- **kwargs
Additional keyword arguments are forwarded to the
lsst.pipe.base.Task
constructor.
- config
Notes
Each instance of
RawIngestTask
writes to the same Butler. Each invocation ofRawIngestTask.run
ingests a list of files.Methods Summary
Empty (clear) the metadata for this Task and all sub-Tasks.
expandDataIds
(data)Expand the data IDs associated with a raw exposure.
extractMetadata
(filename)Extract and process metadata from a single raw file.
Return the default DatasetType of the datasets ingested by this Task.
Get metadata for all tasks.
Get the task name as a hierarchical name including parent task names.
getName
()Get the name of the task.
Return subsets of fields in the
ObservationInfo
that we care about.Get a dictionary of all tasks as a shallow copy.
groupByExposure
(files)Group an iterable of
RawFileData
by exposure.ingestExposureDatasets
(exposure, ...[, ...])Ingest all raw files in one exposure.
ingestFiles
(files, *[, pool, processes, ...])Ingest files into a Butler data repository.
locateAndReadIndexFiles
(files)Given a list of files, look for index files and read them.
makeDependencyRecords
(obsInfo, universe)Construct dependency records.
makeExposureRecord
(obsInfo, universe, **kwargs)Construct a registry record for an exposure.
makeField
(doc)Make a
lsst.pex.config.ConfigurableField
for this task.makeSubtask
(name, **keyArgs)Create a subtask as a new instance as the
name
attribute of this task.prep
(files, *[, pool])Perform all non-database-updating ingest preprocessing steps.
processIndexEntries
(index_entries)Convert index entries to RawFileData.
run
(files, *[, pool, processes, run, ...])Ingest files into a Butler data repository.
timer
(name[, logLevel])Context manager to log performance data for an arbitrary block of code.
Methods Documentation
- expandDataIds(data: RawExposureData) RawExposureData ¶
Expand the data IDs associated with a raw exposure.
This adds the metadata records.
- Parameters:
- exposure
RawExposureData
A structure containing information about the exposure to be ingested. Must have
RawExposureData.record
populated. Should be considered consumed upon return.
- exposure
- Returns:
- exposure
RawExposureData
An updated version of the input structure, with
RawExposureData.dataId
and nestedRawFileData.dataId
attributes updated to data IDs for whichhasRecords
returnsTrue
.
- exposure
- extractMetadata(filename: ResourcePath) RawFileData ¶
Extract and process metadata from a single raw file.
- Parameters:
- filename
lsst.resources.ResourcePath
URI to the file.
- filename
- Returns:
- data
RawFileData
A structure containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the
RawFileData.dataId
attribute will be a minimal (unexpanded)DataCoordinate
instance. Theinstrument
field will beNone
if there is a problem with metadata extraction.
- data
Notes
Assumes that there is a single dataset associated with the given file. Instruments using a single file to store multiple datasets must implement their own version of this method.
By default the method will catch all exceptions unless the
failFast
configuration item isTrue
. If an error is encountered the_on_metadata_failure()
method will be called. If no exceptions result and an error was encountered the returned object will have a null-instrument class and no datasets.This method supports sidecar JSON files which can be used to extract metadata without having to read the data file itself. The sidecar file is always used if found.
- getDatasetType() DatasetType ¶
Return the default DatasetType of the datasets ingested by this Task.
- Returns:
- datasetType
DatasetType
The default dataset type to use for the data being ingested. This is only used if the relevant
Instrument
does not define an override.
- datasetType
- getFullMetadata() TaskMetadata ¶
Get metadata for all tasks.
- Returns:
- metadata
TaskMetadata
The keys are the full task name. Values are metadata for the top-level task and all subtasks, sub-subtasks, etc.
- metadata
Notes
The returned metadata includes timing information (if
@timer.timeMethod
is used) and any metadata set by the task. The name of each item consists of the full task name with.
replaced by:
, followed by.
and the name of the item, e.g.:topLevelTaskName:subtaskName:subsubtaskName.itemName
using
:
in the full task name disambiguates the rare situation that a task has a subtask and a metadata item with the same name.
- getFullName() str ¶
Get the task name as a hierarchical name including parent task names.
- Returns:
- fullName
str
The full name consists of the name of the parent task and each subtask separated by periods. For example:
The full name of top-level task “top” is simply “top”.
The full name of subtask “sub” of top-level task “top” is “top.sub”.
The full name of subtask “sub2” of subtask “sub” of top-level task “top” is “top.sub.sub2”.
- fullName
- getName() str ¶
Get the name of the task.
- Returns:
- taskName
str
Name of the task.
- taskName
See also
getFullName
Get the full name of the task.
- classmethod getObservationInfoSubsets() tuple[set, set] ¶
Return subsets of fields in the
ObservationInfo
that we care about.These fields will be used in constructing an exposure record.
- getTaskDict() dict[str, weakref.ReferenceType[lsst.pipe.base.task.Task]] ¶
Get a dictionary of all tasks as a shallow copy.
- Returns:
- taskDict
dict
Dictionary containing full task name: task object for the top-level task and all subtasks, sub-subtasks, etc.
- taskDict
- groupByExposure(files: Iterable[RawFileData]) list[lsst.obs.base.ingest.RawExposureData] ¶
Group an iterable of
RawFileData
by exposure.- Parameters:
- filesiterable of
RawFileData
File-level information to group.
- filesiterable of
- Returns:
- exposures
list
ofRawExposureData
A list of structures that group the file-level information by exposure. All fields will be populated. The
RawExposureData.dataId
attributes will be minimal (unexpanded)DataCoordinate
instances.
- exposures
- ingestExposureDatasets(exposure: RawExposureData, datasetType: DatasetType, *, run: str, skip_existing_exposures: bool = False, track_file_attrs: bool = True) list[lsst.daf.butler._file_dataset.FileDataset] ¶
Ingest all raw files in one exposure.
- Parameters:
- exposure
RawExposureData
A structure containing information about the exposure to be ingested. Must have
RawExposureData.records
populated and all data ID attributes expanded.- datasetType
DatasetType
The dataset type associated with this exposure.
- run
str
Name of a RUN-type collection to write to.
- skip_existing_exposures
bool
, optional If
True
(False
is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.- track_file_attrs
bool
, optional Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implementation.
- exposure
- Returns:
- datasets
list
oflsst.daf.butler.FileDataset
Per-file structures identifying the files ingested and their dataset representation in the data repository.
- datasets
- ingestFiles(files: Iterable[ResourcePath], *, pool: Any | None = None, processes: int = 1, run: str | None = None, skip_existing_exposures: bool = False, update_exposure_records: bool = False, track_file_attrs: bool = True) tuple[list[lsst.daf.butler._dataset_ref.DatasetRef], list[lsst.resources._resourcePath.ResourcePath], int, int, int] ¶
Ingest files into a Butler data repository.
This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before
run
is called.- Parameters:
- filesiterable over
lsst.resources.ResourcePath
URIs to the files to be ingested.
- pool
multiprocessing.Pool
, optional If not
None
, a process pool with which to parallelize some operations.- processes
int
, optional The number of processes to use. Ignored if
pool
is notNone
.- run
str
, optional Name of a RUN-type collection to write to, overriding the default derived from the instrument name.
- skip_existing_exposures
bool
, optional If
True
(False
is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.- update_exposure_records
bool
, optional If
True
(False
is default), update existing exposure records that conflict with the new ones instead of rejecting them. THIS IS AN ADVANCED OPTION THAT SHOULD ONLY BE USED TO FIX METADATA THAT IS KNOWN TO BE BAD. This should usually be combined withskip_existing_exposures=True
.- track_file_attrs
bool
, optional Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implementation.
- filesiterable over
- Returns:
- refs
list
oflsst.daf.butler.DatasetRef
Dataset references for ingested raws.
- bad_files
list
ofResourcePath
Given paths that could not be ingested.
- n_exposures
int
Number of exposures successfully ingested.
- n_exposures_failed
int
Number of exposures that failed when inserting dimension data.
- n_ingests_failed
int
Number of exposures that failed when ingesting raw datasets.
- refs
- locateAndReadIndexFiles(files: Iterable[ResourcePath]) tuple[dict[lsst.resources._resourcePath.ResourcePath, Any], list[lsst.resources._resourcePath.ResourcePath], set[lsst.resources._resourcePath.ResourcePath], set[lsst.resources._resourcePath.ResourcePath]] ¶
Given a list of files, look for index files and read them.
Index files can either be explicitly in the list of files to ingest, or else located in the same directory as a file to ingest. Index entries are always used if present.
- Parameters:
- filesiterable over
lsst.resources.ResourcePath
URIs to the files to be ingested.
- filesiterable over
- Returns:
- index
dict
[ResourcePath
, Any] Merged contents of all relevant index files found. These can be explicitly specified index files or ones found in the directory alongside a data file to be ingested.
- updated_files
list
ofResourcePath
Updated list of the input files with entries removed that were found listed in an index file. Order is not guaranteed to match the order of the files given to this routine.
- good_index_files:
set
[ResourcePath
] Index files that were successfully read.
- bad_index_files:
set
[ResourcePath
] Files that looked like index files but failed to read properly.
- index
- makeDependencyRecords(obsInfo: ObservationInfo, universe: DimensionUniverse) dict[str, lsst.daf.butler.dimensions._records.DimensionRecord] ¶
Construct dependency records.
These dependency records will be inserted into the
Registry
before the exposure records, because they are dependencies of the exposure. This allows an opportunity to satisfy foreign key constraints that exist because of dimensions related to the exposure.This is a method that subclasses may want to customize, if they’ve added dimensions that relate to an exposure.
- makeExposureRecord(obsInfo: ObservationInfo, universe: DimensionUniverse, **kwargs: Any) DimensionRecord ¶
Construct a registry record for an exposure.
This is a method that subclasses will often want to customize. This can often be done by calling this base class implementation with additional
kwargs
.- Parameters:
- obsInfo
ObservationInfo
Observation details for (one of the components of) the exposure.
- universe
DimensionUniverse
Set of all known dimensions.
- **kwargs
Additional field values for this record.
- obsInfo
- Returns:
- record
DimensionRecord
The exposure record that must be inserted into the
Registry
prior to file-level ingest.
- record
- classmethod makeField(doc: str) ConfigurableField ¶
Make a
lsst.pex.config.ConfigurableField
for this task.- Parameters:
- doc
str
Help text for the field.
- doc
- Returns:
- configurableField
lsst.pex.config.ConfigurableField
A
ConfigurableField
for this task.
- configurableField
Examples
Provides a convenient way to specify this task is a subtask of another task.
Here is an example of use:
class OtherTaskConfig(lsst.pex.config.Config): aSubtask = ATaskClass.makeField("brief description of task")
- makeSubtask(name: str, **keyArgs: Any) None ¶
Create a subtask as a new instance as the
name
attribute of this task.- Parameters:
- name
str
Brief name of the subtask.
- **keyArgs
Extra keyword arguments used to construct the task. The following arguments are automatically provided and cannot be overridden:
config
.parentTask
.
- name
Notes
The subtask must be defined by
Task.config.name
, an instance ofConfigurableField
orRegistryField
.
- prep(files: Iterable[ResourcePath], *, pool: Any | None = None) tuple[collections.abc.Iterator[lsst.obs.base.ingest.RawExposureData], list[lsst.resources._resourcePath.ResourcePath]] ¶
Perform all non-database-updating ingest preprocessing steps.
- Parameters:
- Returns:
- processIndexEntries(index_entries: dict[lsst.resources._resourcePath.ResourcePath, Any]) list[lsst.obs.base.ingest.RawFileData] ¶
Convert index entries to RawFileData.
- Parameters:
- index_entries
dict
[ResourcePath
, Any] Dict indexed by name of file to ingest and with keys either raw metadata or translated
ObservationInfo
.
- index_entries
- Returns:
- data
list
[RawFileData
] Structures containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the
RawFileData.dataId
attributes will be minimal (unexpanded)DataCoordinate
instances.
- data
- run(files: Iterable[str | ParseResult | ResourcePath | Path], *, pool: Any | None = None, processes: int = 1, run: str | None = None, file_filter: str | Pattern = '\\.fit[s]?\\b', group_files: bool = True, skip_existing_exposures: bool = False, update_exposure_records: bool = False, track_file_attrs: bool = True) list[lsst.daf.butler._dataset_ref.DatasetRef] ¶
Ingest files into a Butler data repository.
This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before
run
is called.- Parameters:
- filesiterable
lsst.resources.ResourcePath
,str
or path-like Paths to the files to be ingested. Can refer to directories. Will be made absolute if they are not already.
- pool
multiprocessing.Pool
, optional If not
None
, a process pool with which to parallelize some operations.- processes
int
, optional The number of processes to use. Ignored if
pool
is notNone
.- run
str
, optional Name of a RUN-type collection to write to, overriding the default derived from the instrument name.
- file_filter
str
orre.Pattern
, optional Pattern to use to discover files to ingest within directories. The default is to search for FITS files. The regex applies to files within the directory.
- group_files
bool
, optional Group files by directory if they have been discovered in directories. Will not affect files explicitly provided.
- skip_existing_exposures
bool
, optional If
True
(False
is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.- update_exposure_records
bool
, optional If
True
(False
is default), update existing exposure records that conflict with the new ones instead of rejecting them. THIS IS AN ADVANCED OPTION THAT SHOULD ONLY BE USED TO FIX METADATA THAT IS KNOWN TO BE BAD. This should usually be combined withskip_existing_exposures=True
.- track_file_attrs
bool
, optional Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implementation.
- filesiterable
- Returns:
- refs
list
oflsst.daf.butler.DatasetRef
Dataset references for ingested raws.
- refs
Notes
This method inserts all datasets for an exposure within a transaction, guaranteeing that partial exposures are never ingested. The exposure dimension record is inserted with
Registry.syncDimensionData
first (in its own transaction), which inserts only if a record with the same primary key does not already exist. This allows different files within the same exposure to be ingested in different runs.