RawIngestTask

class lsst.obs.base.RawIngestTask(config: ~lsst.obs.base.ingest.RawIngestConfig, *, butler: ~lsst.daf.butler._butler.Butler, on_success: ~typing.Callable[[~typing.List[~lsst.daf.butler.core.fileDataset.FileDataset]], ~typing.Any] = <function _do_nothing>, on_metadata_failure: ~typing.Callable[[~lsst.resources._resourcePath.ResourcePath, Exception], ~typing.Any] = <function _do_nothing>, on_ingest_failure: ~typing.Callable[[~lsst.obs.base.ingest.RawExposureData, Exception], ~typing.Any] = <function _do_nothing>, **kwargs: ~typing.Any)

Bases: Task

Driver Task for ingesting raw data into Gen3 Butler repositories.

Parameters:

configRawIngestConfig

Configuration for the task.

butlerButler

Writeable butler instance, with butler.run set to the appropriate RUN collection for these raw datasets.

on_successCallable, optional

A callback invoked when all of the raws associated with an exposure are ingested. Will be passed a list of FileDataset objects, each containing one or more resolved DatasetRef objects. If this callback raises it will interrupt the entire ingest process, even if RawIngestConfig.failFast is False.

on_metadata_failureCallable, optional

A callback invoked when a failure occurs trying to translate the metadata for a file. Will be passed the URI and the exception, in that order, as positional arguments. Guaranteed to be called in an except block, allowing the callback to re-raise or replace (with raise ... from) to override the task’s usual error handling (before RawIngestConfig.failFast logic occurs).

on_ingest_failureCallable, optional

A callback invoked when dimension record or dataset insertion into the database fails for an exposure. Will be passed a RawExposureData instance and the exception, in that order, as positional arguments. Guaranteed to be called in an except block, allowing the callback to re-raise or replace (with raise ... from) to override the task’s usual error handling (before RawIngestConfig.failFast logic occurs).

**kwargs

Additional keyword arguments are forwarded to the lsst.pipe.base.Task constructor.

Notes

Each instance of RawIngestTask writes to the same Butler. Each invocation of RawIngestTask.run ingests a list of files.

Methods Summary

emptyMetadata()	Empty (clear) the metadata for this Task and all sub-Tasks.
expandDataIds(data)	Expand the data IDs associated with a raw exposure.
extractMetadata(filename)	Extract and process metadata from a single raw file.
getDatasetType()	Return the DatasetType of the datasets ingested by this Task.
getFullMetadata()	Get metadata for all tasks.
getFullName()	Get the task name as a hierarchical name including parent task names.
getName()	Get the name of the task.
getObservationInfoSubsets()	Return subsets of fields in the ObservationInfo that we care about
getTaskDict()	Get a dictionary of all tasks as a shallow copy.
groupByExposure(files)	Group an iterable of RawFileData by exposure.
ingestExposureDatasets(exposure, *[, run, ...])	Ingest all raw files in one exposure.
ingestFiles(files, *[, pool, processes, ...])	Ingest files into a Butler data repository.
locateAndReadIndexFiles(files)	Given a list of files, look for index files and read them.
makeDependencyRecords(obsInfo, universe)	Construct dependency records
makeExposureRecord(obsInfo, universe, **kwargs)	Construct a registry record for an exposure
makeField(doc)	Make a lsst.pex.config.ConfigurableField for this task.
makeSubtask(name, **keyArgs)	Create a subtask as a new instance as the name attribute of this task.
prep(files, *[, pool, processes])	Perform all non-database-updating ingest preprocessing steps.
processIndexEntries(index_entries)	Convert index entries to RawFileData.
run(files, *[, pool, processes, run, ...])	Ingest files into a Butler data repository.
timer(name[, logLevel])	Context manager to log performance data for an arbitrary block of code.

Methods Documentation

emptyMetadata() → None

Empty (clear) the metadata for this Task and all sub-Tasks.

expandDataIds(data: RawExposureData) → RawExposureData

Expand the data IDs associated with a raw exposure.

This adds the metadata records.

Parameters:

exposureRawExposureData

A structure containing information about the exposure to be ingested. Must have RawExposureData.record populated. Should be considered consumed upon return.

Returns:

exposureRawExposureData

An updated version of the input structure, with RawExposureData.dataId and nested RawFileData.dataId attributes updated to data IDs for which hasRecords returns True.

extractMetadata(filename: ResourcePath) → RawFileData

Extract and process metadata from a single raw file.

Parameters:

filenamelsst.resources.ResourcePath

URI to the file.

Returns:

dataRawFileData

A structure containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the RawFileData.dataId attribute will be a minimal (unexpanded) DataCoordinate instance. The instrument field will be None if there is a problem with metadata extraction.

Notes

Assumes that there is a single dataset associated with the given file. Instruments using a single file to store multiple datasets must implement their own version of this method.

By default the method will catch all exceptions unless the failFast configuration item is True. If an error is encountered the _on_metadata_failure() method will be called. If no exceptions result and an error was encountered the returned object will have a null-instrument class and no datasets.

This method supports sidecar JSON files which can be used to extract metadata without having to read the data file itself. The sidecar file is always used if found.

getDatasetType() → DatasetType

Return the DatasetType of the datasets ingested by this Task.

getFullMetadata() → TaskMetadata

Get metadata for all tasks.

Returns:

metadataTaskMetadata

The keys are the full task name. Values are metadata for the top-level task and all subtasks, sub-subtasks, etc.

Notes

The returned metadata includes timing information (if @timer.timeMethod is used) and any metadata set by the task. The name of each item consists of the full task name with . replaced by :, followed by . and the name of the item, e.g.:

topLevelTaskName:subtaskName:subsubtaskName.itemName

using : in the full task name disambiguates the rare situation that a task has a subtask and a metadata item with the same name.

getFullName() → str

Get the task name as a hierarchical name including parent task names.

Returns:

fullNamestr

The full name consists of the name of the parent task and each subtask separated by periods. For example:

• The full name of top-level task “top” is simply “top”.

• The full name of subtask “sub” of top-level task “top” is “top.sub”.

• The full name of subtask “sub2” of subtask “sub” of top-level task “top” is “top.sub.sub2”.

getName() → str

Get the name of the task.

Returns:

taskNamestr

Name of the task.

See also

getFullName

classmethod getObservationInfoSubsets() → Tuple[Set, Set]

Return subsets of fields in the ObservationInfo that we care about

These fields will be used in constructing an exposure record.

Returns:

requiredset

Set of ObservationInfo field names that are required.

optionalset

Set of ObservationInfo field names we will use if they are available.

getTaskDict() → Dict[str, ReferenceType[Task]]

Get a dictionary of all tasks as a shallow copy.

Returns:

taskDictdict

Dictionary containing full task name: task object for the top-level task and all subtasks, sub-subtasks, etc.

groupByExposure(files: Iterable[RawFileData]) → List[RawExposureData]

Group an iterable of RawFileData by exposure.

Parameters:

filesiterable of RawFileData

File-level information to group.

Returns:

exposureslist of RawExposureData

A list of structures that group the file-level information by exposure. All fields will be populated. The RawExposureData.dataId attributes will be minimal (unexpanded) DataCoordinate instances.

ingestExposureDatasets(exposure: RawExposureData, *, run: str | None = None, skip_existing_exposures: bool = False, track_file_attrs: bool = True) → List[FileDataset]

Ingest all raw files in one exposure.

Parameters:

exposureRawExposureData

A structure containing information about the exposure to be ingested. Must have RawExposureData.records populated and all data ID attributes expanded.

runstr, optional

Name of a RUN-type collection to write to, overriding self.butler.run.

skip_existing_exposuresbool, optional

If True (False is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.

track_file_attrsbool, optional

Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implentation.

Returns:

datasetslist of lsst.daf.butler.FileDataset

Per-file structures identifying the files ingested and their dataset representation in the data repository.

ingestFiles(files: Iterable[ResourcePath], *, pool: Any | None = None, processes: int = 1, run: str | None = None, skip_existing_exposures: bool = False, update_exposure_records: bool = False, track_file_attrs: bool = True) → Tuple[List[DatasetRef], List[ResourcePath], int, int, int]

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before run is called.

Parameters:

filesiterable over lsst.resources.ResourcePath

URIs to the files to be ingested.

poolmultiprocessing.Pool, optional

If not None, a process pool with which to parallelize some operations.

processesint, optional

The number of processes to use. Ignored if pool is not None.

runstr, optional

Name of a RUN-type collection to write to, overriding the default derived from the instrument name.

skip_existing_exposuresbool, optional

If True (False is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.

update_exposure_recordsbool, optional

If True (False is default), update existing exposure records that conflict with the new ones instead of rejecting them. THIS IS AN ADVANCED OPTION THAT SHOULD ONLY BE USED TO FIX METADATA THAT IS KNOWN TO BE BAD. This should usually be combined with skip_existing_exposures=True.

track_file_attrsbool, optional

Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implentation.

Returns:

refslist of lsst.daf.butler.DatasetRef

Dataset references for ingested raws.

bad_fileslist of ResourcePath

Given paths that could not be ingested.

n_exposuresint

Number of exposures successfully ingested.

n_exposures_failedint

Number of exposures that failed when inserting dimension data.

n_ingests_failedint

Number of exposures that failed when ingesting raw datasets.

locateAndReadIndexFiles(files: Iterable[ResourcePath]) → Tuple[Dict[ResourcePath, Any], List[ResourcePath], Set[ResourcePath], Set[ResourcePath]]

Given a list of files, look for index files and read them.

Index files can either be explicitly in the list of files to ingest, or else located in the same directory as a file to ingest. Index entries are always used if present.

Parameters:

filesiterable over lsst.resources.ResourcePath

URIs to the files to be ingested.

Returns:

indexdict [ResourcePath, Any]

Merged contents of all relevant index files found. These can be explicitly specified index files or ones found in the directory alongside a data file to be ingested.

updated_fileslist of ResourcePath

Updated list of the input files with entries removed that were found listed in an index file. Order is not guaranteed to match the order of the files given to this routine.

good_index_files: set [ ResourcePath ]

Index files that were successfully read.

bad_index_files: set [ ResourcePath ]

Files that looked like index files but failed to read properly.

makeDependencyRecords(obsInfo: ObservationInfo, universe: DimensionUniverse) → Dict[str, DimensionRecord]

Construct dependency records

These dependency records will be inserted into the Registry before the exposure records, because they are dependencies of the exposure. This allows an opportunity to satisfy foreign key constraints that exist because of dimensions related to the exposure.

This is a method that subclasses may want to customize, if they’ve added dimensions that relate to an exposure.

Parameters:

obsInfoObservationInfo

Observation details for (one of the components of) the exposure.

universeDimensionUniverse

Set of all known dimensions.

Returns:

recordsdict [str, DimensionRecord]

The records to insert, indexed by dimension name.

makeExposureRecord(obsInfo: ObservationInfo, universe: DimensionUniverse, **kwargs: Any) → DimensionRecord

Construct a registry record for an exposure

This is a method that subclasses will often want to customize. This can often be done by calling this base class implementation with additional kwargs.

Parameters:

obsInfoObservationInfo

Observation details for (one of the components of) the exposure.

universeDimensionUniverse

Set of all known dimensions.

**kwargs

Additional field values for this record.

Returns:

recordDimensionRecord

The exposure record that must be inserted into the Registry prior to file-level ingest.

classmethod makeField(doc: str) → ConfigurableField

Make a lsst.pex.config.ConfigurableField for this task.

Parameters:

docstr

Help text for the field.

Returns:

configurableFieldlsst.pex.config.ConfigurableField

A ConfigurableField for this task.

Examples

Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use:

class OtherTaskConfig(lsst.pex.config.Config):
    aSubtask = ATaskClass.makeField("brief description of task")

makeSubtask(name: str, **keyArgs: Any) → None

Create a subtask as a new instance as the name attribute of this task.

Parameters:

namestr

Brief name of the subtask.

keyArgs

Extra keyword arguments used to construct the task. The following arguments are automatically provided and cannot be overridden:

• “config”.

• “parentTask”.

Notes

The subtask must be defined by Task.config.name, an instance of ConfigurableField or RegistryField.

prep(files: Iterable[ResourcePath], *, pool: Any | None = None, processes: int = 1) → Tuple[Iterator[RawExposureData], List[ResourcePath]]

Perform all non-database-updating ingest preprocessing steps.

Parameters:

filesiterable over str or path-like objects

Paths to the files to be ingested. Will be made absolute if they are not already.

poolmultiprocessing.Pool, optional

If not None, a process pool with which to parallelize some operations.

processesint, optional

The number of processes to use. Ignored if pool is not None.

Returns:

exposuresIterator [ RawExposureData ]

Data structures containing dimension records, filenames, and data IDs to be ingested (one structure for each exposure).

bad_fileslist of str

List of all the files that could not have metadata extracted.

processIndexEntries(index_entries: Dict[ResourcePath, Any]) → List[RawFileData]

Convert index entries to RawFileData.

Parameters:

index_entriesdict [ResourcePath, Any]

Dict indexed by name of file to ingest and with keys either raw metadata or translated ObservationInfo.

Returns:

datalist [ RawFileData ]

Structures containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the RawFileData.dataId attributes will be minimal (unexpanded) DataCoordinate instances.

run(files: Iterable[str | ParseResult | ResourcePath | Path], *, pool: Any | None = None, processes: int = 1, run: str | None = None, file_filter: str | Pattern = '\\.fit[s]?\\b', group_files: bool = True, skip_existing_exposures: bool = False, update_exposure_records: bool = False, track_file_attrs: bool = True) → List[DatasetRef]

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before run is called.

Parameters:

filesiterable lsst.resources.ResourcePath, str or path-like

Paths to the files to be ingested. Can refer to directories. Will be made absolute if they are not already.

poolmultiprocessing.Pool, optional

If not None, a process pool with which to parallelize some operations.

processesint, optional

The number of processes to use. Ignored if pool is not None.

runstr, optional

Name of a RUN-type collection to write to, overriding the default derived from the instrument name.

file_filterstr or re.Pattern, optional

Pattern to use to discover files to ingest within directories. The default is to search for FITS files. The regex applies to files within the directory.

group_filesbool, optional

Group files by directory if they have been discovered in directories. Will not affect files explicitly provided.

skip_existing_exposuresbool, optional

If True (False is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.

update_exposure_recordsbool, optional

If True (False is default), update existing exposure records that conflict with the new ones instead of rejecting them. THIS IS AN ADVANCED OPTION THAT SHOULD ONLY BE USED TO FIX METADATA THAT IS KNOWN TO BE BAD. This should usually be combined with skip_existing_exposures=True.

track_file_attrsbool, optional

Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implentation.

Returns:

refslist of lsst.daf.butler.DatasetRef

Dataset references for ingested raws.

Notes

This method inserts all datasets for an exposure within a transaction, guaranteeing that partial exposures are never ingested. The exposure dimension record is inserted with Registry.syncDimensionData first (in its own transaction), which inserts only if a record with the same primary key does not already exist. This allows different files within the same exposure to be ingested in different runs.

timer(name: str, logLevel: int = 10) → Iterator[None]

Context manager to log performance data for an arbitrary block of code.

Parameters:

namestr

Name of code being timed; data will be logged using item name: Start and End.

logLevel

A logging level constant.

See also

timer.logInfo

Examples

Creating a timer context:

with self.timer("someCodeToTime"):
    pass  # code to time