RawIngestTask¶

class lsst.obs.base.RawIngestTask(config: lsst.obs.base.ingest.RawIngestConfig, *, butler: lsst.daf.butler._butler.Butler, on_success: Callable[[List[lsst.daf.butler.core.fileDataset.FileDataset]], Any] = <function _do_nothing>, on_metadata_failure: Callable[[lsst.resources._resourcePath.ResourcePath, Exception], Any] = <function _do_nothing>, on_ingest_failure: Callable[[lsst.obs.base.ingest.RawExposureData, Exception], Any] = <function _do_nothing>, **kwargs)¶

Bases: lsst.pipe.base.Task

Driver Task for ingesting raw data into Gen3 Butler repositories.

Parameters:

config : RawIngestConfig: Configuration for the task.
butler : Butler: Writeable butler instance, with butler.run set to the appropriate RUN collection for these raw datasets.
on_success : Callable, optional: A callback invoked when all of the raws associated with an exposure are ingested. Will be passed a list of FileDataset objects, each containing one or more resolved DatasetRef objects. If this callback raises it will interrupt the entire ingest process, even if RawIngestConfig.failFast is False.
on_metadata_failure : Callable, optional: A callback invoked when a failure occurs trying to translate the metadata for a file. Will be passed the URI and the exception, in that order, as positional arguments. Guaranteed to be called in an except block, allowing the callback to re-raise or replace (with raise ... from) to override the task’s usual error handling (before RawIngestConfig.failFast logic occurs).
on_ingest_failure : Callable, optional: A callback invoked when dimension record or dataset insertion into the database fails for an exposure. Will be passed a RawExposureData instance and the exception, in that order, as positional arguments. Guaranteed to be called in an except block, allowing the callback to re-raise or replace (with raise ... from) to override the task’s usual error handling (before RawIngestConfig.failFast logic occurs).
**kwargs: Additional keyword arguments are forwarded to the lsst.pipe.base.Task constructor.

Notes

Each instance of RawIngestTask writes to the same Butler. Each invocation of RawIngestTask.run ingests a list of files.

Methods Summary

`emptyMetadata`()	Empty (clear) the metadata for this Task and all sub-Tasks.
`expandDataIds`(data)	Expand the data IDs associated with a raw exposure.
`extractMetadata`(filename)	Extract and process metadata from a single raw file.
`getAllSchemaCatalogs`()	Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
`getDatasetType`()	Return the DatasetType of the datasets ingested by this Task.
`getFullMetadata`()	Get metadata for all tasks.
`getFullName`()	Get the task name as a hierarchical name including parent task names.
`getName`()	Get the name of the task.
`getObservationInfoSubsets`()	Return subsets of fields in the `ObservationInfo` that we care about
`getSchemaCatalogs`()	Get the schemas generated by this task.
`getTaskDict`()	Get a dictionary of all tasks as a shallow copy.
`groupByExposure`(files)	Group an iterable of `RawFileData` by exposure.
`ingestExposureDatasets`(exposure, *, run, …)	Ingest all raw files in one exposure.
`ingestFiles`(files, *, pool, processes, run, …)	Ingest files into a Butler data repository.
`locateAndReadIndexFiles`(files)	Given a list of files, look for index files and read them.
`makeDependencyRecords`(obsInfo, universe)	Construct dependency records
`makeExposureRecord`(obsInfo, universe, **kwargs)	Construct a registry record for an exposure
`makeField`(doc)	Make a `lsst.pex.config.ConfigurableField` for this task.
`makeSubtask`(name, **keyArgs)	Create a subtask as a new instance as the `name` attribute of this task.
`prep`(files, *, pool, processes)	Perform all non-database-updating ingest preprocessing steps.
`processIndexEntries`(index_entries, Any])	Convert index entries to RawFileData.
`run`(files, urllib.parse.ParseResult, …)	Ingest files into a Butler data repository.
`timer`(name, logLevel)	Context manager to log performance data for an arbitrary block of code.

Methods Documentation

emptyMetadata() → None¶: Empty (clear) the metadata for this Task and all sub-Tasks.

expandDataIds(data: lsst.obs.base.ingest.RawExposureData) → lsst.obs.base.ingest.RawExposureData¶

Expand the data IDs associated with a raw exposure.

This adds the metadata records.

Parameters:	exposure : `RawExposureData` A structure containing information about the exposure to be ingested. Must have `RawExposureData.record` populated. Should be considered consumed upon return.
Returns:	exposure : `RawExposureData` An updated version of the input structure, with `RawExposureData.dataId` and nested `RawFileData.dataId` attributes updated to data IDs for which `hasRecords` returns `True`.

extractMetadata(filename: lsst.resources._resourcePath.ResourcePath) → lsst.obs.base.ingest.RawFileData¶

Extract and process metadata from a single raw file.

Parameters:	filename : `lsst.resources.ResourcePath` URI to the file.
Returns:	data : `RawFileData` A structure containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the `RawFileData.dataId` attribute will be a minimal (unexpanded) `DataCoordinate` instance. The `instrument` field will be `None` if there is a problem with metadata extraction.

Notes

Assumes that there is a single dataset associated with the given file. Instruments using a single file to store multiple datasets must implement their own version of this method.

By default the method will catch all exceptions unless the failFast configuration item is True. If an error is encountered the _on_metadata_failure() method will be called. If no exceptions result and an error was encountered the returned object will have a null-instrument class and no datasets.

This method supports sidecar JSON files which can be used to extract metadata without having to read the data file itself. The sidecar file is always used if found.

getAllSchemaCatalogs() → Dict[str, Any]¶

Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

Returns:	schemacatalogs : `dict` Keys are butler dataset type, values are a empty catalog (an instance of the appropriate `lsst.afw.table` Catalog type) for all tasks in the hierarchy, from the top-level task down through all subtasks.

Notes

This method may be called on any task in the hierarchy; it will return the same answer, regardless.

The default implementation should always suffice. If your subtask uses schemas the override Task.getSchemaCatalogs, not this method.

getDatasetType() → lsst.daf.butler.core.datasets.type.DatasetType¶: Return the DatasetType of the datasets ingested by this Task.

getFullMetadata() → lsst.pipe.base._task_metadata.TaskMetadata¶

Get metadata for all tasks.

Returns:	metadata : `TaskMetadata` The keys are the full task name. Values are metadata for the top-level task and all subtasks, sub-subtasks, etc.

Notes

The returned metadata includes timing information (if @timer.timeMethod is used) and any metadata set by the task. The name of each item consists of the full task name with . replaced by :, followed by . and the name of the item, e.g.:

topLevelTaskName:subtaskName:subsubtaskName.itemName

using : in the full task name disambiguates the rare situation that a task has a subtask and a metadata item with the same name.

getFullName() → str¶

Get the task name as a hierarchical name including parent task names.

Returns:	fullName : `str` The full name consists of the name of the parent task and each subtask separated by periods. For example: The full name of top-level task “top” is simply “top”. The full name of subtask “sub” of top-level task “top” is “top.sub”. The full name of subtask “sub2” of subtask “sub” of top-level task “top” is “top.sub.sub2”.

getName() → str¶

Get the name of the task.

Returns:	taskName : `str` Name of the task.

See also

Task.getAllSchemaCatalogs

Notes

Warning

Subclasses that use schemas must override this method. The default implementation returns an empty dict.

This method may be called at any time after the Task is constructed, which means that all task schemas should be computed at construction time, not when data is actually processed. This reflects the philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

getTaskDict() → Dict[str, weakref.ReferenceType[Task]]¶

Get a dictionary of all tasks as a shallow copy.

Returns:	taskDict : `dict` Dictionary containing full task name: task object for the top-level task and all subtasks, sub-subtasks, etc.

groupByExposure(files: Iterable[lsst.obs.base.ingest.RawFileData]) → List[lsst.obs.base.ingest.RawExposureData]¶

Group an iterable of RawFileData by exposure.

Parameters:	files : iterable of `RawFileData` File-level information to group.
Returns:	exposures : `list` of `RawExposureData` A list of structures that group the file-level information by exposure. All fields will be populated. The `RawExposureData.dataId` attributes will be minimal (unexpanded) `DataCoordinate` instances.

ingestExposureDatasets(exposure: lsst.obs.base.ingest.RawExposureData, *, run: Optional[str] = None, skip_existing_exposures: bool = False, track_file_attrs: bool = True) → List[lsst.daf.butler.core.fileDataset.FileDataset]¶

Ingest all raw files in one exposure.

Parameters:

exposure : RawExposureData: A structure containing information about the exposure to be ingested. Must have RawExposureData.records populated and all data ID attributes expanded.
run : str, optional: Name of a RUN-type collection to write to, overriding self.butler.run.
skip_existing_exposures : bool, optional: If True (False is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.
track_file_attrs : bool, optional: Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implentation.

Returns:

datasets : list of lsst.daf.butler.FileDataset: Per-file structures identifying the files ingested and their dataset representation in the data repository.

ingestFiles(files: Iterable[lsst.resources._resourcePath.ResourcePath], *, pool: Optional[Any] = None, processes: int = 1, run: Optional[str] = None, skip_existing_exposures: bool = False, update_exposure_records: bool = False, track_file_attrs: bool = True) → Tuple[List[lsst.daf.butler.core.datasets.ref.DatasetRef], List[lsst.resources._resourcePath.ResourcePath], int, int, int]¶

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before run is called.

Parameters:

files : iterable over lsst.resources.ResourcePath: URIs to the files to be ingested.
pool : multiprocessing.Pool, optional: If not None, a process pool with which to parallelize some operations.
processes : int, optional: The number of processes to use. Ignored if pool is not None.
run : str, optional: Name of a RUN-type collection to write to, overriding the default derived from the instrument name.
skip_existing_exposures : bool, optional: If True (False is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.
update_exposure_records : bool, optional: If True (False is default), update existing exposure records that conflict with the new ones instead of rejecting them. THIS IS AN ADVANCED OPTION THAT SHOULD ONLY BE USED TO FIX METADATA THAT IS KNOWN TO BE BAD. This should usually be combined with skip_existing_exposures=True.
track_file_attrs : bool, optional: Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implentation.

Returns:

refs : list of lsst.daf.butler.DatasetRef: Dataset references for ingested raws.
bad_files : list of ResourcePath: Given paths that could not be ingested.
n_exposures : int: Number of exposures successfully ingested.
n_exposures_failed : int: Number of exposures that failed when inserting dimension data.
n_ingests_failed : int: Number of exposures that failed when ingesting raw datasets.

locateAndReadIndexFiles(files: Iterable[lsst.resources._resourcePath.ResourcePath]) → Tuple[Dict[lsst.resources._resourcePath.ResourcePath, Any], List[lsst.resources._resourcePath.ResourcePath], Set[lsst.resources._resourcePath.ResourcePath], Set[lsst.resources._resourcePath.ResourcePath]]¶

Given a list of files, look for index files and read them.

Index files can either be explicitly in the list of files to ingest, or else located in the same directory as a file to ingest. Index entries are always used if present.

Parameters:

files : iterable over lsst.resources.ResourcePath: URIs to the files to be ingested.

Returns:

index : dict [ResourcePath, Any]: Merged contents of all relevant index files found. These can be explicitly specified index files or ones found in the directory alongside a data file to be ingested.
updated_files : list of ResourcePath: Updated list of the input files with entries removed that were found listed in an index file. Order is not guaranteed to match the order of the files given to this routine.
good_index_files: `set` [ `ResourcePath` ]: Index files that were successfully read.
bad_index_files: `set` [ `ResourcePath` ]: Files that looked like index files but failed to read properly.

makeDependencyRecords(obsInfo: astro_metadata_translator.observationInfo.ObservationInfo, universe: lsst.daf.butler.core.dimensions._universe.DimensionUniverse) → Dict[str, lsst.daf.butler.core.dimensions._records.DimensionRecord]¶

Construct dependency records

These dependency records will be inserted into the Registry before the exposure records, because they are dependencies of the exposure. This allows an opportunity to satisfy foreign key constraints that exist because of dimensions related to the exposure.

This is a method that subclasses may want to customize, if they’ve added dimensions that relate to an exposure.

Parameters:	obsInfo : `ObservationInfo` Observation details for (one of the components of) the exposure. universe : `DimensionUniverse` Set of all known dimensions.
Returns:	records : `dict` [`str`, `DimensionRecord`] The records to insert, indexed by dimension name.

makeExposureRecord(obsInfo: astro_metadata_translator.observationInfo.ObservationInfo, universe: lsst.daf.butler.core.dimensions._universe.DimensionUniverse, **kwargs) → lsst.daf.butler.core.dimensions._records.DimensionRecord¶

Construct a registry record for an exposure

This is a method that subclasses will often want to customize. This can often be done by calling this base class implementation with additional kwargs.

Parameters:	obsInfo : `ObservationInfo` Observation details for (one of the components of) the exposure. universe : `DimensionUniverse` Set of all known dimensions. **kwargs Additional field values for this record.
Returns:	record : `DimensionRecord` The exposure record that must be inserted into the `Registry` prior to file-level ingest.

classmethod makeField(doc: str) → lsst.pex.config.configurableField.ConfigurableField¶

Make a lsst.pex.config.ConfigurableField for this task.

Parameters:	doc : `str` Help text for the field.
Returns:	configurableField : `lsst.pex.config.ConfigurableField` A `ConfigurableField` for this task.

Examples

Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use:

class OtherTaskConfig(lsst.pex.config.Config):
    aSubtask = ATaskClass.makeField("brief description of task")

makeSubtask(name: str, **keyArgs) → None¶

Create a subtask as a new instance as the name attribute of this task.

Parameters:	name : `str` Brief name of the subtask. keyArgs Extra keyword arguments used to construct the task. The following arguments are automatically provided and cannot be overridden: “config”. “parentTask”.

Notes

The subtask must be defined by Task.config.name, an instance of ConfigurableField or RegistryField.

prep(files: Iterable[lsst.resources._resourcePath.ResourcePath], *, pool: Optional[Any] = None, processes: int = 1) → Tuple[Iterator[lsst.obs.base.ingest.RawExposureData], List[lsst.resources._resourcePath.ResourcePath]]¶

Perform all non-database-updating ingest preprocessing steps.

Parameters:	files : iterable over `str` or path-like objects Paths to the files to be ingested. Will be made absolute if they are not already. pool : `multiprocessing.Pool`, optional If not `None`, a process pool with which to parallelize some operations. processes : `int`, optional The number of processes to use. Ignored if `pool` is not `None`.
Returns:	exposures : `Iterator` [ `RawExposureData` ] Data structures containing dimension records, filenames, and data IDs to be ingested (one structure for each exposure). bad_files : `list` of `str` List of all the files that could not have metadata extracted.

processIndexEntries(index_entries: Dict[lsst.resources._resourcePath.ResourcePath, Any]) → List[lsst.obs.base.ingest.RawFileData]¶

Convert index entries to RawFileData.

Parameters:	index_entries : `dict` [`ResourcePath`, Any] Dict indexed by name of file to ingest and with keys either raw metadata or translated `ObservationInfo`.
Returns:	data : `list` [ `RawFileData` ] Structures containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the `RawFileData.dataId` attributes will be minimal (unexpanded) `DataCoordinate` instances.

run(files: Iterable[Union[str, urllib.parse.ParseResult, ResourcePath, pathlib.Path]], *, pool: Optional[Any] = None, processes: int = 1, run: Optional[str] = None, file_filter: Union[str, re.Pattern] = '\\.fit[s]?\\b', group_files: bool = True, skip_existing_exposures: bool = False, update_exposure_records: bool = False, track_file_attrs: bool = True) → List[lsst.daf.butler.core.datasets.ref.DatasetRef]¶

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before run is called.

Parameters:

files : iterable lsst.resources.ResourcePath, str or path-like: Paths to the files to be ingested. Can refer to directories. Will be made absolute if they are not already.
pool : multiprocessing.Pool, optional: If not None, a process pool with which to parallelize some operations.
processes : int, optional: The number of processes to use. Ignored if pool is not None.
run : str, optional: Name of a RUN-type collection to write to, overriding the default derived from the instrument name.
file_filter : str or re.Pattern, optional: Pattern to use to discover files to ingest within directories. The default is to search for FITS files. The regex applies to files within the directory.
group_files : bool, optional: Group files by directory if they have been discovered in directories. Will not affect files explicitly provided.
skip_existing_exposures : bool, optional: If True (False is default), skip raws that have already been ingested (i.e. raws for which we already have a dataset with the same data ID in the target collection, even if from another file). Note that this is much slower than just not passing already-ingested files as inputs, because we still need to read and process metadata to identify which exposures to search for. It also will not work reliably if multiple processes are attempting to ingest raws from the same exposure concurrently, in that different processes may still attempt to ingest the same raw and conflict, causing a failure that prevents other raws from the same exposure from being ingested.
update_exposure_records : bool, optional: If True (False is default), update existing exposure records that conflict with the new ones instead of rejecting them. THIS IS AN ADVANCED OPTION THAT SHOULD ONLY BE USED TO FIX METADATA THAT IS KNOWN TO BE BAD. This should usually be combined with skip_existing_exposures=True.
track_file_attrs : bool, optional: Control whether file attributes such as the size or checksum should be tracked by the datastore. Whether this parameter is honored depends on the specific datastore implentation.

Returns:

refs : list of lsst.daf.butler.DatasetRef: Dataset references for ingested raws.

Notes

This method inserts all datasets for an exposure within a transaction, guaranteeing that partial exposures are never ingested. The exposure dimension record is inserted with Registry.syncDimensionData first (in its own transaction), which inserts only if a record with the same primary key does not already exist. This allows different files within the same exposure to be ingested in different runs.

timer(name: str, logLevel: int = 10) → Iterator[None]¶

Context manager to log performance data for an arbitrary block of code.

Parameters:	name : `str` Name of code being timed; data will be logged using item name: `Start` and `End`. logLevel A `logging` level constant.

Navigation

RawIngestTask¶