MegaPrimeRawIngestTask¶

class lsst.obs.cfht.ingest.MegaPrimeRawIngestTask(*args, **kwargs)¶

Bases: lsst.obs.base.RawIngestTask

Task for ingesting raw MegaPrime multi-extension FITS data into Gen3.

Deprecated since version 22.0: MegaPrime no longer requires a specialist gen3 ingest task. Please use the default. Will be removed after v23.

Methods Summary

`emptyMetadata`()	Empty (clear) the metadata for this Task and all sub-Tasks.
`expandDataIds`(data)	Expand the data IDs associated with a raw exposure.
`extractMetadata`(filename)	Extract and process metadata from a single raw file.
`getAllSchemaCatalogs`()	Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
`getDatasetType`()	Return the DatasetType of the datasets ingested by this Task.
`getFullMetadata`()	Get metadata for all tasks.
`getFullName`()	Get the task name as a hierarchical name including parent task names.
`getName`()	Get the name of the task.
`getSchemaCatalogs`()	Get the schemas generated by this task.
`getTaskDict`()	Get a dictionary of all tasks as a shallow copy.
`groupByExposure`(files)	Group an iterable of `RawFileData` by exposure.
`ingestExposureDatasets`(exposure, *[, run])	Ingest all raw files in one exposure.
`ingestFiles`(files, *[, pool, processes, run])	Ingest files into a Butler data repository.
`locateAndReadIndexFiles`(files)	Given a list of files, look for index files and read them.
`makeField`(doc)	Make a `lsst.pex.config.ConfigurableField` for this task.
`makeSubtask`(name, **keyArgs)	Create a subtask as a new instance as the `name` attribute of this task.
`prep`(files, *[, pool, processes])	Perform all non-database-updating ingest preprocessing steps.
`processIndexEntries`(index_entries)	Convert index entries to RawFileData.
`run`(files, *[, pool, processes, run, …])	Ingest files into a Butler data repository.
`timer`(name[, logLevel])	Context manager to log performance data for an arbitrary block of code.

Methods Documentation

emptyMetadata()¶: Empty (clear) the metadata for this Task and all sub-Tasks.

expandDataIds(data: lsst.obs.base.ingest.RawExposureData) → lsst.obs.base.ingest.RawExposureData¶

Expand the data IDs associated with a raw exposure.

This adds the metadata records.

Parameters

exposureRawExposureData: A structure containing information about the exposure to be ingested. Must have RawExposureData.records populated. Should be considered consumed upon return.

Returns

exposureRawExposureData: An updated version of the input structure, with RawExposureData.dataId and nested RawFileData.dataId attributes updated to data IDs for which hasRecords returns True.

extractMetadata(filename: lsst.daf.butler.ButlerURI) → lsst.obs.base.ingest.RawFileData¶

Extract and process metadata from a single raw file.

Parameters

filenameButlerURI: URI to the file.

Returns

dataRawFileData: A structure containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the RawFileData.dataId attribute will be a minimal (unexpanded) DataCoordinate instance. The instrument field will be None if there is a problem with metadata extraction.

Notes

Assumes that there is a single dataset associated with the given file. Instruments using a single file to store multiple datasets must implement their own version of this method.

By default the method will catch all exceptions unless the failFast configuration item is True. If an error is encountered the _on_metadata_failure() method will be called. If no exceptions result and an error was encountered the returned object will have a null-instrument class and no datasets.

This method supports sidecar JSON files which can be used to extract metadata without having to read the data file itself. The sidecar file is always used if found.

getAllSchemaCatalogs()¶

Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

Returns

schemacatalogsdict: Keys are butler dataset type, values are a empty catalog (an instance of the appropriate lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down through all subtasks.

Notes

This method may be called on any task in the hierarchy; it will return the same answer, regardless.

The default implementation should always suffice. If your subtask uses schemas the override Task.getSchemaCatalogs, not this method.

getDatasetType()¶: Return the DatasetType of the datasets ingested by this Task.

getFullMetadata()¶

Get metadata for all tasks.

Returns

metadatalsst.daf.base.PropertySet: The PropertySet keys are the full task name. Values are metadata for the top-level task and all subtasks, sub-subtasks, etc.

Notes

The returned metadata includes timing information (if @timer.timeMethod is used) and any metadata set by the task. The name of each item consists of the full task name with . replaced by :, followed by . and the name of the item, e.g.:

topLevelTaskName:subtaskName:subsubtaskName.itemName

using : in the full task name disambiguates the rare situation that a task has a subtask and a metadata item with the same name.

getFullName()¶

Get the task name as a hierarchical name including parent task names.

Returns

fullNamestr

The full name consists of the name of the parent task and each subtask separated by periods. For example:

The full name of top-level task “top” is simply “top”.
The full name of subtask “sub” of top-level task “top” is “top.sub”.
The full name of subtask “sub2” of subtask “sub” of top-level task “top” is “top.sub.sub2”.

getName()¶

Get the name of the task.

Returns

taskNamestr: Name of the task.

See also

Task.getAllSchemaCatalogs

Notes

Warning

Subclasses that use schemas must override this method. The default implementation returns an empty dict.

This method may be called at any time after the Task is constructed, which means that all task schemas should be computed at construction time, not when data is actually processed. This reflects the philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

getTaskDict()¶

Get a dictionary of all tasks as a shallow copy.

Returns

taskDictdict: Dictionary containing full task name: task object for the top-level task and all subtasks, sub-subtasks, etc.

groupByExposure(files: Iterable[lsst.obs.base.ingest.RawFileData]) → List[lsst.obs.base.ingest.RawExposureData]¶

Group an iterable of RawFileData by exposure.

Parameters

filesiterable of RawFileData: File-level information to group.

Returns

exposureslist of RawExposureData: A list of structures that group the file-level information by exposure. All fields will be populated. The RawExposureData.dataId attributes will be minimal (unexpanded) DataCoordinate instances.

ingestExposureDatasets(exposure: lsst.obs.base.ingest.RawExposureData, *, run: Optional[str] = None) → List[lsst.daf.butler.FileDataset]¶

Ingest all raw files in one exposure.

Parameters

exposureRawExposureData: A structure containing information about the exposure to be ingested. Must have RawExposureData.records populated and all data ID attributes expanded.
runstr, optional: Name of a RUN-type collection to write to, overriding self.butler.run.

Returns

datasetslist of lsst.daf.butler.FileDataset: Per-file structures identifying the files ingested and their dataset representation in the data repository.

ingestFiles(files, *, pool: Optional[multiprocessing.context.BaseContext.Pool] = None, processes: int = 1, run: Optional[str] = None)¶

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before run is called.

Parameters

filesiterable over ButlerURI: URIs to the files to be ingested.
poolmultiprocessing.Pool, optional: If not None, a process pool with which to parallelize some operations.
processesint, optional: The number of processes to use. Ignored if pool is not None.
runstr, optional: Name of a RUN-type collection to write to, overriding the default derived from the instrument name.

Returns

refslist of lsst.daf.butler.DatasetRef: Dataset references for ingested raws.

locateAndReadIndexFiles(files)¶

Given a list of files, look for index files and read them.

Index files can either be explicitly in the list of files to ingest, or else located in the same directory as a file to ingest. Index entries are always used if present.

Parameters

filesiterable over ButlerURI: URIs to the files to be ingested.

Returns

indexdict [str, Any]: Merged contents of all relevant index files found. These can be explicitly specified index files or ones found in the directory alongside a data file to be ingested.
updated_filesiterable of str: Updated list of the input files with entries removed that were found listed in an index file. Order is not guaranteed to match the order of the files given to this routine.
bad_index_files: set[str]: Files that looked like index files but failed to read properly.

classmethod makeField(doc)¶

Make a lsst.pex.config.ConfigurableField for this task.

Parameters

docstr: Help text for the field.

Returns

configurableFieldlsst.pex.config.ConfigurableField: A ConfigurableField for this task.

Examples

Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use:

class OtherTaskConfig(lsst.pex.config.Config):
    aSubtask = ATaskClass.makeField("brief description of task")

makeSubtask(name, **keyArgs)¶

Create a subtask as a new instance as the name attribute of this task.

Parameters

namestr

Brief name of the subtask.

keyArgs

Extra keyword arguments used to construct the task. The following arguments are automatically provided and cannot be overridden:

“config”.
“parentTask”.

Notes

The subtask must be defined by Task.config.name, an instance of ConfigurableField or RegistryField.

prep(files, *, pool: Optional[multiprocessing.context.BaseContext.Pool] = None, processes: int = 1) → Tuple[Iterator[lsst.obs.base.ingest.RawExposureData], List[str]]¶

Perform all non-database-updating ingest preprocessing steps.

Parameters

filesiterable over str or path-like objects: Paths to the files to be ingested. Will be made absolute if they are not already.
poolmultiprocessing.Pool, optional: If not None, a process pool with which to parallelize some operations.
processesint, optional: The number of processes to use. Ignored if pool is not None.

Returns

exposuresIterator [ RawExposureData ]: Data structures containing dimension records, filenames, and data IDs to be ingested (one structure for each exposure).
bad_fileslist of str: List of all the files that could not have metadata extracted.

processIndexEntries(index_entries)¶

Convert index entries to RawFileData.

Parameters

index_entriesdict [str, Any]: Dict indexed by name of file to ingest and with keys either raw metadata or translated ObservationInfo.

Returns

dataRawFileData: A structure containing the metadata extracted from the file, as well as the original filename. All fields will be populated, but the RawFileData.dataId attribute will be a minimal (unexpanded) DataCoordinate instance.

run(files, *, pool: Optional[multiprocessing.context.BaseContext.Pool] = None, processes: int = 1, run: Optional[str] = None, file_filter: Union[str, re.Pattern] = '\\.fit[s]?\\b', group_files: bool = True)¶

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before run is called.

Parameters

filesiterable over ButlerURI, str or path-like objects: Paths to the files to be ingested. Can refer to directories. Will be made absolute if they are not already.
poolmultiprocessing.Pool, optional: If not None, a process pool with which to parallelize some operations.
processesint, optional: The number of processes to use. Ignored if pool is not None.
runstr, optional: Name of a RUN-type collection to write to, overriding the default derived from the instrument name.
file_filterstr or re.Pattern, optional: Pattern to use to discover files to ingest within directories. The default is to search for FITS files. The regex applies to files within the directory.
group_filesbool, optional: Group files by directory if they have been discovered in directories. Will not affect files explicitly provided.

Returns

refslist of lsst.daf.butler.DatasetRef: Dataset references for ingested raws.

Notes

This method inserts all datasets for an exposure within a transaction, guaranteeing that partial exposures are never ingested. The exposure dimension record is inserted with Registry.syncDimensionData first (in its own transaction), which inserts only if a record with the same primary key does not already exist. This allows different files within the same exposure to be incremented in different runs.

timer(name, logLevel=10000)¶

Context manager to log performance data for an arbitrary block of code.

Parameters

namestr: Name of code being timed; data will be logged using item name: Start and End.
logLevel: A lsst.log level constant.

Navigation

MegaPrimeRawIngestTask¶