obs_base v27.0.0 (2024-06-05)

New Features

    • Added support for ingesting raw data into a version 6 dimension universe. This universe includes day_obs and group as dimensions.

    • Modified visit definition to allow a visit to be defined by group dimension.

    • Added Instrument.translatorClass class property that can be used to specify the relevant astro_metadata_translator.MetadataTranslator to use.

    • Added Instrument.group_name_to_group_id method to convert a group name string to an integer suitable for use as a visit ID. (DM-42636)

  • Added a --update-records option to the butler ingest-raws command-line tool. This can be used if there has been a change in the metadata translator resulting in a change of definition of the exposure record. Only use this option if you understand why a change has occurred. (DM-43135)

  • Corrected and clarified docstrings for read_curated_calibs, read_one_calib and read_all functions, ensuring variable name consistency between these two and check_metadata. (DM-22465)

obs_base v26.0.0 (2023-08-03)

New Features

  • Added support for defining visits incrementally as exposures are ingested. This allows files from the telescope to be ingested one at a time whilst redefining the existing visits. Additionally --update-records and --incremental have been added to the butler define-visits command-line. (DM-36395)

    • Added transmission_ curve dataset types to the set of curated calibrations.

    • Updated the curated calibration code in the Instrument definition to allow for flexibility in required dimensions.

    • Updated the read curated calibration code to allow for the same flexibility in dimensions. (DM-36597)

  • Raw ingest can now ask the Instrument class for the raw dataset type definition. This means it is no longer required to subclass the getDatasetType method and allows various instruments to be ingested with the base class implementation. (DM-37950)

  • DefineVisitsTask now calls ObsCore table manager to update exposure regions after visit is defined. New configuration field updateObsCoreTable for that task can be set to False to disable exposure updates. (DM-38205)

An API Removal or Deprecation

  • Deprecated ExposureIdInfo in favor of lsst.meas.base.IdGenerator. (DM-31924)

  • Removed deprecated getInstrument function from lsst.obs.base.utils. (DM-37534)

Bug Fixes

  • Fixed curated calibration reading to check parent directory if there are no sub-directories. (DM-36598)

Other Changes and Additions

  • Modified the raw ingest task to use resolved DatasetRef. (DM-38779)

obs_base v25.0.0 (2023-03-02)

New Features

    • Removed Gen2 code from package, including 2to3 conversion code. Use an older release to convert any remaining Gen2 repositories to Gen3.

    • Moved support for reading curated calibrations for obs_*_data packages from pipe_tasks and added unit tests for this code. (DM-35035)

    • Added focusZ to MakeRawVisitInfoViaObsInfo. (DM-35186)

API Changes

  • The ingest-raws and define-visits subcommands no longer allow multiple config settings within a single --config option. We have decided that it is too dangerous to split on comma in the general case and so have removed that facility to be consistent with other commands. Use multiple --config options instead. (DM-35917)

An API Removal or Deprecation

  • Removed deprecated filterLabel exposure component access. (DM-27811)

obs_base v24.0.0 (2022-08-30)

New Features

    • Visits will now be defined for all on-sky observations regardless of the observation type.

    • Changed butler define-visits so that it now takes a --where option. This can be used to restrict the visit definition to specific exposures. (DM-33848)

  • Add a --fail-fast option to butler ingest-raws (DM-33891)

    • Modify ingest-raws to support new schema for exposure records.

    • Change define-visits to support new and old schema.

    • Visit system is now an enum rather than a configuration value.

    • Add new visit system to group by seq_start and seq_end and also allocate one-to-one visits for every exposure. (DM-33942)

    • lsst.obs.base.Instrument is now a subclass of lsst.pipe.base.Instrument. This simplifies the dependencies of ctrl_mpexec by removing any need to understand camera geometry or curated calibrations.

    • As part of this move the butler register-instrument command has been moved to pipe_base.

    • The PackagesFormatter has been moved to daf_butler and the PexConfigFormatter has been moved to pipe_base since both of these are required by ctrl_mpexec.

    • lsst.obs.base.utils.getInstrument has been replaced with Instrument.from_string. (DM-34105)

    • Made choice of required ObservationInfo properties configurable through RawIngestTask.getObservationInfoSubsets.

    • Added the concept of “dependency” records to be added to the registry before adding the exposure record; this makes it easier to satisfy foreign key constraints when the exposure relates to dimensions beyond the standard set.

    • Added RawIngestTask methods makeExposureRecord and makeDependencyRecords to provide hooks for subclasses to provide values for additional columns. (DM-34175)

API Changes

  • Add a new option --track-file-attrs to butler ingest-raws. This controls whether the ingested files should have file sizes and checksums tracked by the datastore. Use --no-track-files-attrs to disable size tracking. (DM-33086)

An API Removal or Deprecation

  • FilterDefinition no longer supports lsst.afw.image.Filter. The defineFilters and reset methods have been removed, as have all wavelength parameters to the FilterDefinition constructor.

    The old filter component for exposures has been removed, and replaced with a new filter component backed by lsst.afw.image.FilterLabel. It functions identically to the filterLabel component, which has been deprecated. (DM-27177)

  • Remove the processes and pool arguments and the --processes command-line argument from lsst.obs.base.DefineVisitsTask.run and butler define-visits (respectively). These were already broken for processes > 1, and internal parallelization here is no longer useful now that this task just does database I/O, not raw metadata reads. (DM-33783)

obs_base v23.0.0 (2021-12-10)

New Features

  • 2to3 conversion has been improved to add a dry run facility, to defer dataId expansion when not required, and to allow templates to be overridden. (DM-28636)

  • Reorganize the base Exposure and raw formatters to improve efficiency and clarify component handling. (DM-28698)

  • Add amp parameter to the formatters for the Exposure StorageClass, allowing single-amplifier subimage reads. (DM-29370)

  • Change raw ingest to use a reproducible UUID5 dataset ID. This means that the dataset ID for a raw ingested in one repository will be identical to that used in another. For integer-based registries this change will have no effect. (DM-29950)

  • Add support for updating exposure and visit definitions in RawIngestTask and DefineVisitsTask. (DM-30866)

  • Add support for forced updates of instrument, detector, and physical_filter definitions during instrument registration. (DM-31903)

Bug Fixes

  • Not all PSFs are persistable and now if one is encountered as part of composite disassembly it will be ignored. These types of PSFs were already silently dropped when writing a full Exposure. (DM-29794)

  • The butler define-visits command now correctly uses the --collections option to constrain the exposures that will be processed into visits. (DM-31079)

obs_base v22.0 (2021-04-01)

New Feature

  • Enhance raw data ingest such that there is no longer a need for a special subclass when ingesting DECam data. The metadata translator can now find additional headers itself. [DM-29166]

  • Add progress reporting to raw ingest, visit definition, and 2to3 conversion.

  • Change raw data ingest to support remote object stores. [DM-25965]

  • Raw data ingest now supports external metadata sidecar files or JSON per-directory index files. Creating these sidecar files in advance (using astrometadata write-index or astrometadata write-sidecar) can significantly improve ingest performance. This is especially useful if a particular test data set is commonly re-ingested. [DM-27476]

  • Raw data ingest has been modified to provide a callback feature when files fail to be ingested or are successfully ingested. This allows reporting tools to make detailed reports when doing bulk ingest. [DM-29071]

  • 2to3 conversion has been significantly improved. [DM-27147]

Other

  • When reading exposures the formatter now checks that the filter label in the DataId is consistent with the filter label read from the file. [DM-28583]