obs_base v24.0.0 (2022-08-30)

New Features

    • Visits will now be defined for all on-sky observations regardless of the observation type.

    • Changed butler define-visits so that it now takes a --where option. This can be used to restrict the visit definition to specific exposures. (DM-33848)

  • Add a --fail-fast option to butler ingest-raws (DM-33891)

    • Modify ingest-raws to support new schema for exposure records.

    • Change define-visits to support new and old schema.

    • Visit system is now an enum rather than a configuration value.

    • Add new visit system to group by seq_start and seq_end and also allocate one-to-one visits for every exposure. (DM-33942)

    • lsst.obs.base.Instrument is now a subclass of lsst.pipe.base.Instrument. This simplifies the dependencies of ctrl_mpexec by removing any need to understand camera geometry or curated calibrations.

    • As part of this move the butler register-instrument command has been moved to pipe_base.

    • The PackagesFormatter has been moved to daf_butler and the PexConfigFormatter has been moved to pipe_base since both of these are required by ctrl_mpexec.

    • lsst.obs.base.utils.getInstrument has been replaced with Instrument.from_string. (DM-34105)

    • Made choice of required ObservationInfo properties configurable through RawIngestTask.getObservationInfoSubsets.

    • Added the concept of “dependency” records to be added to the registry before adding the exposure record; this makes it easier to satisfy foreign key constraints when the exposure relates to dimensions beyond the standard set.

    • Added RawIngestTask methods makeExposureRecord and makeDependencyRecords to provide hooks for subclasses to provide values for additional columns. (DM-34175)

API Changes

  • Add a new option --track-file-attrs to butler ingest-raws. This controls whether the ingested files should have file sizes and checksums tracked by the datastore. Use --no-track-files-attrs to disable size tracking. (DM-33086)

An API Removal or Deprecation

  • FilterDefinition no longer supports Filter. The defineFilters and reset methods have been removed, as have all wavelength parameters to the FilterDefinition constructor.

    The old filter component for exposures has been removed, and replaced with a new filter component backed by FilterLabel. It functions identically to the filterLabel component, which has been deprecated. (DM-27177)

  • Remove the processes and pool arguments and the --processes command-line argument from lsst.obs.base.DefineVisitsTask.run and butler define-visits (respectively). These were already broken for processes > 1, and internal parallelization here is no longer useful now that this task just does database I/O, not raw metadata reads. (DM-33783)

obs_base v23.0.0 (2021-12-10)

New Features

  • 2to3 conversion has been improved to add a dry run facility, to defer dataId expansion when not required, and to allow templates to be overridden. (DM-28636)

  • Reorganize the base Exposure and raw formatters to improve efficiency and clarify component handling. (DM-28698)

  • Add amp parameter to the formatters for the Exposure StorageClass, allowing single-amplifier subimage reads. (DM-29370)

  • Change raw ingest to use a reproducible UUID5 dataset ID. This means that the dataset ID for a raw ingested in one repository will be identical to that used in another. For integer-based registries this change will have no effect. (DM-29950)

  • Add support for updating exposure and visit definitions in RawIngestTask and DefineVisitsTask. (DM-30866)

  • Add support for forced updates of instrument, detector, and physical_filter definitions during instrument registration. (DM-31903)

Bug Fixes

  • Not all PSFs are persistable and now if one is encountered as part of composite disassembly it will be ignored. These types of PSFs were already silently dropped when writing a full Exposure. (DM-29794)

  • The butler define-visits command now correctly uses the --collections option to constrain the exposures that will be processed into visits. (DM-31079)

obs_base v22.0 (2021-04-01)

New Feature

  • Enhance raw data ingest such that there is no longer a need for a special subclass when ingesting DECam data. The metadata translator can now find additional headers itself. [DM-29166]

  • Add progress reporting to raw ingest, visit definition, and 2to3 conversion.

  • Change raw data ingest to support remote object stores. [DM-25965]

  • Raw data ingest now supports external metadata sidecar files or JSON per-directory index files. Creating these sidecar files in advance (using astrometadata write-index or astrometadata write-sidecar) can significantly improve ingest performance. This is especially useful if a particular test data set is commonly re-ingested. [DM-27476]

  • Raw data ingest has been modified to provide a callback feature when files fail to be ingested or are successfully ingested. This allows reporting tools to make detailed reports when doing bulk ingest. [DM-29071]

  • 2to3 conversion has been significantly improved. [DM-27147]

Other

  • When reading exposures the formatter now checks that the filter label in the DataId is consistent with the filter label read from the file. [DM-28583]