.. _release-v23-0-0-tickets: ################################### Tickets Addressed in Release 23.0.0 ################################### Tickets merged after w_2021_40 and backported --------------------------------------------- - `DM-32714 `_: Fix backwards compatibility broken by DM-32649 [pipe_tasks] - `DM-32666 `_: Add GaussianFlux columns to SourceTable [obs_lsst, obs_subaru] - `DM-32649 `_: Move psf quality statistics calculation from selectImages to computeExposureSummaryStats [afw, pipe_tasks] - `DM-32624 `_: Override source selection criteria in SFM calibration stages for LSSTCam-imSim [meas_algorithms, obs_lsst] - `DM-32594 `_: Remove dependency on inflection in bps_config [ctrl_bps] - `DM-32504 `_: SingleQuantumExecutor should log exception messages [ctrl_mpexec] - `DM-32448 `_: Backport incorrect GAaP field name [obs_lsst] - `DM-32435 `_: Silent failure in executing memory-hungry job? [ctrl_bps] - `DM-32333 `_: SourceTable should use LocalNanojansky functor on apFluxes [obs_lsst, obs_subaru] - `DM-32324 `_: Add forced source table to sdm_schemas and ci_imsim [sdm_schemas] - `DM-32306 `_: refBand functor doesn't know that u-band exists [pipe_tasks] - `DM-32305 `_: lsst.daf.butler.Config slowness with large numbers of lookups [daf_butler] - `DM-32290 `_: A create-table failure led to registry corruption [daf_butler] - `DM-32241 `_: bps submit can take multiple hours [ctrl_bps] - `DM-32238 `_: Fix unexpected floating point values in drpAssociation task [pipe_tasks] - `DM-32220 `_: HTCondor plugin copying execution butler back from job [ctrl_bps] - `DM-32217 `_: BPS displays cumulative timings instead of time intervals during a submission [ctrl_bps] - `DM-32201 `_: HTCondor plugin bug if output collection contains period [ctrl_bps] - `DM-32198 `_: Set sentinel values for flag/integer columns for missing bands in Object tables [pipe_tasks] - `DM-32129 `_: Add a maximum mean offset threshold for SFM astrometry fit success [meas_astrom, obs_lsst] - `DM-32124 `_: ValueError: Only one node is allowed to produce forcedSource [pipe_tasks] - `DM-32074 `_: Modify how HTCondor plugin handles transfer of execution butler [ctrl_bps] - `DM-32066 `_: BPS jobs with memory autoscaling enabled remain idle after the first run attempt [ctrl_bps] - `DM-32030 `_: Fixup Jointcal after DM-31825 [jointcal] - `DM-32027 `_: Rename outCollection to outputRun [ctrl_bps] - `DM-31990 `_: Add configuration option to set coverage nside for survey-wide HealSparsePropertyMaps [pipe_tasks] - `DM-31976 `_: Change handling of collections and dataset types in dataset transfer [ctrl_mpexec, daf_butler] - `DM-31956 `_: mergeExecutionButler task hits database deadlock intermittently [daf_butler] - `DM-31777 `_: Determine and use correct position for PSF shape determination in ImageDifferenceTask [ip_diffim, pipe_tasks] Tickets merged before w_2021_40 ------------------------------- - `DM-6056 `_: Remove PTR and CONST_PTR macros [afw, base, daf_persistence, ip_diffim, meas_algorithms, meas_astrom, meas_base, meas_deblender, meas_extensions_photometryKron, meas_extensions_psfex, meas_modelfit, obs_subaru, shapelet] - `DM-11419 `_: afwTable rows define offsets as int32, causing problems with very large (>2Gb) rows [afw] - `DM-13089 `_: Use new visitInfo parallactic angle calculations in jointcal [jointcal] - `DM-13365 `_: Minimal on-disk caching Datastore [daf_butler] - `DM-13944 `_: add id to VisitInfo [afw, obs_base] - `DM-15361 `_: Make lsst.log outputs appear in Jupyter notebook cells [log] - `DM-15552 `_: Convert meas_deblender to numpydoc status [meas_deblender] - `DM-16556 `_: calcRmsDistances is comparing objects in correct distances due to indexing error [faro] - `DM-17471 `_: Save intermediate data product with pre-interpolation pixel values [ip_isr] - `DM-19392 `_: Interval class and geom Python enhancements: Stage 1 [afw, geom, sphgeom] - `DM-19634 `_: Longlog cmdline option suppresses DEBUG and TRACE messages issued by lsst.log [pipe_base] - `DM-22136 `_: inspect_job.py crashes on empty Job [verify] - `DM-22486 `_: afwTable will seg-fault if you add a new record to a table with schema=None [afw] - `DM-22534 `_: Implement cpSkyTask [cp_pipe, daf_butler] - `DM-23330 `_: Doxygen warnings in pipe_base [cp_pipe, ctrl_pool, meas_algorithms, pipe_base] - `DM-23694 `_: Create script for producing distortion model from Jointcal output [afw, geom, jointcal] - `DM-24278 `_: Apply proper motion to matched sources in Jointcal [jointcal] - `DM-24389 `_: Implement amp-to-amp offset correction task [ip_isr, obs_subaru] - `DM-24704 `_: Make brighter-fatter correction a subclass of lsst.ip.isr.IsrCalib [cp_pipe, daf_butler, ip_isr, obs_base, obs_lsst] - `DM-24762 `_: Add option to PTC task to correct for sigma clipping bias [cp_pipe] - `DM-24806 `_: Lift key ap_verify classes into lsst.ap.verify namespace [ap_verify] - `DM-24913 `_: obs_decam README is confusing [obs_decam] - `DM-25013 `_: Unclear error messages in pipetask run [ctrl_mpexec, daf_butler, pipe_base] - `DM-25789 `_: Update PA1 metric implementation [faro] - `DM-26031 `_: Fix broken measureCrosstalk test [cp_pipe] - `DM-26037 `_: Add option to ignore nPixels from edge in PTC task [cp_pipe] - `DM-26214 `_: Switch to using sourceTable and objectTable parquet files instead of the src FITS files [faro] - `DM-26397 `_: Add explicit DAG export from ctrl_bps [ctrl_bps] - `DM-26651 `_: Persist individual bright stars' warping Transforms, origin and number of rotations [meas_algorithms] - `DM-26719 `_: Resync with Spectractor upstream [obs_base, obs_lsst, obs_test] - `DM-27010 `_: Allow IsrTask to use PTC dataset as gain source [ip_isr] - `DM-27031 `_: Use primary flag in source count metrics [ip_diffim] - `DM-27032 `_: Create deblended source metric [ap_verify, ip_diffim, verify_metrics] - `DM-27083 `_: Improve jointcal outlier rejection [jointcal] - `DM-27103 `_: Not all IsrCalib tests round trip calibrations [ip_isr] - `DM-27241 `_: Add butler API for downloading file artifacts from datastore [daf_butler] - `DM-27260 `_: Change panDA code distribution to use preexisting mechanisms [ctrl_bps] - `DM-27387 `_: Implement scattering magnitudes for ApFakes [ap_pipe, ap_verify, pipe_tasks] - `DM-27412 `_: Rework daf.butler.tests.makeTestCollection [daf_butler] - `DM-27416 `_: Add doumentation on creating Pipelines [pipe_base] - `DM-27492 `_: Improvements to pipetask execution options [ctrl_mpexec, pipe_base] - `DM-27612 `_: Applying fringe correction adds numerous mask features [ip_isr] - `DM-27650 `_: Allow irlsFit function in cp_pipe/utils.py to use a general iterative weight function. [cp_pipe] - `DM-27675 `_: Add ability to pass single large QuantumGraph to jobs. [ctrl_bps] - `DM-27676 `_: Add ability to remove submitted runs [ctrl_bps] - `DM-27800 `_: (Optionally) Match catalogs on (parentID, peakID) in pipe_analysis [afw] - `DM-28111 `_: Include RUN name in QG dot diagrams for resolved DatasetRefs [ctrl_mpexec] - `DM-28148 `_: Make a gen 3 version of the colour colour plots in color_analysis [obs_subaru, sdm_schemas] - `DM-28315 `_: Design simplified Registry ABC for butler [daf_butler] - `DM-28394 `_: write write|transform|consolidateForcedSourceTable Tasks [ap_association, obs_lsst, obs_subaru, pipe_tasks] - `DM-28636 `_: Set up initial "main" shared data repository at NCSA [daf_butler, obs_base, obs_lsst] - `DM-28646 `_: Add ability to create trimmed registry during quantum graph build [pipe_base] - `DM-28649 `_: Modify pipetask to use standalone minimal registry [ctrl_mpexec, pipe_base] - `DM-28650 `_: Support efficient transfer of registry content from minimal registry to shared registry [daf_butler] - `DM-28653 `_: Modify BPS to support execution butler [ctrl_bps] - `DM-28668 `_: PipelineTask unit test framework bypasses dimensions checks [ap_pipe, pipe_base, verify] - `DM-28678 `_: Investigate pydantic for JSON serialization of Butler [daf_butler] - `DM-28698 `_: Restructure image formatter relationships [obs_base, obs_lsst] - `DM-28716 `_: Add pybind wrapper to afw archive that handles adding set of HDUs [afw] - `DM-28794 `_: Defects instantiation creates undefined/infinite bbox [ip_isr] - `DM-28803 `_: Add capability to discard stars with NaN annuli [meas_algorithms] - `DM-28955 `_: Allow passing `alardSigGauss` parameter to `modelPsfMatch` task at runtime [ip_diffim] - `DM-28961 `_: DiaSource centroids outside of image causing crash in cuttout creation in PackageAlerts. [ap_association] - `DM-29007 `_: Lazy command line creation [ctrl_bps] - `DM-29069 `_: statsControl.setNumIter(0) kills python/kernels [afw] - `DM-29147 `_: Switch from using mapApData to new pipeline task and functors. [ap_association, ap_pipe] - `DM-29196 `_: Use UUIDs as dataset_ids in registry [daf_butler] - `DM-29221 `_: Add ApPipe.yaml pipelines to appropriate repos [ap_pipe] - `DM-29242 `_: Numpydoc conversion of meas_algorithms through psfSelectionFromMatchList.py [meas_algorithms] - `DM-29249 `_: Strip passwords from database URIs before we print them [daf_butler] - `DM-29272 `_: PTC task: Validate that the variance calculation provided by awf and the C_00 entry of the covariance matrix produced via FFT using Astier's code is the same [cp_pipe] - `DM-29273 `_: Fix deprecation warnings reported by sqlalchemy 1.4 [daf_butler] - `DM-29279 `_: Add initial healsparse "input map" generation to coaddition tasks [daf_butler] - `DM-29310 `_: Use CoaddPsfs from all warps in getTemplate task [ip_diffim, meas_algorithms, pipe_tasks] - `DM-29314 `_: Add debiased PSF moments to HSM [meas_extensions_shapeHSM] - `DM-29325 `_: Update mask name in for vignetted region to match established convention [cp_pipe] - `DM-29338 `_: Incorporate Gen 3 crosstalk in HiTS runs [ap_pipe, ap_verify, cp_pipe, obs_decam] - `DM-29341 `_: Enable running Fakes in CI for ap_verify [ap_verify] - `DM-29344 `_: Investigate the CI differences between Gen 2 and 3 in COSMOS field [ap_pipe, ap_verify] - `DM-29346 `_: Compute PSFlux chi squared metric [ap_association, meas_base] - `DM-29348 `_: Refactor fgcmFitCycle.py task for multiple cycles at once. [fgcm, fgcmcal, obs_subaru] - `DM-29363 `_: High-level faro documentation [faro] - `DM-29364 `_: Reorganize and document faro pipelines to clarify for users [faro] - `DM-29370 `_: Allow butler gen3 to retrieve amplifiers from raw [afw, daf_butler, obs_base, obs_lsst, utils] - `DM-29384 `_: Improve exception handling in ExecFixupDataId [ctrl_mpexec] - `DM-29431 `_: Change envAppend to envPrepend [fgcm, jointcal, meas_algorithms, meas_extensions_convolved] - `DM-29490 `_: Debug decorrelation afterburner when convolveTemplate=False [ip_diffim] - `DM-29500 `_: Link to afw Box docs from geom [afw, geom] - `DM-29501 `_: replace unary_function and binary_function, make other changes to pass compilation with C++17 enabled [afw, geom, ip_diffim, meas_algorithms, meas_base] - `DM-29511 `_: Cleanup code on afw using clang-tidy [afw] - `DM-29522 `_: Add spatially varying ZOGY subtraction [ip_diffim] - `DM-29530 `_: Config Paths hardcoded during graph building [obs_lsst, obs_subaru] - `DM-29531 `_: Update IsrCalib so it can serve as StorageClass for subclasses [daf_butler, ip_isr] - `DM-29543 `_: Gen3 refcat converter [meas_algorithms] - `DM-29562 `_: Fix index usage in super-common butler query [daf_butler] - `DM-29563 `_: Replace boost::variant, fix clang compilation [afw] - `DM-29569 `_: PTC maxMeanSignal no longer works [cp_pipe] - `DM-29573 `_: Gen3 sourceTable_visit use "detector" which breaks fgcmcal [fgcmcal] - `DM-29580 `_: Can't redo part of a failed pipeline in bps with changed configs [ctrl_mpexec] - `DM-29583 `_: Create dataset class for extended PSF models [daf_butler] - `DM-29589 `_: Fix cp_pipe pipelines to avoid dimension conflict [cp_pipe, obs_lsst] - `DM-29615 `_: Fix connections problems in running Gen3 jointcal on HSC RC2 [jointcal, obs_subaru] - `DM-29617 `_: Write prototype remote butler Registry [daf_butler] - `DM-29633 `_: Investigate scarlet log propagation [meas_extensions_scarlet] - `DM-29694 `_: Allow retrieval of dimensions as JSON directly from database [daf_butler] - `DM-29699 `_: CalibCombineConnections changes its quantum dimensions at construction [cp_pipe, obs_lsst] - `DM-29701 `_: Latest weekly build fails when building utils [utils] - `DM-29703 `_: Add difference imaging to DRP pipeline [ip_diffim] - `DM-29708 `_: Make ButlerHttpURI compatible with non-webDAV HTTP backends [daf_butler] - `DM-29727 `_: Fix syntax bug introduced on DM-28636 [obs_base] - `DM-29735 `_: Felis schema file for DP0.1 [sdm_schemas] - `DM-29737 `_: Check if a SourceCatalog is sorted when calling `find` [afw] - `DM-29750 `_: Extend DRP pipeline definitions and labeled subsets [obs_subaru] - `DM-29756 `_: BPS - Automatic retry with larger memory [ctrl_bps] - `DM-29767 `_: Test if absolute deviation in estimated flux is large [meas_base] - `DM-29775 `_: Fix style issues in DM-29737 [afw] - `DM-29776 `_: Attempt complete two-QG HSC RC2 run on w16 on one tract [obs_subaru] - `DM-29790 `_: Replace boost::filesystem with std::filesystem [afw, meas_algorithms, meas_modelfit] - `DM-29794 `_: Investigate ci_hsc_gen3 image differencing failure with disassembly [daf_butler, obs_base, pipelines_check] - `DM-29800 `_: Fix compilation with clang 11.1.0 on Linux [daf_base] - `DM-29804 `_: Add debiased HSM moments to default obs_* configs [obs_decam, obs_lsst, obs_subaru] - `DM-29805 `_: Add Arm64 support to sphgeom [sphgeom] - `DM-29808 `_: Various fixes to MetadataMetricTask [verify] - `DM-29810 `_: Numpydoc conversion of meas_algorithms through simple_curve.py [meas_algorithms] - `DM-29812 `_: Remove mapApData.py. [ap_association] - `DM-29830 `_: Add DC2 defaults to obs_lsst imsim configs [obs_lsst] - `DM-29841 `_: Update faro application of external calibs [faro] - `DM-29849 `_: Speed up many-dataset deletion [daf_butler] - `DM-29853 `_: Update autoconf files for psfex package [psfex] - `DM-29856 `_: Switch ap_verify.py to Gen 3 by default [ap_verify] - `DM-29863 `_: ShapeHSM fails when psf bbox size is position dependent. [meas_extensions_shapeHSM] - `DM-29869 `_: Make doScaleDiffimVariance=True the default in ImageDifferenceTask [ap_pipe] - `DM-29877 `_: Avoid avoidable logic error in measureCrosstalk [cp_pipe] - `DM-29881 `_: Investigate differences in gen2 vs. gen3 SFP products for HSC-Y [ip_isr, obs_decam] - `DM-29888 `_: Add config field(s) to meas_extensions_scarlet to run on a subset of an input catalog [meas_deblender, meas_extensions_scarlet] - `DM-29892 `_: Attribute error for PF1Task [faro] - `DM-29893 `_: un-hardcode bps report output [ctrl_bps] - `DM-29908 `_: Registry collection loading can fail due to concurrent deletes [daf_butler] - `DM-29916 `_: Single-shot, multi-cycle FGCM is memory-inefficient [fgcm, fgcmcal] - `DM-29921 `_: Remove erroneous debug item from datastore config [daf_butler] - `DM-29933 `_: Add base classes for single-detector source catalog metrics [faro] - `DM-29936 `_: Enable getting Children without repeatedly checking if the SourceCatalog is sorted [afw, meas_base] - `DM-29940 `_: Update DP0.1 Felis with table_index values [sdm_schemas] - `DM-29944 `_: Add some narrow-band filters to skymap's tract+patch+band data ID packers [skymap] - `DM-29950 `_: Use UUID5 for raw ingest when using UUIDs [daf_butler, obs_base] - `DM-29953 `_: bps submit TypeError about SerializedDataCoordinate [ctrl_bps] - `DM-29966 `_: Fix logic defaulting butler.yaml in butler config [daf_butler] - `DM-29970 `_: Cleanup histogram classes [jointcal] - `DM-29981 `_: Migrate cp_pipe pipelines as defined in RFC-775 [cp_pipe, obs_lsst, obs_subaru] - `DM-29987 `_: Save Detectors with master calibrations [cp_pipe] - `DM-30004 `_: Replace boost::regex with std::regex [afw, daf_base, daf_persistence] - `DM-30023 `_: Replace C++17 deprecated uncaught_exception [utils] - `DM-30030 `_: Make calibration source selection consistent between gen2 and gen3 middleware - part deux [ap_association, meas_algorithms, meas_astrom, obs_base, obs_subaru] - `DM-30046 `_: Investigate memory usage of ForcedPhotCcd [meas_base] - `DM-30056 `_: requestCpus not ending up in HTCondor submit files [ctrl_bps] - `DM-30057 `_: AP timing metrics out of date [ap_association, ap_verify] - `DM-30061 `_: PipelineTasks do not use pipeline label as name [ctrl_mpexec, pipe_base] - `DM-30071 `_: FgcmBuildStarsTableConnections sourceSchema should be an InitInput [fgcmcal] - `DM-30076 `_: Fix missing config imports in obs_lsst [obs_lsst] - `DM-30079 `_: Corrupted documentation breaks documentation builds [ip_diffim] - `DM-30093 `_: Astropy Header object does not behave dict-like. [astro_metadata_translator] - `DM-30104 `_: Make pickling the clustered quantum graph optional [ctrl_bps] - `DM-30105 `_: Get children sources without repeatedly checking if the afw SourceCatalog is sorted by parent [afw, meas_base] - `DM-30109 `_: Fix error introduced in DM-29841 (external calibs) implementation [faro] - `DM-30124 `_: error trying to specify query for ingest_date [daf_butler] - `DM-30125 `_: Add simple unit tests for image differencing [ip_diffim] - `DM-30130 `_: Establish a 1-1 correspondence between exposures and input dimensions in cpPtcExtract [cp_pipe] - `DM-30139 `_: Make position Optional when manually instantiating Stamps [meas_algorithms] - `DM-30140 `_: Add bulk datastore trash API [daf_butler] - `DM-30142 `_: Support P and Q for simulated images [obs_lsst] - `DM-30145 `_: Allow sqlalchemy to use IN operator for DELETE [daf_butler] - `DM-30147 `_: Butler warning about TypeDecorator Base64Bytes() [daf_butler] - `DM-30148 `_: PipelineTasks use wrong label as name [ctrl_mpexec] - `DM-30153 `_: Make jointcal debug output files not collide [jointcal] - `DM-30161 `_: Remove deprecated configs in fgcmFitCycle [fgcmcal] - `DM-30178 `_: Please make daf_butler compatible with sqlalchemy 2 [daf_butler] - `DM-30200 `_: Support glob strings in registry query APIs for collections and dataset types [daf_butler] - `DM-30202 `_: Add "online" coaddition code to meas_algorithms and pipe_tasks [meas_algorithms, pipe_tasks] - `DM-30204 `_: Add support for filter N708 to DECam [obs_decam] - `DM-30245 `_: Document cp_pipe [cp_pipe] - `DM-30254 `_: Fix jointcal crash when doing outlier rejection on only the model [jointcal] - `DM-30257 `_: Remove eotest dependency from cp_pipe [cp_pipe] - `DM-30270 `_: Make doFilterMorphological=True default in obs_subaru [obs_subaru] - `DM-30272 `_: Change default for dataset_id in registry to UUID [daf_butler] - `DM-30281 `_: Demonstrate the effect of random padding regeneration and fix unit test by making padding deterministic [ip_diffim] - `DM-30286 `_: Define a verbose log level for daf_butler [daf_butler] - `DM-30287 `_: Add LoadIndexedReferenceObjectsTask RST doc [meas_algorithms] - `DM-30291 `_: Setup towncrier on daf_butler [daf_butler] - `DM-30296 `_: ap_verify HSC Gen 3 ingestion crashes on missing defineVisits config [obs_subaru] - `DM-30301 `_: Switch Task to use python logging rather than lsst.log [meas_base, obs_base, obs_test, pipe_base, pipe_tasks] - `DM-30316 `_: Write UUID migration script for sqlite [daf_butler] - `DM-30321 `_: Validate the schema in ci_hsc_gen3 [sdm_schemas] - `DM-30335 `_: Demonstrate process for use of execution butler [daf_butler, pipe_base] - `DM-30339 `_: Replace boost::any with std::any [afw, daf_base] - `DM-30342 `_: Propagate flags to Object tables [obs_lsst, obs_subaru, sdm_schemas] - `DM-30346 `_: Remove boost:serialization [afw] - `DM-30349 `_: Source count metrics include fake sources [ap_verify, pipe_tasks] - `DM-30350 `_: Replace individual graph files to a single per workflow [ctrl_bps] - `DM-30351 `_: Make sure brighter-fatter correction is working on DC2 data for DP0.2 [ip_isr, obs_base, obs_lsst, obs_lsst_data] - `DM-30365 `_: Gen3 RC2 reprocessing with w_2021_22 and bps [skymap] - `DM-30372 `_: Specifying a manager in a butler seed prevents other managers from being specified [daf_butler] - `DM-30373 `_: Add butler command line tool for creating collection chains [daf_butler] - `DM-30386 `_: Make DRP wrapper task for DiaCalculation task [ap_association, meas_base] - `DM-30425 `_: Error in daf_base EPOCH definition [daf_base] - `DM-30426 `_: Out-of-bounds array access in shapeHSM [afw, meas_extensions_piff, meas_extensions_shapeHSM] - `DM-30444 `_: Require X% template coverage to attempt image subtraction [ip_diffim] - `DM-30445 `_: Have Stamps contain an optional Archive element [meas_algorithms] - `DM-30448 `_: Use correct shape measurement for ellipticity residual calculation [faro] - `DM-30465 `_: Implement decorrelation afterburner for maximum likelihood images [ip_diffim, pipe_tasks] - `DM-30490 `_: Add delta-magnitude check to AstrometryTask [meas_astrom] - `DM-30497 `_: Bug: DiaSource table cannot find Ixx column [ap_association] - `DM-30534 `_: Test and Implement Default Parameters for Jointcal Outlier Rejection [jointcal, obs_subaru] - `DM-30563 `_: ap_verify failing to find dataset type fakes_deepDiff_warpedExp [ap_pipe, ap_verify] - `DM-30564 `_: Improve task label uniqueness exception message [pipe_base] - `DM-30630 `_: Filter look-up for in isrTask is broken in a few places [ip_isr] - `DM-30633 `_: Add meas_extensions_trailedSources as setupOptional to lsst_distrib [lsst_distrib] - `DM-30649 `_: Exit with new gen3 "expected failure" exit code when its available [ctrl_mpexec, ip_isr, obs_lsst, pipe_base] - `DM-30653 `_: Write some release notes for v22 release of middleware [ctrl_mpexec, daf_butler, obs_base, pipe_base] - `DM-30665 `_: Add Source and Object schema_checks to ci_imsim [sdm_schemas] - `DM-30683 `_: Remove diaSourceAnalysis.py [ip_diffim] - `DM-30685 `_: Remove physical_filter+detector+exposure entry from fileDatastore.yaml templates [daf_butler] - `DM-30692 `_: Add refObjLoader lookups for DECam N708 filter [obs_decam] - `DM-30693 `_: Add meas_extensions_gaap to lsst_distrib [lsst_distrib] - `DM-30696 `_: Add task to translate gen2 fgcmcal photocalibs to gen3 photocalib catalog [fgcmcal] - `DM-30701 `_: Propagate astrometry offset stats in AstrometryTask and store in ExposureSummaryStats/VisitSummary [afw, meas_astrom] - `DM-30702 `_: Add provenance information to quantum graph [ctrl_mpexec, pipe_base] - `DM-30718 `_: butler.get fails to recognize full dataId [daf_butler] - `DM-30733 `_: Fix bug in BrightStarStamps' initAndNormalize method [meas_algorithms] - `DM-30738 `_: Backport gen3 imSim BF-kernels for gen2 [ip_isr, obs_lsst] - `DM-30743 `_: Create temporary cache directory only when needed [daf_butler] - `DM-30753 `_: Automated publishing pipeline of TAP_SCHEMA metadata [sdm_schemas] - `DM-30761 `_: "bind" keyword argument not forward in queryDatasets recursion [daf_butler] - `DM-30767 `_: Execution butler builder needs to insert initOutput datasets [ctrl_mpexec, daf_butler, pipe_base] - `DM-30771 `_: Execution butler must be able to skip components on get [daf_butler] - `DM-30776 `_: Move MatchApFakesTask to pipe_tasks [ap_pipe, ap_verify] - `DM-30777 `_: Remove unused includes in afw [afw, jointcal] - `DM-30784 `_: butler transfer datasets must create dataset types if necessary [daf_butler] - `DM-30787 `_: Translate getting started tutorials to use gen3 and the new RC2 subset [pipe_base] - `DM-30795 `_: Fix ap_verify breakage due to DM-30649 [ap_verify] - `DM-30798 `_: Fix ap_association breakage due to DM-30649 [ap_association] - `DM-30801 `_: Pass memory requirements to PanDA for automatic queue brokerage decision [ctrl_bps] - `DM-30815 `_: Update expBits used in gen2 bulter to match value computed for gen3 [obs_lsst] - `DM-30820 `_: Investigate effect of reducing background fit Chebyshev polynomial order in DC2 [obs_lsst] - `DM-30823 `_: Add modelPhotRep metrics to faro [faro] - `DM-30829 `_: Add persistence method for WarpedPsf [afw, meas_algorithms] - `DM-30838 `_: Disable bright-object-mask-based catalog flags for ImSim/DC2 [obs_lsst] - `DM-30842 `_: Add healSparsePropertyMaps task to pipeline subsets [obs_subaru] - `DM-30855 `_: Butler unit tests fail with click8 [daf_butler] - `DM-30858 `_: Add GAaP measurements to obs packages [obs_lsst, obs_subaru, sdm_schemas] - `DM-30863 `_: Update LATISS visit region padding to 4k pixels [obs_lsst] - `DM-30866 `_: Add support for replacing bad DimensionRecords, use it in DefineVisits and RawIngest, and fix old LATISS visit regions [daf_butler, obs_base] - `DM-30883 `_: Incorrect triplet initialization can result in crash [jointcal] - `DM-30884 `_: Fix schema inconsistency in hsc.yaml [sdm_schemas] - `DM-30900 `_: Stop using obs_* configurations in fgcmcal [fgcmcal] - `DM-30902 `_: Numpydoc conversion of meas_algorithms through stamps.py [meas_algorithms] - `DM-30905 `_: Numpydoc conversion of meas_algorithms through utils.py [meas_algorithms] - `DM-30926 `_: Add SkySources to ImageDifferenceTask [ap_association, ap_pipe] - `DM-30935 `_: Add generic butler ingest command [daf_butler] - `DM-30939 `_: Fixup image type headers in old LATISS data [obs_lsst] - `DM-30940 `_: Add the --date-created command line option to dispatch_verify.py [verify] - `DM-30943 `_: Turn on delta mag rejection for astrometry matching in configs for LSSTCam-imSim [obs_lsst] - `DM-30951 `_: patchInfo.getIndex() returns the second index with a float instead of an int [skymap] - `DM-30965 `_: Update PanDA plugin to support lazy commands generation [ctrl_bps] - `DM-30967 `_: Fix segfault due to MKL [base] - `DM-30977 `_: Write pipeline task log output to butler [ctrl_mpexec, daf_butler, pipe_base] - `DM-30985 `_: Second attempt to fix hard coded paths in functorFile config [ap_association, ap_pipe, dax_apdb] - `DM-30992 `_: dax_apdb's uint64 is incompatible with pandas 1.3.0 [dax_apdb] - `DM-30996 `_: Implement log4cxx appender to Python logging [log] - `DM-31016 `_: Replace all print statements in faro with log statements [faro] - `DM-31023 `_: Add wise_01 schema description to sdm_schemas [sdm_schemas] - `DM-31026 `_: Create Ephemeris Cache Precomputation Task for SSO attribution pipeline [ap_association] - `DM-31027 `_: statsCtrl.getMaskPropagationThreshold has buffer overrun and can segfault [afw] - `DM-31031 `_: Define for DC2 DRP pipeline a step1, 2, and 3 (and redefine for HSC) [obs_lsst, obs_subaru] - `DM-31033 `_: Move DiaCalculation from a subtask of AssociationTask to a subtask of DiaPipe. [ap_association] - `DM-31043 `_: Update ctrl_bps setup.cfg with flake8's max-doc-length=79 [ctrl_bps] - `DM-31054 `_: Make aggregation steps robust to expected failures in RC2/DC2 [ctrl_mpexec] - `DM-31055 `_: The length is not long enough for the "skymap" column [sdm_schemas] - `DM-31060 `_: Clean up some log usage in Tasks [ip_diffim, ip_isr, meas_algorithms, meas_base, obs_base] - `DM-31064 `_: Create metric to capture the total number of fake sources used [ap_pipe, ap_verify, verify_metrics] - `DM-31066 `_: Move "convolution mode" log messages [ip_diffim] - `DM-31077 `_: Implement principal-column flagging for DP0.1 tables [sdm_schemas] - `DM-31079 `_: Butler define-visits ignores collection [obs_base] - `DM-31093 `_: Trigger deployment of wise_01 to TAP_SCHEMA in NCSA RSP instances [sdm_schemas] - `DM-31096 `_: Fix bitrot in gen3 linearity solver [cp_pipe] - `DM-31100 `_: Alter code for Networkx compatibility [pipe_base] - `DM-31103 `_: Add some missing python logging APIs to lsst.log [log] - `DM-31105 `_: Update faro pipelines to completely separate those with/without applying external calibrations [faro] - `DM-31116 `_: Deleting non-existant files from S3 ButlerURI triggers backoff [daf_butler] - `DM-31119 `_: Investigate all-NaN columns in objectTable_tract [obs_lsst, obs_subaru, sdm_schemas] - `DM-31120 `_: Change butler logger to python logging [ctrl_mpexec, daf_butler, log] - `DM-31131 `_: lsst.verify outputs non-standard NaN in JSON [verify] - `DM-31135 `_: Add DM license preamble for each of the python files [faro] - `DM-31136 `_: Consistently adopt DM convention for class names involving Task, Config, and Connections [faro] - `DM-31141 `_: Add test decorators for cartesian product [utils] - `DM-31146 `_: Add cp_verify to lsst_distrib [lsst_distrib] - `DM-31152 `_: Fix line/file reporting for lsst.log.warning [log] - `DM-31156 `_: Investigate GAaP colors in DC2 involving g band [obs_lsst, obs_subaru] - `DM-31163 `_: Add section to Object Table spec for flags taken from forced_src [obs_lsst, obs_subaru] - `DM-31174 `_: str() method failure on lsst.ip.isr.Defects class [ip_isr] - `DM-31180 `_: ap_verify loses logs from pipeline [ap_verify] - `DM-31188 `_: Fix LoadReferenceCatalogTask so it doesn't temporarily clobber its own config [pipe_tasks] - `DM-31217 `_: Add task to consolidate healsparse property maps [obs_lsst, obs_subaru, pipe_tasks, skymap] - `DM-31220 `_: API mismatch in Gen 2 ap_pipe [ap_association, ap_pipe] - `DM-31228 `_: Migrate obs packages to python logging where appropriate [obs_cfht, obs_decam, obs_lsst, obs_subaru] - `DM-31238 `_: Remove phosimToRafts from obs_lsst [obs_lsst] - `DM-31240 `_: Add support of Execution Butler into the PanDA plugin [ctrl_bps] - `DM-31249 `_: Fix ButlerURI.join confusion with schemeless URI [daf_butler] - `DM-31251 `_: Add execution butler example to pipelines_check [daf_butler, pipe_base, pipelines_check] - `DM-31255 `_: Fix log-to-butler failutres with --skip-existing [ctrl_mpexec] - `DM-31262 `_: Execution butler creation should not forward namespace value [pipe_base] - `DM-31267 `_: Jointcal failures in w_2021_30 [jointcal] - `DM-31272 `_: Allow for curation of schema_index values "above" the Felis-file level [sdm_schemas] - `DM-31275 `_: w_2021_30 deblend or mergeDetections not playing nice with missing bands [pipe_tasks] - `DM-31278 `_: transformDiaSourceCatalog.py not expanding environment variables after weekly 29 [ap_association] - `DM-31282 `_: Fetch only last-produced metric in chained collection in gen3_to_job [verify] - `DM-31285 `_: Enable meas_algorithms python API docs [meas_algorithms] - `DM-31291 `_: Documentation for CatalogMeasurementBase broken [faro] - `DM-31294 `_: Re-run faro on DC2 and re-dispatch with appropriate dates [faro] - `DM-31295 `_: Add timers for datastore activities and improve S3 performance [daf_butler] - `DM-31301 `_: Fix VignetteExposure name [cp_pipe] - `DM-31306 `_: Create single SAL script that produces biases, darks, and flats [cp_pipe] - `DM-31313 `_: Add htmIndex computation for DiaSources [ap_association, pipe_tasks] - `DM-31324 `_: Rearrange steps in DRP.yaml for more complete outputs and weave in diffm [obs_lsst, obs_subaru] - `DM-31326 `_: Run faro on gen3_RC2_subset and set it up in Jenkins [faro] - `DM-31331 `_: Add new log options to bps [ctrl_bps] - `DM-31337 `_: Allow skipping calibration collections in registry search methods [daf_butler] - `DM-31344 `_: Execution butler creation fails on s3 [daf_butler] - `DM-31355 `_: Switch from the deprecated log.getName() to log.name [ap_verify, ip_diffim, meas_extensions_piff] - `DM-31359 `_: MaskStreaks sending bad matrix to scipy.linalg.cho_factor [pipe_tasks] - `DM-31360 `_: w_2021_30 deblend still not playing nice with missing bands [pipe_tasks] - `DM-31366 `_: Add confirmation request to butler prune-collection [daf_butler] - `DM-31367 `_: Add butler register-dataset-type subcommand [daf_butler] - `DM-31376 `_: assembleCoadd can't handle when selectVisit selects a visit with no Warp [pipe_tasks] - `DM-31381 `_: No connection called refDataset in CatalogMeasurementBaseConnections [faro] - `DM-31382 `_: Run faro in ci_hsc and ci_imsim [faro] - `DM-31384 `_: Add pipetask log option to add arbitrary MDC records [ctrl_bps, ctrl_mpexec, daf_butler] - `DM-31389 `_: Move new DiaObject creation out of AssociationTask [ap_association] - `DM-31394 `_: Overscan subtraction can create negative variance [ip_isr] - `DM-31398 `_: Error with execution butler on IDF's DC2 repo [daf_butler, pipe_base] - `DM-31412 `_: Add config/imsim/charImage.py to obs_lsst [obs_lsst] - `DM-31417 `_: Add makeHpxWcs to afwGeom to make an HPX SkyWcs object. [afw] - `DM-31419 `_: ImageDifferenceTask no longer timed in AP [ap_pipe, ap_verify, verify] - `DM-31429 `_: UNION ALL in subqueries is causing catastrophic database pessimizations [daf_butler] - `DM-31434 `_: Placeholder PSF WARN in charImage should be INFO [pipe_tasks] - `DM-31442 `_: Pass logName for undeblended plugins [meas_base] - `DM-31443 `_: Fix typo in quickFrameMeasurement task [pipe_tasks] - `DM-31445 `_: Ensure GAaP measurements are aperture corrected [obs_lsst, obs_subaru] - `DM-31448 `_: Allow gen2 butler lookups even when fields in the dataId are not in the registry [daf_persistence] - `DM-31462 `_: Add random seed config to make fgcm density downsampling repeatable [fgcm, fgcmcal, obs_subaru] - `DM-31466 `_: segfault when spatialOrder is zero in psfDetermination [meas_extensions_psfex] - `DM-31472 `_: Execution Butler merge failed with large workflows [daf_butler] - `DM-31476 `_: Clean up PanDA's example bps yaml file [ctrl_bps] - `DM-31489 `_: Update StrayLightData to use FitsGenericFormatter with a deferred data set [ip_isr, obs_subaru] - `DM-31505 `_: Update fgcm to better handle survey edges and reference star outliers [fgcm, fgcmcal] - `DM-31507 `_: Adjust stack objects to accommodate reference counting [ctrl_mpexec, daf_butler, pex_config, pipe_base, pipe_tasks] - `DM-31520 `_: Request butler collection-chain to take comma-separated list [daf_butler, pipelines_check] - `DM-31528 `_: Add more log messages to the measure task [meas_base, meas_extensions_scarlet] - `DM-31530 `_: Log outputs are being swallowed when running pipe_drivers scripts [ctrl_pool] - `DM-31535 `_: Deprecate PSF methods default position argument [afw] - `DM-31540 `_: Remove butlerQC.registry usage from fgcmcal [fgcmcal] - `DM-31541 `_: Change pipetaskInit job to not read full QuantumGraph [ctrl_bps] - `DM-31542 `_: Examine why disassembled composites take up much more space in datastore [afw, daf_butler] - `DM-31543 `_: Update sdm_schemas to make DP0.1 catalogs on NCSA "small" qserv visible to TAP [sdm_schemas] - `DM-31545 `_: Fix cp_pipe/defects debug error [cp_pipe] - `DM-31548 `_: patch_skypix_overlap table needs an additional index [daf_butler] - `DM-31549 `_: Fix clang compilation in lsst_distrib [log] - `DM-31564 `_: Fix sconsUtils to allow running CodeChecker [sconsUtils] - `DM-31571 `_: Gen3 isrTask could be made to take a default None camera object [ip_isr] - `DM-31579 `_: Fix permissions of butler log datasets [ctrl_mpexec] - `DM-31580 `_: Split "step5" into a per-tract step5 and a per-visit step6 [obs_lsst, obs_subaru] - `DM-31583 `_: Add interfaces to dataset summary table data and use it to improve query generation [daf_butler, pipe_base] - `DM-31591 `_: Update developer guide on logging practices [meas_base] - `DM-31596 `_: Export DM-28920 calibrations for use at the summit [daf_butler] - `DM-31601 `_: Add tasks to forced phot diaObjects on calexps,diffim,coadds [ap_association, meas_base, obs_lsst, obs_subaru, pipe_tasks] - `DM-31609 `_: lsst.verify.Measurement needs a useful __repr__() [verify] - `DM-31623 `_: Support dataId rewriting on butler.put [daf_butler] - `DM-31631 `_: Forward job creation message on to dispatch_verify [verify] - `DM-31640 `_: New filter installed in LATISS, needs to be declared in filters.py [obs_lsst] - `DM-31659 `_: Logging loses part of message in assembly.py [obs_lsst] - `DM-31663 `_: fgcm 3.6.7 (with fgcmcal 779eefa) fails on macOS Catalina [fgcm] - `DM-31664 `_: Fix clang compilation warnings [jointcal, meas_extensions_psfex] - `DM-31676 `_: Add genTemplate to obs_subaru/obs_lsst subsets for DC2/RC2 batch reruns [obs_lsst, obs_subaru, pipe_tasks] - `DM-31691 `_: pipetask doesn't find dataset in execution butler from previous run. [ctrl_mpexec] - `DM-31696 `_: VERBOSE log-level in pipetask run sometimes issues DEBUG log messages [daf_butler, ip_isr] - `DM-31700 `_: Butler parquet formatter fails to load table with no indexes [daf_butler] - `DM-31710 `_: Optimize crosstalkSources lookup in IsrTask [ip_isr] - `DM-31711 `_: Problems pipetask --save-execution-butler w_2021_37 [ctrl_mpexec] - `DM-31721 `_: Create cpputils package [afw, ap_pipe, ap_verify, daf_base, geom, jointcal, meas_algorithms, meas_base, meas_modelfit, obs_cfht, obs_decam, obs_lsst, obs_subaru, utils] - `DM-31722 `_: Move utility code out of pipe_base and daf_butler into utils package [utils] - `DM-31760 `_: Single amp reads for obs_lsst have the wrong geometry. [obs_lsst] - `DM-31765 `_: Make ForcedSourceOnDiaObjectsTable [obs_lsst, obs_subaru, pipe_tasks] - `DM-31783 `_: Register DECam N419 and N540 filters [obs_decam] - `DM-31785 `_: make butler transfer-datasets more efficient [daf_butler] - `DM-31796 `_: Add SDSS filters for LATISS in filters.py [obs_lsst] - `DM-31822 `_: github pages based schema browser [sdm_schemas] - `DM-31825 `_: Parquet Table clean up before DP0.2 Preliminary run. [ap_association, fgcmcal, jointcal, meas_base, obs_lsst, obs_subaru, pipe_tasks, sdm_schemas] - `DM-31826 `_: FileDatastore.transfer_from is replacing physical files with circular symlinks in the execution butler final job [daf_butler] - `DM-31827 `_: Stop providing fgcm photoCalib solutions when dubious [fgcmcal] - `DM-31832 `_: Change default LoadDiaCatalogs behavior to spatial loading of DiaSources. [ap_association] - `DM-31841 `_: The queue parameter submitted on per task level is not available in the GenericWorkflowJob [ctrl_bps] - `DM-31846 `_: Ongoing Jenkins issues [faro] - `DM-31848 `_: Add faro steps to DRP.yaml in obs_lsst/imsim [obs_lsst] - `DM-31849 `_: ForcedPhotCcdOnDataFrame outputs all NaNs [meas_base, pipe_tasks] - `DM-31859 `_: Add user-specified clustering to bps [ctrl_bps] - `DM-31861 `_: HSC DRP.yaml should not import from jointcal tests [obs_subaru] - `DM-31869 `_: Remove computeShape calls without argument in GAaP [meas_base] - `DM-31887 `_: Make execution butler default in ctrl_bps [ctrl_bps] - `DM-31891 `_: Fix cp_pipe documentation build error [cp_pipe] - `DM-31900 `_: Use per task cloud setting in PanDA plugin [ctrl_bps] - `DM-31903 `_: Add update kwargs/option to instrument registration [obs_base, obs_cfht, obs_decam, obs_lsst, obs_subaru] - `DM-31912 `_: Add additional GAaP circular apertures for DP0.2 [obs_lsst, sdm_schemas] - `DM-31941 `_: Log dataset does not overwrite on rerun with execution butler [ctrl_mpexec, daf_butler] - `DM-31944 `_: concurrency_limit split in htcondor submit file [ctrl_bps] - `DM-31948 `_: Fix regression in AP catalog metrics [ip_diffim] - `DM-31970 `_: Fix submission AttributeError in htcondor plugin when not using execution butler [ctrl_bps]