Tickets Addressed in Release 23.0.0

Tickets merged after w_2021_40 and backported

  • DM-32714: Fix backwards compatibility broken by DM-32649 [pipe_tasks]
  • DM-32666: Add GaussianFlux columns to SourceTable [obs_lsst, obs_subaru]
  • DM-32649: Move psf quality statistics calculation from selectImages to computeExposureSummaryStats [afw, pipe_tasks]
  • DM-32624: Override source selection criteria in SFM calibration stages for LSSTCam-imSim [meas_algorithms, obs_lsst]
  • DM-32594: Remove dependency on inflection in bps_config [ctrl_bps]
  • DM-32504: SingleQuantumExecutor should log exception messages [ctrl_mpexec]
  • DM-32448: Backport incorrect GAaP field name [obs_lsst]
  • DM-32435: Silent failure in executing memory-hungry job? [ctrl_bps]
  • DM-32333: SourceTable should use LocalNanojansky functor on apFluxes [obs_lsst, obs_subaru]
  • DM-32324: Add forced source table to sdm_schemas and ci_imsim [sdm_schemas]
  • DM-32306: refBand functor doesn’t know that u-band exists [pipe_tasks]
  • DM-32305: lsst.daf.butler.Config slowness with large numbers of lookups [daf_butler]
  • DM-32290: A create-table failure led to registry corruption [daf_butler]
  • DM-32241: bps submit can take multiple hours [ctrl_bps]
  • DM-32238: Fix unexpected floating point values in drpAssociation task [pipe_tasks]
  • DM-32220: HTCondor plugin copying execution butler back from job [ctrl_bps]
  • DM-32217: BPS displays cumulative timings instead of time intervals during a submission [ctrl_bps]
  • DM-32201: HTCondor plugin bug if output collection contains period [ctrl_bps]
  • DM-32198: Set sentinel values for flag/integer columns for missing bands in Object tables [pipe_tasks]
  • DM-32129: Add a maximum mean offset threshold for SFM astrometry fit success [meas_astrom, obs_lsst]
  • DM-32124: ValueError: Only one node is allowed to produce forcedSource [pipe_tasks]
  • DM-32074: Modify how HTCondor plugin handles transfer of execution butler [ctrl_bps]
  • DM-32066: BPS jobs with memory autoscaling enabled remain idle after the first run attempt [ctrl_bps]
  • DM-32030: Fixup Jointcal after DM-31825 [jointcal]
  • DM-32027: Rename outCollection to outputRun [ctrl_bps]
  • DM-31990: Add configuration option to set coverage nside for survey-wide HealSparsePropertyMaps [pipe_tasks]
  • DM-31976: Change handling of collections and dataset types in dataset transfer [ctrl_mpexec, daf_butler]
  • DM-31956: mergeExecutionButler task hits database deadlock intermittently [daf_butler]
  • DM-31777: Determine and use correct position for PSF shape determination in ImageDifferenceTask [ip_diffim, pipe_tasks]

Tickets merged before w_2021_40

  • DM-6056: Remove PTR and CONST_PTR macros [afw, base, daf_persistence, ip_diffim, meas_algorithms, meas_astrom, meas_base, meas_deblender, meas_extensions_photometryKron, meas_extensions_psfex, meas_modelfit, obs_subaru, shapelet]
  • DM-11419: afwTable rows define offsets as int32, causing problems with very large (>2Gb) rows [afw]
  • DM-13089: Use new visitInfo parallactic angle calculations in jointcal [jointcal]
  • DM-13365: Minimal on-disk caching Datastore [daf_butler]
  • DM-13944: add id to VisitInfo [afw, obs_base]
  • DM-15361: Make lsst.log outputs appear in Jupyter notebook cells [log]
  • DM-15552: Convert meas_deblender to numpydoc status [meas_deblender]
  • DM-16556: calcRmsDistances is comparing objects in correct distances due to indexing error [faro]
  • DM-17471: Save intermediate data product with pre-interpolation pixel values [ip_isr]
  • DM-19392: Interval class and geom Python enhancements: Stage 1 [afw, geom, sphgeom]
  • DM-19634: Longlog cmdline option suppresses DEBUG and TRACE messages issued by lsst.log [pipe_base]
  • DM-22136: inspect_job.py crashes on empty Job [verify]
  • DM-22486: afwTable will seg-fault if you add a new record to a table with schema=None [afw]
  • DM-22534: Implement cpSkyTask [cp_pipe, daf_butler]
  • DM-23330: Doxygen warnings in pipe_base [cp_pipe, ctrl_pool, meas_algorithms, pipe_base]
  • DM-23694: Create script for producing distortion model from Jointcal output [afw, geom, jointcal]
  • DM-24278: Apply proper motion to matched sources in Jointcal [jointcal]
  • DM-24389: Implement amp-to-amp offset correction task [ip_isr, obs_subaru]
  • DM-24704: Make brighter-fatter correction a subclass of lsst.ip.isr.IsrCalib [cp_pipe, daf_butler, ip_isr, obs_base, obs_lsst]
  • DM-24762: Add option to PTC task to correct for sigma clipping bias [cp_pipe]
  • DM-24806: Lift key ap_verify classes into lsst.ap.verify namespace [ap_verify]
  • DM-24913: obs_decam README is confusing [obs_decam]
  • DM-25013: Unclear error messages in pipetask run [ctrl_mpexec, daf_butler, pipe_base]
  • DM-25789: Update PA1 metric implementation [faro]
  • DM-26031: Fix broken measureCrosstalk test [cp_pipe]
  • DM-26037: Add option to ignore nPixels from edge in PTC task [cp_pipe]
  • DM-26214: Switch to using sourceTable and objectTable parquet files instead of the src FITS files [faro]
  • DM-26397: Add explicit DAG export from ctrl_bps [ctrl_bps]
  • DM-26651: Persist individual bright stars’ warping Transforms, origin and number of rotations [meas_algorithms]
  • DM-26719: Resync with Spectractor upstream [obs_base, obs_lsst, obs_test]
  • DM-27010: Allow IsrTask to use PTC dataset as gain source [ip_isr]
  • DM-27031: Use primary flag in source count metrics [ip_diffim]
  • DM-27032: Create deblended source metric [ap_verify, ip_diffim, verify_metrics]
  • DM-27083: Improve jointcal outlier rejection [jointcal]
  • DM-27103: Not all IsrCalib tests round trip calibrations [ip_isr]
  • DM-27241: Add butler API for downloading file artifacts from datastore [daf_butler]
  • DM-27260: Change panDA code distribution to use preexisting mechanisms [ctrl_bps]
  • DM-27387: Implement scattering magnitudes for ApFakes [ap_pipe, ap_verify, pipe_tasks]
  • DM-27412: Rework daf.butler.tests.makeTestCollection [daf_butler]
  • DM-27416: Add doumentation on creating Pipelines [pipe_base]
  • DM-27492: Improvements to pipetask execution options [ctrl_mpexec, pipe_base]
  • DM-27612: Applying fringe correction adds numerous mask features [ip_isr]
  • DM-27650: Allow irlsFit function in cp_pipe/utils.py to use a general iterative weight function. [cp_pipe]
  • DM-27675: Add ability to pass single large QuantumGraph to jobs. [ctrl_bps]
  • DM-27676: Add ability to remove submitted runs [ctrl_bps]
  • DM-27800: (Optionally) Match catalogs on (parentID, peakID) in pipe_analysis [afw]
  • DM-28111: Include RUN name in QG dot diagrams for resolved DatasetRefs [ctrl_mpexec]
  • DM-28148: Make a gen 3 version of the colour colour plots in color_analysis [obs_subaru, sdm_schemas]
  • DM-28315: Design simplified Registry ABC for butler [daf_butler]
  • DM-28394: write write|transform|consolidateForcedSourceTable Tasks [ap_association, obs_lsst, obs_subaru, pipe_tasks]
  • DM-28636: Set up initial “main” shared data repository at NCSA [daf_butler, obs_base, obs_lsst]
  • DM-28646: Add ability to create trimmed registry during quantum graph build [pipe_base]
  • DM-28649: Modify pipetask to use standalone minimal registry [ctrl_mpexec, pipe_base]
  • DM-28650: Support efficient transfer of registry content from minimal registry to shared registry [daf_butler]
  • DM-28653: Modify BPS to support execution butler [ctrl_bps]
  • DM-28668: PipelineTask unit test framework bypasses dimensions checks [ap_pipe, pipe_base, verify]
  • DM-28678: Investigate pydantic for JSON serialization of Butler [daf_butler]
  • DM-28698: Restructure image formatter relationships [obs_base, obs_lsst]
  • DM-28716: Add pybind wrapper to afw archive that handles adding set of HDUs [afw]
  • DM-28794: Defects instantiation creates undefined/infinite bbox [ip_isr]
  • DM-28803: Add capability to discard stars with NaN annuli [meas_algorithms]
  • DM-28955: Allow passing alardSigGauss parameter to modelPsfMatch task at runtime [ip_diffim]
  • DM-28961: DiaSource centroids outside of image causing crash in cuttout creation in PackageAlerts. [ap_association]
  • DM-29007: Lazy command line creation [ctrl_bps]
  • DM-29069: statsControl.setNumIter(0) kills python/kernels [afw]
  • DM-29147: Switch from using mapApData to new pipeline task and functors. [ap_association, ap_pipe]
  • DM-29196: Use UUIDs as dataset_ids in registry [daf_butler]
  • DM-29221: Add ApPipe.yaml pipelines to appropriate repos [ap_pipe]
  • DM-29242: Numpydoc conversion of meas_algorithms through psfSelectionFromMatchList.py [meas_algorithms]
  • DM-29249: Strip passwords from database URIs before we print them [daf_butler]
  • DM-29272: PTC task: Validate that the variance calculation provided by awf and the C_00 entry of the covariance matrix produced via FFT using Astier’s code is the same [cp_pipe]
  • DM-29273: Fix deprecation warnings reported by sqlalchemy 1.4 [daf_butler]
  • DM-29279: Add initial healsparse “input map” generation to coaddition tasks [daf_butler]
  • DM-29310: Use CoaddPsfs from all warps in getTemplate task [ip_diffim, meas_algorithms, pipe_tasks]
  • DM-29314: Add debiased PSF moments to HSM [meas_extensions_shapeHSM]
  • DM-29325: Update mask name in for vignetted region to match established convention [cp_pipe]
  • DM-29338: Incorporate Gen 3 crosstalk in HiTS runs [ap_pipe, ap_verify, cp_pipe, obs_decam]
  • DM-29341: Enable running Fakes in CI for ap_verify [ap_verify]
  • DM-29344: Investigate the CI differences between Gen 2 and 3 in COSMOS field [ap_pipe, ap_verify]
  • DM-29346: Compute PSFlux chi squared metric [ap_association, meas_base]
  • DM-29348: Refactor fgcmFitCycle.py task for multiple cycles at once. [fgcm, fgcmcal, obs_subaru]
  • DM-29363: High-level faro documentation [faro]
  • DM-29364: Reorganize and document faro pipelines to clarify for users [faro]
  • DM-29370: Allow butler gen3 to retrieve amplifiers from raw [afw, daf_butler, obs_base, obs_lsst, utils]
  • DM-29384: Improve exception handling in ExecFixupDataId [ctrl_mpexec]
  • DM-29431: Change envAppend to envPrepend [fgcm, jointcal, meas_algorithms, meas_extensions_convolved]
  • DM-29490: Debug decorrelation afterburner when convolveTemplate=False [ip_diffim]
  • DM-29500: Link to afw Box docs from geom [afw, geom]
  • DM-29501: replace unary_function and binary_function, make other changes to pass compilation with C++17 enabled [afw, geom, ip_diffim, meas_algorithms, meas_base]
  • DM-29511: Cleanup code on afw using clang-tidy [afw]
  • DM-29522: Add spatially varying ZOGY subtraction [ip_diffim]
  • DM-29530: Config Paths hardcoded during graph building [obs_lsst, obs_subaru]
  • DM-29531: Update IsrCalib so it can serve as StorageClass for subclasses [daf_butler, ip_isr]
  • DM-29543: Gen3 refcat converter [meas_algorithms]
  • DM-29562: Fix index usage in super-common butler query [daf_butler]
  • DM-29563: Replace boost::variant, fix clang compilation [afw]
  • DM-29569: PTC maxMeanSignal no longer works [cp_pipe]
  • DM-29573: Gen3 sourceTable_visit use “detector” which breaks fgcmcal [fgcmcal]
  • DM-29580: Can’t redo part of a failed pipeline in bps with changed configs [ctrl_mpexec]
  • DM-29583: Create dataset class for extended PSF models [daf_butler]
  • DM-29589: Fix cp_pipe pipelines to avoid dimension conflict [cp_pipe, obs_lsst]
  • DM-29615: Fix connections problems in running Gen3 jointcal on HSC RC2 [jointcal, obs_subaru]
  • DM-29617: Write prototype remote butler Registry [daf_butler]
  • DM-29633: Investigate scarlet log propagation [meas_extensions_scarlet]
  • DM-29694: Allow retrieval of dimensions as JSON directly from database [daf_butler]
  • DM-29699: CalibCombineConnections changes its quantum dimensions at construction [cp_pipe, obs_lsst]
  • DM-29701: Latest weekly build fails when building utils [utils]
  • DM-29703: Add difference imaging to DRP pipeline [ip_diffim]
  • DM-29708: Make ButlerHttpURI compatible with non-webDAV HTTP backends [daf_butler]
  • DM-29727: Fix syntax bug introduced on DM-28636 [obs_base]
  • DM-29735: Felis schema file for DP0.1 [sdm_schemas]
  • DM-29737: Check if a SourceCatalog is sorted when calling find [afw]
  • DM-29750: Extend DRP pipeline definitions and labeled subsets [obs_subaru]
  • DM-29756: BPS - Automatic retry with larger memory [ctrl_bps]
  • DM-29767: Test if absolute deviation in estimated flux is large [meas_base]
  • DM-29775: Fix style issues in DM-29737 [afw]
  • DM-29776: Attempt complete two-QG HSC RC2 run on w16 on one tract [obs_subaru]
  • DM-29790: Replace boost::filesystem with std::filesystem [afw, meas_algorithms, meas_modelfit]
  • DM-29794: Investigate ci_hsc_gen3 image differencing failure with disassembly [daf_butler, obs_base, pipelines_check]
  • DM-29800: Fix compilation with clang 11.1.0 on Linux [daf_base]
  • DM-29804: Add debiased HSM moments to default obs_* configs [obs_decam, obs_lsst, obs_subaru]
  • DM-29805: Add Arm64 support to sphgeom [sphgeom]
  • DM-29808: Various fixes to MetadataMetricTask [verify]
  • DM-29810: Numpydoc conversion of meas_algorithms through simple_curve.py [meas_algorithms]
  • DM-29812: Remove mapApData.py. [ap_association]
  • DM-29830: Add DC2 defaults to obs_lsst imsim configs [obs_lsst]
  • DM-29841: Update faro application of external calibs [faro]
  • DM-29849: Speed up many-dataset deletion [daf_butler]
  • DM-29853: Update autoconf files for psfex package [psfex]
  • DM-29856: Switch ap_verify.py to Gen 3 by default [ap_verify]
  • DM-29863: ShapeHSM fails when psf bbox size is position dependent. [meas_extensions_shapeHSM]
  • DM-29869: Make doScaleDiffimVariance=True the default in ImageDifferenceTask [ap_pipe]
  • DM-29877: Avoid avoidable logic error in measureCrosstalk [cp_pipe]
  • DM-29881: Investigate differences in gen2 vs. gen3 SFP products for HSC-Y [ip_isr, obs_decam]
  • DM-29888: Add config field(s) to meas_extensions_scarlet to run on a subset of an input catalog [meas_deblender, meas_extensions_scarlet]
  • DM-29892: Attribute error for PF1Task [faro]
  • DM-29893: un-hardcode bps report output [ctrl_bps]
  • DM-29908: Registry collection loading can fail due to concurrent deletes [daf_butler]
  • DM-29916: Single-shot, multi-cycle FGCM is memory-inefficient [fgcm, fgcmcal]
  • DM-29921: Remove erroneous debug item from datastore config [daf_butler]
  • DM-29933: Add base classes for single-detector source catalog metrics [faro]
  • DM-29936: Enable getting Children without repeatedly checking if the SourceCatalog is sorted [afw, meas_base]
  • DM-29940: Update DP0.1 Felis with table_index values [sdm_schemas]
  • DM-29944: Add some narrow-band filters to skymap’s tract+patch+band data ID packers [skymap]
  • DM-29950: Use UUID5 for raw ingest when using UUIDs [daf_butler, obs_base]
  • DM-29953: bps submit TypeError about SerializedDataCoordinate [ctrl_bps]
  • DM-29966: Fix logic defaulting butler.yaml in butler config [daf_butler]
  • DM-29970: Cleanup histogram classes [jointcal]
  • DM-29981: Migrate cp_pipe pipelines as defined in RFC-775 [cp_pipe, obs_lsst, obs_subaru]
  • DM-29987: Save Detectors with master calibrations [cp_pipe]
  • DM-30004: Replace boost::regex with std::regex [afw, daf_base, daf_persistence]
  • DM-30023: Replace C++17 deprecated uncaught_exception [utils]
  • DM-30030: Make calibration source selection consistent between gen2 and gen3 middleware - part deux [ap_association, meas_algorithms, meas_astrom, obs_base, obs_subaru]
  • DM-30046: Investigate memory usage of ForcedPhotCcd [meas_base]
  • DM-30056: requestCpus not ending up in HTCondor submit files [ctrl_bps]
  • DM-30057: AP timing metrics out of date [ap_association, ap_verify]
  • DM-30061: PipelineTasks do not use pipeline label as name [ctrl_mpexec, pipe_base]
  • DM-30071: FgcmBuildStarsTableConnections sourceSchema should be an InitInput [fgcmcal]
  • DM-30076: Fix missing config imports in obs_lsst [obs_lsst]
  • DM-30079: Corrupted documentation breaks documentation builds [ip_diffim]
  • DM-30093: Astropy Header object does not behave dict-like. [astro_metadata_translator]
  • DM-30104: Make pickling the clustered quantum graph optional [ctrl_bps]
  • DM-30105: Get children sources without repeatedly checking if the afw SourceCatalog is sorted by parent [afw, meas_base]
  • DM-30109: Fix error introduced in DM-29841 (external calibs) implementation [faro]
  • DM-30124: error trying to specify query for ingest_date [daf_butler]
  • DM-30125: Add simple unit tests for image differencing [ip_diffim]
  • DM-30130: Establish a 1-1 correspondence between exposures and input dimensions in cpPtcExtract [cp_pipe]
  • DM-30139: Make position Optional when manually instantiating Stamps [meas_algorithms]
  • DM-30140: Add bulk datastore trash API [daf_butler]
  • DM-30142: Support P and Q for simulated images [obs_lsst]
  • DM-30145: Allow sqlalchemy to use IN operator for DELETE [daf_butler]
  • DM-30147: Butler warning about TypeDecorator Base64Bytes() [daf_butler]
  • DM-30148: PipelineTasks use wrong label as name [ctrl_mpexec]
  • DM-30153: Make jointcal debug output files not collide [jointcal]
  • DM-30161: Remove deprecated configs in fgcmFitCycle [fgcmcal]
  • DM-30178: Please make daf_butler compatible with sqlalchemy 2 [daf_butler]
  • DM-30200: Support glob strings in registry query APIs for collections and dataset types [daf_butler]
  • DM-30202: Add “online” coaddition code to meas_algorithms and pipe_tasks [meas_algorithms, pipe_tasks]
  • DM-30204: Add support for filter N708 to DECam [obs_decam]
  • DM-30245: Document cp_pipe [cp_pipe]
  • DM-30254: Fix jointcal crash when doing outlier rejection on only the model [jointcal]
  • DM-30257: Remove eotest dependency from cp_pipe [cp_pipe]
  • DM-30270: Make doFilterMorphological=True default in obs_subaru [obs_subaru]
  • DM-30272: Change default for dataset_id in registry to UUID [daf_butler]
  • DM-30281: Demonstrate the effect of random padding regeneration and fix unit test by making padding deterministic [ip_diffim]
  • DM-30286: Define a verbose log level for daf_butler [daf_butler]
  • DM-30287: Add LoadIndexedReferenceObjectsTask RST doc [meas_algorithms]
  • DM-30291: Setup towncrier on daf_butler [daf_butler]
  • DM-30296: ap_verify HSC Gen 3 ingestion crashes on missing defineVisits config [obs_subaru]
  • DM-30301: Switch Task to use python logging rather than lsst.log [meas_base, obs_base, obs_test, pipe_base, pipe_tasks]
  • DM-30316: Write UUID migration script for sqlite [daf_butler]
  • DM-30321: Validate the schema in ci_hsc_gen3 [sdm_schemas]
  • DM-30335: Demonstrate process for use of execution butler [daf_butler, pipe_base]
  • DM-30339: Replace boost::any with std::any [afw, daf_base]
  • DM-30342: Propagate flags to Object tables [obs_lsst, obs_subaru, sdm_schemas]
  • DM-30346: Remove boost:serialization [afw]
  • DM-30349: Source count metrics include fake sources [ap_verify, pipe_tasks]
  • DM-30350: Replace individual graph files to a single per workflow [ctrl_bps]
  • DM-30351: Make sure brighter-fatter correction is working on DC2 data for DP0.2 [ip_isr, obs_base, obs_lsst, obs_lsst_data]
  • DM-30365: Gen3 RC2 reprocessing with w_2021_22 and bps [skymap]
  • DM-30372: Specifying a manager in a butler seed prevents other managers from being specified [daf_butler]
  • DM-30373: Add butler command line tool for creating collection chains [daf_butler]
  • DM-30386: Make DRP wrapper task for DiaCalculation task [ap_association, meas_base]
  • DM-30425: Error in daf_base EPOCH definition [daf_base]
  • DM-30426: Out-of-bounds array access in shapeHSM [afw, meas_extensions_piff, meas_extensions_shapeHSM]
  • DM-30444: Require X% template coverage to attempt image subtraction [ip_diffim]
  • DM-30445: Have Stamps contain an optional Archive element [meas_algorithms]
  • DM-30448: Use correct shape measurement for ellipticity residual calculation [faro]
  • DM-30465: Implement decorrelation afterburner for maximum likelihood images [ip_diffim, pipe_tasks]
  • DM-30490: Add delta-magnitude check to AstrometryTask [meas_astrom]
  • DM-30497: Bug: DiaSource table cannot find Ixx column [ap_association]
  • DM-30534: Test and Implement Default Parameters for Jointcal Outlier Rejection [jointcal, obs_subaru]
  • DM-30563: ap_verify failing to find dataset type fakes_deepDiff_warpedExp [ap_pipe, ap_verify]
  • DM-30564: Improve task label uniqueness exception message [pipe_base]
  • DM-30630: Filter look-up for in isrTask is broken in a few places [ip_isr]
  • DM-30633: Add meas_extensions_trailedSources as setupOptional to lsst_distrib [lsst_distrib]
  • DM-30649: Exit with new gen3 “expected failure” exit code when its available [ctrl_mpexec, ip_isr, obs_lsst, pipe_base]
  • DM-30653: Write some release notes for v22 release of middleware [ctrl_mpexec, daf_butler, obs_base, pipe_base]
  • DM-30665: Add Source and Object schema_checks to ci_imsim [sdm_schemas]
  • DM-30683: Remove diaSourceAnalysis.py [ip_diffim]
  • DM-30685: Remove physical_filter+detector+exposure entry from fileDatastore.yaml templates [daf_butler]
  • DM-30692: Add refObjLoader lookups for DECam N708 filter [obs_decam]
  • DM-30693: Add meas_extensions_gaap to lsst_distrib [lsst_distrib]
  • DM-30696: Add task to translate gen2 fgcmcal photocalibs to gen3 photocalib catalog [fgcmcal]
  • DM-30701: Propagate astrometry offset stats in AstrometryTask and store in ExposureSummaryStats/VisitSummary [afw, meas_astrom]
  • DM-30702: Add provenance information to quantum graph [ctrl_mpexec, pipe_base]
  • DM-30718: butler.get fails to recognize full dataId [daf_butler]
  • DM-30733: Fix bug in BrightStarStamps’ initAndNormalize method [meas_algorithms]
  • DM-30738: Backport gen3 imSim BF-kernels for gen2 [ip_isr, obs_lsst]
  • DM-30743: Create temporary cache directory only when needed [daf_butler]
  • DM-30753: Automated publishing pipeline of TAP_SCHEMA metadata [sdm_schemas]
  • DM-30761: “bind” keyword argument not forward in queryDatasets recursion [daf_butler]
  • DM-30767: Execution butler builder needs to insert initOutput datasets [ctrl_mpexec, daf_butler, pipe_base]
  • DM-30771: Execution butler must be able to skip components on get [daf_butler]
  • DM-30776: Move MatchApFakesTask to pipe_tasks [ap_pipe, ap_verify]
  • DM-30777: Remove unused includes in afw [afw, jointcal]
  • DM-30784: butler transfer datasets must create dataset types if necessary [daf_butler]
  • DM-30787: Translate getting started tutorials to use gen3 and the new RC2 subset [pipe_base]
  • DM-30795: Fix ap_verify breakage due to DM-30649 [ap_verify]
  • DM-30798: Fix ap_association breakage due to DM-30649 [ap_association]
  • DM-30801: Pass memory requirements to PanDA for automatic queue brokerage decision [ctrl_bps]
  • DM-30815: Update expBits used in gen2 bulter to match value computed for gen3 [obs_lsst]
  • DM-30820: Investigate effect of reducing background fit Chebyshev polynomial order in DC2 [obs_lsst]
  • DM-30823: Add modelPhotRep metrics to faro [faro]
  • DM-30829: Add persistence method for WarpedPsf [afw, meas_algorithms]
  • DM-30838: Disable bright-object-mask-based catalog flags for ImSim/DC2 [obs_lsst]
  • DM-30842: Add healSparsePropertyMaps task to pipeline subsets [obs_subaru]
  • DM-30855: Butler unit tests fail with click8 [daf_butler]
  • DM-30858: Add GAaP measurements to obs packages [obs_lsst, obs_subaru, sdm_schemas]
  • DM-30863: Update LATISS visit region padding to 4k pixels [obs_lsst]
  • DM-30866: Add support for replacing bad DimensionRecords, use it in DefineVisits and RawIngest, and fix old LATISS visit regions [daf_butler, obs_base]
  • DM-30883: Incorrect triplet initialization can result in crash [jointcal]
  • DM-30884: Fix schema inconsistency in hsc.yaml [sdm_schemas]
  • DM-30900: Stop using obs_* configurations in fgcmcal [fgcmcal]
  • DM-30902: Numpydoc conversion of meas_algorithms through stamps.py [meas_algorithms]
  • DM-30905: Numpydoc conversion of meas_algorithms through utils.py [meas_algorithms]
  • DM-30926: Add SkySources to ImageDifferenceTask [ap_association, ap_pipe]
  • DM-30935: Add generic butler ingest command [daf_butler]
  • DM-30939: Fixup image type headers in old LATISS data [obs_lsst]
  • DM-30940: Add the –date-created command line option to dispatch_verify.py [verify]
  • DM-30943: Turn on delta mag rejection for astrometry matching in configs for LSSTCam-imSim [obs_lsst]
  • DM-30951: patchInfo.getIndex() returns the second index with a float instead of an int [skymap]
  • DM-30965: Update PanDA plugin to support lazy commands generation [ctrl_bps]
  • DM-30967: Fix segfault due to MKL [base]
  • DM-30977: Write pipeline task log output to butler [ctrl_mpexec, daf_butler, pipe_base]
  • DM-30985: Second attempt to fix hard coded paths in functorFile config [ap_association, ap_pipe, dax_apdb]
  • DM-30992: dax_apdb’s uint64 is incompatible with pandas 1.3.0 [dax_apdb]
  • DM-30996: Implement log4cxx appender to Python logging [log]
  • DM-31016: Replace all print statements in faro with log statements [faro]
  • DM-31023: Add wise_01 schema description to sdm_schemas [sdm_schemas]
  • DM-31026: Create Ephemeris Cache Precomputation Task for SSO attribution pipeline [ap_association]
  • DM-31027: statsCtrl.getMaskPropagationThreshold has buffer overrun and can segfault [afw]
  • DM-31031: Define for DC2 DRP pipeline a step1, 2, and 3 (and redefine for HSC) [obs_lsst, obs_subaru]
  • DM-31033: Move DiaCalculation from a subtask of AssociationTask to a subtask of DiaPipe. [ap_association]
  • DM-31043: Update ctrl_bps setup.cfg with flake8’s max-doc-length=79 [ctrl_bps]
  • DM-31054: Make aggregation steps robust to expected failures in RC2/DC2 [ctrl_mpexec]
  • DM-31055: The length is not long enough for the “skymap” column [sdm_schemas]
  • DM-31060: Clean up some log usage in Tasks [ip_diffim, ip_isr, meas_algorithms, meas_base, obs_base]
  • DM-31064: Create metric to capture the total number of fake sources used [ap_pipe, ap_verify, verify_metrics]
  • DM-31066: Move “convolution mode” log messages [ip_diffim]
  • DM-31077: Implement principal-column flagging for DP0.1 tables [sdm_schemas]
  • DM-31079: Butler define-visits ignores collection [obs_base]
  • DM-31093: Trigger deployment of wise_01 to TAP_SCHEMA in NCSA RSP instances [sdm_schemas]
  • DM-31096: Fix bitrot in gen3 linearity solver [cp_pipe]
  • DM-31100: Alter code for Networkx compatibility [pipe_base]
  • DM-31103: Add some missing python logging APIs to lsst.log [log]
  • DM-31105: Update faro pipelines to completely separate those with/without applying external calibrations [faro]
  • DM-31116: Deleting non-existant files from S3 ButlerURI triggers backoff [daf_butler]
  • DM-31119: Investigate all-NaN columns in objectTable_tract [obs_lsst, obs_subaru, sdm_schemas]
  • DM-31120: Change butler logger to python logging [ctrl_mpexec, daf_butler, log]
  • DM-31131: lsst.verify outputs non-standard NaN in JSON [verify]
  • DM-31135: Add DM license preamble for each of the python files [faro]
  • DM-31136: Consistently adopt DM convention for class names involving Task, Config, and Connections [faro]
  • DM-31141: Add test decorators for cartesian product [utils]
  • DM-31146: Add cp_verify to lsst_distrib [lsst_distrib]
  • DM-31152: Fix line/file reporting for lsst.log.warning [log]
  • DM-31156: Investigate GAaP colors in DC2 involving g band [obs_lsst, obs_subaru]
  • DM-31163: Add section to Object Table spec for flags taken from forced_src [obs_lsst, obs_subaru]
  • DM-31174: str() method failure on lsst.ip.isr.Defects class [ip_isr]
  • DM-31180: ap_verify loses logs from pipeline [ap_verify]
  • DM-31188: Fix LoadReferenceCatalogTask so it doesn’t temporarily clobber its own config [pipe_tasks]
  • DM-31217: Add task to consolidate healsparse property maps [obs_lsst, obs_subaru, pipe_tasks, skymap]
  • DM-31220: API mismatch in Gen 2 ap_pipe [ap_association, ap_pipe]
  • DM-31228: Migrate obs packages to python logging where appropriate [obs_cfht, obs_decam, obs_lsst, obs_subaru]
  • DM-31238: Remove phosimToRafts from obs_lsst [obs_lsst]
  • DM-31240: Add support of Execution Butler into the PanDA plugin [ctrl_bps]
  • DM-31249: Fix ButlerURI.join confusion with schemeless URI [daf_butler]
  • DM-31251: Add execution butler example to pipelines_check [daf_butler, pipe_base, pipelines_check]
  • DM-31255: Fix log-to-butler failutres with –skip-existing [ctrl_mpexec]
  • DM-31262: Execution butler creation should not forward namespace value [pipe_base]
  • DM-31267: Jointcal failures in w_2021_30 [jointcal]
  • DM-31272: Allow for curation of schema_index values “above” the Felis-file level [sdm_schemas]
  • DM-31275: w_2021_30 deblend or mergeDetections not playing nice with missing bands [pipe_tasks]
  • DM-31278: transformDiaSourceCatalog.py not expanding environment variables after weekly 29 [ap_association]
  • DM-31282: Fetch only last-produced metric in chained collection in gen3_to_job [verify]
  • DM-31285: Enable meas_algorithms python API docs [meas_algorithms]
  • DM-31291: Documentation for CatalogMeasurementBase broken [faro]
  • DM-31294: Re-run faro on DC2 and re-dispatch with appropriate dates [faro]
  • DM-31295: Add timers for datastore activities and improve S3 performance [daf_butler]
  • DM-31301: Fix VignetteExposure name [cp_pipe]
  • DM-31306: Create single SAL script that produces biases, darks, and flats [cp_pipe]
  • DM-31313: Add htmIndex computation for DiaSources [ap_association, pipe_tasks]
  • DM-31324: Rearrange steps in DRP.yaml for more complete outputs and weave in diffm [obs_lsst, obs_subaru]
  • DM-31326: Run faro on gen3_RC2_subset and set it up in Jenkins [faro]
  • DM-31331: Add new log options to bps [ctrl_bps]
  • DM-31337: Allow skipping calibration collections in registry search methods [daf_butler]
  • DM-31344: Execution butler creation fails on s3 [daf_butler]
  • DM-31355: Switch from the deprecated log.getName() to log.name [ap_verify, ip_diffim, meas_extensions_piff]
  • DM-31359: MaskStreaks sending bad matrix to scipy.linalg.cho_factor [pipe_tasks]
  • DM-31360: w_2021_30 deblend still not playing nice with missing bands [pipe_tasks]
  • DM-31366: Add confirmation request to butler prune-collection [daf_butler]
  • DM-31367: Add butler register-dataset-type subcommand [daf_butler]
  • DM-31376: assembleCoadd can’t handle when selectVisit selects a visit with no Warp [pipe_tasks]
  • DM-31381: No connection called refDataset in CatalogMeasurementBaseConnections [faro]
  • DM-31382: Run faro in ci_hsc and ci_imsim [faro]
  • DM-31384: Add pipetask log option to add arbitrary MDC records [ctrl_bps, ctrl_mpexec, daf_butler]
  • DM-31389: Move new DiaObject creation out of AssociationTask [ap_association]
  • DM-31394: Overscan subtraction can create negative variance [ip_isr]
  • DM-31398: Error with execution butler on IDF’s DC2 repo [daf_butler, pipe_base]
  • DM-31412: Add config/imsim/charImage.py to obs_lsst [obs_lsst]
  • DM-31417: Add makeHpxWcs to afwGeom to make an HPX SkyWcs object. [afw]
  • DM-31419: ImageDifferenceTask no longer timed in AP [ap_pipe, ap_verify, verify]
  • DM-31429: UNION ALL in subqueries is causing catastrophic database pessimizations [daf_butler]
  • DM-31434: Placeholder PSF WARN in charImage should be INFO [pipe_tasks]
  • DM-31442: Pass logName for undeblended plugins [meas_base]
  • DM-31443: Fix typo in quickFrameMeasurement task [pipe_tasks]
  • DM-31445: Ensure GAaP measurements are aperture corrected [obs_lsst, obs_subaru]
  • DM-31448: Allow gen2 butler lookups even when fields in the dataId are not in the registry [daf_persistence]
  • DM-31462: Add random seed config to make fgcm density downsampling repeatable [fgcm, fgcmcal, obs_subaru]
  • DM-31466: segfault when spatialOrder is zero in psfDetermination [meas_extensions_psfex]
  • DM-31472: Execution Butler merge failed with large workflows [daf_butler]
  • DM-31476: Clean up PanDA’s example bps yaml file [ctrl_bps]
  • DM-31489: Update StrayLightData to use FitsGenericFormatter with a deferred data set [ip_isr, obs_subaru]
  • DM-31505: Update fgcm to better handle survey edges and reference star outliers [fgcm, fgcmcal]
  • DM-31507: Adjust stack objects to accommodate reference counting [ctrl_mpexec, daf_butler, pex_config, pipe_base, pipe_tasks]
  • DM-31520: Request butler collection-chain to take comma-separated list [daf_butler, pipelines_check]
  • DM-31528: Add more log messages to the measure task [meas_base, meas_extensions_scarlet]
  • DM-31530: Log outputs are being swallowed when running pipe_drivers scripts [ctrl_pool]
  • DM-31535: Deprecate PSF methods default position argument [afw]
  • DM-31540: Remove butlerQC.registry usage from fgcmcal [fgcmcal]
  • DM-31541: Change pipetaskInit job to not read full QuantumGraph [ctrl_bps]
  • DM-31542: Examine why disassembled composites take up much more space in datastore [afw, daf_butler]
  • DM-31543: Update sdm_schemas to make DP0.1 catalogs on NCSA “small” qserv visible to TAP [sdm_schemas]
  • DM-31545: Fix cp_pipe/defects debug error [cp_pipe]
  • DM-31548: patch_skypix_overlap table needs an additional index [daf_butler]
  • DM-31549: Fix clang compilation in lsst_distrib [log]
  • DM-31564: Fix sconsUtils to allow running CodeChecker [sconsUtils]
  • DM-31571: Gen3 isrTask could be made to take a default None camera object [ip_isr]
  • DM-31579: Fix permissions of butler log datasets [ctrl_mpexec]
  • DM-31580: Split “step5” into a per-tract step5 and a per-visit step6 [obs_lsst, obs_subaru]
  • DM-31583: Add interfaces to dataset summary table data and use it to improve query generation [daf_butler, pipe_base]
  • DM-31591: Update developer guide on logging practices [meas_base]
  • DM-31596: Export DM-28920 calibrations for use at the summit [daf_butler]
  • DM-31601: Add tasks to forced phot diaObjects on calexps,diffim,coadds [ap_association, meas_base, obs_lsst, obs_subaru, pipe_tasks]
  • DM-31609: lsst.verify.Measurement needs a useful __repr__() [verify]
  • DM-31623: Support dataId rewriting on butler.put [daf_butler]
  • DM-31631: Forward job creation message on to dispatch_verify [verify]
  • DM-31640: New filter installed in LATISS, needs to be declared in filters.py [obs_lsst]
  • DM-31659: Logging loses part of message in assembly.py [obs_lsst]
  • DM-31663: fgcm 3.6.7 (with fgcmcal 779eefa) fails on macOS Catalina [fgcm]
  • DM-31664: Fix clang compilation warnings [jointcal, meas_extensions_psfex]
  • DM-31676: Add genTemplate to obs_subaru/obs_lsst subsets for DC2/RC2 batch reruns [obs_lsst, obs_subaru, pipe_tasks]
  • DM-31691: pipetask doesn’t find dataset in execution butler from previous run. [ctrl_mpexec]
  • DM-31696: VERBOSE log-level in pipetask run sometimes issues DEBUG log messages [daf_butler, ip_isr]
  • DM-31700: Butler parquet formatter fails to load table with no indexes [daf_butler]
  • DM-31710: Optimize crosstalkSources lookup in IsrTask [ip_isr]
  • DM-31711: Problems pipetask –save-execution-butler w_2021_37 [ctrl_mpexec]
  • DM-31721: Create cpputils package [afw, ap_pipe, ap_verify, daf_base, geom, jointcal, meas_algorithms, meas_base, meas_modelfit, obs_cfht, obs_decam, obs_lsst, obs_subaru, utils]
  • DM-31722: Move utility code out of pipe_base and daf_butler into utils package [utils]
  • DM-31760: Single amp reads for obs_lsst have the wrong geometry. [obs_lsst]
  • DM-31765: Make ForcedSourceOnDiaObjectsTable [obs_lsst, obs_subaru, pipe_tasks]
  • DM-31783: Register DECam N419 and N540 filters [obs_decam]
  • DM-31785: make butler transfer-datasets more efficient [daf_butler]
  • DM-31796: Add SDSS filters for LATISS in filters.py [obs_lsst]
  • DM-31822: github pages based schema browser [sdm_schemas]
  • DM-31825: Parquet Table clean up before DP0.2 Preliminary run. [ap_association, fgcmcal, jointcal, meas_base, obs_lsst, obs_subaru, pipe_tasks, sdm_schemas]
  • DM-31826: FileDatastore.transfer_from is replacing physical files with circular symlinks in the execution butler final job [daf_butler]
  • DM-31827: Stop providing fgcm photoCalib solutions when dubious [fgcmcal]
  • DM-31832: Change default LoadDiaCatalogs behavior to spatial loading of DiaSources. [ap_association]
  • DM-31841: The queue parameter submitted on per task level is not available in the GenericWorkflowJob [ctrl_bps]
  • DM-31846: Ongoing Jenkins issues [faro]
  • DM-31848: Add faro steps to DRP.yaml in obs_lsst/imsim [obs_lsst]
  • DM-31849: ForcedPhotCcdOnDataFrame outputs all NaNs [meas_base, pipe_tasks]
  • DM-31859: Add user-specified clustering to bps [ctrl_bps]
  • DM-31861: HSC DRP.yaml should not import from jointcal tests [obs_subaru]
  • DM-31869: Remove computeShape calls without argument in GAaP [meas_base]
  • DM-31887: Make execution butler default in ctrl_bps [ctrl_bps]
  • DM-31891: Fix cp_pipe documentation build error [cp_pipe]
  • DM-31900: Use per task cloud setting in PanDA plugin [ctrl_bps]
  • DM-31903: Add update kwargs/option to instrument registration [obs_base, obs_cfht, obs_decam, obs_lsst, obs_subaru]
  • DM-31912: Add additional GAaP circular apertures for DP0.2 [obs_lsst, sdm_schemas]
  • DM-31941: Log dataset does not overwrite on rerun with execution butler [ctrl_mpexec, daf_butler]
  • DM-31944: concurrency_limit split in htcondor submit file [ctrl_bps]
  • DM-31948: Fix regression in AP catalog metrics [ip_diffim]
  • DM-31970: Fix submission AttributeError in htcondor plugin when not using execution butler [ctrl_bps]