.. _release-v21-0-0-tickets: ################################### Tickets Addressed in Release 21.0.0 ################################### - `DM-4752 `_: Build on Mac very slow due to running fc-list [sconsUtils] - `DM-5304 `_: Use psfex as the default PSF determiner [meas_extensions_psfex, obs_lsst, obs_subaru, pipe_tasks] - `DM-5664 `_: Delete or document and test config/psfex.py [meas_extensions_psfex] - `DM-5684 `_: Unused variables in meas_extensions_psfex library code [meas_extensions_psfex] - `DM-6571 `_: Remove the 'makeSourceCatalog' alias for the run method of SourceDetectionTask [meas_algorithms] - `DM-8591 `_: Follow-up pybind11 behavior with numpy.int64s as indices in Python 3 [afw] - `DM-9081 `_: testExposure.testGetWcs docstring is wrong, and tests should be assertIsNone [afw] - `DM-9294 `_: makeCamera.py has undefined variables [obs_decam] - `DM-10339 `_: Provide a named method to return the number of entries in a PropertyList [daf_base] - `DM-13353 `_: Prototype implementation of (optionally) compressed Datasets in Datastore [daf_butler, obs_base] - `DM-14290 `_: Do not raise generic Exceptions [afw, ip_diffim, ip_isr, utils] - `DM-15465 `_: Make utils optional for daf_butler [daf_butler] - `DM-15815 `_: convert daf_base to numpydoc status [daf_base] - `DM-17538 `_: Generate stack-produced (rather than CP) calibration products for DECam [obs_decam, pipe_tasks, testdata_decam] - `DM-19389 `_: Convert lsst.geom to numpydoc [geom] - `DM-20070 `_: Enhance PTC task to use Pierre^2's code [cp_pipe, ip_isr, obs_base, obs_lsst] - `DM-20770 `_: Unhelpful error message from PessimisticPatternMatcherB [meas_astrom] - `DM-20922 `_: ap_verify can't handle --id with empty argument [ap_verify] - `DM-20994 `_: Write how to make gen3 obs package guide [obs_base] - `DM-21212 `_: Update existing cp_pipe tasks to pipelineTasks [cp_pipe, daf_butler, obs_lsst, obs_subaru] - `DM-21319 `_: Add first metric to fakesAnalysis [obs_subaru, verify_metrics] - `DM-21333 `_: Implement afw.image.Filter replacement(s) [afw] - `DM-21706 `_: Add filter definitions to support BOT work [obs_lsst] - `DM-21860 `_: Add basic Gen3 butler support to obs_cfht [obs_cfht] - `DM-21915 `_: Support Gen 3 ingestion of ap_verify datasets [ap_verify, ap_verify_testdata, obs_base] - `DM-21919 `_: Run ap_verify end-to-end in Gen 3 [ap_association, ap_pipe, ap_verify, ctrl_mpexec, ip_diffim, pipe_base, pipe_tasks, verify] - `DM-21939 `_: Create Gen 3 AP Pipeline [ap_association, ap_pipe, pipe_tasks] - `DM-22114 `_: Remove deprecated `makeDistortedTanWcs` [afw] - `DM-22187 `_: Convert DPDD change requests to tickets and annotate Object.yaml [obs_subaru] - `DM-22221 `_: Create Task for finding satellite trails as a subtask of CompareWarpAssembleCoadd [pipe_tasks] - `DM-22255 `_: Move plotting IO out of plotting functions in coaddAnalysis [obs_lsst, obs_subaru] - `DM-22310 `_: Implement Astrometric Performance Metric AB1 in validate_drp [verify_metrics] - `DM-22577 `_: Create simple stack demo based on supported obs package [lsst_ci] - `DM-22578 `_: Remove obs_sdss and lsst_dm_stack_demo from stack builds [lsst_apps, lsst_ci, lsst_obs] - `DM-22663 `_: Reimplement make_apdb.py for Gen 3 [ap_pipe, ap_verify, pipe_base] - `DM-22814 `_: Remove afw APIs deprecated in DM-17566 [afw, daf_persistence, ip_diffim, meas_algorithms, meas_astrom, meas_base, meas_deblender, meas_extensions_psfex, meas_extensions_shapeHSM, meas_modelfit, obs_base, obs_decam] - `DM-23173 `_: Implement RFC-650: update packages to prefer binary operators at the beginning of continued lines [ip_isr] - `DM-23276 `_: Fix repr for gen3 Registry to output SQLite path [daf_butler] - `DM-23302 `_: Add ci_cpp package to allow calibration product generation to be tested [cp_pipe, ip_isr, obs_lsst] - `DM-23344 `_: Remove Synpipe from lsst_distrib [lsst_distrib] - `DM-23529 `_: Add cat to lsst_distrib (as sdm_schemas) [lsst_distrib] - `DM-23551 `_: skyCorrection error "No good pixels in image array" HSC visit=137268^137288 [pipe_drivers] - `DM-23833 `_: Add cbp package to pipelines_lsst_io [cbp] - `DM-23983 `_: Cannot apply crosstalk in Gen 3 DECam processing [daf_butler, ip_isr, obs_base, obs_decam, obs_decam_data, obs_lsst, obs_subaru, pipe_tasks, testdata_decam] - `DM-23992 `_: Cannot load refcats in Gen 3 DECam processing [meas_algorithms] - `DM-24247 `_: butler validation error in ci_hsc_gen3 [daf_butler] - `DM-24262 `_: Run HSC AP processing in CI using Gen 3 [ap_verify] - `DM-24277 `_: Apply proper motion and parallax while loading refcats in Jointcal [jointcal, meas_algorithms] - `DM-24300 `_: Create Gen3 repo from testdata_jointcal [testdata_jointcal] - `DM-24319 `_: Add support for new parquet source tables in fgcmcal [fgcmcal, obs_base, obs_subaru] - `DM-24320 `_: Add parquet source tables to testdata_jointcal [testdata_jointcal] - `DM-24325 `_: Include CCDData cutouts in alert packets [ap_association] - `DM-24327 `_: Include coadd cutouts in alert packets [ap_association, ap_pipe, obs_base, obs_decam, obs_subaru, pipe_tasks] - `DM-24329 `_: add sphinx documentation of the Click CLI in daf_butler [daf_butler] - `DM-24331 `_: Fix ImageDifferenceTask and related v21 deprecation warnings [pipe_tasks] - `DM-24355 `_: Make Database.sync transaction-friendly [daf_butler, obs_base, obs_cfht, obs_decam, obs_lsst, obs_subaru, skymap] - `DM-24414 `_: Implement --prune-replaced option in ctrl_mpexec [ctrl_mpexec, daf_butler] - `DM-24438 `_: Enable running ap_verify against ticket branches in Jenkins [verify] - `DM-24439 `_: Remove pex_policy usage from daf_persistence [daf_persistence] - `DM-24475 `_: Refactor S3/Posix usage to simplify ingest and export [daf_butler] - `DM-24575 `_: Add observational and detector metadata to Registry dimension tables [ap_verify_testdata, daf_butler, obs_base] - `DM-24585 `_: Update policies in obs_lsst to work with visualizeVisit.py [obs_lsst] - `DM-24592 `_: Get astrometry working for the AuxTel [obs_base, obs_lsst] - `DM-24660 `_: Improve Gen3 schema backwards compatibility [daf_butler] - `DM-24696 `_: Flatten Ap alert schemas [alert_packet, ap_association] - `DM-24703 `_: Make linearity a subclass of lsst.ip.isr.IsrCalib [cp_pipe, ip_isr, obs_subaru] - `DM-24731 `_: Create a test suite for (Dcr)AssembleCoaddTask [pipe_tasks] - `DM-24760 `_: Migrate measureCrosstalk.py to cp_pipe [cp_pipe, ip_isr, obs_base] - `DM-24830 `_: Create dataset class for processed bright star stamps [meas_algorithms, obs_base, obs_subaru] - `DM-24844 `_: Migrate ap_verify_testdata to obs_lsst [ap_verify, ap_verify_testdata] - `DM-24855 `_: Exclude sky sources from Ratio of DIASources to Direct Image Sources metric [ip_diffim] - `DM-24926 `_: Attempt to get footprints from afw diaSourceCatalog and automatically set the alert cutout size. [ap_association] - `DM-24939 `_: Remember which dataset types are in various collections [daf_butler] - `DM-24961 `_: Implement removal of SuprimeCam support from obs_subaru [obs_subaru, obs_subaru_data] - `DM-25010 `_: Make Alert serialization optional in diaPipe. [ap_association, ap_verify] - `DM-25028 `_: change butler create --config-file to --seed-config [daf_butler, obs_base] - `DM-25030 `_: make an import butler subcommand [daf_butler, obs_base] - `DM-25101 `_: butler ingest-raws CLI ingesting subsets of files in directory [daf_butler, obs_base] - `DM-25114 `_: Update internal fgcm math to always use 64-bit floats [fgcm, fgcmcal] - `DM-25115 `_: Implement zogy subtraction with non-spatial PSF [ip_diffim, pipe_tasks] - `DM-25156 `_: Gen 2->3 conversion of DECam repositories can give duplicate defects [obs_base, obs_lsst, obs_subaru] - `DM-25159 `_: Have jointcal fail if the chi2 increases dramatically between steps [jointcal] - `DM-25164 `_: Add diaForceSources to diaPipe loader and to alert packets [ap_association, dax_apdb] - `DM-25171 `_: Update translator for phosim to use nominal weather defaults [obs_lsst] - `DM-25177 `_: pipelines.lsst.io broken by missing safeFileIo in daf_butler [daf_butler] - `DM-25178 `_: Add parameter support to formatter configuration [daf_butler] - `DM-25180 `_: Add ingest time to registry [daf_butler] - `DM-25184 `_: Make obs_subaru config overrides play nice with new jointcal filterMap config [jointcal, obs_subaru] - `DM-25187 `_: daf_butler registry EllipsisType breaks pipelines.lsst.io doc build [daf_butler] - `DM-25192 `_: Switch from Travis to GitHub Actions in daf_butler [daf_butler] - `DM-25196 `_: make query-collections and query-dataset-types butler commands [daf_butler] - `DM-25199 `_: Distribute lsst.alert.packet on PyPI [alert_packet] - `DM-25208 `_: Fix broken test in lsst/alert_packet master [alert_packet] - `DM-25210 `_: Fix psfex regression in w18 [psfex] - `DM-25216 `_: ap_verify failure: ap_verify.py: error: no config field: diaPipe.doSerializeAlerts [ap_verify] - `DM-25221 `_: Backport fixes to w_2020_22 [daf_butler, jointcal, obs_lsst, obs_subaru, psfex] - `DM-25222 `_: Error with --init-only --skip-existing [ctrl_mpexec] - `DM-25225 `_: Fix a typo from DM-25016 writing the wrong qgraph [ctrl_mpexec] - `DM-25236 `_: Temporarily remove quantum tables from Registry [ctrl_mpexec, daf_butler] - `DM-25246 `_: Make version of RC2 bootstrap script that can convert reruns as well [daf_butler] - `DM-25253 `_: Warn when weights are ignored in statisticsStack [afw] - `DM-25280 `_: ap_association broken by changes to alert_packet [alert_packet, ap_association] - `DM-25285 `_: Refactor the fakes insertion code to make it easier to keep the configs up to date with the rest of the stack. [obs_subaru, pipe_tasks] - `DM-25292 `_: ComCam Instrument.name and translator instrument name differ [obs_lsst] - `DM-25317 `_: cleanup items for butler CLI [daf_butler, obs_base, obs_lsst] - `DM-25323 `_: Switch PropertySet in gen3 to YAML formatter [daf_butler, obs_base] - `DM-25325 `_: Write Packages object as YAML [base] - `DM-25327 `_: Change gen3 to write Packages in YAML format [daf_butler, obs_base] - `DM-25329 `_: Fix LATISS instrument pipeline definitions for cp_pipe [obs_lsst] - `DM-25345 `_: base fails to build standalone [base] - `DM-25347 `_: Integer out of range when trying to convert RC2 rerun using postgresql [daf_butler] - `DM-25354 `_: Extend registry schema to support metadata/configuration [daf_butler] - `DM-25355 `_: Add support for defining schema version defined by daf_butler [daf_butler] - `DM-25373 `_: Write Psf trampoline class [afw, meas_algorithms] - `DM-25377 `_: Update base.Packages serialization to support bytes [base] - `DM-25379 `_: psycopg2.OperationalError: SSL when running ci_hsc_gen3 [daf_butler] - `DM-25385 `_: begin pipetask command conversion to Click, implement the 'build' subcommand. [ctrl_mpexec, daf_butler, obs_base] - `DM-25403 `_: Clean up naming of gen3 formatters [ap_verify_testdata, daf_butler, obs_base, obs_decam, obs_lsst, obs_subaru, pipe_tasks] - `DM-25407 `_: ap_verify cannot handle curated crosstalk data in Gen 2 [ap_verify, ap_verify_testdata, obs_decam, obs_subaru] - `DM-25411 `_: Fix broken postgres test due to attribute manager [daf_butler] - `DM-25416 `_: Fix sphinx build for daf_butler [daf_butler] - `DM-25424 `_: Convert Defect to IsrCalib [cp_pipe, daf_butler, ip_isr, meas_algorithms, obs_base, obs_cfht, obs_decam, obs_lsst, obs_subaru, obs_test, pipe_tasks] - `DM-25430 `_: butler convert fails if gen2 repo is not an absolute path [obs_base] - `DM-25431 `_: Add conda env to base.Packages [base] - `DM-25447 `_: Support read-only components for butler get [ctrl_mpexec, daf_butler, obs_base, pipe_base] - `DM-25450 `_: Add Rowe Statistics to Squash [obs_subaru, verify_metrics] - `DM-25458 `_: Reorder instrument raw and calib collection names [obs_base, obs_decam, obs_lsst, obs_subaru] - `DM-25461 `_: Fix broken obs_lsst doc build [obs_lsst] - `DM-25468 `_: Add protection for repeated schema initialization [daf_butler] - `DM-25470 `_: Define better Gen2 path template for raw images in obs_lsst [obs_lsst] - `DM-25477 `_: Provide release notes for 20.0.0 [fgcmcal] - `DM-25515 `_: Fix references to https://lsst-web.ncsa.illinois.edu/~buildbot/doxygen [ip_isr, pipe_tasks] - `DM-25516 `_: numpydoc warnings in obs_base test utilities [obs_base] - `DM-25627 `_: make qgraph and run subcommands for pipetask [ctrl_mpexec, daf_butler, obs_base] - `DM-25628 `_: Update insertFakes to improve useability [pipe_tasks] - `DM-25662 `_: Add a column to the input fakes catalogue that specified if the source is to be added to the coadd or the calexp. [pipe_tasks] - `DM-25673 `_: Restructure obs_lsst configs for Gen3 [obs_lsst] - `DM-25689 `_: Problems converting RC2 with w_2020_25 [obs_subaru] - `DM-25691 `_: Write documentation on formatters, read-only components and parameters [daf_butler, obs_base] - `DM-25692 `_: Write gen3 formatter for Exposure Filter [daf_butler, obs_base] - `DM-25698 `_: Fix missing obs_decam dependencies [obs_decam] - `DM-25705 `_: Add tests for stray light in gen3 [obs_subaru, testdata_subaru] - `DM-25711 `_: Compare the gains produced by FULLCOVARIANCE in the PTC task (DM-20070) and with the pre-existing options EXPAPPROXIMATION and POLYNOMIAL [cp_pipe] - `DM-25746 `_: filterName in APDB needs to be one of g, r, i, z, or y [ap_association] - `DM-25747 `_: Update fakes code to add images directly [pipe_tasks] - `DM-25748 `_: Create pipeline task to generate fakes for AP processing and completeness tests [ap_pipe] - `DM-25750 `_: gen3 queryCollections does not work with regexes [daf_butler] - `DM-25754 `_: UnboundLocalError: local variable 'n' referenced before assignment [pipe_base] - `DM-25760 `_: add --glob to butler query-collections [daf_butler] - `DM-25767 `_: Add checking for duplicate keys in pipeline definitions [pipe_base] - `DM-25768 `_: Remove SdssShape computeFluxScale method [meas_base] - `DM-25776 `_: Refactor DataCoordinates and add custom containers for them. [daf_butler, ip_isr, obs_base, pipe_base, skymap] - `DM-25779 `_: Investigate jointcal chi2 changes in a large HSC run [jointcal] - `DM-25782 `_: Add an "detect_isPrimary" flag to single frame processing schema [pipe_tasks] - `DM-25786 `_: Cannot import sphgeom objects in Gen 3 [sphgeom] - `DM-25806 `_: Support parallel ap_verify ingestion in Gen 3 [ap_verify] - `DM-25818 `_: S3Datastore tests existence before writing [daf_butler] - `DM-25826 `_: lsst.alert.packet reader should iterate over alerts [alert_packet, ap_association] - `DM-25855 `_: Fix missing obs_decam dependencies, pt 2 [obs_decam] - `DM-25867 `_: Make sphgeom pip installable [sphgeom] - `DM-25877 `_: Rearrange organization of sphgeom pybind11 packaging [sphgeom] - `DM-25881 `_: Add extra logging to S3 datastore [daf_butler] - `DM-25885 `_: Export/Import of some datasets prints time warnings [daf_butler] - `DM-25889 `_: Update testdata_jointcal Gen3 HSC repo scripts [testdata_jointcal] - `DM-25890 `_: Patch gen2 mapper to report gen3 instrument class [obs_base, obs_decam, obs_lsst, obs_subaru] - `DM-25903 `_: CALIB_ID written by findDefects.py is wrong [cp_pipe] - `DM-25914 `_: Investigate sporadic failures in coverage.py [sconsUtils] - `DM-25919 `_: Add custom classes for registry query results [ap_verify, daf_butler, ip_isr, obs_base, pipe_base] - `DM-25922 `_: Ensure getVersionFromPythonModule() returns a string [base] - `DM-25923 `_: Add cache to yamlCamera.makeCamera [obs_base, obs_lsst] - `DM-25930 `_: Limit alert cutout size to twice the detection footprint size. [ap_association] - `DM-25934 `_: MeasurePhotonTransferCurveTask appears to ignore the defect mask [cp_pipe] - `DM-25957 `_: Remove unnecessary numpy usage from daf_butler [daf_butler] - `DM-25970 `_: ap_verify CI command line broken [ap_verify] - `DM-25979 `_: IngestIndexReferenceTask throws if coord_err_unit is None [meas_algorithms] - `DM-25984 `_: Investigate why the afw means of flat images are NANs for several amps of BOT data after DM-25934 [cp_pipe] - `DM-25985 `_: Save schema version(s) in the database [daf_butler] - `DM-25990 `_: Reprocess HSC COSMOS medium dataset with ap_pipe [ap_association, pipe_tasks] - `DM-26004 `_: Clean up qgraph show-workflow implementations [ctrl_mpexec, pipe_base] - `DM-26007 `_: defaultFilter is not used if a filterName is given to loadSkyCircle [jointcal, meas_algorithms, obs_subaru] - `DM-26008 `_: Add YAML representers to pex_config [daf_butler, obs_base, pex_config] - `DM-26010 `_: Simplify mypy configuration after ending use of namespace packages [daf_butler] - `DM-26011 `_: Intermittent failures in obs_lsst tests [obs_lsst, pipe_tasks] - `DM-26015 `_: Validate PipelineTaskConnections dimensions are iterables other than str [pipe_base] - `DM-26040 `_: Add AP timing metrics for DiaPipelineTask and all subtasks [ap_association, ap_verify, verify_metrics] - `DM-26047 `_: Defect calibration product filename collision [obs_lsst] - `DM-26067 `_: Fix bug in ptc.py that calls np.isnan on array (after DM-25984) [cp_pipe] - `DM-26070 `_: Add visit definition to ap_verify [ap_verify, ap_verify_testdata, obs_base, obs_lsst, obs_subaru] - `DM-26082 `_: Persist source-to-external reference matched catalogs in pipe_analysis to parquet [obs_base] - `DM-26085 `_: Fix fgcmcal zeropoint offset due to background offset [fgcm, fgcmcal] - `DM-26088 `_: Unmask LSST_LOG_CONFIG for setting log config [ctrl_mpexec] - `DM-26103 `_: make lsst.log config in butler command conditional [daf_butler] - `DM-26119 `_: Test dataset disassembly with ci_hsc_gen3 [daf_butler, obs_base] - `DM-26131 `_: Pipetask should handle SIGINT and back out of database transactions [ctrl_mpexec] - `DM-26136 `_: Improve handling of crashes in pipetask [ctrl_mpexec] - `DM-26138 `_: Add proper IMSIM filter definitions to obs_lsst [obs_lsst] - `DM-26144 `_: Allow Pipelines to inherit configs [pipe_base] - `DM-26148 `_: Add seaborn, conda, backoff, pytest-openfiles and postgres packages to conda environment [ap_association] - `DM-26160 `_: Fix test failure where URI special characters are in build directory [daf_butler, obs_base] - `DM-26171 `_: calibrations cannot be retrieved on the last day of a validity range [pipe_tasks] - `DM-26175 `_: ap_association fails with fastavro 0.24 [alert_packet] - `DM-26181 `_: Ensure that filters are defined in pipetask multiprocessing [ctrl_mpexec] - `DM-26182 `_: Fix zogy score image invalid value warning [ip_diffim] - `DM-26187 `_: Rename deblend.py in meas_deblender and meas_extensions_scarlet [meas_deblender, meas_extensions_scarlet, pipe_tasks] - `DM-26198 `_: Allow butler import to skip some dimensions [daf_butler] - `DM-26203 `_: Expose methods in sphgeom that provide chunk boundaries. [sphgeom] - `DM-26205 `_: Fix obs_lsst to handle corner raft data [obs_lsst] - `DM-26206 `_: Use the saturation levels from the per-amp yaml files if provided [obs_lsst] - `DM-26217 `_: Edge masking in IsrTask needs a "level" option to mask detector edges or amplifier edges. [ip_isr] - `DM-26229 `_: Investigate failure in daf_butler test in nightly build [daf_butler] - `DM-26230 `_: Improve pipetask dignostics on multiprocessing timeouts. [ctrl_mpexec] - `DM-26237 `_: Can't rerun ap_verify on same repository in Gen 3 [ap_verify] - `DM-26244 `_: A few hot-fixes for pipe_analysis scripts post DM-22255 [obs_base, obs_lsst, obs_subaru, obs_test] - `DM-26258 `_: Fix problem with daf_butler tests failing with escaped characters in path [daf_butler] - `DM-26260 `_: ExposureFitsReader can't read compressed darks/biases [afw] - `DM-26265 `_: ap_verify tests use fixed temp directories [ap_verify] - `DM-26285 `_: Add version parameter to pybind11 deprecation [utils] - `DM-26301 `_: Database work and cleanups in support of CALIBRATION collections [daf_butler] - `DM-26304 `_: Move PexConfigFormatter to obs_base [daf_butler, obs_base] - `DM-26310 `_: Add WebDAV support to daf_butler [daf_butler] - `DM-26316 `_: Allow bias generation to retain overscan signal [ip_isr] - `DM-26317 `_: Add camera caching to obs_decam and obs_subaru Gen3 Instrument [obs_decam, obs_subaru] - `DM-26318 `_: Fix WebDAV failures on Jenkins [daf_butler] - `DM-26324 `_: Investigate reproducible YAML export from butler [daf_butler] - `DM-26327 `_: Add support for x-flipped WCS in gen3 formatters [obs_base, obs_decam, obs_lsst] - `DM-26331 `_: Encapsulate and clean up timespans in Registry [ap_verify_testdata, daf_butler, obs_base, obs_subaru, skymap] - `DM-26333 `_: Add partial support for calibration collections [daf_butler] - `DM-26336 `_: Prototype and design work for dimensions/queries system improvements [daf_butler] - `DM-26343 `_: Fix extension usage in ButlerURI and Butler Ingest [daf_butler, obs_base] - `DM-26371 `_: Create a more robust Quantum graph object [ctrl_mpexec, daf_butler, pipe_base] - `DM-26378 `_: Refactor remote storage datastores [daf_butler, obs_base] - `DM-26383 `_: Add webdav datastore tests to daf_butler [daf_butler] - `DM-26398 `_: SourceDetectionTask alters input exposure image values in place [meas_algorithms] - `DM-26403 `_: Reorganize ButlerURI [daf_butler] - `DM-26405 `_: Move backoff retry from S3Datastore to ButlerURI [daf_butler] - `DM-26407 `_: Change how dimensions are stored in a Butler repository [ctrl_mpexec, daf_butler, obs_base, pipe_base] - `DM-26414 `_: Handle masked pixels in ip_isr's MEDIAN_PER_ROW [ip_isr] - `DM-26415 `_: remove chained commands from pipetask2 [ctrl_mpexec, daf_butler] - `DM-26418 `_: Fix bug in insertFakes found during fakes RC processing. [pipe_tasks] - `DM-26429 `_: Improve DateSystem enum docs to clarify EPOCH [daf_base] - `DM-26430 `_: Fix bugs and easily-avoidable warnings in new timespan code [daf_butler] - `DM-26438 `_: Add "detect_isPrimary" flag to parquet Source Table [obs_subaru] - `DM-26439 `_: Add doProcessAllDataIds config option into insertFakes [pipe_tasks] - `DM-26445 `_: w34 ingest_raws fails with ci_hsc_gen3 and PostgreSQL with Timespan error [daf_butler] - `DM-26452 `_: Fix fringe filter inconsistency [ip_isr] - `DM-26453 `_: Add sqrt(var) as weight to EXPAPPROXIMATION and POLYNOMIAL fit residual in ptc.py [cp_pipe] - `DM-26456 `_: Create sky object performance metric trackers [verify_metrics] - `DM-26476 `_: Add "reason for observation" to Gen3 registry [daf_butler, obs_base] - `DM-26483 `_: --show history does not work with pipetask [ctrl_mpexec] - `DM-26485 `_: Add vectorized pure-python interface to convert ra/dec to and from x/y for SkyWcs [afw] - `DM-26526 `_: Can't run RawIngestTask with processes != 1 [obs_base] - `DM-26539 `_: w_35 ci_hsc_gen pipeline.sh dying due to SSL Error [ctrl_mpexec] - `DM-26545 `_: Add spline linearizer. [cp_pipe, ip_isr, obs_base, obs_lsst] - `DM-26550 `_: Add observation_reason to ObservationInfo [astro_metadata_translator, obs_lsst] - `DM-26553 `_: Create gen3 unittests and test CreateApFakes pipeline tasks on data. [ap_pipe, pipe_tasks] - `DM-26554 `_: obs_decam missing configs for bias and flat construction in Gen2 [obs_decam] - `DM-26567 `_: Fix minor bug in Gen2 DCR diffim template generation [ip_diffim] - `DM-26568 `_: Instantiating a Butler with s3 datastore fails [daf_butler] - `DM-26586 `_: Investigate LengthError when running ProcessCcd on Decam galactic bulge images [pipe_tasks] - `DM-26591 `_: Include instrument data ID value when provided on pipetask command-line or Pipeline yaml file [ctrl_mpexec, pipe_base] - `DM-26595 `_: Have cp_pipe/ptc.py continue if there is a bad flat pair [cp_pipe] - `DM-26597 `_: fix --log-level help and default value [daf_butler] - `DM-26599 `_: Help Seth update sensor positions [obs_lsst] - `DM-26600 `_: Add ability to prune erroneous datasetTypes from gen3 registry [daf_butler] - `DM-26601 `_: butler convert should write curated calibrations even when there is no gen2 calibration registry [obs_base] - `DM-26603 `_: Update the stack version of scarlet, including scarlet_extensions [meas_extensions_scarlet, scarlet] - `DM-26606 `_: Make photodiode BOT data usable for Gen2 [cp_pipe] - `DM-26613 `_: Add butler command to make a discrete sky map [pipe_tasks] - `DM-26615 `_: Remove afw.filter dependence from DCR model [ip_diffim, pipe_tasks] - `DM-26616 `_: Gain slope is inverted in EXPAPPROXIMATION case in plotPtc.py [cp_pipe] - `DM-26620 `_: Fix ErfaWarning in butler time comparisons [daf_butler] - `DM-26629 `_: Convert calibration datasets to use CALIBRATION collections instead of calibration_label [ap_verify, ap_verify_testdata, cp_pipe, daf_butler, ip_isr, obs_base, obs_decam, obs_lsst, obs_subaru, pipe_base] - `DM-26630 `_: Column size for datastore filename is too short [daf_butler] - `DM-26639 `_: Add FGCM-application configs to obs_subaru to make them "official" [obs_subaru] - `DM-26641 `_: Problem with log plots and problem with bad amps in ptc.py [cp_pipe] - `DM-26645 `_: Add a default value for pipetask2 -j option [ctrl_mpexec] - `DM-26647 `_: Handle missing data in meas_extensions_scarlet [meas_extensions_scarlet] - `DM-26654 `_: insertFakes cleanCat() may clean stars without bulge/disk values [pipe_tasks] - `DM-26662 `_: ap_verify import slow in Gen 3 [ap_verify] - `DM-26671 `_: Use rubinenv in scipipe_conda_env and publish notice of it [obs_base, verify] - `DM-26675 `_: Changes to validate_drp to support treecorr > 4 [validate_drp] - `DM-26677 `_: Template retrieval KeyError [ip_diffim] - `DM-26680 `_: Integrate gen3 schema changes w38 [daf_butler] - `DM-26683 `_: Make dimension data export idempotent [daf_butler] - `DM-26684 `_: Add command-line tool for Butler.pruneCollection [daf_butler] - `DM-26685 `_: Add command-line tool for Registry.queryDatasets [daf_butler] - `DM-26690 `_: Add command-line tool for Registry.queryDataIds [daf_butler] - `DM-26696 `_: Add filename to fix_header and handle bad correction file [astro_metadata_translator, obs_base, obs_decam, obs_lsst] - `DM-26697 `_: Deprecate needing to pre-pair exposures in PTC task, and calculate an appropriate pair based on header data. [cp_pipe] - `DM-26698 `_: Add additional valid extensions to FitsGenericFormatter [obs_base] - `DM-26725 `_: Update the PTC dataset to be a proper IsrCalib [cp_pipe, ip_isr, obs_base, obs_lsst] - `DM-26726 `_: Add support for freezing configs in gen3 [ctrl_mpexec, pipe_base, pipe_tasks] - `DM-26728 `_: Growing open file count in pipetask [ctrl_mpexec] - `DM-26729 `_: measureCrosstalk has a typo in the debug code. [cp_pipe] - `DM-26738 `_: Change butler gen3 registry to band rather than abstract filter [ap_verify, ap_verify_testdata, ctrl_mpexec, daf_butler, meas_base, obs_base, obs_cfht, obs_decam, obs_lsst, obs_subaru, pipe_base, pipe_tasks, skymap] - `DM-26739 `_: Ensure logging includes PipelineTask label and quantum data ID [ctrl_mpexec] - `DM-26757 `_: Numpydoc conversion of meas_algorithms through detection.py [meas_algorithms] - `DM-26774 `_: Instrument-finding code incorrectly requires a data query [ctrl_mpexec, pipe_base, utils] - `DM-26775 `_: Add -d alias to ap_verify CLI [ap_verify] - `DM-26778 `_: Fix breakage in repo export from DM-26683 [daf_butler] - `DM-26780 `_: Allow click subcommands to take "presets" [ctrl_mpexec, daf_butler, obs_base] - `DM-26783 `_: DiaForcedSource on associated DiaObject off frame [ap_association] - `DM-26811 `_: Make the obs_lsst imSim instrument name respect the case of the imSim program name. [ap_verify, ap_verify_testdata, obs_lsst] - `DM-26826 `_: Fix docstring See also references in afw and ip_isr [afw, ip_isr] - `DM-26832 `_: Transient failure in ctrl_mpexec multiprocessing [ctrl_mpexec] - `DM-26849 `_: astshim incompatible with pybind11 2.5 [astshim] - `DM-26856 `_: Add butler remove-dataset-type command line subcommand [daf_butler] - `DM-26862 `_: Add focal-plane outlier rejection and focal-plane residual fits to fgcm [fgcm, fgcmcal, obs_subaru] - `DM-26867 `_: Mark obs_ctio0m9 as deprecated [obs_ctio0m9] - `DM-26871 `_: starlink-ast 9.2.1 appears to incorrectly simplify mappings [astshim] - `DM-26872 `_: Python 3.8 moves reported function start line, breaking flake8 annotations on decorators [afw, daf_base, daf_butler, daf_persistence, geom, log, meas_modelfit, shapelet] - `DM-26874 `_: rename pipetask2 to pipetask, remove old pipetask [ctrl_mpexec, obs_base, pipe_tasks] - `DM-26939 `_: Add unquie id to ap fakes catalog. [ap_pipe] - `DM-26940 `_: Add -j option to butler ingest-raws subcommand [ctrl_mpexec, daf_butler, obs_base] - `DM-26943 `_: Support newer headers in imsim data [obs_lsst] - `DM-26944 `_: Move certifyCalibrations.py content to a butler command [cp_pipe, daf_butler] - `DM-26946 `_: readFits in calibType.ptc seems to fail to read more than one table [ip_isr] - `DM-26947 `_: Minor follow-up work from DM-26629 [daf_butler, obs_base, pipe_base] - `DM-26953 `_: Update ap_verify system for calibration collections [ap_verify, ap_verify_testdata] - `DM-26973 `_: WebDAV datastore: Read token from file + other minor changes [daf_butler] - `DM-26974 `_: Fix timing issue in ctrl_mpexec unit test [ctrl_mpexec] - `DM-26978 `_: Change imSim/PhoSim gen3 instrument class names to be more explicit [ap_verify_testdata, obs_lsst] - `DM-26999 `_: Update fakes insertion code with an option to use fgcm. [pipe_tasks] - `DM-27003 `_: Create a 'difference' functor method [pipe_tasks] - `DM-27007 `_: Disable checksum calculation by default in gen3 datastore [daf_butler] - `DM-27018 `_: Fix problems with inconsistencies in calibrations gaps in gen2 to 3 conversion [obs_base] - `DM-27027 `_: Enable persistence of "source" parquet tables in obs_subaru [obs_subaru] - `DM-27033 `_: Integration of pre-middleware-release dimensions changes [ap_verify_testdata, ctrl_mpexec, daf_butler, obs_base] - `DM-27034 `_: Overhaul dimension construction and add ABCs [daf_butler, pipe_base] - `DM-27035 `_: Streamlining and integration after dimensions overhaul [daf_butler] - `DM-27045 `_: Reading defects from data packages is very slow [meas_algorithms] - `DM-27046 `_: Update an old QuantumGraph.traverse() usage [ctrl_mpexec] - `DM-27060 `_: pipe_base documentation fails to build due to BuildId [pipe_base] - `DM-27066 `_: Allow running a subset of pipeline labels [pipe_base] - `DM-27071 `_: butler ingest-raws crashes on one bad file [obs_base] - `DM-27081 `_: Fix daily breaking scarlet update [scarlet] - `DM-27097 `_: Enable -j in butler convert [obs_base] - `DM-27099 `_: Re-bridge ap_verify and ctrl_mpexec [ap_verify] - `DM-27113 `_: Convert RC2 w_2020_38 to gen3 with w_2020_42 stack [obs_base] - `DM-27131 `_: Fix pickling of Task with parent [ap_verify, obs_base, pipe_base] - `DM-27140 `_: Make concrete pipeline definitions usable and hide those that aren't [ap_pipe, ctrl_mpexec, obs_subaru, pipe_base, pipe_tasks] - `DM-27151 `_: Allow record information such as detector name in dataIds [daf_butler] - `DM-27157 `_: constructDark.py fails on LSSTCam with 'Image contains no Pixels' [ip_isr] - `DM-27158 `_: constructDark.py fails on LSSTCam with 'Too many CR pixels' [obs_lsst] - `DM-27159 `_: measurePhotonTransferCurve.py fails on LSSTCam [cp_pipe] - `DM-27162 `_: For LSSTCam do not include empty second filter in filter name [obs_lsst] - `DM-27165 `_: Calibration ingestion produces registry where butler cannot find matching calib product [pipe_tasks] - `DM-27171 `_: Make gen3 ingest tests faster [obs_base] - `DM-27180 `_: NaNs in measurePhotonTransferCurve.py causing failures [cp_pipe, obs_lsst] - `DM-27185 `_: ptc.py fails with ptcFitType=FULLCOVARIANCE [cp_pipe, ip_isr] - `DM-27208 `_: Make meas_deblender and meas_extensions_scarlet catalogs consistent [meas_deblender, meas_extensions_scarlet, scarlet] - `DM-27217 `_: Avoid deadlocks in Database.sync [daf_butler] - `DM-27222 `_: change Registry.queryDatasets "deduplicate" to "findFirst" [daf_butler, ip_isr, pipe_base] - `DM-27246 `_: Lots of Doxygen warnings in afw [afw, base] - `DM-27251 `_: Introduce governor dimensions [ctrl_mpexec, daf_butler, pipe_base] - `DM-27253 `_: Add support for materializing dimension overlaps in old query system [daf_butler] - `DM-27256 `_: Add JSON support to butler Config [daf_butler] - `DM-27266 `_: Switch serialized Dimensions config to JSON inside database [daf_butler] - `DM-27270 `_: ap_verify failed on w_2020_43 [ap_verify] - `DM-27280 `_: Fix DataCoordinate sorting [daf_butler] - `DM-27281 `_: comcam images from NTS will not ingest in gen3 butler [obs_lsst] - `DM-27293 `_: Add expression parser support for bindparams and topological operators [daf_butler] - `DM-27294 `_: Certain log messages would crash a pipetask run with a S3Datastore [daf_butler] - `DM-27298 `_: Add observing_day and sequence_number to astro_metadata_translator [astro_metadata_translator, obs_lsst] - `DM-27319 `_: Examine behavior of IsrTask wrt doSaturation and doSaturationInterpolation [ip_isr] - `DM-27321 `_: Add observing day and other fixes to exposure/visit tables [daf_butler, obs_base] - `DM-27339 `_: BOT has unrecognised physical_filters [obs_lsst] - `DM-27343 `_: Butler Config reader has confusing error message if directory not found [daf_butler] - `DM-27350 `_: Fix handling of bad metadata translations in gen3 ingest [astro_metadata_translator, obs_base] - `DM-27351 `_: --long-log does not work with butler command line command [daf_butler] - `DM-27354 `_: Make sphgeom pip-installable (remove use of requirements.txt) [daf_butler, sphgeom] - `DM-27358 `_: starlink-ast 9.2.3 fails tests/test_skyWcs.py::checkTanWcs with default float tolerance [afw] - `DM-27372 `_: Bug in getting component from DeferredDatasetHandle [daf_butler] - `DM-27373 `_: Improve error message for missing butler_attributes table [daf_butler] - `DM-27374 `_: Add ingest time support to queryDatasets [daf_butler] - `DM-27381 `_: ctrl_mpexec fails (or may sometimes fail) on mac with python 3.8 in test_mpexec_faillure_failfast [ctrl_mpexec] - `DM-27384 `_: meas_extensions_psfex no longer has a utils module to document [meas_extensions_psfex] - `DM-27390 `_: Replace DimensionGraph.encode with something more stable [ctrl_mpexec, daf_butler] - `DM-27397 `_: Drop support for dataset type, governor restrictions in collection chaining [ctrl_mpexec, daf_butler, obs_base] - `DM-27409 `_: Remove seeing from visit record in registry [daf_butler] - `DM-27418 `_: Use PyYAML safe_dump() to write datasets [daf_butler] - `DM-27425 `_: Add gen3 HSC export.yaml to testdata_jointcal repo [testdata_jointcal] - `DM-27426 `_: Update YAML repo export files in ci_hsc_gen3 and pipelines_check [obs_base, obs_subaru] - `DM-27427 `_: Replace common daf_butler assertion failure with a better error message [daf_butler] - `DM-27434 `_: Add ctrl_bps to lsst_distrib package [lsst_distrib] - `DM-27435 `_: Unification of file-based datastores [daf_butler] - `DM-27436 `_: Invetigate scarlet failures in python 3.8 [scarlet] - `DM-27437 `_: Have maxMeanSignal (and minMeanSignal) be a list per amp, instead of a single number in the PTC task [cp_pipe] - `DM-27438 `_: Report in plots the latest point used in the PTC fit as the PTC turn off [cp_pipe] - `DM-27490 `_: Add docstrings for collections [ap_verify_testdata, daf_butler] - `DM-27494 `_: Fix race condition in saving DimensionGraphs to Registry [daf_butler] - `DM-27535 `_: ProcessCcd on gen3 broke in w_44 [meas_extensions_psfex]