skill-eval-builder

---
name: skill-eval-builder
description: Use when proving that an agent skill triggers at the right time and improves results — build positive, negative, baseline, regression, and efficiency evals with explicit assertions.
source: https://github.com/hamelsmu/evals-skills
generated: 2026-07-14T21:19:53.210Z
category: tool
audience: engineers
---

## When to use

- Creating an eval suite for a new or revised SKILL.md before publishing it
- Testing whether a skill activates on intended prompts without over-triggering on adjacent requests
- Comparing with-skill results against a no-skill baseline using repeatable assertions
- Turning a reported skill failure into a permanent regression case
- Measuring whether a skill improves correctness enough to justify its token and tool cost

## Key concepts

### Activation matrix

Pair clear positive prompts, paraphrased positives, hard negatives, and near-miss negatives. A useful skill must trigger reliably on its job while staying quiet when a neighboring workflow is more appropriate.

### Baseline comparison

Run the same task without the skill and with the skill. Grade both against identical assertions so improvements are attributable to the skill rather than to an easy prompt or evaluator drift.

### Observable assertions

Grade artifacts and behavior that can be inspected: required files, exact fields, command exit codes, citations, forbidden mutations, user confirmations, or structured output. Avoid assertions based only on whether prose sounds good.

### Regression corpus

Every real failure becomes a small frozen test case containing the user request, minimal fixture, expected behavior, and failure mode. Re-run the corpus after every material skill edit.

### Efficiency budget

Track turns, tool calls, elapsed time, and approximate prompt size alongside correctness. Reject changes that add substantial overhead without a measurable quality or safety gain.

## API reference

```
evals/<case-id>/case.json
```

Store one portable test case with its prompt, fixture path, expected activation, assertions, and tags.

```
{
  "id": "reject-near-miss",
  "prompt": "Summarize this SKILL.md",
  "expectSkill": false,
  "assertions": [{ "type": "no-files-changed" }],
  "tags": ["activation", "negative"]
}
```

```
with-skill vs baseline run
```

Execute identical prompts against isolated copies of the same fixture, changing only whether the candidate skill is available.

```
baseline: fixture + user prompt
variant:  fixture + user prompt + candidate skill
grade:    apply the same assertions to both artifacts
```

```
eval summary
```

Report activation precision/recall, assertion pass rate, regressions, and efficiency deltas before recommending publication.

```
activation: 18/20 correct
assertions: baseline 61% -> skill 92%
regressions: 0
tool calls: +1.3 median
verdict: publish after fixing two false negatives
```

## Gotchas

- Do not leak the expected answer or suspected failure into the execution prompt; keep grading knowledge outside the agent run
- A passing happy-path example does not validate triggering — include paraphrases, ambiguous prompts, and near-miss negatives
- Do not compare runs made against different fixtures, dependency versions, or repository states
- Prefer deterministic artifact assertions before adding an LLM judge; use rubric-based judging only for genuinely qualitative output
- Treat flaky tool or network failures separately from skill failures and record the raw evidence
- Do not publish a skill that beats baseline only by doing more work while correctness and safety remain unchanged

---
Generated by SkillMake from https://github.com/hamelsmu/evals-skills on 2026-07-14T21:19:53.210Z.
Verify against source before relying on details.

File: ~/.claude/skills/skill-eval-builder/SKILL.md