# Testing the skill itself

Do not assume the skill improves outcomes. Measure it against a no-skill baseline and against the
previous version. This is the same idea as Anthropic's `skill-creator`: run baseline and with-skill
runs on realistic prompts, compare quality and cost, then revise.

## What to measure

| Metric | How |
|---|---|
| Acceptance | Did the artifact pass the task's own checks? |
| Human quality (1–5) | Blind review by someone who did not run the task |
| Time | Wall-clock from prompt to accepted artifact |
| Tokens / requests | Harness usage; record input/output/cache if available |
| Agents spawned | Count; detect over-orchestration |
| Process artifacts | Files under `.swarm/`; are they read or just produced? |
| Regression | Tests, UX, performance, baseline comparison |
| Performance | For UI: release p95 frame time, input latency, startup |
| Trigger | Did the skill activate when appropriate, and stay out of small work? |

## Runs

For each task, run at least:

1. **baseline** — no skill;
2. **candidate** — current `agent-swarm`;
3. optionally, the previous version.

Use fresh contexts, the same model/settings, the same repository commit and the same time limit.
Run twice if variance is high. Do not let the baseline see the skill's artifacts.

## Tasks

- [`tasks/small-fix.md`](tasks/small-fix.md) — should **not** trigger a swarm.
- [`tasks/product-ui.md`](tasks/product-ui.md) — should trigger a small team and product gate.
- [`tasks/broad-migration.md`](tasks/broad-migration.md) — should trigger a real swarm.

## Pass condition

The skill is useful only if it improves at least one of:

- acceptance/quality without a large time or token penalty;
- time/tokens without reducing quality;
- product quality (performance/UX/baseline) on the product task.

It fails if it adds agents/artifacts without improving outcomes, or if it triggers on the small fix.
Record results in a copy of [`scorecard.md`](scorecard.md).