48 lines
1.9 KiB
Markdown
48 lines
1.9 KiB
Markdown
# Testing the skill itself
|
||
|
||
Do not assume the skill improves outcomes. Measure it against a no-skill baseline and against the
|
||
previous version. This is the same idea as Anthropic's `skill-creator`: run baseline and with-skill
|
||
runs on realistic prompts, compare quality and cost, then revise.
|
||
|
||
## What to measure
|
||
|
||
| Metric | How |
|
||
|---|---|
|
||
| Acceptance | Did the artifact pass the task's own checks? |
|
||
| Human quality (1–5) | Blind review by someone who did not run the task |
|
||
| Time | Wall-clock from prompt to accepted artifact |
|
||
| Tokens / requests | Harness usage; record input/output/cache if available |
|
||
| Agents spawned | Count; detect over-orchestration |
|
||
| Process artifacts | Files under `.swarm/`; are they read or just produced? |
|
||
| Regression | Tests, UX, performance, baseline comparison |
|
||
| Performance | For UI: release p95 frame time, input latency, startup |
|
||
| Trigger | Did the skill activate when appropriate, and stay out of small work? |
|
||
|
||
## Runs
|
||
|
||
For each task, run at least:
|
||
|
||
1. **baseline** — no skill;
|
||
2. **candidate** — current `agent-swarm`;
|
||
3. optionally, the previous version.
|
||
|
||
Use fresh contexts, the same model/settings, the same repository commit and the same time limit.
|
||
Run twice if variance is high. Do not let the baseline see the skill's artifacts.
|
||
|
||
## Tasks
|
||
|
||
- [`tasks/small-fix.md`](tasks/small-fix.md) — should **not** trigger a swarm.
|
||
- [`tasks/product-ui.md`](tasks/product-ui.md) — should trigger a small team and product gate.
|
||
- [`tasks/broad-migration.md`](tasks/broad-migration.md) — should trigger a real swarm.
|
||
|
||
## Pass condition
|
||
|
||
The skill is useful only if it improves at least one of:
|
||
|
||
- acceptance/quality without a large time or token penalty;
|
||
- time/tokens without reducing quality;
|
||
- product quality (performance/UX/baseline) on the product task.
|
||
|
||
It fails if it adds agents/artifacts without improving outcomes, or if it triggers on the small fix.
|
||
Record results in a copy of [`scorecard.md`](scorecard.md).
|