1.9 KiB
1.9 KiB
Testing the skill itself
Do not assume the skill improves outcomes. Measure it against a no-skill baseline and against the
previous version. This is the same idea as Anthropic's skill-creator: run baseline and with-skill
runs on realistic prompts, compare quality and cost, then revise.
What to measure
| Metric | How |
|---|---|
| Acceptance | Did the artifact pass the task's own checks? |
| Human quality (1–5) | Blind review by someone who did not run the task |
| Time | Wall-clock from prompt to accepted artifact |
| Tokens / requests | Harness usage; record input/output/cache if available |
| Agents spawned | Count; detect over-orchestration |
| Process artifacts | Files under .swarm/; are they read or just produced? |
| Regression | Tests, UX, performance, baseline comparison |
| Performance | For UI: release p95 frame time, input latency, startup |
| Trigger | Did the skill activate when appropriate, and stay out of small work? |
Runs
For each task, run at least:
- baseline — no skill;
- candidate — current
agent-swarm; - optionally, the previous version.
Use fresh contexts, the same model/settings, the same repository commit and the same time limit. Run twice if variance is high. Do not let the baseline see the skill's artifacts.
Tasks
tasks/small-fix.md— should not trigger a swarm.tasks/product-ui.md— should trigger a small team and product gate.tasks/broad-migration.md— should trigger a real swarm.
Pass condition
The skill is useful only if it improves at least one of:
- acceptance/quality without a large time or token penalty;
- time/tokens without reducing quality;
- product quality (performance/UX/baseline) on the product task.
It fails if it adds agents/artifacts without improving outcomes, or if it triggers on the small fix.
Record results in a copy of scorecard.md.