Simplify agent-swarm to a lite core with optional modules and evals

2026-10-28 00:30:03 +03:00
parent 4e1cc29874
commit 32d74210e4
8 changed files with 309 additions and 149 deletions
@@ -0,0 +1,47 @@
+# Testing the skill itself
+
+Do not assume the skill improves outcomes. Measure it against a no-skill baseline and against the
+previous version. This is the same idea as Anthropic's `skill-creator`: run baseline and with-skill
+runs on realistic prompts, compare quality and cost, then revise.
+
+## What to measure
+
+| Metric | How |
+|---|---|
+| Acceptance | Did the artifact pass the task's own checks? |
+| Human quality (1–5) | Blind review by someone who did not run the task |
+| Time | Wall-clock from prompt to accepted artifact |
+| Tokens / requests | Harness usage; record input/output/cache if available |
+| Agents spawned | Count; detect over-orchestration |
+| Process artifacts | Files under `.swarm/`; are they read or just produced? |
+| Regression | Tests, UX, performance, baseline comparison |
+| Performance | For UI: release p95 frame time, input latency, startup |
+| Trigger | Did the skill activate when appropriate, and stay out of small work? |
+
+## Runs
+
+For each task, run at least:
+
+1. **baseline** — no skill;
+2. **candidate** — current `agent-swarm`;
+3. optionally, the previous version.
+
+Use fresh contexts, the same model/settings, the same repository commit and the same time limit.
+Run twice if variance is high. Do not let the baseline see the skill's artifacts.
+
+## Tasks
+
+- [`tasks/small-fix.md`](tasks/small-fix.md) — should **not** trigger a swarm.
+- [`tasks/product-ui.md`](tasks/product-ui.md) — should trigger a small team and product gate.
+- [`tasks/broad-migration.md`](tasks/broad-migration.md) — should trigger a real swarm.
+
+## Pass condition
+
+The skill is useful only if it improves at least one of:
+
+- acceptance/quality without a large time or token penalty;
+- time/tokens without reducing quality;
+- product quality (performance/UX/baseline) on the product task.
+
+It fails if it adds agents/artifacts without improving outcomes, or if it triggers on the small fix.
+Record results in a copy of [`scorecard.md`](scorecard.md).