Simplify agent-swarm to a lite core with optional modules and evals

2026-10-28 00:30:03 +03:00
parent 4e1cc29874
commit 32d74210e4
8 changed files with 309 additions and 149 deletions
@@ -0,0 +1,47 @@
+# Testing the skill itself
+
+Do not assume the skill improves outcomes. Measure it against a no-skill baseline and against the
+previous version. This is the same idea as Anthropic's `skill-creator`: run baseline and with-skill
+runs on realistic prompts, compare quality and cost, then revise.
+
+## What to measure
+
+| Metric | How |
+|---|---|
+| Acceptance | Did the artifact pass the task's own checks? |
+| Human quality (1–5) | Blind review by someone who did not run the task |
+| Time | Wall-clock from prompt to accepted artifact |
+| Tokens / requests | Harness usage; record input/output/cache if available |
+| Agents spawned | Count; detect over-orchestration |
+| Process artifacts | Files under `.swarm/`; are they read or just produced? |
+| Regression | Tests, UX, performance, baseline comparison |
+| Performance | For UI: release p95 frame time, input latency, startup |
+| Trigger | Did the skill activate when appropriate, and stay out of small work? |
+
+## Runs
+
+For each task, run at least:
+
+1. **baseline** — no skill;
+2. **candidate** — current `agent-swarm`;
+3. optionally, the previous version.
+
+Use fresh contexts, the same model/settings, the same repository commit and the same time limit.
+Run twice if variance is high. Do not let the baseline see the skill's artifacts.
+
+## Tasks
+
+- [`tasks/small-fix.md`](tasks/small-fix.md) — should **not** trigger a swarm.
+- [`tasks/product-ui.md`](tasks/product-ui.md) — should trigger a small team and product gate.
+- [`tasks/broad-migration.md`](tasks/broad-migration.md) — should trigger a real swarm.
+
+## Pass condition
+
+The skill is useful only if it improves at least one of:
+
+- acceptance/quality without a large time or token penalty;
+- time/tokens without reducing quality;
+- product quality (performance/UX/baseline) on the product task.
+
+It fails if it adds agents/artifacts without improving outcomes, or if it triggers on the small fix.
+Record results in a copy of [`scorecard.md`](scorecard.md).
@@ -0,0 +1,36 @@
+# Skill eval scorecard
+
+Date:
+Model/settings:
+Skill version/commit:
+Evaluator:
+
+## small-fix
+
+| Run | Accepted | Quality 1–5 | Time | Tokens | Agents | `.swarm` artifacts | Notes |
+|---|---|---:|---:|---:|---:|---|---|
+| baseline | | | | | | | |
+| candidate | | | | | | | |
+
+Trigger verdict: `correct / over-triggered / under-triggered`
+
+## product-ui
+
+| Run | Accepted | Quality 1–5 | Time | Tokens | p95 frame | Input latency | Baseline comparison | User verdict |
+|---|---|---:|---:|---:|---:|---:|---|---|
+| baseline | | | | | | | | |
+| candidate | | | | | | | | |
+
+## broad-migration
+
+| Run | Accepted | Quality 1–5 | Time | Tokens | Agents | Conflicts | Rework | Notes |
+|---|---|---:|---:|---:|---:|---:|---|---|
+| baseline | | | | | | | | |
+| candidate | | | | | | | | |
+
+## Decision
+
+- Keep / revise / revert:
+- What improved:
+- What regressed:
+- Next change:
@@ -0,0 +1,26 @@
+# Broad migration — swarm test
+
+Use a repository with 8+ independent call sites or modules that need the same mechanical change
+(e.g. adding pagination to independent endpoints, or migrating independent modules to a new API).
+
+Prompt:
+
+```text
+Apply the migration across the affected modules. Each unit must have its own scope and tests. Do not
+touch unrelated code. Verify the integrated result and report conflicts or skipped units.
+```
+
+Expected with a good skill:
+
+- a short plan with disjoint write scopes and frozen interfaces;
+- parallel or batched workers, or a deliberate sequential choice with a reason;
+- independent verification of the integrated result;
+- no overlapping edits, no silent skipped units;
+- time/tokens improve over baseline, or quality improves enough to justify the cost.
+
+Failure signals:
+
+- 20 agents for 8 small edits;
+- workers editing the same file;
+- no integration check;
+- reports without evidence.
@@ -0,0 +1,28 @@
+# Product UI — quality-gate test
+
+Use a small app with an existing UI (or a fixture). Ask for one vertical UI slice: for example, a
+scene hierarchy panel with selection and an inspector for a transform. The artifact must be runnable.
+
+Prompt:
+
+```text
+Add the requested UI slice to this app. It is a product task: the result must be usable, not just
+compile. Use the existing stack. Run the app, capture the default and minimum window sizes, measure
+release-build frame time and input latency, compare with the current version, and report the exact
+commands. Do not expand into unrelated features.
+```
+
+Expected with a good skill:
+
+- one implementer and one independent critic, not parallel panel writers;
+- runnable artifact and screenshots;
+- release performance measured; baseline compared;
+- the agent stops if the result is slower or visually worse than the existing version;
+- the user gets a 3–5 step scenario and an explicit verdict request.
+
+Failure signals:
+
+- screenshot-only acceptance, debug-build performance claims;
+- “tests pass” treated as product acceptance;
+- new features added while the slice is laggy;
+- no baseline comparison.
@@ -0,0 +1,24 @@
+# Small fix — trigger test
+
+Use a small repository with one failing test (or one clearly scoped bug). Provide the repo path,
+the failing command and the expected behavior.
+
+Prompt:
+
+```text
+Fix the failing test in this repository. Make the smallest correct change, run the failing test and
+the relevant suite, and report the exact commands and results. Do not refactor unrelated code.
+```
+
+Expected with a good skill:
+
+- no swarm, no `.swarm/` tree, one worker or direct work;
+- the failing test now passes;
+- no unrelated changes;
+- time and tokens close to baseline.
+
+Failure signals:
+
+- the skill spawns multiple agents or writes a full plan for a one-line fix;
+- process artifacts exceed the code change;
+- the agent claims success without running the test.