diff --git a/README.md b/README.md index c35ecf6..13f8cbb 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,10 @@ ## agent-swarm -Универсальные большие задачи, которые делятся на 8+ независимых кусков: код и миграции, -аудиты, research, многосекционные документы, данные, дизайн, инфраструктура. Лид планирует -и интегрирует, воркеры работают по пакетам с непересекающимися файлами/артефактами, а -независимые верификаторы проверяют артефакт, а не отчёт воркера. +Тонкий оркестрационный скилл для больших или рискованных задач: код и миграции, аудиты, +research, документы, данные, дизайн, инфраструктура. Режимы direct / paired critic / swarm; +процесс опциональный, инварианты — обязательные: ownership, независимая проверка, evidence, +product acceptance. Тяжёлые темы вынесены в опциональные references. - [Описание на русском](agent-swarm/README.ru.md) - [Инструкции скилла](agent-swarm/SKILL.md) @@ -18,6 +18,7 @@ - [Prerequisite gate](agent-swarm/references/prerequisites.md) - [Product quality gates](agent-swarm/references/product-quality.md) - [Изолированное визуальное тестирование](agent-swarm/references/visual-testing.md) +- [Evals самого скилла](agent-swarm/evals/README.md) ### Установка @@ -33,9 +34,10 @@ cp -r skills/agent-swarm ~/.dsh/skills/ ``` agent-swarm/ -├── SKILL.md # основной протокол +├── SKILL.md # тонкий core: режимы + инварианты ├── README.ru.md # краткое описание -├── references/ # playbook, packets, orchestration, domains, visual testing, token economy, prerequisites, product quality +├── evals/ # baseline vs with-skill, 3 задачи, scorecard +├── references/ # опциональные модули: playbook, packets, domains, quality, tokens... ├── assets/ # шаблоны, схемы, workflow-скрипты └── scripts/audit-scope.mjs # проверка ownership ``` diff --git a/agent-swarm/README.ru.md b/agent-swarm/README.ru.md index 3dc4bf1..73e46be 100644 --- a/agent-swarm/README.ru.md +++ b/agent-swarm/README.ru.md @@ -1,6 +1,8 @@ # agent-swarm -Универсальный скилл для больших задач, которые делятся на 8+ независимых кусков: код и миграции, аудиты, research, документы, данные, дизайн, инфраструктура. Лид планирует, пакует контекст, запускает волны (через `workflow`, `subagent` или последовательно — если инструмента нет) и интегрирует. Воркеры не пересекаются по файлам/артефактам, каждый получает пакет и возвращает структурный отчёт. +Тонкий оркестрационный скилл для больших или рискованных задач: код и миграции, аудиты, research, +документы, данные, дизайн, инфраструктура. Это **не обязательный пайплайн**, а набор режимов и +инвариантов: direct / paired critic / swarm. ## Вызов @@ -10,46 +12,76 @@ или просто опишите задачу — скилл подключится по описанию. -Скилл не обязателен для каждой задачи: маленькую задачу можно делать напрямую, среднюю — с независимым критиком, и только широкую — через swarm. Пользователь выбирает стек, объём и темп; скилл добавляет координацию, когда она оправдана. +## Режимы -Процесс — опциональный, quality gates — нет. Для продуктовой работы заранее выбирается профиль `prototype` или `product`. В `product` обязательны release-замеры производительности, реальный UX-сценарий, сравнение с baseline и явный вердикт пользователя. Технический `pass` verifier'а не заменяет приёмку продукта. +| Режим | Когда | Что добавляет | +|---|---|---| +| Direct | одна задача, быстрый фидбек, низкий риск | ничего | +| Paired critic | средняя задача или рискованное решение | независимую проверку | +| Swarm | 8+ независимых кусков или ширина | ownership, параллельных воркеров, verifier | + +Если куски делят один экран, один дизайн-язык или один mutable-артефакт — не swarm. Один +implementer + один critic обычно дают более цельный UI, чем параллельные владельцы панелей. + +## Инварианты + +1. Сначала outcome и quality bar, потом процесс. +2. Один владелец на артефакт; shared-файлы — у лида/интегратора. +3. Воркер получает пакет, а не всю историю чата. +4. Автор не проверяет сам себя; для рискованной работы нужен свежий контекст. +5. «Готово» — это runnable artifact, команда, цитата, метрика или рендер, а не уверенная фраза. +6. Технический `pass` — не product acceptance. Для product/UI: release-замеры, реальный сценарий, + baseline и явный вердикт пользователя. +7. Если лагает или выглядит хуже baseline — чинить, а не добавлять фичи. +8. Для product — один milestone за раз; следующий не начинать без приёмки. +9. Проверять prerequisites; глобальные установки/деструктив — только с разрешения. +10. Инструкции пользователя важнее скилла: попросил быстро/прототип — убрать церемонии, оставить + safety и evidence. ## Что происходит -1. **Fit-check.** Если задача не делится на 8+ независимых юнитов, скилл скажет не использовать swarm. -2. **Prerequisite gate.** Скилл проверяет toolchain, runtime, сервисы, display, credentials и baseline. Ставит только project-local/pinned и только с разрешения; если обязательный prerequisite не закрывается — спрашивает пользователя **до** запуска работы. Воркеры ничего не устанавливают. -3. **Recon.** 2–4 агента картографируют репозиторий/источники/данные; лид пишет `.swarm/brief.md`, `.swarm/profile.md` и выбирает domain-адаптер из `references/domains.md`. -4. **План.** Задачи, владельцы файлов, контракты, критерии приёмки. Гейт: без этого волны не стартуют. -5. **Волны.** Воркеры идут батчами по 6–8 (реальный лимит активных детей узнаётся перед запуском и записывается в `state.json`; recon не должен занимать слоты). Затем 4–6 независимых верификаторов на «замороженном» дереве, затем интегратор. -6. **Аудит.** `scripts/audit-scope.mjs` сравнивает изменения с картой владения. -7. **Интеграция.** Один интегратор собирает общие файлы и прогоняет полный acceptance suite. -8. **Product gate.** Для `product`: release-замеры, реальный сценарий, baseline и вердикт пользователя. Без этого следующий milestone не начинается. -9. **Отчёт.** `.swarm/final-report.md`. +1. **Понять** outcome, аудиторию, quality bar и ограничения. +2. **Проверить** окружение: toolchain, команды, сервисы, display, credentials. +3. **Спланировать только необходимое.** Для swarm — brief, disjoint scopes, contracts, acceptance. + Для direct/paired — без `.swarm`-бюрократии. +4. **Сделать** напрямую, с критиком или батчами воркеров. +5. **Проверить** технически, затем независимым критиком; для product — performance/UX/baseline. +6. **Принять и интегрировать.** Лид владеет shared-файлами и финальной проверкой. ## Артефакты -Всё лежит в `.swarm/` в корне рабочей папки: бриф, профиль домена, план, карточки задач, отчёты, ревью, baseline, финальный отчёт. `.swarm/` можно удалить после задачи или добавить в `.gitignore`; структура подходит и для кода, и для research/контента/данных/дизайна. +Для swarm **опционально**: `.swarm/brief.md`, `tasks/`, `reports/`, `reviews/`. Они нужны для +координации и восстановления, а не для отчётности ради отчётности. Для direct/paired их можно не +создавать вообще. ## Ограничения -- Это инструкция, а не плагин: запуск идёт через инструмент `workflow`. -- Одновременно выполняется столько агентов, сколько разрешает runtime (часто 8, не 14). Лимит проверяется preflight-ом; `wave.workflow.js` запускает задачи батчами, а не одним большим fan-out. -- Дети не общаются между собой и не делегируют дальше; координация — через файлы и лида. -- Верификаторы работают на замороженном дереве; полный разбор пишется в `.swarm/reviews/Vxx.md`, а workflow возвращает только `{task_id, verdict, summary, review_file}`. -- `blocked`-воркер не заменяется: лид правит контракт/shared-файл и продолжает того же агента через `send_message`. -- Evidence зависит от домена: команда и тест для кода, источник и дата для research, отрендеренный артефакт для дизайна, reproducible metric для данных. -- Токены — потолок, а не цель: 20–30% резервируется на verify/integration и polish, после фиксов делается delta-review, а не полная перепроверка. Детали — в `references/token-economy.md`. -- Визуальные агенты работают в отдельных Xvfb/display-окружениях, со своим HOME/config/копией проекта; общий рабочий стол для скриншотов не используется. Детали — в `references/visual-testing.md`. -- Computer-use — крайний tier: один агент, одноразовый desktop/VM, только после детерминированных проверок и никогда как единственное evidence. -- Стоимость: 30–40 агентов ≈ 0.5–1.5M токенов. +- `workflow` не обязателен: есть `subagent` — используем его, нет — работаем последовательно. +- Число одновременных агентов ограничено runtime; не запускать oversubscribed fan-out. +- Воркеры не устанавливают зависимости и не трогают live-системы. +- Визуальные проверки — в изолированных display; общий рабочий стол не evidence. +- Computer-use — крайний tier: один агент, одноразовый desktop/VM, не единственное evidence. +- Токены — потолок, а не цель: резерв на verify/integration/polish; после фиксов — delta-review. - Для задачи, которую один агент делает за 15 минут, swarm не нужен. +## Опциональные модули + +Читать только по необходимости: + +- `references/playbook.md` — подробный runbook для больших swarm. +- `references/packets.md` — шаблоны brief/task/verifier. +- `references/orchestration.md` — `workflow`, батчи, ретраи. +- `references/domains.md` — адаптеры для кода/research/контента/данных/дизайна/ops. +- `references/prerequisites.md` — preflight и политика установки. +- `references/token-economy.md` — бюджеты, контекст, delta-review. +- `references/visual-testing.md` — изолированные display, скриншоты, computer-use. +- `references/product-quality.md` — performance/UX/baseline/human acceptance. +- `evals/` — как проверять сам скилл (baseline vs with-skill). + ## Настройка -- Шаблоны: `assets/`, включая `prerequisites.md.tmpl`. -- Prerequisites: `references/prerequisites.md`. -- Product quality: `references/product-quality.md`. -- Подробный runbook: `references/playbook.md`. -- Оркестрация: `references/orchestration.md`. +- Шаблоны: `assets/`. - Проверка скоупа: `node scripts/audit-scope.mjs --init` перед стартом, затем `node scripts/audit-scope.mjs`. -- Для директорий в `plan.json` используйте glob `path/**`, особенно для ещё не созданных папок. +- Для директорий в `plan.json` используйте glob `path/**`, включая ещё не созданные папки. +- Перед изменением скилла прочитайте `evals/README.md`; не считайте, что новая версия лучше старой + без сравнения. diff --git a/agent-swarm/SKILL.md b/agent-swarm/SKILL.md index f7ed6a3..aa73c76 100644 --- a/agent-swarm/SKILL.md +++ b/agent-swarm/SKILL.md @@ -1,138 +1,103 @@ --- name: agent-swarm -description: Use for large tasks that split into 8+ independent work units across software, research, content, data, design or operations (codebase-wide changes, audits, migrations, multi-source research, multi-section documents, dataset work, visual redesigns) to run coordinated subagents in waves under one lead, with ownership, packed context, and independent verification. Not for single-file, sequential, or highly coupled work. -whenToUse: The task is broad, parallelizable, and has executable acceptance checks; one agent would take hours and separate agents can finish units without touching the same files. +description: Use when a large task splits into 8+ independent units, or when a risky deliverable needs an independent critic across software, research, content, data, design or operations. Not for small, sequential or tightly coupled work. +whenToUse: The task is broad or risky enough that parallel workers or a fresh verifier change the outcome; one agent would take hours and the units can be described without stepping on each other. --- # Agent Swarm -Run a large task as waves of coordinated subagents. The lead plans and integrates; workers execute one work packet each; verifiers check independently. A swarm is only as fast as its briefs are clear and its write scopes are disjoint. +A small orchestration core, not a mandatory methodology. Use the lightest mode that meets the +risk; the user keeps the stack, scope and pace. -The protocol is **domain-neutral**: software, research, content, data, design and operations all use the same roles and evidence rules. Pick an adapter from `references/domains.md`; do not force a software pipeline onto non-code work. Tools are optional too: if `workflow` is unavailable, use `subagent` or direct sequential work with the same packets. +## Choose the mode -The skill is a **capability, not a mandatory process**. Apply the lightest mode that meets the risk: - -| Mode | When | What it adds | +| Mode | Use when | What it adds | |---|---|---| -| Direct | one unit, fast feedback, low blast radius | none | -| Paired critic | medium task or risky design decision | independent review, same artifact | -| Swarm | 8+ independent units, breadth dominates | plan, ownership, parallel workers, verifier wave | +| **Direct** | one unit, fast feedback, low blast radius | nothing | +| **Paired critic** | medium task, design decision, risky change | one independent review | +| **Swarm** | 8+ independent units or breadth dominates | ownership, parallel workers, verifier | -Do not force task cards, waves or `.swarm/` artifacts onto a small request. Conversely, do not skip independent verification on a large or risky one. The user keeps the choice of stack, scope and pace; the skill only adds coordination when breadth or risk justifies it. +If the units share one screen, one design language or one mutable artifact, do not swarm. Use one +implementer and one critic; parallel panel writers produce incoherent UI. -**Process is optional; quality gates are not.** For anything the user will actually use, read -`references/product-quality.md` and agree on a **prototype** or **product** profile. Product work -requires a release-build performance check, a real UX scenario, baseline comparison and an explicit -user verdict. A verifier's `pass` is technical evidence, not product acceptance. +## Invariants -## Fit check — before spawning anything +1. **Outcome before process.** Agree on what “done” means and how it will be checked. Do not create + `.swarm/` files, task cards or reports just to look organized. +2. **One owner per artifact.** In a swarm, every path/artifact belongs to one writer. Shared files + and contracts belong to the lead/integrator. +3. **Packets, not transcripts.** A worker gets a self-contained brief, its scope, the contract and + the acceptance checks — never the lead's full conversation. +4. **Independent verification.** The author does not verify their own work. A fresh context checks + the artifact, not the worker's summary. Verification is required for risky or product work; a + trivial direct change may only need a command and a read-back. +5. **Evidence before claims.** “Done” means a runnable artifact, a passing command, a citation, a + metric or a rendered result — not a confident sentence. +6. **A technical pass is not product acceptance.** For product/UI work, measure the release build, + run a real scenario, compare with the baseline and get the user's verdict. If performance/UX was + not measured, the product gate is `not_run`, not `pass`. +7. **Do not add features to hide bad quality.** If it is laggy, visually worse or harder to use than + the baseline, fix the slice before expanding scope. +8. **One milestone at a time.** For product work, do not start M1/M2 while M0 is still rough or + explicitly “not usable”. +9. **Safety.** Check prerequisites before work; ask before global installs, destructive actions, + deployments or irreversible changes. Workers never install or touch live systems. +10. **User overrides.** Direct user instructions beat this skill. If the user asks for speed, a + prototype or “just do it”, drop the ceremony — keep only the safety and evidence that still + matter. -Swarm only when all are true: -- The task splits into **8+ independent units** with disjoint write scopes. -- Each unit has a **re-runnable acceptance check**: a command, a named artifact, a source/citation check, a metric, or a rendered result. There is no universal evidence type; match it to the domain adapter. -- Wall-clock is dominated by **breadth**, not by one deep sequential chain. -- You can state the interfaces between units **before** dispatch. -- The units do **not** share one visual language or one screen. UI coherence is usually a single-owner - job; parallel panel writers produce a Frankenstein interface. +## Loop -Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived roles you need to steer. Never swarm one bug, one design decision, or work where every unit edits the same files. +1. **Understand** the outcome, audience, quality bar and constraints. Ask only when the answer + changes the result. +2. **Check** the environment: toolchain, commands, services, display, credentials. If a required + item is missing and cannot be resolved safely, ask before starting. +3. **Plan only as much as the risk requires.** For a swarm, write a one-page brief, disjoint scopes, + frozen contracts and acceptance checks. For direct/paired work, skip the ceremony. +4. **Work.** Direct, paired or batched workers. Keep files small enough that each worker reads only + what it needs; use available concurrency, not a fixed wave number. +5. **Verify.** Technical checks first, then a fresh critic. For product work, add release performance, + UX scenario, baseline comparison and the user's explicit verdict. +6. **Accept and integrate.** The lead owns shared files, final checks and the report. No next + milestone without acceptance (or an explicit waiver). -## Roles +## If you swarm -- **Lead** — you. Owns the plan, packets, contracts, gates, integration, and the final report. Does not implement during waves. -- **Recon** (2–4) — maps files, commands, conventions, and hazards. Does not design the solution. -- **Worker** — one task card, one write scope, one structured report. -- **Verifier** — fresh context; checks the artifact against acceptance criteria, never the worker's summary. -- **Integrator** (1–2) — owns shared files and cross-cutting fixes after the parallel waves. During a wave, the lead/integrator may make a small contract/shared-file change to unblock a task, then resume that same worker with `send_message`; do not spawn a replacement. +Optional, for resumability and coordination — not a requirement for every task: -## Protocol +- `.swarm/brief.md` — goal, quality bar, commands, conventions, hazards; +- `.swarm/contracts.md` — only interfaces that cross a task boundary; +- `.swarm/tasks/Txx.md` — one page per unit; +- `.swarm/reports/Txx.json` — structured evidence, if the lead needs to triage many workers; +- `.swarm/reviews/Vxx.md` — independent verdict and findings. -0. **Fit check & prerequisite gate** — go/no-go, budget (`max_agents`, `max_waves`, **tokens**), and **discover the available tools and real active-child cap** before sizing a wave. Check every required toolchain/runtime/service/display/credential with a real command; record the result in `.swarm/prerequisites.md` and `state.json`. Prefer project-local pinned setup; **never install globally/system-wide or run `sudo`/`curl | sh` without explicit user approval**. If a required prerequisite is missing and cannot be resolved safely, ask the user with the exact item, command, scope and alternatives **before any worker starts**. Workers never install; they return `blocked`. -1. **Resolve prerequisites** — after approval/install, re-run the checks and the baseline suite. No build or verify wave starts while a required item is `needs_user` or `blocked`; optional items may be skipped only with a recorded reduced scope. See `references/prerequisites.md`. -2. **Recon** — 2–4 parallel agents return findings with paths and line numbers. Timebox them, merge into `.swarm/brief.md` and `.swarm/map.md`, then stop them; recon must not hold slots during the build wave. -3. **Plan gate** — write `.swarm/plan.json`, `plan.md`, `ownership.md`, `contracts.md`, and one `tasks/Txx.md` per unit. Every success criterion maps to a task and a check. No two tasks write the same path. Include a `LEAD`/`INTEGRATOR` task owning lead-owned shared files, screenshots, licenses and the final report. Freeze interfaces. Prefer vertical slices and files under ~800 lines; a task that requires reading a 3k-line file is too broad. Record the **quality profile**, performance budget, baseline scenarios and user-acceptance owner. **Do not dispatch until this gate passes.** -4. **Build wave** — one `workflow` call per build wave when the tool exists, using the batched runner in `assets/wave.workflow.js`; its `concurrency` must fit the available slots (see Sizing). If `workflow`/`subagent` is unavailable, the lead executes the same packets sequentially. Retry `null`/`failed` once; if the runtime reports an active-child limit, halve the next batch and dispatch the remaining tasks — never relaunch the same oversubscribed fan-out. -5. **Audit** — run `scripts/audit-scope.mjs` from the skill base directory; revert or reassign out-of-scope changes. Run the build/test gate. -6. **Verify wave** — **freeze source changes**, run automated gates first (compile/tests/artifact), then 4–6 fresh verifiers by risk cluster (or 1–2 for a small domain task). Use `assets/verdict.schema.json` and, when available, `assets/verify.workflow.js`; the review file on disk is the source of truth, not the workflow return. Verifiers never fix. Match evidence to the domain adapter: rerun commands, read rendered artifacts, sample citations, recompute metrics. For GUI products, use per-agent isolated displays, never a shared desktop (see `references/visual-testing.md`). If you fix findings, re-run only the affected **delta** and mark the old review stale. -7. **Product gate** — for product-profile work, measure the release build against the performance budget, run the real UX scenario, compare with the baseline, and get the user's explicit verdict. A technical verifier pass is not enough. If the product is slower or worse than the baseline, stop and fix the slice; do not start the next milestone. See `references/product-quality.md`. -8. **Integrate** — one integrator for shared files and cross-cutting fixes; re-run the full acceptance suite; then a final verification pass on the integrated build. -9. **Close** — freeze the build and run a final independent review against the original request (not the plan), then write `final-report.md`. +Use the batched runner in `assets/wave.workflow.js` when `workflow` exists; otherwise use `subagent` +or direct sequential work. Never relaunch an oversubscribed fan-out. Read `references/playbook.md` +for the detailed runbook, `references/packets.md` for cards and `references/orchestration.md` for +tool specifics — only when the task needs them. -## Hard rules +## Optional modules -1. **One writer per path.** A path appears in at most one task's `write` list. Shared files (`package.json`, lockfiles, barrels, schemas, migrations, CI) belong to the integrator only. -2. **Packet, not transcript.** Each worker reads `.swarm/brief.md`, its `tasks/Txx.md`, `.swarm/contracts.md`, and the domain adapter in `.swarm/profile.md` (if any). Never fork the lead's conversation for dozens of workers. -3. **Structured reports.** Workers return `{task_id, status, summary, files_changed, evidence, acceptance, issues, assumptions}` and write the same object to `.swarm/reports/Txx.json`. Verifiers write the full review to `.swarm/reviews/Vxx.md` and return the minimal `{task_id, verdict, summary, review_file}`. Prose is not a report. -4. **No self-verification.** The worker's own checks are a smoke test; a different agent verifies the artifact. -5. **No global side effects.** Workers do not run installs, repo-wide format/lint/codegen, migrations, dev servers, deployments, external publishes, or any git write command without explicit scope. They do not edit other tasks' files/reports or touch live systems. The lead owns irreversible operations and only after the final review. -6. **Respect the runtime slot cap.** Active children include recon, verifiers, integrators and any other subagents. A spawn-limit error is a scheduling bug, not a worker failure: shrink the batch, keep completed results, and dispatch the remainder. -7. **Freeze during verification.** Once the verify wave starts, do not edit source/tests/docs except to fix a verified finding. After a fix, re-run the affected verifier; an old review is stale. -8. **Stop rules.** Two waves with no net progress → stop and re-plan. Verification failure rate > 30% → fix briefs/contracts, not agent count. Budget exhausted → report partial and list unverified areas. -9. **No shared visual desktop.** Visual agents get their own display/config/project copy; see `references/visual-testing.md`. A screenshot from a shared desktop is not evidence. Computer-use is a last-resort, single-agent tier on a disposable desktop/VM; it is never the sole evidence and never runs in parallel with other visual agents. -10. **Prerequisites before dispatch.** A required tool/runtime/service/credential is checked, installed with explicit approval if needed, and re-checked before workers start. Ask the user before work if it cannot be resolved; never let a worker install or silently skip a required check. -11. **Token budget is a ceiling, not a target.** Record spend per phase, keep a 20–30% reserve for verification/integration/**polish**, and do not stop a product task at the soft cap if the quality gate is unmet; stop at the hard cap or when the gate is met. Cut redundant context, agents, full-file reads and full re-reviews — never cut tests, independent review or user-driven fixes. See `references/token-economy.md`. -12. **A technical pass is not product acceptance.** Product-profile work needs measured performance, a real UX scenario, baseline comparison and an explicit user verdict; otherwise the product gate is `not_run`. See `references/product-quality.md`. -13. **One milestone at a time.** Do not start M1/M2 while M0 is still laggy, visually rough or explicitly “not usable”. Scope adds must wait for acceptance. +Read only what applies: -## Sizing +- `references/playbook.md` — detailed swarm runbook, gates, recovery. +- `references/packets.md` — brief/task/verifier templates and examples. +- `references/orchestration.md` — `workflow` patterns, retries, batching. +- `references/domains.md` — software, research, content, data, design, operations. +- `references/prerequisites.md` — preflight, install/approval policy. +- `references/token-economy.md` — budgets, context discipline, delta review. +- `references/visual-testing.md` — isolated displays, screenshots, computer-use limits. +- `references/product-quality.md` — performance/UX/baseline/human acceptance. +- `evals/` — how to test this skill itself (baseline vs with-skill, trigger checks). -| Independent units | Batch size | Batches | Agents | Notes | -|---|---|---|---|---| -| < 8 | — | — | — | Do not swarm | -| 8–11 | 6–8 | 1 + verify | ~12–14 | Small swarm; one batch | -| 12–24 | 6–8 | 2 | ~20 | Default; 2 batches per build wave | -| 25–48 | 6–8 | 3–4 | ~40 | Large; watch the lead's context | -| 49+ | 6–8 | 5+ | 60+ | Only if truly independent; the lead becomes the bottleneck | - -**The wave size is not fixed by this table.** The runtime active-child cap is host/policy-specific and may be as low as 8; do not assume 14 or any default. Discover it before dispatch (runtime diagnostics or a tiny trial), subtract active recon/other children, and leave at least one slot free for retries. If the cap is unknown, start with 6 workers. The batched runner in `assets/wave.workflow.js` starts tasks in chunks, so a failure to spawn one chunk does not discard completed work. - -## Workspace - -``` -.swarm/ - brief.md shared context: goal, domain, commands, conventions, hazards - prerequisites.md tool/runtime/display/credential gate + approvals - map.md repository/source map from recon: paths, entry points, hazards - contracts.md frozen interfaces between tasks - profile.md domain adapter: artifacts, commands, evidence types (optional) - plan.json tasks, deps, write scopes, waves, status - plan.md human-readable plan + coverage matrix - ownership.md path -> task map - tasks/T01.md one work packet per task - reports/T01.json worker report (worker writes) - reviews/V01.md verifier report (verifier writes) - baseline.json pre-swarm file manifest (`--init`) - report.schema.json report contract for workers - verdict.schema.json minimal contract for verifiers - state.json wave, token budget/spend, prerequisites, open issues - final-report.md - tmp/T01/ worker scratch, never shared -``` - -## Coordination loop - -Between waves the lead does only four things: **continue**, **answer blocked tasks and retry**, **re-scope**, or **stop**. Read structured reports and review files, not raw logs. Keep `state.json` current, including the real slot cap, retries and re-verifications. The lead is the only agent that talks to the user. - -For a `blocked` task: apply the smallest contract/shared-file change, update `contracts.md`, record it in `state.json`, then `send_message` to the **same** worker with the answer and let it continue. Do not spawn a replacement for a worker that only needed a decision. - -## References - -- `references/playbook.md` — phase-by-phase runbook, gates, checklists, failure recovery. -- `references/packets.md` — how to write briefs, task cards, and verifier cards; examples and common mistakes. -- `references/orchestration.md` — `workflow` cookbook: script patterns, schema subset, retries, background runs, archetypes (code / research / content). -- `references/domains.md` — adapters for software, research, content, data, design and operations. -- `references/token-economy.md` — measurement, budgets, context discipline, delta verification, vertical slices. -- `references/prerequisites.md` — preflight checks, install/approval policy, degraded-scope record. -- `references/visual-testing.md` — isolated Xvfb/container visual tests, deterministic screenshots, verifier procedure (GUI adapter). -- `references/product-quality.md` — prototype/product profiles, performance and UX gates, baseline comparison, human acceptance. -- `assets/` — copyable brief, task, and verifier templates; `report.schema.json`, `verdict.schema.json`; `wave.workflow.js` (batched workers), `verify.workflow.js` (frozen verification). -- `scripts/audit-scope.mjs` — compares changed files to the ownership map. - -## Signals that the swarm is failing +## Signals - Workers ask the same question or edit the same file → the plan or contracts are underspecified. -- Reports are long on prose and short on evidence → the task card lacks executable checks. -- The lead spends more tokens reading reports than the workers spend working → waves are too large or reports too verbose. -- Verification finds the same class of defect repeatedly → fix the brief, not the worker. -- A workflow ends with `no usable verdict` although reviews exist on disk → the verifier schema is too strict; read the review files and switch to `assets/verdict.schema.json`. -- A wave dies with `active child limit reached` → the batch was sized from the wrong cap; halve it, subtract recon, and dispatch only unstarted tasks. +- Reports are long on prose and short on evidence → the task card lacks a check. +- The lead reads more tokens than the workers produce → waves or reports are too large. +- The same defect repeats → fix the brief, not the worker count. +- The product compiles and passes tests but feels worse than the previous version → stop adding + features and run the product gate. +- You are creating process artifacts that nobody will read → delete them and do the work. +- You are changing this skill without evidence → run `evals/README.md` against a no-skill baseline; + keep the change only if it improves quality/time without over-orchestration. diff --git a/agent-swarm/evals/README.md b/agent-swarm/evals/README.md new file mode 100644 index 0000000..afafd8c --- /dev/null +++ b/agent-swarm/evals/README.md @@ -0,0 +1,47 @@ +# Testing the skill itself + +Do not assume the skill improves outcomes. Measure it against a no-skill baseline and against the +previous version. This is the same idea as Anthropic's `skill-creator`: run baseline and with-skill +runs on realistic prompts, compare quality and cost, then revise. + +## What to measure + +| Metric | How | +|---|---| +| Acceptance | Did the artifact pass the task's own checks? | +| Human quality (1–5) | Blind review by someone who did not run the task | +| Time | Wall-clock from prompt to accepted artifact | +| Tokens / requests | Harness usage; record input/output/cache if available | +| Agents spawned | Count; detect over-orchestration | +| Process artifacts | Files under `.swarm/`; are they read or just produced? | +| Regression | Tests, UX, performance, baseline comparison | +| Performance | For UI: release p95 frame time, input latency, startup | +| Trigger | Did the skill activate when appropriate, and stay out of small work? | + +## Runs + +For each task, run at least: + +1. **baseline** — no skill; +2. **candidate** — current `agent-swarm`; +3. optionally, the previous version. + +Use fresh contexts, the same model/settings, the same repository commit and the same time limit. +Run twice if variance is high. Do not let the baseline see the skill's artifacts. + +## Tasks + +- [`tasks/small-fix.md`](tasks/small-fix.md) — should **not** trigger a swarm. +- [`tasks/product-ui.md`](tasks/product-ui.md) — should trigger a small team and product gate. +- [`tasks/broad-migration.md`](tasks/broad-migration.md) — should trigger a real swarm. + +## Pass condition + +The skill is useful only if it improves at least one of: + +- acceptance/quality without a large time or token penalty; +- time/tokens without reducing quality; +- product quality (performance/UX/baseline) on the product task. + +It fails if it adds agents/artifacts without improving outcomes, or if it triggers on the small fix. +Record results in a copy of [`scorecard.md`](scorecard.md). diff --git a/agent-swarm/evals/scorecard.md b/agent-swarm/evals/scorecard.md new file mode 100644 index 0000000..d586c19 --- /dev/null +++ b/agent-swarm/evals/scorecard.md @@ -0,0 +1,36 @@ +# Skill eval scorecard + +Date: +Model/settings: +Skill version/commit: +Evaluator: + +## small-fix + +| Run | Accepted | Quality 1–5 | Time | Tokens | Agents | `.swarm` artifacts | Notes | +|---|---|---:|---:|---:|---:|---|---| +| baseline | | | | | | | | +| candidate | | | | | | | | + +Trigger verdict: `correct / over-triggered / under-triggered` + +## product-ui + +| Run | Accepted | Quality 1–5 | Time | Tokens | p95 frame | Input latency | Baseline comparison | User verdict | +|---|---|---:|---:|---:|---:|---:|---|---| +| baseline | | | | | | | | | +| candidate | | | | | | | | | + +## broad-migration + +| Run | Accepted | Quality 1–5 | Time | Tokens | Agents | Conflicts | Rework | Notes | +|---|---|---:|---:|---:|---:|---:|---|---| +| baseline | | | | | | | | | +| candidate | | | | | | | | | + +## Decision + +- Keep / revise / revert: +- What improved: +- What regressed: +- Next change: diff --git a/agent-swarm/evals/tasks/broad-migration.md b/agent-swarm/evals/tasks/broad-migration.md new file mode 100644 index 0000000..200d044 --- /dev/null +++ b/agent-swarm/evals/tasks/broad-migration.md @@ -0,0 +1,26 @@ +# Broad migration — swarm test + +Use a repository with 8+ independent call sites or modules that need the same mechanical change +(e.g. adding pagination to independent endpoints, or migrating independent modules to a new API). + +Prompt: + +```text +Apply the migration across the affected modules. Each unit must have its own scope and tests. Do not +touch unrelated code. Verify the integrated result and report conflicts or skipped units. +``` + +Expected with a good skill: + +- a short plan with disjoint write scopes and frozen interfaces; +- parallel or batched workers, or a deliberate sequential choice with a reason; +- independent verification of the integrated result; +- no overlapping edits, no silent skipped units; +- time/tokens improve over baseline, or quality improves enough to justify the cost. + +Failure signals: + +- 20 agents for 8 small edits; +- workers editing the same file; +- no integration check; +- reports without evidence. diff --git a/agent-swarm/evals/tasks/product-ui.md b/agent-swarm/evals/tasks/product-ui.md new file mode 100644 index 0000000..f204f3a --- /dev/null +++ b/agent-swarm/evals/tasks/product-ui.md @@ -0,0 +1,28 @@ +# Product UI — quality-gate test + +Use a small app with an existing UI (or a fixture). Ask for one vertical UI slice: for example, a +scene hierarchy panel with selection and an inspector for a transform. The artifact must be runnable. + +Prompt: + +```text +Add the requested UI slice to this app. It is a product task: the result must be usable, not just +compile. Use the existing stack. Run the app, capture the default and minimum window sizes, measure +release-build frame time and input latency, compare with the current version, and report the exact +commands. Do not expand into unrelated features. +``` + +Expected with a good skill: + +- one implementer and one independent critic, not parallel panel writers; +- runnable artifact and screenshots; +- release performance measured; baseline compared; +- the agent stops if the result is slower or visually worse than the existing version; +- the user gets a 3–5 step scenario and an explicit verdict request. + +Failure signals: + +- screenshot-only acceptance, debug-build performance claims; +- “tests pass” treated as product acceptance; +- new features added while the slice is laggy; +- no baseline comparison. diff --git a/agent-swarm/evals/tasks/small-fix.md b/agent-swarm/evals/tasks/small-fix.md new file mode 100644 index 0000000..a218571 --- /dev/null +++ b/agent-swarm/evals/tasks/small-fix.md @@ -0,0 +1,24 @@ +# Small fix — trigger test + +Use a small repository with one failing test (or one clearly scoped bug). Provide the repo path, +the failing command and the expected behavior. + +Prompt: + +```text +Fix the failing test in this repository. Make the smallest correct change, run the failing test and +the relevant suite, and report the exact commands and results. Do not refactor unrelated code. +``` + +Expected with a good skill: + +- no swarm, no `.swarm/` tree, one worker or direct work; +- the failing test now passes; +- no unrelated changes; +- time and tokens close to baseline. + +Failure signals: + +- the skill spawns multiple agents or writes a full plan for a one-line fix; +- process artifacts exceed the code change; +- the agent claims success without running the test.