Add token budgets, telemetry and delta verification
This commit is contained in:
@@ -14,6 +14,7 @@
|
||||
- [Runbook](agent-swarm/references/playbook.md)
|
||||
- [Cookbook по `workflow`](agent-swarm/references/orchestration.md)
|
||||
- [Domain-адаптеры](agent-swarm/references/domains.md)
|
||||
- [Token economy](agent-swarm/references/token-economy.md)
|
||||
- [Изолированное визуальное тестирование](agent-swarm/references/visual-testing.md)
|
||||
|
||||
### Установка
|
||||
@@ -32,7 +33,7 @@ cp -r skills/agent-swarm ~/.dsh/skills/
|
||||
agent-swarm/
|
||||
├── SKILL.md # основной протокол
|
||||
├── README.ru.md # краткое описание
|
||||
├── references/ # playbook, packets, orchestration, domains, visual testing
|
||||
├── references/ # playbook, packets, orchestration, domains, visual testing, token economy
|
||||
├── assets/ # шаблоны, схемы, workflow-скрипты
|
||||
└── scripts/audit-scope.mjs # проверка ownership
|
||||
```
|
||||
|
||||
@@ -33,6 +33,7 @@
|
||||
- Верификаторы работают на замороженном дереве; полный разбор пишется в `.swarm/reviews/Vxx.md`, а workflow возвращает только `{task_id, verdict, summary, review_file}`.
|
||||
- `blocked`-воркер не заменяется: лид правит контракт/shared-файл и продолжает того же агента через `send_message`.
|
||||
- Evidence зависит от домена: команда и тест для кода, источник и дата для research, отрендеренный артефакт для дизайна, reproducible metric для данных.
|
||||
- Токены — такой же бюджет, как время: 20–30% резервируется на verify/integration, после фиксов делается delta-review, а не полная перепроверка. Детали — в `references/token-economy.md`.
|
||||
- Визуальные агенты работают в отдельных Xvfb/display-окружениях, со своим HOME/config/копией проекта; общий рабочий стол для скриншотов не используется. Детали — в `references/visual-testing.md`.
|
||||
- Computer-use — крайний tier: один агент, одноразовый desktop/VM, только после детерминированных проверок и никогда как единственное evidence.
|
||||
- Стоимость: 30–40 агентов ≈ 0.5–1.5M токенов.
|
||||
|
||||
@@ -40,12 +40,12 @@ Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived
|
||||
|
||||
## Protocol
|
||||
|
||||
0. **Fit check & preflight** — go/no-go, budget (`max_agents`, `max_waves`), and **discover the available tools and real active-child cap** before sizing a wave. Record in `state.json`: tool availability (`workflow`, `subagent`, display/artifact command, build commands), slot cap, active children, planned batch size, and the chosen domain adapter.
|
||||
0. **Fit check & preflight** — go/no-go, budget (`max_agents`, `max_waves`, **tokens**), and **discover the available tools and real active-child cap** before sizing a wave. Record in `state.json`: tool availability (`workflow`, `subagent`, display/artifact command, build commands), slot cap, active children, planned batch size, chosen domain adapter, token budget/soft cap/reserve, and whether usage is measured or estimated (see `references/token-economy.md`).
|
||||
1. **Recon** — 2–4 parallel agents return findings with paths and line numbers. Timebox them, merge into `.swarm/brief.md` and `.swarm/map.md`, then stop them; recon must not hold slots during the build wave.
|
||||
2. **Plan gate** — write `.swarm/plan.json`, `plan.md`, `ownership.md`, `contracts.md`, and one `tasks/Txx.md` per unit. Every success criterion maps to a task and a check. No two tasks write the same path. Include a `LEAD`/`INTEGRATOR` task owning lead-owned shared files, screenshots, licenses and the final report. Freeze interfaces. **Do not dispatch until this gate passes.**
|
||||
2. **Plan gate** — write `.swarm/plan.json`, `plan.md`, `ownership.md`, `contracts.md`, and one `tasks/Txx.md` per unit. Every success criterion maps to a task and a check. No two tasks write the same path. Include a `LEAD`/`INTEGRATOR` task owning lead-owned shared files, screenshots, licenses and the final report. Freeze interfaces. Prefer vertical slices and files under ~800 lines; a task that requires reading a 3k-line file is too broad. **Do not dispatch until this gate passes.**
|
||||
3. **Build wave** — one `workflow` call per build wave when the tool exists, using the batched runner in `assets/wave.workflow.js`; its `concurrency` must fit the available slots (see Sizing). If `workflow`/`subagent` is unavailable, the lead executes the same packets sequentially. Retry `null`/`failed` once; if the runtime reports an active-child limit, halve the next batch and dispatch the remaining tasks — never relaunch the same oversubscribed fan-out.
|
||||
4. **Audit** — run `scripts/audit-scope.mjs` from the skill base directory; revert or reassign out-of-scope changes. Run the build/test gate.
|
||||
5. **Verify wave** — **freeze source changes**, then run 4–6 fresh verifiers by risk cluster (or 1–2 for a small domain task). Use `assets/verdict.schema.json` and, when available, `assets/verify.workflow.js`; the review file on disk is the source of truth, not the workflow return. Verifiers never fix. Match evidence to the domain adapter: rerun commands, read rendered artifacts, sample citations, recompute metrics. For GUI products, use per-agent isolated displays, never a shared desktop (see `references/visual-testing.md`). If you fix findings, re-run the affected verifier and mark the old review stale.
|
||||
5. **Verify wave** — **freeze source changes**, run automated gates first (compile/tests/artifact), then 4–6 fresh verifiers by risk cluster (or 1–2 for a small domain task). Use `assets/verdict.schema.json` and, when available, `assets/verify.workflow.js`; the review file on disk is the source of truth, not the workflow return. Verifiers never fix. Match evidence to the domain adapter: rerun commands, read rendered artifacts, sample citations, recompute metrics. For GUI products, use per-agent isolated displays, never a shared desktop (see `references/visual-testing.md`). If you fix findings, re-run only the affected **delta** and mark the old review stale.
|
||||
6. **Integrate** — one integrator for shared files and cross-cutting fixes; re-run the full acceptance suite; then a final verification pass on the integrated build.
|
||||
7. **Close** — freeze the build and run a final independent review against the original request (not the plan), then write `final-report.md`.
|
||||
|
||||
@@ -60,6 +60,7 @@ Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived
|
||||
7. **Freeze during verification.** Once the verify wave starts, do not edit source/tests/docs except to fix a verified finding. After a fix, re-run the affected verifier; an old review is stale.
|
||||
8. **Stop rules.** Two waves with no net progress → stop and re-plan. Verification failure rate > 30% → fix briefs/contracts, not agent count. Budget exhausted → report partial and list unverified areas.
|
||||
9. **No shared visual desktop.** Visual agents get their own display/config/project copy; see `references/visual-testing.md`. A screenshot from a shared desktop is not evidence. Computer-use is a last-resort, single-agent tier on a disposable desktop/VM; it is never the sole evidence and never runs in parallel with other visual agents.
|
||||
10. **Token budget is a quality constraint.** Record spend per phase, keep a 20–30% reserve for verification/integration, and stop at the hard cap with a partial report. Cut redundant context, agents, full-file reads and full re-reviews — never cut tests or independent review. See `references/token-economy.md`.
|
||||
|
||||
## Sizing
|
||||
|
||||
@@ -90,7 +91,7 @@ Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived
|
||||
baseline.json pre-swarm file manifest (`--init`)
|
||||
report.schema.json report contract for workers
|
||||
verdict.schema.json minimal contract for verifiers
|
||||
state.json wave, budget spent, open issues
|
||||
state.json wave, token budget/spend per phase, open issues
|
||||
final-report.md
|
||||
tmp/T01/ worker scratch, never shared
|
||||
```
|
||||
@@ -107,6 +108,7 @@ For a `blocked` task: apply the smallest contract/shared-file change, update `co
|
||||
- `references/packets.md` — how to write briefs, task cards, and verifier cards; examples and common mistakes.
|
||||
- `references/orchestration.md` — `workflow` cookbook: script patterns, schema subset, retries, background runs, archetypes (code / research / content).
|
||||
- `references/domains.md` — adapters for software, research, content, data, design and operations.
|
||||
- `references/token-economy.md` — measurement, budgets, context discipline, delta verification, vertical slices.
|
||||
- `references/visual-testing.md` — isolated Xvfb/container visual tests, deterministic screenshots, verifier procedure (GUI adapter).
|
||||
- `assets/` — copyable brief, task, and verifier templates; `report.schema.json`, `verdict.schema.json`; `wave.workflow.js` (batched workers), `verify.workflow.js` (frozen verification).
|
||||
- `scripts/audit-scope.mjs` — compares changed files to the ownership map.
|
||||
|
||||
@@ -41,6 +41,11 @@ Scope is checked by the lead with the skill's `scripts/audit-scope.mjs`; workers
|
||||
- {{Generated files, migrations, flaky tests, shared ports, secrets, caches.}}
|
||||
- {{Generated or vendored paths that must not be edited.}}
|
||||
|
||||
## Token / effort budget
|
||||
- Hard budget: {{tokens or n/a}}; reserve {{20–30%}} for verification and integration.
|
||||
- Prefer vertical slices and files under ~800 lines.
|
||||
- Workers read the listed inputs/ranges, not the whole repository.
|
||||
|
||||
## Frozen contracts
|
||||
See `.swarm/contracts.md`. Do not change an interface without returning `blocked`.
|
||||
|
||||
|
||||
@@ -28,6 +28,8 @@ Role: worker | Wave: {{n}} | Depends on: {{ids or —}} | Effort: {{S/M/L}}
|
||||
|
||||
## Constraints
|
||||
- {{No new dependencies, style, performance, compatibility.}}
|
||||
- Context budget: read only the inputs above; if you need more than {{N}} additional files, return `blocked` instead of exploring the whole repo.
|
||||
- Do not run the full test suite unless the card explicitly asks; run targeted checks.
|
||||
|
||||
## Acceptance (re-runnable)
|
||||
1. `{{command or artifact path}}` → {{expected result/state}}
|
||||
@@ -36,6 +38,7 @@ Role: worker | Wave: {{n}} | Depends on: {{ids or —}} | Effort: {{S/M/L}}
|
||||
|
||||
## Report
|
||||
`.swarm/reports/{{ID}}.json` per `.swarm/report.schema.json`; status `done|partial|blocked|failed`.
|
||||
Keep the report to summary + evidence lines; raw logs go to `.swarm/tmp/{{ID}}/`.
|
||||
|
||||
## Stop conditions
|
||||
- Time box: 25 minutes for S/M, 45–60 minutes for L, or 3 failed attempts at the same step.
|
||||
|
||||
@@ -34,6 +34,7 @@ Computer-use (mouse/keyboard) is an optional last-resort tier: one agent, dispos
|
||||
`pass` / `pass_with_risks` / `fail` — with the top risks and exact repro commands.
|
||||
|
||||
## Report
|
||||
Keep the review to findings + commands + evidence (target 5–8k tokens); raw logs stay in `.swarm/tmp/`. Review the diff and affected checks, not the whole repository.
|
||||
FIRST write the full review to `.swarm/reviews/{{ID}}.md`. Then return only the minimal structured verdict:
|
||||
`{task_id, verdict, summary, review_file}` per `.swarm/verdict.schema.json`.
|
||||
The review file is the source of truth; it must exist even if the structured reply is lost. Do not return a long findings array in the workflow schema.
|
||||
|
||||
@@ -208,4 +208,4 @@ If the workflow returns `null`/`no usable verdict`, check the review file before
|
||||
|
||||
## Cost
|
||||
|
||||
Each child carries the system prompt, tool schemas, packet, code reads, and its output: typically 10–40k tokens. A 12-agent wave is roughly 150–400k tokens, but a host capped at 8 cannot run 12 at once; batches reduce peak concurrency, not total cost. Budget before dispatch and count retries/re-verifications; stop at `max_agents`.
|
||||
Each child carries the system prompt, tool schemas, packet, code reads, and its output: typically 10–25k tokens of fixed overhead before any real work. A 12-agent wave can exceed 300k tokens, and a host capped at 8 cannot run 12 at once; batching reduces peak concurrency, not total cost. Budget before dispatch and count retries, delta re-checks and reviews; stop at the token cap. Eight substantial workers are often cheaper and better than twenty tiny ones. See `references/token-economy.md`.
|
||||
|
||||
@@ -14,9 +14,11 @@ Answer these before spawning:
|
||||
|
||||
Budget:
|
||||
- `agents ≈ units + ceil(units / 5) verifiers + 1–2 integrators + 2–4 recon`; add retries and re-verifications explicitly. Set `max_agents` only after checking available tools and slot cap.
|
||||
- Tokens ≈ `agents × 20k` (10–40k per agent depending on code reads).
|
||||
- Tokens ≈ `agents × fixed overhead (10–25k) + context reads + output + reviews`. Treat this as an estimate, then calibrate from measured usage. A 3k-line file read by several agents can cost more than the code they write.
|
||||
- Reserve **20–30%** for verification, integration and one fix round. Set a soft cap at ~70% and a hard cap; stop at the hard cap with a partial report.
|
||||
- Waves ≈ `ceil(units / batch_size)` + verify + integrate, where `batch_size` comes from the **real** active-child cap, not from a documented default.
|
||||
- Stop at `max_agents` or `max_waves`, whichever comes first. Record both, plus `slot_cap`, `active_at_dispatch`, `batch_size`, retries and re-verifications in `state.json`.
|
||||
- Stop at `max_agents`, `max_waves` or the token cap, whichever comes first. Record all caps, `slot_cap`, `active_at_dispatch`, `batch_size`, measured/estimated usage per phase, retries and re-verifications in `state.json`.
|
||||
- See `references/token-economy.md`; do not invent precise token numbers without measurement.
|
||||
|
||||
Go: a 40-file migration with per-module tests, a 30-source research report, a 20-section document with a style guide, a multi-dataset analysis, a multi-screen visual redesign, a multi-service runbook.
|
||||
No-go: one failing test, one architectural decision, a detail pass where every unit touches the same artifact.
|
||||
@@ -47,7 +49,11 @@ node <skill>/scripts/audit-scope.mjs --init
|
||||
|
||||
Write the plan before any worker starts. Keep the plan proportionate: a small domain task may need only a brief and a verifier, not a full `.swarm` tree.
|
||||
|
||||
**Decomposition.** One task = one deliverable, one writer, one acceptance check, at most one page. The deliverable may be a module, a research question, a document section, a dataset, a design artifact, or a runbook. Budget **5–25 minutes for S/M** and **40–60 minutes for L**; if a task is expected to produce a single source file over ~1,500 lines (or a very large document/dataset), split it or accept an integration-heavy task. If it needs two writers, split it; if it needs the whole repo, it is not a task.
|
||||
**Decomposition.** One task = one deliverable, one writer, one acceptance check, at most one page. The deliverable may be a module, a research question, a document section, a dataset, a design artifact, or a runbook. Budget **5–25 minutes for S/M** and **40–60 minutes for L**; if a task is expected to produce a single source file over ~800 lines (or a very large document/dataset), split it or accept an integration-heavy task. If it needs two writers, split it; if it needs the whole repo, it is not a task.
|
||||
|
||||
**Vertical slices.** Prefer one end-to-end capability at a time (selection → transform → undo → save) over a big-bang layer-by-layer build. A slice has its own checkpoint, tests and independent review; it limits late integration and repeated full reviews.
|
||||
|
||||
**Token shape.** Estimate the fixed cost per child, the context each task must read, and the review cost. Merge tiny tasks that pay fixed overhead for little work; split giant files that force repeated large reads. The goal is quality per token, not the smallest number of agents.
|
||||
|
||||
**Ownership.** Every artifact/path in exactly one `write` list: files, sections, datasets, source notes, design files. Add an explicit `LEAD` task whose list contains lead/integrator-owned shared artifacts, screenshots, licenses and the final report; without it, audit-scope reports the lead's own integration edits as unowned. Shared files are integrator-only: manifests, lockfiles, indexes/barrels, schemas, shared style/token files, migrations, i18n catalogs, snapshots, generated code. If two tasks must touch one artifact, serialize them with `depends_on` or merge them.
|
||||
|
||||
@@ -60,7 +66,10 @@ Write the plan before any worker starts. Keep the plan proportionate: a small do
|
||||
**Waves.** Topologically sort by `depends_on`, then balance effort across batches. Do **not** plan 8–12 workers from a documented default: discover the runtime active-child cap first. Compute `batch_size = clamp(slot_cap - active_children - 1, 2, 8)`; recon and any other subagents count as active children. If the cap is unknown, use 6. Put foundational or risky tasks in batch 1; verify after the last build batch.
|
||||
|
||||
**Gate checklist:**
|
||||
- [ ] 8+ tasks, each with objective, inputs, write scope, acceptance, stop condition.
|
||||
- [ ] 8+ tasks, each with objective, inputs, write scope, acceptance, stop condition, and a token/effort budget.
|
||||
- [ ] Files/modules over ~800 lines are split, or the task explicitly owns the integration cost.
|
||||
- [ ] Token budget, soft cap and verification reserve are recorded; units are merged/split to avoid fixed-overhead waste.
|
||||
- [ ] The plan uses vertical slices/checkpoints for a multi-milestone product.
|
||||
- [ ] No path in two write scopes.
|
||||
- [ ] Contracts frozen for every cross-task interface.
|
||||
- [ ] Coverage matrix complete.
|
||||
@@ -78,6 +87,7 @@ Minimal `plan.json` shape:
|
||||
```json
|
||||
{
|
||||
"goal": "one line",
|
||||
"budget": { "tokens": 600000, "soft_cap": 0.7, "reserved_for_verify": 0.25 },
|
||||
"tasks": [
|
||||
{
|
||||
"id": "LEAD",
|
||||
@@ -109,13 +119,15 @@ Minimal `plan.json` shape:
|
||||
Dispatch one `workflow` call per build wave (see `references/orchestration.md`). Use `assets/wave.workflow.js`, which splits tasks into batches, catches spawn errors instead of discarding the whole wave, and carries `args.concurrency`.
|
||||
|
||||
Before the call:
|
||||
- [ ] Task cards are on disk and reference real paths.
|
||||
- [ ] Task cards are on disk and reference real paths/ranges, not pasted context.
|
||||
- [ ] `args.tasks` carries `{id, write}` for each task.
|
||||
- [ ] The report schema is passed as `args.schema`, and `args.concurrency` is at most the free slot count.
|
||||
- [ ] Every task has a stop condition.
|
||||
- [ ] Baseline exists and `state.json` records the cap.
|
||||
- [ ] Every task has a stop condition and a size/effort budget.
|
||||
- [ ] Workers run targeted checks, not the full suite; the lead owns the full gate.
|
||||
- [ ] Baseline exists and `state.json` records the cap and token budget.
|
||||
|
||||
After the call:
|
||||
- [ ] Record measured/estimated token usage per phase if available.
|
||||
- [ ] Triage structured reports: done / partial / blocked / failed.
|
||||
- [ ] Treat `spawn/runtime error` and `active child limit` results as scheduling failures, not worker failures: keep completed results and dispatch only the unstarted remainder in a smaller batch.
|
||||
- [ ] Run `node <skill>/scripts/audit-scope.mjs` (the lead runs it; workers do not); revert or reassign out-of-scope changes.
|
||||
@@ -137,7 +149,9 @@ If the block reveals an underspecified interface shared by several tasks, update
|
||||
|
||||
## 4. Verify wave
|
||||
|
||||
Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
|
||||
Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks, fewer for a small domain task. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
|
||||
|
||||
**Cheap gates first.** Run compile/typecheck/targeted tests/artifact checks before spending verifier tokens. A verifier does not review a red build; it reviews a green diff and evidence. Keep reviews to findings + commands + evidence (target 5–8k tokens); raw logs stay in artifacts.
|
||||
|
||||
For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence. Computer-use is a final, single-agent exploratory tier on a disposable desktop/VM — never a parallel test mechanism and never the only evidence.
|
||||
|
||||
@@ -145,9 +159,9 @@ For a GUI product, give each visual verifier its own Xvfb/display and project co
|
||||
|
||||
Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed.
|
||||
|
||||
Evidence standard: for every criterion, a command and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
|
||||
Evidence standard: for every criterion, a command/artifact/citation/metric and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
|
||||
|
||||
On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run the affected verifier and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run a final integrated verification pass.
|
||||
On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run only the affected **delta** (diff + findings + acceptance commands), not the whole repository, and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run one final integrated verification pass on the frozen build.
|
||||
|
||||
## 5. Integrate
|
||||
|
||||
@@ -185,9 +199,14 @@ Final review is against the **original request**, not the plan: a fresh agent ch
|
||||
| Verifiers find the same defect | brief missing a convention | fix the brief, add a check, re-run affected tasks |
|
||||
| Lead context filling up | reading full reports | read summary + evidence; open only exceptions |
|
||||
| A worker hangs | no per-agent timeout exists | kill the job, split the task, add a tighter stop condition |
|
||||
| Token spend explodes | huge files, too many agents, full re-reads/reviews | split files, right-size batch, delta-review, record usage |
|
||||
| Verifier repeats a full review after a small fix | no delta procedure | review diff + findings + affected checks only |
|
||||
| Reports/reviews dominate input | oversized prose | cap to evidence lines; logs in artifacts |
|
||||
| Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots |
|
||||
| A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command |
|
||||
|
||||
## Time and cost
|
||||
|
||||
Parallel speedup is bounded by the sequential share and by the real slot cap, not by the number of tasks. Push sequential work down: recon once, contracts before dispatch, verification in parallel. A 36-unit task with 10-minute units and a 6-slot batch is ~6 batches × 10 min ≈ 60 min of agent work, plus plan, verify and integration; do not promise a 3× speedup when the host caps active children at 8 or fewer.
|
||||
Parallel speedup is bounded by the sequential share, the real slot cap and token cost, not by the number of tasks. Push sequential work down: recon once, contracts before dispatch, verification in parallel. A 36-unit task with 10-minute units and a 6-slot batch is ~6 batches × 10 min ≈ 60 min of agent work, plus plan, verify and integration; do not promise a 3× speedup when the host caps active children at 8 or fewer.
|
||||
|
||||
Token cost is dominated by fixed per-child overhead and repeated context reads. Eight substantial workers are often cheaper and better than twenty tiny ones; one delta re-check is much cheaper than a second full review. Track tokens per accepted task and per confirmed defect; optimize the ratio, not raw spend. See `references/token-economy.md`.
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# Token economy
|
||||
|
||||
Quality per token beats raw token spend. The expensive part of a swarm is usually **input context and
|
||||
repeated reading**, not the code a worker writes. Measure, budget, then cut redundancy — never cut
|
||||
the independent checks that catch real defects.
|
||||
|
||||
## Measure before optimizing
|
||||
|
||||
Record usage per phase and per agent in `.swarm/state.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"budget": { "tokens": 600000, "soft_cap": 0.7, "reserved_for_verify": 0.2 },
|
||||
"usage": [
|
||||
{ "phase": "recon", "agent": "R01", "tokens_in": 18000, "tokens_out": 4000 },
|
||||
{ "phase": "build", "agent": "T03", "tokens_in": 24000, "tokens_out": 9000 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
If the runtime does not expose usage, estimate from context size and output length and mark the value
|
||||
as `estimated`. The point is a feedback loop, not accounting precision.
|
||||
|
||||
Useful ratios:
|
||||
|
||||
- tokens per accepted task;
|
||||
- verifier tokens per confirmed defect;
|
||||
- rework tokens caused by a missed contract;
|
||||
- share of spend on recon / build / verify / integration.
|
||||
|
||||
## Budgets and reserves
|
||||
|
||||
- Set a **hard budget** for the milestone and a **soft cap** at ~70%. At the soft cap, stop expanding
|
||||
scope; finish the current slice and verify.
|
||||
- Reserve **20–30%** for verification, integration and one round of fixes. Do not spend the whole
|
||||
budget on generation.
|
||||
- Budget per phase, not only per wave: recon, build, verify, integrate.
|
||||
- Stop at the cap and report partial results plus unverified areas. Silent overspend is worse than a
|
||||
partial result.
|
||||
- Calibrate from the first measured milestone; do not copy invented numbers from a template.
|
||||
|
||||
## Context discipline
|
||||
|
||||
The biggest savings without quality loss:
|
||||
|
||||
1. **Small files.** Split modules above ~800 lines (UI panels, viewport, inspector). A worker should
|
||||
read 300–800 relevant lines, not 3,000. Smaller modules also compile and verify faster.
|
||||
2. **Read ranges, not repositories.** Use `path:line`, symbol search and grep first. Do not read a
|
||||
whole crate to change one function.
|
||||
3. **Point to artifacts, do not paste them.** Task cards reference `.swarm/brief.md`,
|
||||
`.swarm/contracts.md` and exact ranges; they do not duplicate them in the prompt.
|
||||
4. **Cap reports and reviews.** Worker report: summary, files/artifacts, evidence, issues,
|
||||
assumptions. Review: findings + commands + evidence; target 5–8k tokens, raw logs stay in files.
|
||||
5. **One page per task.** If a card needs more, the task is too broad.
|
||||
6. **No full-suite runs per worker.** Workers run targeted checks; the lead runs the full gate once
|
||||
per wave.
|
||||
7. **Fresh context only where independence matters.** The final verifier needs a clean read. A fix
|
||||
re-check can be a **delta review**: diff + findings + acceptance commands, not the whole repo.
|
||||
|
||||
## Verification ladders
|
||||
|
||||
Do not spend LLM review tokens on a red build:
|
||||
|
||||
1. compile / typecheck / lint;
|
||||
2. targeted tests;
|
||||
3. artifact/render/source check;
|
||||
4. independent verifier on the diff and evidence;
|
||||
5. final full review once, on a frozen integrated build.
|
||||
|
||||
Freeze before step 4. If code changes, re-check only the affected delta; mark the old review stale.
|
||||
This preserves quality while avoiding repeated full reviews.
|
||||
|
||||
## Orchestration shape
|
||||
|
||||
- **Fixed overhead dominates small tasks.** Each child carries system prompt, tools, brief and task
|
||||
context. 30 small agents can cost more than 8 substantial ones.
|
||||
- **Right-size:** 2–4 implementers + 1 verifier for a vertical slice; 6–10 agents for a broad
|
||||
milestone; 30 only for genuinely independent breadth.
|
||||
- **Model routing:** strong model for architecture, difficult fixes and final review; cheaper/faster
|
||||
model for tests, docs, screenshots and mechanical edits.
|
||||
- **Vertical slices:** ship one end-to-end feature at a time (selection → transform → undo → save),
|
||||
with a checkpoint and verification after each. Do not generate the whole product and verify at the
|
||||
end.
|
||||
- **Reuse:** do not regenerate a working core, format or codegen. Extend it.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
| Pattern | Cost | Fix |
|
||||
|---|---|---|
|
||||
| 3k-line file edited by several agents | repeated reads, merge conflicts | split before dispatch |
|
||||
| Full repo read for a one-function change | huge input | path:line + grep |
|
||||
| Full re-review after every fix | duplicate verifier spend | delta review |
|
||||
| Verifier runs on a failing build | wasted review | automated gate first |
|
||||
| 20 tiny tasks | fixed overhead per child | merge into coherent units |
|
||||
| Reports/reviews as essays | output + lead input | schema + evidence lines |
|
||||
| Whole product in one wave | late integration, huge rework | vertical slices + checkpoints |
|
||||
| Vague "max quality" with no budget | unbounded spend | hard/soft caps + reserve |
|
||||
|
||||
## Minimal operating rule
|
||||
|
||||
> Measure per phase, reserve for verification, keep files and packets small, verify the diff after
|
||||
> fixes, and spend the saved tokens on one strong independent review rather than many redundant
|
||||
> agents.
|
||||
Reference in New Issue
Block a user