Add token budgets, telemetry and delta verification

agent-swarm/references/playbook.md

@@ -14,9 +14,11 @@ Answer these before spawning:
 
 Budget:
 - `agents ≈ units + ceil(units / 5) verifiers + 1–2 integrators + 2–4 recon`; add retries and re-verifications explicitly. Set `max_agents` only after checking available tools and slot cap.
-- Tokens ≈ `agents × 20k` (10–40k per agent depending on code reads).
+- Tokens ≈ `agents × fixed overhead (10–25k) + context reads + output + reviews`. Treat this as an estimate, then calibrate from measured usage. A 3k-line file read by several agents can cost more than the code they write.
+- Reserve **20–30%** for verification, integration and one fix round. Set a soft cap at ~70% and a hard cap; stop at the hard cap with a partial report.
 - Waves ≈ `ceil(units / batch_size)` + verify + integrate, where `batch_size` comes from the **real** active-child cap, not from a documented default.
-- Stop at `max_agents` or `max_waves`, whichever comes first. Record both, plus `slot_cap`, `active_at_dispatch`, `batch_size`, retries and re-verifications in `state.json`.
+- Stop at `max_agents`, `max_waves` or the token cap, whichever comes first. Record all caps, `slot_cap`, `active_at_dispatch`, `batch_size`, measured/estimated usage per phase, retries and re-verifications in `state.json`.
+- See `references/token-economy.md`; do not invent precise token numbers without measurement.
 
 Go: a 40-file migration with per-module tests, a 30-source research report, a 20-section document with a style guide, a multi-dataset analysis, a multi-screen visual redesign, a multi-service runbook.
 No-go: one failing test, one architectural decision, a detail pass where every unit touches the same artifact.
@@ -47,7 +49,11 @@ node <skill>/scripts/audit-scope.mjs --init
 
 Write the plan before any worker starts. Keep the plan proportionate: a small domain task may need only a brief and a verifier, not a full `.swarm` tree.
 
-**Decomposition.** One task = one deliverable, one writer, one acceptance check, at most one page. The deliverable may be a module, a research question, a document section, a dataset, a design artifact, or a runbook. Budget **5–25 minutes for S/M** and **40–60 minutes for L**; if a task is expected to produce a single source file over ~1,500 lines (or a very large document/dataset), split it or accept an integration-heavy task. If it needs two writers, split it; if it needs the whole repo, it is not a task.
+**Decomposition.** One task = one deliverable, one writer, one acceptance check, at most one page. The deliverable may be a module, a research question, a document section, a dataset, a design artifact, or a runbook. Budget **5–25 minutes for S/M** and **40–60 minutes for L**; if a task is expected to produce a single source file over ~800 lines (or a very large document/dataset), split it or accept an integration-heavy task. If it needs two writers, split it; if it needs the whole repo, it is not a task.
+
+**Vertical slices.** Prefer one end-to-end capability at a time (selection → transform → undo → save) over a big-bang layer-by-layer build. A slice has its own checkpoint, tests and independent review; it limits late integration and repeated full reviews.
+
+**Token shape.** Estimate the fixed cost per child, the context each task must read, and the review cost. Merge tiny tasks that pay fixed overhead for little work; split giant files that force repeated large reads. The goal is quality per token, not the smallest number of agents.
 
 **Ownership.** Every artifact/path in exactly one `write` list: files, sections, datasets, source notes, design files. Add an explicit `LEAD` task whose list contains lead/integrator-owned shared artifacts, screenshots, licenses and the final report; without it, audit-scope reports the lead's own integration edits as unowned. Shared files are integrator-only: manifests, lockfiles, indexes/barrels, schemas, shared style/token files, migrations, i18n catalogs, snapshots, generated code. If two tasks must touch one artifact, serialize them with `depends_on` or merge them.
 
@@ -60,7 +66,10 @@ Write the plan before any worker starts. Keep the plan proportionate: a small do
 **Waves.** Topologically sort by `depends_on`, then balance effort across batches. Do **not** plan 8–12 workers from a documented default: discover the runtime active-child cap first. Compute `batch_size = clamp(slot_cap - active_children - 1, 2, 8)`; recon and any other subagents count as active children. If the cap is unknown, use 6. Put foundational or risky tasks in batch 1; verify after the last build batch.
 
 **Gate checklist:**
-- [ ] 8+ tasks, each with objective, inputs, write scope, acceptance, stop condition.
+- [ ] 8+ tasks, each with objective, inputs, write scope, acceptance, stop condition, and a token/effort budget.
+- [ ] Files/modules over ~800 lines are split, or the task explicitly owns the integration cost.
+- [ ] Token budget, soft cap and verification reserve are recorded; units are merged/split to avoid fixed-overhead waste.
+- [ ] The plan uses vertical slices/checkpoints for a multi-milestone product.
 - [ ] No path in two write scopes.
 - [ ] Contracts frozen for every cross-task interface.
 - [ ] Coverage matrix complete.
@@ -78,6 +87,7 @@ Minimal `plan.json` shape:
 ```json
 {
   "goal": "one line",
+  "budget": { "tokens": 600000, "soft_cap": 0.7, "reserved_for_verify": 0.25 },
   "tasks": [
     {
       "id": "LEAD",
@@ -109,13 +119,15 @@ Minimal `plan.json` shape:
 Dispatch one `workflow` call per build wave (see `references/orchestration.md`). Use `assets/wave.workflow.js`, which splits tasks into batches, catches spawn errors instead of discarding the whole wave, and carries `args.concurrency`.
 
 Before the call:
-- [ ] Task cards are on disk and reference real paths.
+- [ ] Task cards are on disk and reference real paths/ranges, not pasted context.
 - [ ] `args.tasks` carries `{id, write}` for each task.
 - [ ] The report schema is passed as `args.schema`, and `args.concurrency` is at most the free slot count.
-- [ ] Every task has a stop condition.
-- [ ] Baseline exists and `state.json` records the cap.
+- [ ] Every task has a stop condition and a size/effort budget.
+- [ ] Workers run targeted checks, not the full suite; the lead owns the full gate.
+- [ ] Baseline exists and `state.json` records the cap and token budget.
 
 After the call:
+- [ ] Record measured/estimated token usage per phase if available.
 - [ ] Triage structured reports: done / partial / blocked / failed.
 - [ ] Treat `spawn/runtime error` and `active child limit` results as scheduling failures, not worker failures: keep completed results and dispatch only the unstarted remainder in a smaller batch.
 - [ ] Run `node <skill>/scripts/audit-scope.mjs` (the lead runs it; workers do not); revert or reassign out-of-scope changes.
@@ -137,7 +149,9 @@ If the block reveals an underspecified interface shared by several tasks, update
 
 ## 4. Verify wave
 
-Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
+Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks, fewer for a small domain task. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
+
+**Cheap gates first.** Run compile/typecheck/targeted tests/artifact checks before spending verifier tokens. A verifier does not review a red build; it reviews a green diff and evidence. Keep reviews to findings + commands + evidence (target 5–8k tokens); raw logs stay in artifacts.
 
 For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence. Computer-use is a final, single-agent exploratory tier on a disposable desktop/VM — never a parallel test mechanism and never the only evidence.
 
@@ -145,9 +159,9 @@ For a GUI product, give each visual verifier its own Xvfb/display and project co
 
 Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed.
 
-Evidence standard: for every criterion, a command and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
+Evidence standard: for every criterion, a command/artifact/citation/metric and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
 
-On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run the affected verifier and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run a final integrated verification pass.
+On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run only the affected **delta** (diff + findings + acceptance commands), not the whole repository, and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run one final integrated verification pass on the frozen build.
 
 ## 5. Integrate
 
@@ -185,9 +199,14 @@ Final review is against the **original request**, not the plan: a fresh agent ch
 | Verifiers find the same defect | brief missing a convention | fix the brief, add a check, re-run affected tasks |
 | Lead context filling up | reading full reports | read summary + evidence; open only exceptions |
 | A worker hangs | no per-agent timeout exists | kill the job, split the task, add a tighter stop condition |
+| Token spend explodes | huge files, too many agents, full re-reads/reviews | split files, right-size batch, delta-review, record usage |
+| Verifier repeats a full review after a small fix | no delta procedure | review diff + findings + affected checks only |
+| Reports/reviews dominate input | oversized prose | cap to evidence lines; logs in artifacts |
 | Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots |
 | A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command |
 
 ## Time and cost
 
-Parallel speedup is bounded by the sequential share and by the real slot cap, not by the number of tasks. Push sequential work down: recon once, contracts before dispatch, verification in parallel. A 36-unit task with 10-minute units and a 6-slot batch is ~6 batches × 10 min ≈ 60 min of agent work, plus plan, verify and integration; do not promise a 3× speedup when the host caps active children at 8 or fewer.
+Parallel speedup is bounded by the sequential share, the real slot cap and token cost, not by the number of tasks. Push sequential work down: recon once, contracts before dispatch, verification in parallel. A 36-unit task with 10-minute units and a 6-slot batch is ~6 batches × 10 min ≈ 60 min of agent work, plus plan, verify and integration; do not promise a 3× speedup when the host caps active children at 8 or fewer.
+
+Token cost is dominated by fixed per-child overhead and repeated context reads. Eight substantial workers are often cheaper and better than twenty tiny ones; one delta re-check is much cheaper than a second full review. Track tokens per accepted task and per confirmed defect; optimize the ratio, not raw spend. See `references/token-economy.md`.