Emil_Shanaty/skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add token budgets, telemetry and delta verification

Browse Source
This commit is contained in:
Emil Shanaty
2026-10-27 19:36:02 +03:00
parent 74fdfa458c
commit c3f58ceb40
9 changed files with 152 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent-swarm/references/orchestration.md
+1 -1
View File
@@ -208,4 +208,4 @@ If the workflow returns `null`/`no usable verdict`, check the review file before
## Cost
Each child carries the system prompt, tool schemas, packet, code reads, and its output: typically 10–40k tokens. A 12-agent wave is roughly 150–400k tokens, but a host capped at 8 cannot run 12 at once; batches reduce peak concurrency, not total cost. Budget before dispatch and count retries/re-verifications; stop at `max_agents`.
Each child carries the system prompt, tool schemas, packet, code reads, and its output: typically 10–25k tokens of fixed overhead before any real work. A 12-agent wave can exceed 300k tokens, and a host capped at 8 cannot run 12 at once; batching reduces peak concurrency, not total cost. Budget before dispatch and count retries, delta re-checks and reviews; stop at the token cap. Eight substantial workers are often cheaper and better than twenty tiny ones. See `references/token-economy.md`.
agent-swarm/references/playbook.md
+30 -11
View File
@@ -14,9 +14,11 @@ Answer these before spawning:
Budget:
- `agents ≈ units + ceil(units / 5) verifiers + 1–2 integrators + 2–4 recon`; add retries and re-verifications explicitly. Set `max_agents` only after checking available tools and slot cap.
- Tokens ≈ `agents × 20k` (10–40k per agent depending on code reads).
- Tokens ≈ `agents × fixed overhead (10–25k) + context reads + output + reviews`. Treat this as an estimate, then calibrate from measured usage. A 3k-line file read by several agents can cost more than the code they write.
- Reserve **20–30%** for verification, integration and one fix round. Set a soft cap at ~70% and a hard cap; stop at the hard cap with a partial report.
- Waves ≈ `ceil(units / batch_size)` + verify + integrate, where `batch_size` comes from the **real** active-child cap, not from a documented default.
- Stop at `max_agents` or `max_waves`, whichever comes first. Record both, plus `slot_cap`, `active_at_dispatch`, `batch_size`, retries and re-verifications in `state.json`.
- Stop at `max_agents`, `max_waves` or the token cap, whichever comes first. Record all caps, `slot_cap`, `active_at_dispatch`, `batch_size`, measured/estimated usage per phase, retries and re-verifications in `state.json`.
- See `references/token-economy.md`; do not invent precise token numbers without measurement.
Go: a 40-file migration with per-module tests, a 30-source research report, a 20-section document with a style guide, a multi-dataset analysis, a multi-screen visual redesign, a multi-service runbook.
No-go: one failing test, one architectural decision, a detail pass where every unit touches the same artifact.
@@ -47,7 +49,11 @@ node <skill>/scripts/audit-scope.mjs --init
Write the plan before any worker starts. Keep the plan proportionate: a small domain task may need only a brief and a verifier, not a full `.swarm` tree.
**Decomposition.** One task = one deliverable, one writer, one acceptance check, at most one page. The deliverable may be a module, a research question, a document section, a dataset, a design artifact, or a runbook. Budget **5–25 minutes for S/M** and **40–60 minutes for L**; if a task is expected to produce a single source file over ~1,500 lines (or a very large document/dataset), split it or accept an integration-heavy task. If it needs two writers, split it; if it needs the whole repo, it is not a task.
**Decomposition.** One task = one deliverable, one writer, one acceptance check, at most one page. The deliverable may be a module, a research question, a document section, a dataset, a design artifact, or a runbook. Budget **5–25 minutes for S/M** and **40–60 minutes for L**; if a task is expected to produce a single source file over ~800 lines (or a very large document/dataset), split it or accept an integration-heavy task. If it needs two writers, split it; if it needs the whole repo, it is not a task.
**Vertical slices.** Prefer one end-to-end capability at a time (selection → transform → undo → save) over a big-bang layer-by-layer build. A slice has its own checkpoint, tests and independent review; it limits late integration and repeated full reviews.
**Token shape.** Estimate the fixed cost per child, the context each task must read, and the review cost. Merge tiny tasks that pay fixed overhead for little work; split giant files that force repeated large reads. The goal is quality per token, not the smallest number of agents.
**Ownership.** Every artifact/path in exactly one `write` list: files, sections, datasets, source notes, design files. Add an explicit `LEAD` task whose list contains lead/integrator-owned shared artifacts, screenshots, licenses and the final report; without it, audit-scope reports the lead's own integration edits as unowned. Shared files are integrator-only: manifests, lockfiles, indexes/barrels, schemas, shared style/token files, migrations, i18n catalogs, snapshots, generated code. If two tasks must touch one artifact, serialize them with `depends_on` or merge them.
@@ -60,7 +66,10 @@ Write the plan before any worker starts. Keep the plan proportionate: a small do
**Waves.** Topologically sort by `depends_on`, then balance effort across batches. Do **not** plan 8–12 workers from a documented default: discover the runtime active-child cap first. Compute `batch_size = clamp(slot_cap - active_children - 1, 2, 8)`; recon and any other subagents count as active children. If the cap is unknown, use 6. Put foundational or risky tasks in batch 1; verify after the last build batch.
**Gate checklist:**
- [ ] 8+ tasks, each with objective, inputs, write scope, acceptance, stop condition.
- [ ] 8+ tasks, each with objective, inputs, write scope, acceptance, stop condition, and a token/effort budget.
- [ ] Files/modules over ~800 lines are split, or the task explicitly owns the integration cost.
- [ ] Token budget, soft cap and verification reserve are recorded; units are merged/split to avoid fixed-overhead waste.
- [ ] The plan uses vertical slices/checkpoints for a multi-milestone product.
- [ ] No path in two write scopes.
- [ ] Contracts frozen for every cross-task interface.
- [ ] Coverage matrix complete.
@@ -78,6 +87,7 @@ Minimal `plan.json` shape:
```json
{
"goal": "one line",
"budget": { "tokens": 600000, "soft_cap": 0.7, "reserved_for_verify": 0.25 },
"tasks": [
{
"id": "LEAD",
@@ -109,13 +119,15 @@ Minimal `plan.json` shape:
Dispatch one `workflow` call per build wave (see `references/orchestration.md`). Use `assets/wave.workflow.js`, which splits tasks into batches, catches spawn errors instead of discarding the whole wave, and carries `args.concurrency`.
Before the call:
- [ ] Task cards are on disk and reference real paths.
- [ ] Task cards are on disk and reference real paths/ranges, not pasted context.
- [ ] `args.tasks` carries `{id, write}` for each task.
- [ ] The report schema is passed as `args.schema`, and `args.concurrency` is at most the free slot count.
- [ ] Every task has a stop condition.
- [ ] Baseline exists and `state.json` records the cap.
- [ ] Every task has a stop condition and a size/effort budget.
- [ ] Workers run targeted checks, not the full suite; the lead owns the full gate.
- [ ] Baseline exists and `state.json` records the cap and token budget.
After the call:
- [ ] Record measured/estimated token usage per phase if available.
- [ ] Triage structured reports: done / partial / blocked / failed.
- [ ] Treat `spawn/runtime error` and `active child limit` results as scheduling failures, not worker failures: keep completed results and dispatch only the unstarted remainder in a smaller batch.
- [ ] Run `node <skill>/scripts/audit-scope.mjs` (the lead runs it; workers do not); revert or reassign out-of-scope changes.
@@ -137,7 +149,9 @@ If the block reveals an underspecified interface shared by several tasks, update
## 4. Verify wave
Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks, fewer for a small domain task. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
**Cheap gates first.** Run compile/typecheck/targeted tests/artifact checks before spending verifier tokens. A verifier does not review a red build; it reviews a green diff and evidence. Keep reviews to findings + commands + evidence (target 5–8k tokens); raw logs stay in artifacts.
For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence. Computer-use is a final, single-agent exploratory tier on a disposable desktop/VM — never a parallel test mechanism and never the only evidence.
@@ -145,9 +159,9 @@ For a GUI product, give each visual verifier its own Xvfb/display and project co
Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed.
Evidence standard: for every criterion, a command and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
Evidence standard: for every criterion, a command/artifact/citation/metric and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run the affected verifier and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run a final integrated verification pass.
On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run only the affected **delta** (diff + findings + acceptance commands), not the whole repository, and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run one final integrated verification pass on the frozen build.
## 5. Integrate
@@ -185,9 +199,14 @@ Final review is against the **original request**, not the plan: a fresh agent ch
| Verifiers find the same defect | brief missing a convention | fix the brief, add a check, re-run affected tasks |
| Lead context filling up | reading full reports | read summary + evidence; open only exceptions |
| A worker hangs | no per-agent timeout exists | kill the job, split the task, add a tighter stop condition |
| Token spend explodes | huge files, too many agents, full re-reads/reviews | split files, right-size batch, delta-review, record usage |
| Verifier repeats a full review after a small fix | no delta procedure | review diff + findings + affected checks only |
| Reports/reviews dominate input | oversized prose | cap to evidence lines; logs in artifacts |
| Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots |
| A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command |
## Time and cost
Parallel speedup is bounded by the sequential share and by the real slot cap, not by the number of tasks. Push sequential work down: recon once, contracts before dispatch, verification in parallel. A 36-unit task with 10-minute units and a 6-slot batch is ~6 batches × 10 min ≈ 60 min of agent work, plus plan, verify and integration; do not promise a 3× speedup when the host caps active children at 8 or fewer.
Parallel speedup is bounded by the sequential share, the real slot cap and token cost, not by the number of tasks. Push sequential work down: recon once, contracts before dispatch, verification in parallel. A 36-unit task with 10-minute units and a 6-slot batch is ~6 batches × 10 min ≈ 60 min of agent work, plus plan, verify and integration; do not promise a 3× speedup when the host caps active children at 8 or fewer.
Token cost is dominated by fixed per-child overhead and repeated context reads. Eight substantial workers are often cheaper and better than twenty tiny ones; one delta re-check is much cheaper than a second full review. Track tokens per accepted task and per confirmed defect; optimize the ratio, not raw spend. See `references/token-economy.md`.
agent-swarm/references/token-economy.md
+103
View File
@@ -0,0 +1,103 @@
# Token economy
Quality per token beats raw token spend. The expensive part of a swarm is usually **input context and
repeated reading**, not the code a worker writes. Measure, budget, then cut redundancy — never cut
the independent checks that catch real defects.
## Measure before optimizing
Record usage per phase and per agent in `.swarm/state.json`:
```json
{
"budget": { "tokens": 600000, "soft_cap": 0.7, "reserved_for_verify": 0.2 },
"usage": [
{ "phase": "recon", "agent": "R01", "tokens_in": 18000, "tokens_out": 4000 },
{ "phase": "build", "agent": "T03", "tokens_in": 24000, "tokens_out": 9000 }
]
}
```
If the runtime does not expose usage, estimate from context size and output length and mark the value
as `estimated`. The point is a feedback loop, not accounting precision.
Useful ratios:
- tokens per accepted task;
- verifier tokens per confirmed defect;
- rework tokens caused by a missed contract;
- share of spend on recon / build / verify / integration.
## Budgets and reserves
- Set a **hard budget** for the milestone and a **soft cap** at ~70%. At the soft cap, stop expanding
scope; finish the current slice and verify.
- Reserve **20–30%** for verification, integration and one round of fixes. Do not spend the whole
budget on generation.
- Budget per phase, not only per wave: recon, build, verify, integrate.
- Stop at the cap and report partial results plus unverified areas. Silent overspend is worse than a
partial result.
- Calibrate from the first measured milestone; do not copy invented numbers from a template.
## Context discipline
The biggest savings without quality loss:
1. **Small files.** Split modules above ~800 lines (UI panels, viewport, inspector). A worker should
read 300–800 relevant lines, not 3,000. Smaller modules also compile and verify faster.
2. **Read ranges, not repositories.** Use `path:line`, symbol search and grep first. Do not read a
whole crate to change one function.
3. **Point to artifacts, do not paste them.** Task cards reference `.swarm/brief.md`,
`.swarm/contracts.md` and exact ranges; they do not duplicate them in the prompt.
4. **Cap reports and reviews.** Worker report: summary, files/artifacts, evidence, issues,
assumptions. Review: findings + commands + evidence; target 5–8k tokens, raw logs stay in files.
5. **One page per task.** If a card needs more, the task is too broad.
6. **No full-suite runs per worker.** Workers run targeted checks; the lead runs the full gate once
per wave.
7. **Fresh context only where independence matters.** The final verifier needs a clean read. A fix
re-check can be a **delta review**: diff + findings + acceptance commands, not the whole repo.
## Verification ladders
Do not spend LLM review tokens on a red build:
1. compile / typecheck / lint;
2. targeted tests;
3. artifact/render/source check;
4. independent verifier on the diff and evidence;
5. final full review once, on a frozen integrated build.
Freeze before step 4. If code changes, re-check only the affected delta; mark the old review stale.
This preserves quality while avoiding repeated full reviews.
## Orchestration shape
- **Fixed overhead dominates small tasks.** Each child carries system prompt, tools, brief and task
context. 30 small agents can cost more than 8 substantial ones.
- **Right-size:** 2–4 implementers + 1 verifier for a vertical slice; 6–10 agents for a broad
milestone; 30 only for genuinely independent breadth.
- **Model routing:** strong model for architecture, difficult fixes and final review; cheaper/faster
model for tests, docs, screenshots and mechanical edits.
- **Vertical slices:** ship one end-to-end feature at a time (selection → transform → undo → save),
with a checkpoint and verification after each. Do not generate the whole product and verify at the
end.
- **Reuse:** do not regenerate a working core, format or codegen. Extend it.
## Anti-patterns
| Pattern | Cost | Fix |
|---|---|---|
| 3k-line file edited by several agents | repeated reads, merge conflicts | split before dispatch |
| Full repo read for a one-function change | huge input | path:line + grep |
| Full re-review after every fix | duplicate verifier spend | delta review |
| Verifier runs on a failing build | wasted review | automated gate first |
| 20 tiny tasks | fixed overhead per child | merge into coherent units |
| Reports/reviews as essays | output + lead input | schema + evidence lines |
| Whole product in one wave | late integration, huge rework | vertical slices + checkpoints |
| Vague "max quality" with no budget | unbounded spend | hard/soft caps + reserve |
## Minimal operating rule
> Measure per phase, reserve for verification, keep files and packets small, verify the diff after
> fixes, and spend the saved tokens on one strong independent review rather than many redundant
> agents.