skills/agent-swarm/references/orchestration.md

# Workflow orchestration

The `workflow` tool runs one JavaScript body with top-level `await` and returns a JSON value. It is the only way to fan out dozens of agents without flooding the lead's context.

## Call shape

```json
{
  "meta": {"name": "swarm-wave-1", "description": "Build wave", "phases": [{"title": "wave 1"}]},
  "script": "<script body>",
  "args": {
    "wave": 1,
    "tasks": [{"id": "T01", "write": ["src/a.ts"]}],
    "schema": "<paste the contents of assets/report.schema.json>",
    "concurrency": 6
  },
  "run_in_background": false
}
```

- `meta.phases` is optional; use one phase per wave and `phase()` inside the script.
- `args` is data, never evaluated. The script cannot read files; pass paths and small structured facts.
- Pass `args.schema` as the parsed schema object, not a path or a string.
- Pass `args.concurrency` explicitly. It must be ≤ the free active-child slots; do not rely on the engine to queue.
- The tool blocks until every child settles. Use `run_in_background` for long waves; poll with `job_output`, cancel with `job_kill`.

## agent()

`agent(prompt, opts)` starts one child and resolves with:
- its final text when no schema is passed;
- a validated object when `schema` is passed;
- `null` when the child fails, returns a non-completed stop reason, or produces no structured value.

Options: `label`, `phase`, `schema`, `provider`, `model`. Unknown or deferred options kill the script.

`agent()` can also **throw** when the runtime cannot start a child (for example `active child limit reached`). That throw is not a `null`; if it happens inside `parallel`, it can abort the whole run. Wrap each `agent()` call in `try/catch` inside the batched runner and record a scheduling failure for that task instead of losing the batch.

The child is disposed after the call; there is no follow-up message inside a workflow. Design tasks as one-shot packets.

## parallel vs pipeline

- `parallel([thunks])` — start every thunk together; await all. It is **not a queue**: it does not limit concurrency. Use it only for a chunk whose size fits the free slots.
- `pipeline(items, ...stages)` — per-item chain with no barrier. Use only when each item must pass through stages without waiting for the wave (for example, extract → summarize per source). It multiplies agent count by the number of stages.
- For more items than slots, run batches sequentially and use `parallel` inside each batch (`assets/wave.workflow.js`).

Do not use `parallel` for dependent tasks; use separate waves and let the lead inspect results.

## Schema subset

The engine validates a strict subset. Keep schemas boring:

- Object root (`"type": "object"`).
- `properties`, `required` (names must exist in `properties`), `additionalProperties` boolean.
- Arrays with `items`.
- Strings/numbers/integers/booleans/null, `enum` of scalars, `const`.
- `oneOf` only with exactly-one semantics and no sibling keywords.
- No `minItems`, `maxLength`, `pattern`, `format`, `$ref`, `anyOf`, `allOf`.

A schema miss returns `null`. Verifiers should use the **minimal** schema in `assets/verdict.schema.json`:
`{task_id, verdict, summary, review_file}`. Put the detailed findings in `.swarm/reviews/Vxx.md`; do not make a long findings array required in the workflow schema. If a schema miss happens, read the review file on disk before deciding that the verification failed, then retry with the smaller schema.

## Concurrency and caps

- **Do not hardcode a concurrency default.** The active-child cap is host/policy-specific. Some runtimes allow 14; this environment rejected 8 workers when other children were active. Treat the cap as runtime data.
- **Preflight:** read the runtime diagnostics or `list_agents`; record `slot_cap` and `active_at_dispatch` in `.swarm/state.json`. Recon, verifiers, integrators and any other active children all count.
- **Batch size:** `min(8, slot_cap - active_at_dispatch - 1)`, at least 2. If you do not know the cap, use 6. Leave at least one free slot for retries.
- Each child holds a slot for its whole life. A `parallel` call starts every thunk immediately; if the batch exceeds free slots, the spawns throw.
- Slots are FIFO; a slow task delays later ones. Keep tasks in a batch similar in size.
- `maxTotalAgents`: 1000 per run. `maxItemsPerCall`: 4096 per `parallel`/`pipeline`.

### Batched worker runner

Use `assets/wave.workflow.js`. It splits `args.tasks` into chunks of `args.concurrency`, runs one `parallel` chunk at a time, catches `agent()` throws per task, and returns a structured failure for scheduling errors instead of killing the run. If a chunk reports a limit error, rerun only the unstarted tasks with a smaller concurrency.

## Retry pattern

Full template: `assets/wave.workflow.js`. Retry only when the result is `null` or `status === "failed"`. Do not retry `blocked` — that needs a lead decision. Do not retry a spawn/limit error immediately; shrink the batch or let the lead dispatch the remainder.

```js
async function runOne(t) {
  for (let attempt = 1; attempt <= 2; attempt++) {
    try {
      const r = await agent(promptFor(t, attempt), {
        label: t.id,
        phase: "wave 1",
        schema: REPORT
      });
      if (r && r.status !== "failed") return r;
      log(`${t.id}: attempt ${attempt} produced no usable report`);
    } catch (error) {
      const message = String(error && error.message ? error.message : error);
      if (!/(limit|spawn|could not start|active child|capacity|start a child)/i.test(message)) {
        throw error; // invalid options/schema must fail loudly
      }
      return failed(t, `scheduling error: ${message}`);
    }
  }
  return failed(t);
}
```

On the second attempt, add one line: what failed and what to check. Do not repeat the whole brief. Catch **only** spawn/scheduling failures around `agent()`; invalid hook options or an unsupported schema should still fail loudly so you fix the script.

## Prompt assembly

Build the prompt from `args`, not from prose in the script:

```js
const promptFor = (t, attempt) => [
  `You are agent ${t.id} in a coordinated swarm.`,
  `Read .swarm/brief.md, .swarm/tasks/${t.id}.md, and .swarm/contracts.md first.`,
  `Write only within: ${(t.write || []).join(", ") || "(read-only task)"}, plus your own report, review, and scratch under .swarm/reports/, .swarm/reviews/, and .swarm/tmp/${t.id}/.`,
  `Do not edit other tasks' files or reports; do not run installs, repo-wide format/lint/codegen, migrations, dev servers, or git writes.`,
  `Run the acceptance checks and record exact commands and results.`,
  `Write .swarm/reports/${t.id}.json and return the same object.`,
  `If a load-bearing unknown or a needed file outside scope blocks you, return status "blocked" with blocked_question; do not guess.`,
  attempt > 1
    ? `A previous attempt produced no usable report. Re-read the task and fix the cause or report the exact blocker.`
    : ""
].filter(Boolean).join("\n");
```

The task card carries the details; the prompt carries the role, scope, and protocol. Do not duplicate the card in the prompt.

## Verification runner

Use `assets/verify.workflow.js` for the verify wave (after freezing the tree). It embeds the minimal verdict schema and fetches reviews in batches:

```json
{
  "script": "<contents of assets/verify.workflow.js>",
  "args": {
    "wave": 3,
    "tasks": [{"id": "V01", "review": ".swarm/reviews/V01.md"}],
    "concurrency": 4
  },
  "run_in_background": true
}
```

The verifier writes the review file first and returns only `{task_id, verdict, summary, review_file}`. If the workflow reports `null`/`no usable verdict`, check the review file; then re-run with the minimal schema. Code is frozen during this wave.

## Background runs

Use `run_in_background: true` when a wave may exceed ~10 minutes or you want live progress in the job list. The call returns a job id; `job_output` with `wait: true` blocks until settlement. If a wave hangs, `job_kill` cancels the run and disposes children. A background run's result arrives once, at settlement.

## Tool availability

The protocol does not require the `workflow` tool. At preflight check what exists:

- `workflow` + sufficient slots → batched runner (`wave.workflow.js`).
- `subagent` only → lead dispatches 2–5 long-lived packets directly, or runs sequentially with
  structured reports.
- Neither → the lead executes the packets itself and still uses independent review/evidence; the
  skill becomes a planning/verification checklist, not a parallel runtime.

The same brief, task cards, ownership map and report schema apply in all three modes. Do not
fabricate a workflow if the tool is unavailable, and do not force a full swarm for a task that a
single agent can finish.

## Archetypes

### Code
- Units: module/feature/file cluster, each with its own tests.
- Ownership: source + test files per unit; shared types and barrels are integrator-only.
- Acceptance: targeted test command, typecheck, lint on changed files.
- Verify: a fresh agent runs the tests and reads the diff; no self-review.

### Data / analysis
- Units: dataset, metric, query, notebook or chart per agent.
- Ownership: inputs read-only; one output table/report per unit.
- Acceptance: reproducible script, input hash, row counts, metric definitions.
- Verify: rerun on a sample or full data; recompute key numbers.

### Design / UX
- Units: screen/component/flow/token set per agent.
- Ownership: source design files; shared tokens are integrator-only.
- Acceptance: rendered artifact at required sizes/states, accessibility checks.
- Verify: isolated display, screenshot path, keyboard/focus/contrast notes.

### Research
- Units: one question or one source cluster per agent; output files under `research/`.
- Ownership: one file per agent; no shared notes file.
- Acceptance: source URLs resolve, claims are dated, every criterion covered.
- Verify: citation check and a consistency pass; a second agent samples claims.

### Content / documents
- Units: one section per agent, plus one editor-integrator.
- Ownership: sections are separate files; the editor owns the assembled document.
- Acceptance: section brief, word budget, terminology list, citations.
- Verify: one editor pass for voice and transitions; a fact-checker for claims.

## Verification runs

Use `assets/verify.workflow.js` with `assets/verdict.schema.json` after freezing the tree. Each verifier:
1. writes its full review to `.swarm/reviews/Vxx.md` first;
2. returns only `{task_id, verdict, summary, review_file}`;
3. reports `pass`, `pass_with_risks`, or `fail` plus evidence in the file.

If the workflow returns `null`/`no usable verdict`, check the review file before treating it as a product failure; then retry with the minimal schema or message the verifier for a concise verdict. After any fix, re-run the affected verifier and mark the old review stale.

## Failure handling

- `null` per item: ordinary child failure; retry or mark failed in the result. Do not fail the whole wave.
- Spawn/limit error inside `agent()`: catch per task, keep completed results, and rerun only unstarted tasks with a smaller batch. This is a scheduling failure, not a worker failure.
- A thrown hook error (bad option, unsupported schema, unknown hook) kills the run: fix the script. Do not hide it in a broad try/catch.
- Cancellation: stop the run, inspect partial reports on disk; do not trust the tool result.

## Cost

Each child carries the system prompt, tool schemas, packet, code reads, and its output: typically 10–25k tokens of fixed overhead before any real work. A 12-agent wave can exceed 300k tokens, and a host capped at 8 cannot run 12 at once; batching reduces peak concurrency, not total cost. Budget before dispatch and count retries, delta re-checks and reviews; stop at the token cap. Eight substantial workers are often cheaper and better than twenty tiny ones. See `references/token-economy.md`.