diff --git a/README.md b/README.md index 11be19c..69d34fc 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ - [Инструкции скилла](agent-swarm/SKILL.md) - [Runbook](agent-swarm/references/playbook.md) - [Cookbook по `workflow`](agent-swarm/references/orchestration.md) +- [Изолированное визуальное тестирование](agent-swarm/references/visual-testing.md) ### Установка diff --git a/agent-swarm/README.ru.md b/agent-swarm/README.ru.md index c9b9cbd..9f88da8 100644 --- a/agent-swarm/README.ru.md +++ b/agent-swarm/README.ru.md @@ -30,6 +30,7 @@ - Дети не общаются между собой и не делегируют дальше; координация — через файлы и лида. - Верификаторы работают на замороженном дереве; полный разбор пишется в `.swarm/reviews/Vxx.md`, а workflow возвращает только `{task_id, verdict, summary, review_file}`. - `blocked`-воркер не заменяется: лид правит контракт/shared-файл и продолжает того же агента через `send_message`. +- Визуальные агенты работают в отдельных Xvfb/display-окружениях, со своим HOME/config/копией проекта; общий рабочий стол для скриншотов не используется. Детали — в `references/visual-testing.md`. - Стоимость: 30–40 агентов ≈ 0.5–1.5M токенов. - Для задачи, которую один агент делает за 15 минут, swarm не нужен. diff --git a/agent-swarm/SKILL.md b/agent-swarm/SKILL.md index ff4f80c..ff97e48 100644 --- a/agent-swarm/SKILL.md +++ b/agent-swarm/SKILL.md @@ -33,7 +33,7 @@ Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived 2. **Plan gate** — write `.swarm/plan.json`, `plan.md`, `ownership.md`, `contracts.md`, and one `tasks/Txx.md` per unit. Every success criterion maps to a task and a check. No two tasks write the same path. Include a `LEAD`/`INTEGRATOR` task owning lead-owned shared files, screenshots, licenses and the final report. Freeze interfaces. **Do not dispatch until this gate passes.** 3. **Build wave** — one `workflow` call per build wave, using the batched runner in `assets/wave.workflow.js`; its `concurrency` must fit the available slots (see Sizing). Retry `null`/`failed` once; if the runtime reports an active-child limit, halve the next batch and dispatch the remaining tasks — never relaunch the same oversubscribed fan-out. 4. **Audit** — run `scripts/audit-scope.mjs` from the skill base directory; revert or reassign out-of-scope changes. Run the build/test gate. -5. **Verify wave** — **freeze source changes**, then run 4–6 fresh verifiers by risk cluster. Use `assets/verdict.schema.json` and `assets/verify.workflow.js`; the review file on disk is the source of truth, not the workflow return. Verifiers never fix. If you fix findings, re-run the affected verifier and mark the old review stale. +5. **Verify wave** — **freeze source changes**, then run 4–6 fresh verifiers by risk cluster. Use `assets/verdict.schema.json` and `assets/verify.workflow.js`; the review file on disk is the source of truth, not the workflow return. Verifiers never fix. For GUI products, verify in **per-agent isolated displays**, never a shared desktop (see `references/visual-testing.md`). If you fix findings, re-run the affected verifier and mark the old review stale. 6. **Integrate** — one integrator for shared files and cross-cutting fixes; re-run the full acceptance suite; then a final verification pass on the integrated build. 7. **Close** — freeze the build and run a final independent review against the original request (not the plan), then write `final-report.md`. @@ -47,6 +47,7 @@ Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived 6. **Respect the runtime slot cap.** Active children include recon, verifiers, integrators and any other subagents. A spawn-limit error is a scheduling bug, not a worker failure: shrink the batch, keep completed results, and dispatch the remainder. 7. **Freeze during verification.** Once the verify wave starts, do not edit source/tests/docs except to fix a verified finding. After a fix, re-run the affected verifier; an old review is stale. 8. **Stop rules.** Two waves with no net progress → stop and re-plan. Verification failure rate > 30% → fix briefs/contracts, not agent count. Budget exhausted → report partial and list unverified areas. +9. **No shared visual desktop.** Visual agents get their own display/config/project copy; see `references/visual-testing.md`. A screenshot from a shared desktop is not evidence. ## Sizing @@ -92,6 +93,7 @@ For a `blocked` task: apply the smallest contract/shared-file change, update `co - `references/playbook.md` — phase-by-phase runbook, gates, checklists, failure recovery. - `references/packets.md` — how to write briefs, task cards, and verifier cards; examples and common mistakes. - `references/orchestration.md` — `workflow` cookbook: script patterns, schema subset, retries, background runs, archetypes (code / research / content). +- `references/visual-testing.md` — isolated Xvfb/container visual tests, deterministic screenshots, verifier procedure. - `assets/` — copyable brief, task, and verifier templates; `report.schema.json`, `verdict.schema.json`; `wave.workflow.js` (batched workers), `verify.workflow.js` (frozen verification). - `scripts/audit-scope.mjs` — compares changed files to the ownership map. diff --git a/agent-swarm/assets/verifier.md.tmpl b/agent-swarm/assets/verifier.md.tmpl index eceaae3..50a2040 100644 --- a/agent-swarm/assets/verifier.md.tmpl +++ b/agent-swarm/assets/verifier.md.tmpl @@ -21,6 +21,9 @@ You did not write this work. Do not fix it. Do not trust worker summaries. ## Freeze The tree is frozen during verification. Do not edit product source, tests, docs or manifests. If you need a gate script that rewrites generated files, copy it/hash-check it instead of changing the tree. +## Visual checks +For GUI products, run the app in your own isolated Xvfb/display (never a shared desktop). Capture the default size, the minimum size, empty/error states and every modal. Record the screenshot path and exact command alongside each finding; a screenshot without a command is not evidence. + ## Verdict `pass` / `pass_with_risks` / `fail` — with the top risks and exact repro commands. diff --git a/agent-swarm/references/packets.md b/agent-swarm/references/packets.md index d8d33ed..0921aa8 100644 --- a/agent-swarm/references/packets.md +++ b/agent-swarm/references/packets.md @@ -31,6 +31,7 @@ Copy `assets/task.md.tmpl`. Rules: - Write scope: globs. Everything else is read-only. - Contract: exact signature or output shape. If it is not written, it is not frozen. - Acceptance: 1–3 commands a verifier can re-run. "Code is clean" is not a check. +- GUI: acceptance includes a deterministic screenshot command, fixed window sizes and an isolated display; see `references/visual-testing.md`. - Stop conditions: time/attempt limit (25 min for S/M, 45–60 for L) plus "ask, do not guess" for load-bearing unknowns. - Never include another task's internal details; workers do not need the whole plan. @@ -43,6 +44,7 @@ Copy `assets/verifier.md.tmpl`. Rules: - Require a table: criterion, command, observed, pass/fail. - Ask for scope, contract, and test-quality checks. - Require the full review file **first**, then a minimal `{task_id, verdict, summary, review_file}` return (`assets/verdict.schema.json`). The review file is the source of truth; a schema miss must not erase the verification work. +- GUI: require an isolated Xvfb/display, screenshots at default and minimum sizes, and the exact commands used; shared-desktop screenshots are not evidence. ## Report diff --git a/agent-swarm/references/playbook.md b/agent-swarm/references/playbook.md index 5b3aa77..eca7dc4 100644 --- a/agent-swarm/references/playbook.md +++ b/agent-swarm/references/playbook.md @@ -54,7 +54,7 @@ Write the plan before any worker starts. **Coverage matrix.** Each success criterion → task(s) → check. A criterion with no task is a hole; a task with no criterion is scope creep. -**Visual acceptance.** For a GUI or visual product, define a deterministic screenshot/artifact command before dispatch (fixed window sizes, output path, clean pointer state) and put it in the acceptance matrix. Do not rely on desktop portal capture; it is unreliable on shared sessions and under Wayland. +**Visual acceptance.** For a GUI or visual product, define a deterministic screenshot/artifact command before dispatch (fixed window sizes, output path, clean pointer state) and put it in the acceptance matrix. Do not rely on desktop portal capture; it is unreliable on shared sessions and under Wayland. Visual agents must use isolated displays/configs — see `references/visual-testing.md`; never let two agents share one desktop. **Waves.** Topologically sort by `depends_on`, then balance effort across batches. Do **not** plan 8–12 workers from a documented default: discover the runtime active-child cap first. Compute `batch_size = clamp(slot_cap - active_children - 1, 2, 8)`; recon and any other subagents count as active children. If the cap is unknown, use 6. Put foundational or risky tasks in batch 1; verify after the last build batch. @@ -138,6 +138,8 @@ If the block reveals an underspecified interface shared by several tasks, update Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything. +For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence. + **Freeze before verify.** Once you start this wave, do not edit source, tests or docs. If you already started and then fixed something, mark the in-flight review stale and re-run that verifier on the new build. A verifier's verdict is only valid for the exact tree it saw. Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed. @@ -182,6 +184,8 @@ Final review is against the **original request**, not the plan: a fresh agent ch | Verifiers find the same defect | brief missing a convention | fix the brief, add a check, re-run affected tasks | | Lead context filling up | reading full reports | read summary + evidence; open only exceptions | | A worker hangs | no per-agent timeout exists | kill the job, split the task, add a tighter stop condition | +| Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots | +| A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command | ## Time and cost diff --git a/agent-swarm/references/visual-testing.md b/agent-swarm/references/visual-testing.md new file mode 100644 index 0000000..4c93e9b --- /dev/null +++ b/agent-swarm/references/visual-testing.md @@ -0,0 +1,122 @@ +# Visual testing for GUI agents + +GUI products fail verification for reasons that unit tests cannot see: clipped panels, focus-stealing +modals, dialogs that do not close, wrong DPI, a viewport that renders a blank background, or a +screenshot that captures a different agent's window. Treat visual verification as a separate test +tier with its own infrastructure. + +## Rule zero: never share one desktop + +Do not run several visual agents against the same X11/Wayland session. They steal focus, overlap +windows, race on the clipboard/configuration, and screenshot portals may return another window or +the desktop wallpaper. A common failure mode is "the screenshot looks fine" while it is a different +build's window. + +Each visual agent gets: + +- its own display (`DISPLAY=:N` / Wayland socket), +- its own `HOME`, `XDG_CONFIG_HOME`, `XDG_RUNTIME_DIR`, `TMPDIR`, +- its own project copy/worktree and artifact directory, +- its own `CARGO_TARGET_DIR` (or a read-only prebuilt cache + writable per-agent target), +- its own service port (BRP, dev server, websocket), +- a cleanup trap that kills the app, the X server and any helper processes. + +## Three tiers + +| Tier | Display | Renderer | Parallelism | Use | +|---|---|---|---|---| +| 1. Logic/headless | none | none | high | unit tests, egui `Context::run`, data/actions | +| 2. Visual UI | per-agent Xvfb | llvmpipe/lavapipe | 2–4 agents | layout, dialogs, screenshots, 2D UI | +| 3. Real GPU | one host/display/GPU | real GPU | **1** | PBR, real Bevy render, performance, final screenshot | + +Do not run Tier 3 in parallel with other visual agents. Do not use a shared desktop as Tier 2. + +## Per-agent Xvfb recipe + +Requirements in the image/host: `Xvfb`, `import` (ImageMagick), optionally `xdotool`, +`openbox`/`fluxbox`, Mesa llvmpipe and Vulkan lavapipe. + +```sh +# Pick a free display; never hardcode one that another agent may use. +for n in $(seq 90 140); do + [ ! -e "/tmp/.X11-unix/X$n" ] && DISPLAY_NUM=$n && break +done +SIZE=1440x900 +ART=/tmp/agent-$DISPLAY_NUM + +mkdir -p "$ART"/{home,config,runtime,tmp} +Xvfb ":$DISPLAY_NUM" -screen 0 "${SIZE}x24" -nolisten tcp >"$ART/xvfb.log" 2>&1 & +XVFB_PID=$! +APP_PID="" +cleanup() { + [ -n "$APP_PID" ] && kill "$APP_PID" 2>/dev/null || true + kill "$XVFB_PID" 2>/dev/null || true + wait "$XVFB_PID" 2>/dev/null || true +} +trap cleanup EXIT + +export DISPLAY=":$DISPLAY_NUM" +export HOME="$ART/home" +export XDG_CONFIG_HOME="$ART/config" +export XDG_RUNTIME_DIR="$ART/runtime" +export TMPDIR="$ART/tmp" +export LIBGL_ALWAYS_SOFTWARE=1 +export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json + +# Start the app *after* the X socket exists. A small WM helps focus/IME. +sleep 0.5 +# openbox & # optional +timeout 120s cargo run -p app -- --screenshot-mode "$ART/app.png" & +APP_PID=$! + +# Prefer an in-app deterministic screenshot. Fallback: capture the root window. +wait "$APP_PID" || true +[ -f "$ART/app.png" ] || import -display ":$DISPLAY_NUM" -window root "$ART/fallback.png" +``` + +If the app has no screenshot mode, add one before the UI wave: fixed size, wait N frames, hide +tooltips/animations/cursor, write PNG, exit 0. External root capture is a last resort. + +## Containers + +Containers are useful for reproducibility and cleanup, not as a default for every iteration. + +- Build one pinned image with Rust, Xvfb, Mesa/lavapipe, ImageMagick, xdotool and a minimal WM. +- Run one container per visual agent: `--rm --network=none --userns=keep-id`, no host `$HOME`, a + read-only repo mount plus a writable worktree/artifact mount. +- Keep cargo registry/cache in a named read-only volume; use a per-container target dir. +- GPU: `--device /dev/dri` for Intel/AMD; NVIDIA needs the container toolkit/CDI. Prefer software + rendering for UI tests and reserve GPU containers for one final Tier 3 run. +- Limit visual containers to 2–4 concurrently; llvmpipe uses CPU aggressively. + +## Deterministic screenshots + +- Fixed window size and scale factor (`1.0`) for both small and target sizes, e.g. 1000×640 and + 1440×900. +- Wait for the UI to settle (textures loaded, panels laid out) before capture. +- Disable tooltips, hover highlights, animations, autosave toasts and pointer-dependent chrome in + screenshot mode; move the pointer out of the window. +- Capture the app's own framebuffer, not the compositor. Portal capture is unreliable on Wayland and + shared sessions. +- Compare hashes for determinism and pixel-diff with a tolerance for antialiasing/GPU differences. +- Store the exact command, window size and build commit next to each artifact. + +## Visual verifier procedure + +A visual verifier should: + +1. Launch the exact build in its own isolated display; never reuse another agent's window. +2. Capture at least the default size, the minimum supported size, empty/error states, and every + modal that blocks input. +3. Check clipping/overflow, focus after closing a modal, keyboard shortcuts behind modals, selection + visibility, contrast, and scroll reachability. +4. Record a screenshot path and the observed issue for every finding; "looks off" is not evidence. +5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence; + dragging through a portal may collapse press/move/release into one frame. + +## Failure signs + +- Two visual jobs produce identical screenshots → they shared a display or captured the root. +- A screenshot shows the wrong title/theme/build → focus or portal capture, not a UI bug. +- Only the first visual job is reliable → the display/config isolation is incomplete. +- A "pass" has no screenshot or command → treat it as `not_run`.