Add isolated visual testing guidance
This commit is contained in:
Emil Shanaty
@@ -31,6 +31,7 @@ Copy `assets/task.md.tmpl`. Rules:
|
||||
- Write scope: globs. Everything else is read-only.
|
||||
- Contract: exact signature or output shape. If it is not written, it is not frozen.
|
||||
- Acceptance: 1–3 commands a verifier can re-run. "Code is clean" is not a check.
|
||||
- GUI: acceptance includes a deterministic screenshot command, fixed window sizes and an isolated display; see `references/visual-testing.md`.
|
||||
- Stop conditions: time/attempt limit (25 min for S/M, 45–60 for L) plus "ask, do not guess" for load-bearing unknowns.
|
||||
- Never include another task's internal details; workers do not need the whole plan.
|
||||
|
||||
@@ -43,6 +44,7 @@ Copy `assets/verifier.md.tmpl`. Rules:
|
||||
- Require a table: criterion, command, observed, pass/fail.
|
||||
- Ask for scope, contract, and test-quality checks.
|
||||
- Require the full review file **first**, then a minimal `{task_id, verdict, summary, review_file}` return (`assets/verdict.schema.json`). The review file is the source of truth; a schema miss must not erase the verification work.
|
||||
- GUI: require an isolated Xvfb/display, screenshots at default and minimum sizes, and the exact commands used; shared-desktop screenshots are not evidence.
|
||||
|
||||
## Report
|
||||
|
||||
|
||||
@@ -54,7 +54,7 @@ Write the plan before any worker starts.
|
||||
|
||||
**Coverage matrix.** Each success criterion → task(s) → check. A criterion with no task is a hole; a task with no criterion is scope creep.
|
||||
|
||||
**Visual acceptance.** For a GUI or visual product, define a deterministic screenshot/artifact command before dispatch (fixed window sizes, output path, clean pointer state) and put it in the acceptance matrix. Do not rely on desktop portal capture; it is unreliable on shared sessions and under Wayland.
|
||||
**Visual acceptance.** For a GUI or visual product, define a deterministic screenshot/artifact command before dispatch (fixed window sizes, output path, clean pointer state) and put it in the acceptance matrix. Do not rely on desktop portal capture; it is unreliable on shared sessions and under Wayland. Visual agents must use isolated displays/configs — see `references/visual-testing.md`; never let two agents share one desktop.
|
||||
|
||||
**Waves.** Topologically sort by `depends_on`, then balance effort across batches. Do **not** plan 8–12 workers from a documented default: discover the runtime active-child cap first. Compute `batch_size = clamp(slot_cap - active_children - 1, 2, 8)`; recon and any other subagents count as active children. If the cap is unknown, use 6. Put foundational or risky tasks in batch 1; verify after the last build batch.
|
||||
|
||||
@@ -138,6 +138,8 @@ If the block reveals an underspecified interface shared by several tasks, update
|
||||
|
||||
Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
|
||||
|
||||
For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence.
|
||||
|
||||
**Freeze before verify.** Once you start this wave, do not edit source, tests or docs. If you already started and then fixed something, mark the in-flight review stale and re-run that verifier on the new build. A verifier's verdict is only valid for the exact tree it saw.
|
||||
|
||||
Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed.
|
||||
@@ -182,6 +184,8 @@ Final review is against the **original request**, not the plan: a fresh agent ch
|
||||
| Verifiers find the same defect | brief missing a convention | fix the brief, add a check, re-run affected tasks |
|
||||
| Lead context filling up | reading full reports | read summary + evidence; open only exceptions |
|
||||
| A worker hangs | no per-agent timeout exists | kill the job, split the task, add a tighter stop condition |
|
||||
| Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots |
|
||||
| A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command |
|
||||
|
||||
## Time and cost
|
||||
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# Visual testing for GUI agents
|
||||
|
||||
GUI products fail verification for reasons that unit tests cannot see: clipped panels, focus-stealing
|
||||
modals, dialogs that do not close, wrong DPI, a viewport that renders a blank background, or a
|
||||
screenshot that captures a different agent's window. Treat visual verification as a separate test
|
||||
tier with its own infrastructure.
|
||||
|
||||
## Rule zero: never share one desktop
|
||||
|
||||
Do not run several visual agents against the same X11/Wayland session. They steal focus, overlap
|
||||
windows, race on the clipboard/configuration, and screenshot portals may return another window or
|
||||
the desktop wallpaper. A common failure mode is "the screenshot looks fine" while it is a different
|
||||
build's window.
|
||||
|
||||
Each visual agent gets:
|
||||
|
||||
- its own display (`DISPLAY=:N` / Wayland socket),
|
||||
- its own `HOME`, `XDG_CONFIG_HOME`, `XDG_RUNTIME_DIR`, `TMPDIR`,
|
||||
- its own project copy/worktree and artifact directory,
|
||||
- its own `CARGO_TARGET_DIR` (or a read-only prebuilt cache + writable per-agent target),
|
||||
- its own service port (BRP, dev server, websocket),
|
||||
- a cleanup trap that kills the app, the X server and any helper processes.
|
||||
|
||||
## Three tiers
|
||||
|
||||
| Tier | Display | Renderer | Parallelism | Use |
|
||||
|---|---|---|---|---|
|
||||
| 1. Logic/headless | none | none | high | unit tests, egui `Context::run`, data/actions |
|
||||
| 2. Visual UI | per-agent Xvfb | llvmpipe/lavapipe | 2–4 agents | layout, dialogs, screenshots, 2D UI |
|
||||
| 3. Real GPU | one host/display/GPU | real GPU | **1** | PBR, real Bevy render, performance, final screenshot |
|
||||
|
||||
Do not run Tier 3 in parallel with other visual agents. Do not use a shared desktop as Tier 2.
|
||||
|
||||
## Per-agent Xvfb recipe
|
||||
|
||||
Requirements in the image/host: `Xvfb`, `import` (ImageMagick), optionally `xdotool`,
|
||||
`openbox`/`fluxbox`, Mesa llvmpipe and Vulkan lavapipe.
|
||||
|
||||
```sh
|
||||
# Pick a free display; never hardcode one that another agent may use.
|
||||
for n in $(seq 90 140); do
|
||||
[ ! -e "/tmp/.X11-unix/X$n" ] && DISPLAY_NUM=$n && break
|
||||
done
|
||||
SIZE=1440x900
|
||||
ART=/tmp/agent-$DISPLAY_NUM
|
||||
|
||||
mkdir -p "$ART"/{home,config,runtime,tmp}
|
||||
Xvfb ":$DISPLAY_NUM" -screen 0 "${SIZE}x24" -nolisten tcp >"$ART/xvfb.log" 2>&1 &
|
||||
XVFB_PID=$!
|
||||
APP_PID=""
|
||||
cleanup() {
|
||||
[ -n "$APP_PID" ] && kill "$APP_PID" 2>/dev/null || true
|
||||
kill "$XVFB_PID" 2>/dev/null || true
|
||||
wait "$XVFB_PID" 2>/dev/null || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
export DISPLAY=":$DISPLAY_NUM"
|
||||
export HOME="$ART/home"
|
||||
export XDG_CONFIG_HOME="$ART/config"
|
||||
export XDG_RUNTIME_DIR="$ART/runtime"
|
||||
export TMPDIR="$ART/tmp"
|
||||
export LIBGL_ALWAYS_SOFTWARE=1
|
||||
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json
|
||||
|
||||
# Start the app *after* the X socket exists. A small WM helps focus/IME.
|
||||
sleep 0.5
|
||||
# openbox & # optional
|
||||
timeout 120s cargo run -p app -- --screenshot-mode "$ART/app.png" &
|
||||
APP_PID=$!
|
||||
|
||||
# Prefer an in-app deterministic screenshot. Fallback: capture the root window.
|
||||
wait "$APP_PID" || true
|
||||
[ -f "$ART/app.png" ] || import -display ":$DISPLAY_NUM" -window root "$ART/fallback.png"
|
||||
```
|
||||
|
||||
If the app has no screenshot mode, add one before the UI wave: fixed size, wait N frames, hide
|
||||
tooltips/animations/cursor, write PNG, exit 0. External root capture is a last resort.
|
||||
|
||||
## Containers
|
||||
|
||||
Containers are useful for reproducibility and cleanup, not as a default for every iteration.
|
||||
|
||||
- Build one pinned image with Rust, Xvfb, Mesa/lavapipe, ImageMagick, xdotool and a minimal WM.
|
||||
- Run one container per visual agent: `--rm --network=none --userns=keep-id`, no host `$HOME`, a
|
||||
read-only repo mount plus a writable worktree/artifact mount.
|
||||
- Keep cargo registry/cache in a named read-only volume; use a per-container target dir.
|
||||
- GPU: `--device /dev/dri` for Intel/AMD; NVIDIA needs the container toolkit/CDI. Prefer software
|
||||
rendering for UI tests and reserve GPU containers for one final Tier 3 run.
|
||||
- Limit visual containers to 2–4 concurrently; llvmpipe uses CPU aggressively.
|
||||
|
||||
## Deterministic screenshots
|
||||
|
||||
- Fixed window size and scale factor (`1.0`) for both small and target sizes, e.g. 1000×640 and
|
||||
1440×900.
|
||||
- Wait for the UI to settle (textures loaded, panels laid out) before capture.
|
||||
- Disable tooltips, hover highlights, animations, autosave toasts and pointer-dependent chrome in
|
||||
screenshot mode; move the pointer out of the window.
|
||||
- Capture the app's own framebuffer, not the compositor. Portal capture is unreliable on Wayland and
|
||||
shared sessions.
|
||||
- Compare hashes for determinism and pixel-diff with a tolerance for antialiasing/GPU differences.
|
||||
- Store the exact command, window size and build commit next to each artifact.
|
||||
|
||||
## Visual verifier procedure
|
||||
|
||||
A visual verifier should:
|
||||
|
||||
1. Launch the exact build in its own isolated display; never reuse another agent's window.
|
||||
2. Capture at least the default size, the minimum supported size, empty/error states, and every
|
||||
modal that blocks input.
|
||||
3. Check clipping/overflow, focus after closing a modal, keyboard shortcuts behind modals, selection
|
||||
visibility, contrast, and scroll reachability.
|
||||
4. Record a screenshot path and the observed issue for every finding; "looks off" is not evidence.
|
||||
5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence;
|
||||
dragging through a portal may collapse press/move/release into one frame.
|
||||
|
||||
## Failure signs
|
||||
|
||||
- Two visual jobs produce identical screenshots → they shared a display or captured the root.
|
||||
- A screenshot shows the wrong title/theme/build → focus or portal capture, not a UI bug.
|
||||
- Only the first visual job is reliable → the display/config isolation is incomplete.
|
||||
- A "pass" has no screenshot or command → treat it as `not_run`.
|
||||
Reference in New Issue
Block a user