155 lines
7.0 KiB
Markdown
155 lines
7.0 KiB
Markdown
# Visual testing for GUI agents
|
||
|
||
GUI products fail verification for reasons that unit tests cannot see: clipped panels, focus-stealing
|
||
modals, dialogs that do not close, wrong DPI, a viewport that renders a blank background, or a
|
||
screenshot that captures a different agent's window. Treat visual verification as a separate test
|
||
tier with its own infrastructure.
|
||
|
||
The commands below use Linux Xvfb + ImageMagick as one concrete provider. On macOS/Windows use a
|
||
container or the platform's virtual-display tooling; the invariant is **isolation and deterministic
|
||
capture**, not Xvfb itself. If the app can capture its own framebuffer, that is preferred on every
|
||
platform.
|
||
|
||
## Rule zero: never share one desktop
|
||
|
||
Do not run several visual agents against the same X11/Wayland session. They steal focus, overlap
|
||
windows, race on the clipboard/configuration, and screenshot portals may return another window or
|
||
the desktop wallpaper. A common failure mode is "the screenshot looks fine" while it is a different
|
||
build's window.
|
||
|
||
Each visual agent gets:
|
||
|
||
- its own display (`DISPLAY=:N` / Wayland socket),
|
||
- its own `HOME`, `XDG_CONFIG_HOME`, `XDG_RUNTIME_DIR`, `TMPDIR`,
|
||
- its own project copy/worktree and artifact directory,
|
||
- its own build target/cache directory (or a read-only prebuilt cache + writable per-agent target),
|
||
- its own service port (BRP, dev server, websocket),
|
||
- a cleanup trap that kills the app, the X server and any helper processes.
|
||
|
||
## Three tiers
|
||
|
||
| Tier | Display | Renderer | Parallelism | Use |
|
||
|---|---|---|---|---|
|
||
| 1. Logic/headless | none | none | high | unit tests, egui `Context::run`, data/actions |
|
||
| 2. Visual UI | per-agent Xvfb | llvmpipe/lavapipe | 2–4 agents | layout, dialogs, screenshots, 2D UI |
|
||
| 3. Real GPU | one host/display/GPU | real GPU | **1** | PBR, real Bevy render, performance, final screenshot |
|
||
|
||
Do not run Tier 3 in parallel with other visual agents. Do not use a shared desktop as Tier 2.
|
||
|
||
## Per-agent Xvfb recipe
|
||
|
||
Requirements in the image/host: `Xvfb`, `import` (ImageMagick), optionally `xdotool`,
|
||
`openbox`/`fluxbox`, Mesa llvmpipe and Vulkan lavapipe.
|
||
|
||
```sh
|
||
# Pick a free display; never hardcode one that another agent may use.
|
||
for n in $(seq 90 140); do
|
||
[ ! -e "/tmp/.X11-unix/X$n" ] && DISPLAY_NUM=$n && break
|
||
done
|
||
SIZE=1440x900
|
||
ART=/tmp/agent-$DISPLAY_NUM
|
||
|
||
mkdir -p "$ART"/{home,config,runtime,tmp}
|
||
Xvfb ":$DISPLAY_NUM" -screen 0 "${SIZE}x24" -nolisten tcp >"$ART/xvfb.log" 2>&1 &
|
||
XVFB_PID=$!
|
||
APP_PID=""
|
||
cleanup() {
|
||
[ -n "$APP_PID" ] && kill "$APP_PID" 2>/dev/null || true
|
||
kill "$XVFB_PID" 2>/dev/null || true
|
||
wait "$XVFB_PID" 2>/dev/null || true
|
||
}
|
||
trap cleanup EXIT
|
||
|
||
export DISPLAY=":$DISPLAY_NUM"
|
||
export HOME="$ART/home"
|
||
export XDG_CONFIG_HOME="$ART/config"
|
||
export XDG_RUNTIME_DIR="$ART/runtime"
|
||
export TMPDIR="$ART/tmp"
|
||
export LIBGL_ALWAYS_SOFTWARE=1
|
||
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json
|
||
|
||
# Start the app *after* the X socket exists. A small WM helps focus/IME.
|
||
sleep 0.5
|
||
# openbox & # optional
|
||
timeout 120s <run command> --screenshot-mode "$ART/app.png" &
|
||
APP_PID=$!
|
||
|
||
# Prefer an in-app deterministic screenshot. Fallback: capture the root window.
|
||
wait "$APP_PID" || true
|
||
[ -f "$ART/app.png" ] || import -display ":$DISPLAY_NUM" -window root "$ART/fallback.png"
|
||
```
|
||
|
||
If the app has no screenshot mode, add one before the UI wave: fixed size, wait N frames, hide
|
||
tooltips/animations/cursor, write PNG, exit 0. External root capture is a last resort.
|
||
|
||
## Containers
|
||
|
||
Containers are useful for reproducibility and cleanup, not as a default for every iteration.
|
||
|
||
- Build one pinned image with Rust, Xvfb, Mesa/lavapipe, ImageMagick, xdotool and a minimal WM.
|
||
- Run one container per visual agent: `--rm --network=none --userns=keep-id`, no host `$HOME`, a
|
||
read-only repo mount plus a writable worktree/artifact mount.
|
||
- Keep package-manager/build caches in a named read-only volume; use a per-container target dir.
|
||
- GPU: `--device /dev/dri` for Intel/AMD; NVIDIA needs the container toolkit/CDI. Prefer software
|
||
rendering for UI tests and reserve GPU containers for one final Tier 3 run.
|
||
- Limit visual containers to 2–4 concurrently; llvmpipe uses CPU aggressively.
|
||
|
||
## Deterministic screenshots
|
||
|
||
- Fixed window size and scale factor (`1.0`) for both small and target sizes, e.g. 1000×640 and
|
||
1440×900.
|
||
- Wait for the UI to settle (textures loaded, panels laid out) before capture.
|
||
- Disable tooltips, hover highlights, animations, autosave toasts and pointer-dependent chrome in
|
||
screenshot mode; move the pointer out of the window.
|
||
- Capture the app's own framebuffer, not the compositor. Portal capture is unreliable on Wayland and
|
||
shared sessions.
|
||
- Compare hashes for determinism and pixel-diff with a tolerance for antialiasing/GPU differences.
|
||
- Store the exact command, window size and build commit next to each artifact.
|
||
|
||
## Visual verifier procedure
|
||
|
||
A visual verifier should:
|
||
|
||
1. Launch the exact build in its own isolated display; never reuse another agent's window.
|
||
2. Capture at least the default size, the minimum supported size, empty/error states, and every
|
||
modal that blocks input.
|
||
3. Check clipping/overflow, focus after closing a modal, keyboard shortcuts behind modals, selection
|
||
visibility, contrast, and scroll reachability.
|
||
4. Record a screenshot path and the observed issue for every finding; "looks off" is not evidence.
|
||
5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence;
|
||
dragging through a portal may collapse press/move/release into one frame.
|
||
|
||
## Computer-use: last resort, never a test farm
|
||
|
||
If the runtime grants subagents an MCP `computer_use` tool, they can technically move a mouse and
|
||
type. That does **not** make it a good regression mechanism.
|
||
|
||
Use computer-use only when all of these are true:
|
||
|
||
- the deterministic tiers (headless tests, per-agent Xvfb + scripted screenshots) are already green;
|
||
- the check is exploratory/OS-integration that cannot be scripted (native file dialog, IME, clipboard,
|
||
drag from outside the app);
|
||
- it runs on a dedicated disposable desktop/VM, not the user's live session;
|
||
- exactly **one** agent uses it at a time;
|
||
- secrets, notifications and personal data are not visible;
|
||
- every action is logged and a screenshot artifact is saved.
|
||
|
||
Never:
|
||
|
||
- give computer-use to every worker in a parallel wave;
|
||
- use it as the only evidence for a pass;
|
||
- use it for pixel/regression assertions;
|
||
- let it click/type in the user's desktop without explicit permission;
|
||
- assume an isolated Xvfb session is visible to a compositor/portal-based computer-use backend. If
|
||
the backend targets KWin/Wayland, give it a nested desktop/VM it can actually see.
|
||
|
||
A computer-use result is qualitative input for a human or a vision reviewer, not a substitute for a
|
||
command, an assertion or a deterministic screenshot.
|
||
|
||
## Failure signs
|
||
|
||
- Two visual jobs produce identical screenshots → they shared a display or captured the root.
|
||
- A screenshot shows the wrong title/theme/build → focus or portal capture, not a UI bug.
|
||
- Only the first visual job is reliable → the display/config isolation is incomplete.
|
||
- A "pass" has no screenshot or command → treat it as `not_run`.
|