Visual testing for GUI agents

GUI products fail verification for reasons that unit tests cannot see: clipped panels, focus-stealing modals, dialogs that do not close, wrong DPI, a viewport that renders a blank background, or a screenshot that captures a different agent's window. Treat visual verification as a separate test tier with its own infrastructure.

The commands below use Linux Xvfb + ImageMagick as one concrete provider. On macOS/Windows use a container or the platform's virtual-display tooling; the invariant is isolation and deterministic capture, not Xvfb itself. If the app can capture its own framebuffer, that is preferred on every platform.

Do not run several visual agents against the same X11/Wayland session. They steal focus, overlap windows, race on the clipboard/configuration, and screenshot portals may return another window or the desktop wallpaper. A common failure mode is "the screenshot looks fine" while it is a different build's window.

Each visual agent gets:

its own display (DISPLAY=:N / Wayland socket),
its own HOME, XDG_CONFIG_HOME, XDG_RUNTIME_DIR, TMPDIR,
its own project copy/worktree and artifact directory,
its own build target/cache directory (or a read-only prebuilt cache + writable per-agent target),
its own service port (BRP, dev server, websocket),
a cleanup trap that kills the app, the X server and any helper processes.

Three tiers

Tier	Display	Renderer	Parallelism	Use
1. Logic/headless	none	none	high	unit tests, egui `Context::run`, data/actions
2. Visual UI	per-agent Xvfb	llvmpipe/lavapipe	2–4 agents	layout, dialogs, screenshots, 2D UI
3. Real GPU	one host/display/GPU	real GPU	1	PBR, real Bevy render, performance, final screenshot

Do not run Tier 3 in parallel with other visual agents. Do not use a shared desktop as Tier 2.

Per-agent Xvfb recipe

Requirements in the image/host: Xvfb, import (ImageMagick), optionally xdotool, openbox/fluxbox, Mesa llvmpipe and Vulkan lavapipe.

# Pick a free display; never hardcode one that another agent may use.
for n in $(seq 90 140); do
  [ ! -e "/tmp/.X11-unix/X$n" ] && DISPLAY_NUM=$n && break
done
SIZE=1440x900
ART=/tmp/agent-$DISPLAY_NUM

mkdir -p "$ART"/{home,config,runtime,tmp}
Xvfb ":$DISPLAY_NUM" -screen 0 "${SIZE}x24" -nolisten tcp >"$ART/xvfb.log" 2>&1 &
XVFB_PID=$!
APP_PID=""
cleanup() {
  [ -n "$APP_PID" ] && kill "$APP_PID" 2>/dev/null || true
  kill "$XVFB_PID" 2>/dev/null || true
  wait "$XVFB_PID" 2>/dev/null || true
}
trap cleanup EXIT

export DISPLAY=":$DISPLAY_NUM"
export HOME="$ART/home"
export XDG_CONFIG_HOME="$ART/config"
export XDG_RUNTIME_DIR="$ART/runtime"
export TMPDIR="$ART/tmp"
export LIBGL_ALWAYS_SOFTWARE=1
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json

# Start the app *after* the X socket exists. A small WM helps focus/IME.
sleep 0.5
# openbox &   # optional
timeout 120s <run command> --screenshot-mode "$ART/app.png" &
APP_PID=$!

# Prefer an in-app deterministic screenshot. Fallback: capture the root window.
wait "$APP_PID" || true
[ -f "$ART/app.png" ] || import -display ":$DISPLAY_NUM" -window root "$ART/fallback.png"

If the app has no screenshot mode, add one before the UI wave: fixed size, wait N frames, hide tooltips/animations/cursor, write PNG, exit 0. External root capture is a last resort.

Containers

Containers are useful for reproducibility and cleanup, not as a default for every iteration.

Build one pinned image with Rust, Xvfb, Mesa/lavapipe, ImageMagick, xdotool and a minimal WM.
Run one container per visual agent: --rm --network=none --userns=keep-id, no host $HOME, a read-only repo mount plus a writable worktree/artifact mount.
Keep package-manager/build caches in a named read-only volume; use a per-container target dir.
GPU: --device /dev/dri for Intel/AMD; NVIDIA needs the container toolkit/CDI. Prefer software rendering for UI tests and reserve GPU containers for one final Tier 3 run.
Limit visual containers to 2–4 concurrently; llvmpipe uses CPU aggressively.

Deterministic screenshots

Fixed window size and scale factor (1.0) for both small and target sizes, e.g. 1000×640 and 1440×900.
Wait for the UI to settle (textures loaded, panels laid out) before capture.
Disable tooltips, hover highlights, animations, autosave toasts and pointer-dependent chrome in screenshot mode; move the pointer out of the window.
Capture the app's own framebuffer, not the compositor. Portal capture is unreliable on Wayland and shared sessions.
Compare hashes for determinism and pixel-diff with a tolerance for antialiasing/GPU differences.
Store the exact command, window size and build commit next to each artifact.

Visual verifier procedure

A visual verifier should:

Launch the exact build in its own isolated display; never reuse another agent's window.
Capture at least the default size, the minimum supported size, empty/error states, and every modal that blocks input.
Check clipping/overflow, focus after closing a modal, keyboard shortcuts behind modals, selection visibility, contrast, and scroll reachability.
Record a screenshot path and the observed issue for every finding; "looks off" is not evidence.
For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence; dragging through a portal may collapse press/move/release into one frame.

Computer-use: last resort, never a test farm

If the runtime grants subagents an MCP computer_use tool, they can technically move a mouse and type. That does not make it a good regression mechanism.

Use computer-use only when all of these are true:

the deterministic tiers (headless tests, per-agent Xvfb + scripted screenshots) are already green;
the check is exploratory/OS-integration that cannot be scripted (native file dialog, IME, clipboard, drag from outside the app);
it runs on a dedicated disposable desktop/VM, not the user's live session;
exactly one agent uses it at a time;
secrets, notifications and personal data are not visible;
every action is logged and a screenshot artifact is saved.

Never:

give computer-use to every worker in a parallel wave;
use it as the only evidence for a pass;
use it for pixel/regression assertions;
let it click/type in the user's desktop without explicit permission;
assume an isolated Xvfb session is visible to a compositor/portal-based computer-use backend. If the backend targets KWin/Wayland, give it a nested desktop/VM it can actually see.

A computer-use result is qualitative input for a human or a vision reviewer, not a substitute for a command, an assertion or a deterministic screenshot.

Failure signs

Two visual jobs produce identical screenshots → they shared a display or captured the root.
A screenshot shows the wrong title/theme/build → focus or portal capture, not a UI bug.
Only the first visual job is reliable → the display/config isolation is incomplete.
A "pass" has no screenshot or command → treat it as not_run.

7.0 KiB Raw Permalink Blame History Unescape Escape