# Visual testing for GUI agents

GUI products fail verification for reasons that unit tests cannot see: clipped panels, focus-stealing
modals, dialogs that do not close, wrong DPI, a viewport that renders a blank background, or a
screenshot that captures a different agent's window. Treat visual verification as a separate test
tier with its own infrastructure.

The commands below use Linux Xvfb + ImageMagick as one concrete provider. On macOS/Windows use a
container or the platform's virtual-display tooling; the invariant is **isolation and deterministic
capture**, not Xvfb itself. If the app can capture its own framebuffer, that is preferred on every
platform.

## Rule zero: never share one desktop

Do not run several visual agents against the same X11/Wayland session. They steal focus, overlap
windows, race on the clipboard/configuration, and screenshot portals may return another window or
the desktop wallpaper. A common failure mode is "the screenshot looks fine" while it is a different
build's window.

Each visual agent gets:

- its own display (`DISPLAY=:N` / Wayland socket),
- its own `HOME`, `XDG_CONFIG_HOME`, `XDG_RUNTIME_DIR`, `TMPDIR`,
- its own project copy/worktree and artifact directory,
- its own build target/cache directory (or a read-only prebuilt cache + writable per-agent target),
- its own service port (BRP, dev server, websocket),
- a cleanup trap that kills the app, the X server and any helper processes.

## Three tiers

| Tier | Display | Renderer | Parallelism | Use |
|---|---|---|---|---|
| 1. Logic/headless | none | none | high | unit tests, egui `Context::run`, data/actions |
| 2. Visual UI | per-agent Xvfb | llvmpipe/lavapipe | 2–4 agents | layout, dialogs, screenshots, 2D UI |
| 3. Real GPU | one host/display/GPU | real GPU | **1** | PBR, real Bevy render, performance, final screenshot |

Do not run Tier 3 in parallel with other visual agents. Do not use a shared desktop as Tier 2.

## Per-agent Xvfb recipe

Requirements in the image/host: `Xvfb`, `import` (ImageMagick), optionally `xdotool`,
`openbox`/`fluxbox`, Mesa llvmpipe and Vulkan lavapipe.

```sh
# Pick a free display; never hardcode one that another agent may use.
for n in $(seq 90 140); do
  [ ! -e "/tmp/.X11-unix/X$n" ] && DISPLAY_NUM=$n && break
done
SIZE=1440x900
ART=/tmp/agent-$DISPLAY_NUM

mkdir -p "$ART"/{home,config,runtime,tmp}
Xvfb ":$DISPLAY_NUM" -screen 0 "${SIZE}x24" -nolisten tcp >"$ART/xvfb.log" 2>&1 &
XVFB_PID=$!
APP_PID=""
cleanup() {
  [ -n "$APP_PID" ] && kill "$APP_PID" 2>/dev/null || true
  kill "$XVFB_PID" 2>/dev/null || true
  wait "$XVFB_PID" 2>/dev/null || true
}
trap cleanup EXIT

export DISPLAY=":$DISPLAY_NUM"
export HOME="$ART/home"
export XDG_CONFIG_HOME="$ART/config"
export XDG_RUNTIME_DIR="$ART/runtime"
export TMPDIR="$ART/tmp"
export LIBGL_ALWAYS_SOFTWARE=1
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json

# Start the app *after* the X socket exists. A small WM helps focus/IME.
sleep 0.5
# openbox &   # optional
timeout 120s <run command> --screenshot-mode "$ART/app.png" &
APP_PID=$!

# Prefer an in-app deterministic screenshot. Fallback: capture the root window.
wait "$APP_PID" || true
[ -f "$ART/app.png" ] || import -display ":$DISPLAY_NUM" -window root "$ART/fallback.png"
```

If the app has no screenshot mode, add one before the UI wave: fixed size, wait N frames, hide
tooltips/animations/cursor, write PNG, exit 0. External root capture is a last resort.

## Containers

Containers are useful for reproducibility and cleanup, not as a default for every iteration.

- Build one pinned image with Rust, Xvfb, Mesa/lavapipe, ImageMagick, xdotool and a minimal WM.
- Run one container per visual agent: `--rm --network=none --userns=keep-id`, no host `$HOME`, a
  read-only repo mount plus a writable worktree/artifact mount.
- Keep package-manager/build caches in a named read-only volume; use a per-container target dir.
- GPU: `--device /dev/dri` for Intel/AMD; NVIDIA needs the container toolkit/CDI. Prefer software
  rendering for UI tests and reserve GPU containers for one final Tier 3 run.
- Limit visual containers to 2–4 concurrently; llvmpipe uses CPU aggressively.

## Deterministic screenshots

- Fixed window size and scale factor (`1.0`) for both small and target sizes, e.g. 1000×640 and
  1440×900.
- Wait for the UI to settle (textures loaded, panels laid out) before capture.
- Disable tooltips, hover highlights, animations, autosave toasts and pointer-dependent chrome in
  screenshot mode; move the pointer out of the window.
- Capture the app's own framebuffer, not the compositor. Portal capture is unreliable on Wayland and
  shared sessions.
- Compare hashes for determinism and pixel-diff with a tolerance for antialiasing/GPU differences.
- Store the exact command, window size and build commit next to each artifact.

## Visual verifier procedure

A visual verifier should:

1. Launch the exact build in its own isolated display; never reuse another agent's window.
2. Capture at least the default size, the minimum supported size, empty/error states, and every
   modal that blocks input.
3. Check clipping/overflow, focus after closing a modal, keyboard shortcuts behind modals, selection
   visibility, contrast, and scroll reachability.
4. Record a screenshot path and the observed issue for every finding; "looks off" is not evidence.
5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence;
   dragging through a portal may collapse press/move/release into one frame.

## Failure signs

- Two visual jobs produce identical screenshots → they shared a display or captured the root.
- A screenshot shows the wrong title/theme/build → focus or portal capture, not a UI bug.
- Only the first visual job is reliable → the display/config isolation is incomplete.
- A "pass" has no screenshot or command → treat it as `not_run`.