Document computer-use as a last-resort visual tier

2026-10-27 19:29:50 +03:00
parent 20497d9c0c
commit 74fdfa458c
5 changed files with 32 additions and 2 deletions
@@ -119,6 +119,33 @@ A visual verifier should:
 5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence;
   dragging through a portal may collapse press/move/release into one frame.

+## Computer-use: last resort, never a test farm
+
+If the runtime grants subagents an MCP `computer_use` tool, they can technically move a mouse and
+type. That does **not** make it a good regression mechanism.
+
+Use computer-use only when all of these are true:
+
+- the deterministic tiers (headless tests, per-agent Xvfb + scripted screenshots) are already green;
+- the check is exploratory/OS-integration that cannot be scripted (native file dialog, IME, clipboard,
+  drag from outside the app);
+- it runs on a dedicated disposable desktop/VM, not the user's live session;
+- exactly **one** agent uses it at a time;
+- secrets, notifications and personal data are not visible;
+- every action is logged and a screenshot artifact is saved.
+
+Never:
+
+- give computer-use to every worker in a parallel wave;
+- use it as the only evidence for a pass;
+- use it for pixel/regression assertions;
+- let it click/type in the user's desktop without explicit permission;
+- assume an isolated Xvfb session is visible to a compositor/portal-based computer-use backend. If
+  the backend targets KWin/Wayland, give it a nested desktop/VM it can actually see.
+
+A computer-use result is qualitative input for a human or a vision reviewer, not a substitute for a
+command, an assertion or a deterministic screenshot.
+
 ## Failure signs

 - Two visual jobs produce identical screenshots → they shared a display or captured the root.