Emil_Shanaty/skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document computer-use as a last-resort visual tier

Browse Source
This commit is contained in:
Emil Shanaty
2026-10-27 19:29:50 +03:00
parent 20497d9c0c
commit 74fdfa458c
5 changed files with 32 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent-swarm/references/playbook.md
+1 -1
View File
@@ -139,7 +139,7 @@ If the block reveals an underspecified interface shared by several tasks, update
Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.
For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence.
For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence. Computer-use is a final, single-agent exploratory tier on a disposable desktop/VM — never a parallel test mechanism and never the only evidence.
**Freeze before verify.** Once you start this wave, do not edit source, tests or docs. If you already started and then fixed something, mark the in-flight review stale and re-run that verifier on the new build. A verifier's verdict is only valid for the exact tree it saw.
agent-swarm/references/visual-testing.md
+27
View File
@@ -119,6 +119,33 @@ A visual verifier should:
5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence;
dragging through a portal may collapse press/move/release into one frame.
## Computer-use: last resort, never a test farm
If the runtime grants subagents an MCP `computer_use` tool, they can technically move a mouse and
type. That does **not** make it a good regression mechanism.
Use computer-use only when all of these are true:
- the deterministic tiers (headless tests, per-agent Xvfb + scripted screenshots) are already green;
- the check is exploratory/OS-integration that cannot be scripted (native file dialog, IME, clipboard,
drag from outside the app);
- it runs on a dedicated disposable desktop/VM, not the user's live session;
- exactly **one** agent uses it at a time;
- secrets, notifications and personal data are not visible;
- every action is logged and a screenshot artifact is saved.
Never:
- give computer-use to every worker in a parallel wave;
- use it as the only evidence for a pass;
- use it for pixel/regression assertions;
- let it click/type in the user's desktop without explicit permission;
- assume an isolated Xvfb session is visible to a compositor/portal-based computer-use backend. If
the backend targets KWin/Wayland, give it a nested desktop/VM it can actually see.
A computer-use result is qualitative input for a human or a vision reviewer, not a substitute for a
command, an assertion or a deterministic screenshot.
## Failure signs
- Two visual jobs produce identical screenshots → they shared a display or captured the root.