Document computer-use as a last-resort visual tier

2026-10-27 19:29:50 +03:00
parent 20497d9c0c
commit 74fdfa458c
5 changed files with 32 additions and 2 deletions
@@ -34,6 +34,7 @@
 - `blocked`-воркер не заменяется: лид правит контракт/shared-файл и продолжает того же агента через `send_message`.
 - Evidence зависит от домена: команда и тест для кода, источник и дата для research, отрендеренный артефакт для дизайна, reproducible metric для данных.
 - Визуальные агенты работают в отдельных Xvfb/display-окружениях, со своим HOME/config/копией проекта; общий рабочий стол для скриншотов не используется. Детали — в `references/visual-testing.md`.
+- Computer-use — крайний tier: один агент, одноразовый desktop/VM, только после детерминированных проверок и никогда как единственное evidence.
 - Стоимость: 30–40 агентов ≈ 0.5–1.5M токенов.
 - Для задачи, которую один агент делает за 15 минут, swarm не нужен.

@@ -59,7 +59,7 @@ Otherwise: one agent for small work; 2–5 `subagent` calls for a few long-lived
 6. **Respect the runtime slot cap.** Active children include recon, verifiers, integrators and any other subagents. A spawn-limit error is a scheduling bug, not a worker failure: shrink the batch, keep completed results, and dispatch the remainder.
 7. **Freeze during verification.** Once the verify wave starts, do not edit source/tests/docs except to fix a verified finding. After a fix, re-run the affected verifier; an old review is stale.
 8. **Stop rules.** Two waves with no net progress → stop and re-plan. Verification failure rate > 30% → fix briefs/contracts, not agent count. Budget exhausted → report partial and list unverified areas.
-9. **No shared visual desktop.** Visual agents get their own display/config/project copy; see `references/visual-testing.md`. A screenshot from a shared desktop is not evidence.
+9. **No shared visual desktop.** Visual agents get their own display/config/project copy; see `references/visual-testing.md`. A screenshot from a shared desktop is not evidence. Computer-use is a last-resort, single-agent tier on a disposable desktop/VM; it is never the sole evidence and never runs in parallel with other visual agents.

 ## Sizing

@@ -28,6 +28,8 @@ The tree is frozen during verification. Do not edit product source, tests, docs
 ## Visual checks
 For GUI products, run the app in your own isolated Xvfb/display (never a shared desktop). Capture the default size, the minimum size, empty/error states and every modal. Record the screenshot path and exact command alongside each finding; a screenshot without a command is not evidence.

+Computer-use (mouse/keyboard) is an optional last-resort tier: one agent, disposable desktop/VM, after deterministic checks, never parallel, never the sole evidence. Prefer app screenshot mode or scripted input.
+
 ## Verdict
 `pass` / `pass_with_risks` / `fail` — with the top risks and exact repro commands.

@@ -139,7 +139,7 @@ If the block reveals an underspecified interface shared by several tasks, update

 Choose verifiers by risk cluster, not one per task: 4–6 verifiers for 8–12 tasks. A verifier gets the task cards, the changed files (or diff), and the acceptance commands. It does not get the worker's rationale and does not fix anything.

-For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence.
+For a GUI product, give each visual verifier its own Xvfb/display and project copy (see `references/visual-testing.md`). Capture at least the default size, the minimum supported size, empty/error states and every modal; a shared-desktop screenshot is not evidence. Computer-use is a final, single-agent exploratory tier on a disposable desktop/VM — never a parallel test mechanism and never the only evidence.

 **Freeze before verify.** Once you start this wave, do not edit source, tests or docs. If you already started and then fixed something, mark the in-flight review stale and re-run that verifier on the new build. A verifier's verdict is only valid for the exact tree it saw.

@@ -119,6 +119,33 @@ A visual verifier should:
 5. For animation/drag interactions, use a scripted headless harness or a video/screenshot sequence;
   dragging through a portal may collapse press/move/release into one frame.

+## Computer-use: last resort, never a test farm
+
+If the runtime grants subagents an MCP `computer_use` tool, they can technically move a mouse and
+type. That does **not** make it a good regression mechanism.
+
+Use computer-use only when all of these are true:
+
+- the deterministic tiers (headless tests, per-agent Xvfb + scripted screenshots) are already green;
+- the check is exploratory/OS-integration that cannot be scripted (native file dialog, IME, clipboard,
+  drag from outside the app);
+- it runs on a dedicated disposable desktop/VM, not the user's live session;
+- exactly **one** agent uses it at a time;
+- secrets, notifications and personal data are not visible;
+- every action is logged and a screenshot artifact is saved.
+
+Never:
+
+- give computer-use to every worker in a parallel wave;
+- use it as the only evidence for a pass;
+- use it for pixel/regression assertions;
+- let it click/type in the user's desktop without explicit permission;
+- assume an isolated Xvfb session is visible to a compositor/portal-based computer-use backend. If
+  the backend targets KWin/Wayland, give it a nested desktop/VM it can actually see.
+
+A computer-use result is qualitative input for a human or a vision reviewer, not a substitute for a
+command, an assertion or a deterministic screenshot.
+
 ## Failure signs

 - Two visual jobs produce identical screenshots → they shared a display or captured the root.