Emil_Shanaty/minecraft-builder-mcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add opt-in unattended camera capture without pause menu UI

Browse Source
This commit is contained in:
Emil
2026-09-21 23:35:46 +03:00
parent 224b29f0e7
commit 0fc4b154b6
8 changed files with 169 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
camera-mod/README.md
+28 -5
View File
@@ -29,7 +29,23 @@ export MCB_CAMERA_PORT=8766
Set the same secret in the Paper plugin's camera configuration. Paper accepts 32–512 characters from `A–Z`, `a–z`, `0–9`, `.`, `_`, `~`, and `-`, without spaces or line breaks; the automatically generated value already meets these requirements. All three Paper tokens must differ. Without `MCB_CAMERA_TOKEN`, the HTTP service is disabled. `MCB_CAMERA_PORT` is optional; allowed ports are 1024–65535. The address is always `127.0.0.1`, with no option to bind a public interface.
Launch a separate observer, connect to the intended Paper server, and switch it to spectator using an authorized server mechanism. Configure the observer's UUID in the Paper plugin. A valid separate game session is required if the builder stays on the server simultaneously; the mod does not bypass authentication or account restrictions. Keep client menus closed and do not spectate another entity. A minimized window may stop rendering and cause a timeout.
Launch a separate observer, connect to the intended Paper server, and switch it to spectator using an authorized server mechanism. Configure the observer's UUID in the Paper plugin. A valid separate game session is required if the builder stays on the server simultaneously; the mod does not bypass authentication or account restrictions. In the default interactive mode, keep client menus closed and do not spectate another entity. A minimized window may stop rendering and cause a timeout.
### Unattended observer
Set `MCB_CAMERA_UNATTENDED=1` **in the Minecraft child process environment** and restart that client. Only the literal value `1` enables this mode; it is off by default. Authenticated `GET /health` reports `unattended`, and completed captures include the same flag.
In this mode, an ordinary multiplayer pause menu may remain open and the window may be unfocused. During an active capture the mod clears and skips GUI extraction, including menu blur, HUD, chat, toasts and debug UI. It does not close the menu, grab the mouse, change focus, or change `pauseOnLostFocus`. The menu reappears normally after capture. The capture still hides the HUD and restores its prior visibility, FOV, bobbing, FOV effects, perspective and rotation on success or failure.
Only the exact vanilla `PauseScreen` is allowed. Other screens, including chat, settings, death, loading/resource-pack prompts, disconnect screens, modded pause-screen subclasses and real overlays, remain obstructions. Paused single-player worlds remain blocked. Spectator mode, the observer's own camera, position/view stability, authenticated loopback access, loading checks and the 20-second timeout remain required. If the graphics driver stops rendering, opting in cannot manufacture a fresh frame.
The mod does not create a display or launch another client. On Linux, use an existing virtual graphics display for the dedicated observer. Prism may forward launch requests to an already running launcher and inherit its real display; set the display in the instance's **WrapperCommand**, before Java starts. For example, with a prepared Xvfb display `:99`:
```text
env DISPLAY=:99 MCB_CAMERA_UNATTENDED=1 python3 /path/to/camera-wrapper.py
```
Use a wrapper/config that supplies the intended token and a separate port for each simultaneous observer. Verify the Java process's display and the authenticated health `playerId`/`unattended` before routing captures to it. Keep test captures and machine-specific launcher settings outside this repository.
### One client through Prism
@@ -65,7 +81,7 @@ When positioning through the vanilla console, write explicit decimal X/Z coordin
All requests, including health checks and image retrieval, require `Authorization: Bearer <MCB_CAMERA_TOKEN>`. JSON is not written to the log.
- `GET /health` — cached state from the latest client tick: `status`, `connected`, `spectator`, `busy`, `dimension`, `playerId`, and `updatedAt`. A stale `updatedAt` means the client has stopped updating.
- `GET /health` — cached state from the latest client tick: `status`, `connected`, `spectator`, `busy`, `unattended`, `dimension`, `playerId`, and `updatedAt`. A stale `updatedAt` means the client has stopped updating.
- `POST /v1/capture` — queue one capture. HTTP 202 response: `{"status":"pending","captureId":"<uuid>"}`. A busy camera returns HTTP 409 and `camera_busy`.
- `GET /v1/captures/<uuid>` — retrieve `pending`, `completed`, or `error`. Unknown/expired IDs return HTTP 404. A terminal `error` contains `error` and `message`, without an image.
@@ -97,15 +113,15 @@ Before capture, the mod checks spectator mode, position agreement (±0.05 blocks
2. Three frames with an initialized camera, available chunks, and an empty geometry preparation queue.
3. The pose and window to remain suitable until framebuffer readback.
PNG capture uses `Screenshot.takeScreenshot` after the frame renders, with GPU readback through Blaze3D and no direct OpenGL code. PNG encoding and downscaling run on a separate thread. The HUD, FOV, perspective, view bobbing, and rotation saved when the mod starts handling the capture are restored on the client thread after success or error. Saving starts after the server position arrives; this does not return the player to the position before teleportation. The server remains authoritative over the post-teleport position.
PNG capture uses `Screenshot.takeScreenshot` after the frame renders, with GPU readback through Blaze3D and no direct OpenGL code. Unattended captures count only frames whose GUI extraction was actually suppressed. PNG encoding and downscaling run on a separate thread. The HUD, FOV, perspective, view bobbing, and rotation saved when the mod starts handling the capture are restored on the client thread after success or error. Saving starts after the server position arrives; this does not return the player to the position before teleportation. The server remains authoritative over the post-teleport position.
This is a **checked loading heuristic**, not confirmation of a specific server revision. The response always contains `readiness: "local_chunks_and_render_queue_stable"` and `serverRevisionVerified: false`. `afterOperationId` is for correlation; by itself it does not prove the client received every update from that operation. Do not present this result as revision verification. Strict freshness requires an additional server marker and acknowledgment that the corresponding packets were processed. Distant geometry outside the checked chunks and changes after capture remain limitations.
Loading failures, disconnection, world/position changes, open menus, rotation interference, and missing framebuffer data return an error instead of a stale image. A separate thread enforces the 20-second timeout even if rendering hangs. Another capture is allowed after state restoration on the client thread. At most four results are retained for up to two minutes; PNGs are limited to 8 MiB and request bodies to 8192 bytes. Images remain in memory and are not written to the shared screenshots directory.
Loading failures, disconnection, world/position changes, blocking menus, rotation interference, and missing framebuffer data return an error instead of a stale image. All open menus block the default interactive mode; only the unattended multiplayer pause menu is exempt. A separate thread enforces the 20-second timeout even if rendering hangs. Another capture is allowed after state restoration on the client thread. At most four results are retained for up to two minutes; PNGs are limited to 8 MiB and request bodies to 8192 bytes. Images remain in memory and are not written to the shared screenshots directory.
## Verification and prototype limits
`./gradlew build` compiles the mod against real Minecraft 26.2 dependencies; tests cover HTTP bearer authentication, request-size limits, and parameter validation. These tests do not launch the client or sign in to an account.
`./gradlew build` compiles the mod against real Minecraft 26.2 dependencies; 17 tests cover HTTP bearer authentication, token/request-size limits, parameter validation, auto-connect and the attended/unattended obstruction policy. These unit tests do not launch the client or sign in to an account.
A real graphical test ran on September 12, 2026: one Prism client, the project owner in spectator mode, matching owner and camera UUIDs, Paper 26.2, and a completed 575-block tower. Mixin loading, client connection, server teleportation, framebuffer readback, and PNG delivery through Paper HTTP were verified. The image shows the built tower without the HUD.
@@ -120,3 +136,10 @@ Set `camera-auto-connect: '127.0.0.1:25575'` in the private Paper config used by
Create `config/minecraft-builder-camera.autojoin-disabled` inside the Minecraft instance to pause retries without restarting; remove it to resume. The health endpoint reports enabled/paused state, target and attempt count. Remove the opt-in setting and restart the client to disable permanently. Opt-in deliberately includes reconnecting after a manual disconnect from the configured server.
Auto-connect uses Minecraft's normal connection flow. It does not authenticate new accounts, launch Prism, execute chat commands, change game mode, move the player, or grant editing rights. A trusted local operator can separately use the Paper console for `lobby <player>` and `gamemode spectator <player>`. Graphical rendering is still required for photographs; menus and user input can invalidate a capture.
### Unattended local verification (2026-09-21)
A separate real Minecraft 26.2 spectator client captured a 1280×720 PNG on a virtual display while its window was unfocused and the vanilla pause menu remained open. The image was visually checked for a genuine world frame without menu, HUD or chat. In multiplayer `isPaused()` was false, even with `PauseScreen` open; the menu state, not an assumed paused simulation, was exercised.
Both initially visible and initially hidden HUD states were restored. FOV, view bobbing, FOV effects, perspective and rotation returned to their saved values. A rotation change and a blocking screen introduced during capture produced explicit errors and still restored view settings. The original pause menu, focus state and `pauseOnLostFocus` were left untouched. With unattended mode disabled, the same paused/unfocused client returned `view_obstructed`; after its menu was closed, the default mode still produced a clean 1280×720 image. Machine-specific fixtures, images and receipts stay outside the repository in the local test workspace.