188 lines
8.4 KiB
Markdown
188 lines
8.4 KiB
Markdown
# aicc-capsule testbed
|
|
|
|
Reference implementation of the [AICC Protocol](https://github.com/emil28092005/AICC-Protocol)
|
|
as a 3D room with an AI-controlled capsule. An agent (any LLM with tool
|
|
calling, or the bundled scripted agent) connects over WebSocket, perceives the
|
|
room through sensors, walks to a glowing beacon, and activates it — every step
|
|
over the protocol, no engine hooks.
|
|
|
|
```
|
|
agent (LLM) <--AICC over WebSocket--> RoomBridge <--> Room (world + renderer)
|
|
(testbed/demo.py) (testbed/bridge.py) (testbed/room/)
|
|
```
|
|
|
|
## Layout
|
|
|
|
| Path | Purpose |
|
|
|-------------------------|--------------------------------------------------------------|
|
|
| `room/world.py` | World state: 16x16 room, capsule physics, crates, beacon, audio. Single source of truth. |
|
|
| `room/render.py` | Headless first-person raycaster (Pillow): honest frames from world state. |
|
|
| `room/mapview.py` | Top-down map drawn from sensor data (shared by recorder + live viewer). |
|
|
| `bridge.py` | `RoomBridge(Bridge)`: registers all tools, emits events. |
|
|
| `server.py` entry | `python -m testbed.bridge` — WebSocket server. |
|
|
| `conformance.py` | Runs the 9 core conformance scenarios against this bridge. |
|
|
| `live.py` | Real-time browser viewer (itself an AICC client). |
|
|
| `chat.py` | Interactive chat: natural language -> tool calls. |
|
|
| `llm_agent.py` | Shared LLM driver (controller + autonomous loop). |
|
|
| `demo.py` | Agent demo: LLM driver (OpenAI-compatible) or scripted. |
|
|
| `tests/` | pytest suite (world, renderer, bridge, protocol). |
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
scripts/setup.sh # venv + aicc-py + pillow/websockets/openai
|
|
```
|
|
|
|
The testbed needs the `aicc` SDK installed from `~/Desktop/aicc-py` (the setup
|
|
script does `pip install -e`). The launcher scripts below use the venv
|
|
interpreter directly, so `python` does not need to be on your PATH. To run
|
|
commands by hand instead, activate the venv first:
|
|
`source testbed/.venv/bin/activate`.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
# 1. Start the bridge (headless; keep it running in its own terminal)
|
|
scripts/run_bridge.sh # ws://127.0.0.1:8765
|
|
|
|
# 2. Run the demo — default `auto` tries the LLM, then hands off to the
|
|
# scripted agent so the run always completes
|
|
scripts/run_demo.sh --agent auto
|
|
|
|
# scripted only (deterministic, no LLM needed)
|
|
scripts/run_demo.sh --agent scripted
|
|
|
|
# LLM only (any OpenAI-compatible endpoint; ollama by default)
|
|
scripts/run_demo.sh --agent llm --model gemma4:e2b
|
|
scripts/run_demo.sh --agent llm \
|
|
--base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY
|
|
```
|
|
|
|
If the bridge is already running, `run_bridge.sh` will say so (the port is
|
|
taken); stop the old one with `fuser -k 8765/tcp` or Ctrl-C in its terminal.
|
|
|
|
The demo prints a full transcript of tool calls/results to stdout and saves the
|
|
capsule's final first-person frame to `demo_final_frame.png`.
|
|
|
|
## Interactive chat mode
|
|
|
|
Talk to the capsule's brain in natural language (any language):
|
|
|
|
```bash
|
|
scripts/run_chat.sh --model gemma4:e2b # small + fast
|
|
scripts/run_chat.sh --model gemma4:12b # bigger gemma, slower (~20 s/turn)
|
|
```
|
|
|
|
It starts the bridge, the live viewer and a chat REPL — open
|
|
[http://127.0.0.1:8000](http://127.0.0.1:8000) to watch the capsule while you
|
|
type. The model translates your words into tool calls:
|
|
|
|
```
|
|
you> иди к маяку → look_at + move step by step (auto-continue)
|
|
you> повернись налево → turn(-90)
|
|
you> осмотрись → vision + description
|
|
you> активируй маяк → interact (when close)
|
|
you> /state /look /map /models /model gemma4:12b /steps N /help /exit
|
|
```
|
|
|
|
Each turn's transcript is printed; the current frame lands in `chat_frame.png`
|
|
and the sensor-built map in `chat_map.png`. Any OpenAI-compatible endpoint
|
|
works: `python -m testbed.chat --base-url https://api.openai.com/v1
|
|
--model gpt-4o-mini --api-key $OPENAI_API_KEY`. `--auto-steps N` controls how
|
|
many tool steps the model may chain per request (0 = one action per turn).
|
|
|
|
## Real-time mode
|
|
|
|
Watch the capsule drive live in your browser:
|
|
|
|
```bash
|
|
scripts/run_live.sh --agent scripted
|
|
```
|
|
|
|
This starts the bridge, a viewer server, and the demo; open
|
|
[http://127.0.0.1:8000](http://127.0.0.1:8000) while the agent acts. The page
|
|
shows the first-person frame (`vision`) and a top-down map (`world_query` +
|
|
`proprioception`) updating a few times per second, with the capsule's path,
|
|
heading, and distance to the beacon.
|
|
|
|
The viewer (`testbed/live.py`) is itself a plain AICC client — it sees the
|
|
world only through the protocol sensors, so it works against any bridge, not
|
|
just this one. You can also run it standalone:
|
|
|
|
```bash
|
|
testbed/.venv/bin/python -m testbed.live # then run the demo in another terminal
|
|
```
|
|
|
|
## Visual mode
|
|
|
|
```bash
|
|
scripts/run_demo.sh --agent scripted --frames-dir frames
|
|
```
|
|
|
|
Saves, for every step, the first-person frame (`step_NNN_view.png`) and a
|
|
top-down map of the room with the capsule's path (`step_NNN_map.png`), then
|
|
writes a `demo.gif` animation and a `demo_summary.png` (final map + last
|
|
view). The map is rebuilt purely from sensor data (`world_query`,
|
|
`proprioception`, `vision`) — the same view the agent itself has. Generated
|
|
examples are committed at the repo root (`demo.gif`, `demo_summary.png`).
|
|
|
|
## Conformance
|
|
|
|
```bash
|
|
testbed/.venv/bin/python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios
|
|
# [PASS] core-01..core-09 -> 9/9 scenarios passed
|
|
```
|
|
|
|
The suite runs against the *same* bridge class used by the server and demo.
|
|
|
|
## Tools
|
|
|
|
Registered in the manifest (sensors first, then actuators):
|
|
|
|
| id | class | purpose |
|
|
|-----------------|-----------|----------------------------------------------------|
|
|
| `proprioception`| sensor | position, rotation, velocity, health, tick |
|
|
| `vision` | sensor | first-person RGB frame as base64 PNG (160x120) |
|
|
| `depth` | sensor | aligned depth map (40x30, meters) |
|
|
| `hear` | sensor | audio since last call: beacon hum, collision thuds |
|
|
| `world_query` | sensor | room bounds, obstacle layout, beacon position |
|
|
| `move` | actuator | move forward N meters, collision-aware |
|
|
| `turn` | actuator | rotate yaw/pitch |
|
|
| `look_at` | actuator | aim camera at a named target (`beacon`) |
|
|
| `interact` | actuator | activate the beacon within reach |
|
|
| `echo`/`boom`/`bump` | — | conformance tools (design doc requirement) |
|
|
|
|
World data flows only through sensors: the manifest carries session metadata
|
|
and tool schemas, never world state (protocol §7, single source of truth).
|
|
|
|
## Tick model
|
|
|
|
`tick_mode` is `event`: the world advances one tick per tool call, so every
|
|
observation and event shares a monotonic `tick`. Collision events
|
|
(`topic: collision`, payload `{other, normal, impulse}`) are pushed
|
|
asynchronously when `move` hits a wall or crate.
|
|
|
|
## Demo agents
|
|
|
|
- **LLM agent** (`--agent llm`): generic tool-use loop — manifest tools are
|
|
converted to OpenAI function schemas; every response is executed via
|
|
`AICCClient.call_tool` and fed back as a `tool` message. Vision frames are
|
|
decoded into a coarse color grid so text-only models can navigate. The loop
|
|
keeps a compact `CURRENT STATE` note (agent-side working memory, protocol
|
|
§9) and gently corrects a model that drifts: nudge after text-only replies,
|
|
re-aim corrections when the capsule moves away or faces the wrong way,
|
|
collision guidance. Works with any OpenAI-compatible endpoint (ollama,
|
|
vLLM, OpenAI, ...). Model quality varies — a capable model completes on its
|
|
own; a weak local model may hand off (see `auto`).
|
|
- **Scripted agent** (`--agent scripted`): deterministic bug-algorithm robot —
|
|
sensor-driven steering toward the beacon with detour-on-collision. No LLM,
|
|
always completes. Used as the reference/fallback.
|
|
- **Auto** (`--agent auto`): tries the LLM (bounded steps), then hands off to
|
|
the scripted agent so a demo run always ends with an activated beacon.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
testbed/.venv/bin/python -m pytest -q # 33 tests: physics, renderer, tools, protocol
|
|
```
|