diff --git a/README.md b/README.md index 335e479..042f791 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,11 @@ A 3D room with a capsule that an LLM agent controls through AICC sensor/actuator ## Status -**Not started.** This directory is the context pack for building it. The spec (core.md) and JSON Schema are in `docs/`. The Python SDK (`aicc-py`) is a sibling project at `~/Desktop/aicc-py` — the testbed should build on top of it. +**Working.** The testbed is built: headless 3D room with a capsule, a +protocol-conformant bridge (`9/9` conformance scenarios), and a demo in which +an agent (LLM or scripted) navigates to the beacon and activates it. See +`testbed/README.md` for run instructions; `scripts/BUILDLOG.md` for the build +log. ## Layout diff --git a/scripts/BUILDLOG.md b/scripts/BUILDLOG.md index 6d08ade..0881e52 100644 --- a/scripts/BUILDLOG.md +++ b/scripts/BUILDLOG.md @@ -1,9 +1,9 @@ # aicc-capsule build log -# Fill this in as you go. Keep it short — one line per milestone. -- [ ] 2026-08: context pack created (AGENTS.md, docs, scripts) -- [ ] bridge skeleton: world + capsule + WebSocketServer -- [ ] tools: proprioception, vision, move, turn, look_at, interact (+ echo/boom/bump for conformance) -- [ ] conformance 9/9 green -- [ ] agent demo loop (LLM drives capsule to beacon, interacts) -- [ ] README in testbed/ with run instructions +- [x] 2026-08: context pack created (AGENTS.md, docs, scripts) +- [x] 2026-08: bridge skeleton: world (room + capsule physics + beacon) + headless raycaster + WebSocketServer +- [x] 2026-08: tools: proprioception, vision, depth, hear, world_query, move, turn, look_at, interact (+ echo/boom/bump for conformance) +- [x] 2026-08: conformance 9/9 green against the testbed bridge +- [x] 2026-08: pytest suite (33 tests: world physics, renderer, bridge tools, protocol) +- [x] 2026-08: agent demo loop: LLM (OpenAI-compatible, ollama default) + scripted fallback + auto handoff; E2E verified over WebSocket +- [x] 2026-08: README in testbed/ with run instructions, setup.sh deps, run_bridge.sh diff --git a/scripts/setup.sh b/scripts/setup.sh index be7efc7..66c1fb3 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -8,7 +8,11 @@ source testbed/.venv/bin/activate pip install --upgrade pip # aicc-py SDK from sibling checkout pip install -e "$HOME/Desktop/aicc-py" -# Engine and demo deps go here (e.g. ursina, panda3d, pillow, websockets) -# pip install ... +# Engine + demo deps: Pillow (headless raycaster renderer), websockets (transport), +# openai (LLM demo client), pytest (test suite) +pip install pillow websockets openai pytest pytest-asyncio echo "Setup done. Activate with: source testbed/.venv/bin/activate" +echo "Run the bridge: scripts/run_bridge.sh" +echo "Run the demo: testbed/.venv/bin/python -m testbed.demo --agent auto" +echo "Conformance: testbed/.venv/bin/python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios" diff --git a/testbed/README.md b/testbed/README.md new file mode 100644 index 0000000..ebeceaf --- /dev/null +++ b/testbed/README.md @@ -0,0 +1,116 @@ +# aicc-capsule testbed + +Reference implementation of the [AICC Protocol](https://github.com/emil28092005/AICC-Protocol) +as a 3D room with an AI-controlled capsule. An agent (any LLM with tool +calling, or the bundled scripted agent) connects over WebSocket, perceives the +room through sensors, walks to a glowing beacon, and activates it — every step +over the protocol, no engine hooks. + +``` + agent (LLM) <--AICC over WebSocket--> RoomBridge <--> Room (world + renderer) + (testbed/demo.py) (testbed/bridge.py) (testbed/room/) +``` + +## Layout + +| Path | Purpose | +|-------------------------|--------------------------------------------------------------| +| `room/world.py` | World state: 16x16 room, capsule physics, crates, beacon, audio. Single source of truth. | +| `room/render.py` | Headless first-person raycaster (Pillow): honest frames from world state. | +| `bridge.py` | `RoomBridge(Bridge)`: registers all tools, emits events. | +| `server.py` entry | `python -m testbed.bridge` — WebSocket server. | +| `conformance.py` | Runs the 9 core conformance scenarios against this bridge. | +| `demo.py` | Agent demo: LLM driver (OpenAI-compatible) or scripted. | +| `tests/` | pytest suite (world, renderer, bridge, protocol). | + +## Setup + +```bash +scripts/setup.sh # venv + aicc-py + pillow/websockets/openai +source testbed/.venv/bin/activate +``` + +The testbed needs the `aicc` SDK installed from `~/Desktop/aicc-py` (the setup +script does `pip install -e`). + +## Run + +```bash +# 1. Start the bridge (headless) +scripts/run_bridge.sh # ws://127.0.0.1:8765 + +# 2. Run the demo — default `auto` tries the LLM, then hands off to the +# scripted agent so the run always completes +python -m testbed.demo --agent auto + +# scripted only (deterministic, no LLM needed) +python -m testbed.demo --agent scripted + +# LLM only (any OpenAI-compatible endpoint; ollama by default) +python -m testbed.demo --agent llm --model gemma4:e2b +python -m testbed.demo --agent llm \ + --base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY +``` + +The demo prints a full transcript of tool calls/results to stdout and saves the +capsule's final first-person frame to `demo_final_frame.png`. + +## Conformance + +```bash +python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios +# [PASS] core-01..core-09 -> 9/9 scenarios passed +``` + +The suite runs against the *same* bridge class used by the server and demo. + +## Tools + +Registered in the manifest (sensors first, then actuators): + +| id | class | purpose | +|-----------------|-----------|----------------------------------------------------| +| `proprioception`| sensor | position, rotation, velocity, health, tick | +| `vision` | sensor | first-person RGB frame as base64 PNG (160x120) | +| `depth` | sensor | aligned depth map (40x30, meters) | +| `hear` | sensor | audio since last call: beacon hum, collision thuds | +| `world_query` | sensor | room bounds, obstacle layout, beacon position | +| `move` | actuator | move forward N meters, collision-aware | +| `turn` | actuator | rotate yaw/pitch | +| `look_at` | actuator | aim camera at a named target (`beacon`) | +| `interact` | actuator | activate the beacon within reach | +| `echo`/`boom`/`bump` | — | conformance tools (design doc requirement) | + +World data flows only through sensors: the manifest carries session metadata +and tool schemas, never world state (protocol §7, single source of truth). + +## Tick model + +`tick_mode` is `event`: the world advances one tick per tool call, so every +observation and event shares a monotonic `tick`. Collision events +(`topic: collision`, payload `{other, normal, impulse}`) are pushed +asynchronously when `move` hits a wall or crate. + +## Demo agents + +- **LLM agent** (`--agent llm`): generic tool-use loop — manifest tools are + converted to OpenAI function schemas; every response is executed via + `AICCClient.call_tool` and fed back as a `tool` message. Vision frames are + decoded into a coarse color grid so text-only models can navigate. The loop + keeps a compact `CURRENT STATE` note (agent-side working memory, protocol + §9) and gently corrects a model that drifts: nudge after text-only replies, + re-aim corrections when the capsule moves away or faces the wrong way, + collision guidance. Works with any OpenAI-compatible endpoint (ollama, + vLLM, OpenAI, ...). Model quality varies — a capable model completes on its + own; a weak local model may hand off (see `auto`). +- **Scripted agent** (`--agent scripted`): deterministic bug-algorithm robot — + sensor-driven steering toward the beacon with detour-on-collision. No LLM, + always completes. Used as the reference/fallback. +- **Auto** (`--agent auto`): tries the LLM (bounded steps), then hands off to + the scripted agent so a demo run always ends with an activated beacon. + +## Tests + +```bash +python -m pytest -q # 33 tests: physics, renderer, tools, protocol +```