docs: testbed README (run/agent/conformance), setup.sh deps, build log, project status
This commit is contained in:
@@ -8,7 +8,11 @@ A 3D room with a capsule that an LLM agent controls through AICC sensor/actuator
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
**Not started.** This directory is the context pack for building it. The spec (core.md) and JSON Schema are in `docs/`. The Python SDK (`aicc-py`) is a sibling project at `~/Desktop/aicc-py` — the testbed should build on top of it.
|
**Working.** The testbed is built: headless 3D room with a capsule, a
|
||||||
|
protocol-conformant bridge (`9/9` conformance scenarios), and a demo in which
|
||||||
|
an agent (LLM or scripted) navigates to the beacon and activates it. See
|
||||||
|
`testbed/README.md` for run instructions; `scripts/BUILDLOG.md` for the build
|
||||||
|
log.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
|
|||||||
+7
-7
@@ -1,9 +1,9 @@
|
|||||||
# aicc-capsule build log
|
# aicc-capsule build log
|
||||||
# Fill this in as you go. Keep it short — one line per milestone.
|
|
||||||
|
|
||||||
- [ ] 2026-08: context pack created (AGENTS.md, docs, scripts)
|
- [x] 2026-08: context pack created (AGENTS.md, docs, scripts)
|
||||||
- [ ] bridge skeleton: world + capsule + WebSocketServer
|
- [x] 2026-08: bridge skeleton: world (room + capsule physics + beacon) + headless raycaster + WebSocketServer
|
||||||
- [ ] tools: proprioception, vision, move, turn, look_at, interact (+ echo/boom/bump for conformance)
|
- [x] 2026-08: tools: proprioception, vision, depth, hear, world_query, move, turn, look_at, interact (+ echo/boom/bump for conformance)
|
||||||
- [ ] conformance 9/9 green
|
- [x] 2026-08: conformance 9/9 green against the testbed bridge
|
||||||
- [ ] agent demo loop (LLM drives capsule to beacon, interacts)
|
- [x] 2026-08: pytest suite (33 tests: world physics, renderer, bridge tools, protocol)
|
||||||
- [ ] README in testbed/ with run instructions
|
- [x] 2026-08: agent demo loop: LLM (OpenAI-compatible, ollama default) + scripted fallback + auto handoff; E2E verified over WebSocket
|
||||||
|
- [x] 2026-08: README in testbed/ with run instructions, setup.sh deps, run_bridge.sh
|
||||||
|
|||||||
+6
-2
@@ -8,7 +8,11 @@ source testbed/.venv/bin/activate
|
|||||||
pip install --upgrade pip
|
pip install --upgrade pip
|
||||||
# aicc-py SDK from sibling checkout
|
# aicc-py SDK from sibling checkout
|
||||||
pip install -e "$HOME/Desktop/aicc-py"
|
pip install -e "$HOME/Desktop/aicc-py"
|
||||||
# Engine and demo deps go here (e.g. ursina, panda3d, pillow, websockets)
|
# Engine + demo deps: Pillow (headless raycaster renderer), websockets (transport),
|
||||||
# pip install ...
|
# openai (LLM demo client), pytest (test suite)
|
||||||
|
pip install pillow websockets openai pytest pytest-asyncio
|
||||||
|
|
||||||
echo "Setup done. Activate with: source testbed/.venv/bin/activate"
|
echo "Setup done. Activate with: source testbed/.venv/bin/activate"
|
||||||
|
echo "Run the bridge: scripts/run_bridge.sh"
|
||||||
|
echo "Run the demo: testbed/.venv/bin/python -m testbed.demo --agent auto"
|
||||||
|
echo "Conformance: testbed/.venv/bin/python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios"
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# aicc-capsule testbed
|
||||||
|
|
||||||
|
Reference implementation of the [AICC Protocol](https://github.com/emil28092005/AICC-Protocol)
|
||||||
|
as a 3D room with an AI-controlled capsule. An agent (any LLM with tool
|
||||||
|
calling, or the bundled scripted agent) connects over WebSocket, perceives the
|
||||||
|
room through sensors, walks to a glowing beacon, and activates it — every step
|
||||||
|
over the protocol, no engine hooks.
|
||||||
|
|
||||||
|
```
|
||||||
|
agent (LLM) <--AICC over WebSocket--> RoomBridge <--> Room (world + renderer)
|
||||||
|
(testbed/demo.py) (testbed/bridge.py) (testbed/room/)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
|-------------------------|--------------------------------------------------------------|
|
||||||
|
| `room/world.py` | World state: 16x16 room, capsule physics, crates, beacon, audio. Single source of truth. |
|
||||||
|
| `room/render.py` | Headless first-person raycaster (Pillow): honest frames from world state. |
|
||||||
|
| `bridge.py` | `RoomBridge(Bridge)`: registers all tools, emits events. |
|
||||||
|
| `server.py` entry | `python -m testbed.bridge` — WebSocket server. |
|
||||||
|
| `conformance.py` | Runs the 9 core conformance scenarios against this bridge. |
|
||||||
|
| `demo.py` | Agent demo: LLM driver (OpenAI-compatible) or scripted. |
|
||||||
|
| `tests/` | pytest suite (world, renderer, bridge, protocol). |
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/setup.sh # venv + aicc-py + pillow/websockets/openai
|
||||||
|
source testbed/.venv/bin/activate
|
||||||
|
```
|
||||||
|
|
||||||
|
The testbed needs the `aicc` SDK installed from `~/Desktop/aicc-py` (the setup
|
||||||
|
script does `pip install -e`).
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Start the bridge (headless)
|
||||||
|
scripts/run_bridge.sh # ws://127.0.0.1:8765
|
||||||
|
|
||||||
|
# 2. Run the demo — default `auto` tries the LLM, then hands off to the
|
||||||
|
# scripted agent so the run always completes
|
||||||
|
python -m testbed.demo --agent auto
|
||||||
|
|
||||||
|
# scripted only (deterministic, no LLM needed)
|
||||||
|
python -m testbed.demo --agent scripted
|
||||||
|
|
||||||
|
# LLM only (any OpenAI-compatible endpoint; ollama by default)
|
||||||
|
python -m testbed.demo --agent llm --model gemma4:e2b
|
||||||
|
python -m testbed.demo --agent llm \
|
||||||
|
--base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
The demo prints a full transcript of tool calls/results to stdout and saves the
|
||||||
|
capsule's final first-person frame to `demo_final_frame.png`.
|
||||||
|
|
||||||
|
## Conformance
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios
|
||||||
|
# [PASS] core-01..core-09 -> 9/9 scenarios passed
|
||||||
|
```
|
||||||
|
|
||||||
|
The suite runs against the *same* bridge class used by the server and demo.
|
||||||
|
|
||||||
|
## Tools
|
||||||
|
|
||||||
|
Registered in the manifest (sensors first, then actuators):
|
||||||
|
|
||||||
|
| id | class | purpose |
|
||||||
|
|-----------------|-----------|----------------------------------------------------|
|
||||||
|
| `proprioception`| sensor | position, rotation, velocity, health, tick |
|
||||||
|
| `vision` | sensor | first-person RGB frame as base64 PNG (160x120) |
|
||||||
|
| `depth` | sensor | aligned depth map (40x30, meters) |
|
||||||
|
| `hear` | sensor | audio since last call: beacon hum, collision thuds |
|
||||||
|
| `world_query` | sensor | room bounds, obstacle layout, beacon position |
|
||||||
|
| `move` | actuator | move forward N meters, collision-aware |
|
||||||
|
| `turn` | actuator | rotate yaw/pitch |
|
||||||
|
| `look_at` | actuator | aim camera at a named target (`beacon`) |
|
||||||
|
| `interact` | actuator | activate the beacon within reach |
|
||||||
|
| `echo`/`boom`/`bump` | — | conformance tools (design doc requirement) |
|
||||||
|
|
||||||
|
World data flows only through sensors: the manifest carries session metadata
|
||||||
|
and tool schemas, never world state (protocol §7, single source of truth).
|
||||||
|
|
||||||
|
## Tick model
|
||||||
|
|
||||||
|
`tick_mode` is `event`: the world advances one tick per tool call, so every
|
||||||
|
observation and event shares a monotonic `tick`. Collision events
|
||||||
|
(`topic: collision`, payload `{other, normal, impulse}`) are pushed
|
||||||
|
asynchronously when `move` hits a wall or crate.
|
||||||
|
|
||||||
|
## Demo agents
|
||||||
|
|
||||||
|
- **LLM agent** (`--agent llm`): generic tool-use loop — manifest tools are
|
||||||
|
converted to OpenAI function schemas; every response is executed via
|
||||||
|
`AICCClient.call_tool` and fed back as a `tool` message. Vision frames are
|
||||||
|
decoded into a coarse color grid so text-only models can navigate. The loop
|
||||||
|
keeps a compact `CURRENT STATE` note (agent-side working memory, protocol
|
||||||
|
§9) and gently corrects a model that drifts: nudge after text-only replies,
|
||||||
|
re-aim corrections when the capsule moves away or faces the wrong way,
|
||||||
|
collision guidance. Works with any OpenAI-compatible endpoint (ollama,
|
||||||
|
vLLM, OpenAI, ...). Model quality varies — a capable model completes on its
|
||||||
|
own; a weak local model may hand off (see `auto`).
|
||||||
|
- **Scripted agent** (`--agent scripted`): deterministic bug-algorithm robot —
|
||||||
|
sensor-driven steering toward the beacon with detour-on-collision. No LLM,
|
||||||
|
always completes. Used as the reference/fallback.
|
||||||
|
- **Auto** (`--agent auto`): tries the LLM (bounded steps), then hands off to
|
||||||
|
the scripted agent so a demo run always ends with an activated beacon.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m pytest -q # 33 tests: physics, renderer, tools, protocol
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user