docs: testbed README (run/agent/conformance), setup.sh deps, build log, project status
This commit is contained in:
opencode
@@ -8,7 +8,11 @@ A 3D room with a capsule that an LLM agent controls through AICC sensor/actuator
|
||||
|
||||
## Status
|
||||
|
||||
**Not started.** This directory is the context pack for building it. The spec (core.md) and JSON Schema are in `docs/`. The Python SDK (`aicc-py`) is a sibling project at `~/Desktop/aicc-py` — the testbed should build on top of it.
|
||||
**Working.** The testbed is built: headless 3D room with a capsule, a
|
||||
protocol-conformant bridge (`9/9` conformance scenarios), and a demo in which
|
||||
an agent (LLM or scripted) navigates to the beacon and activates it. See
|
||||
`testbed/README.md` for run instructions; `scripts/BUILDLOG.md` for the build
|
||||
log.
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
+7
-7
@@ -1,9 +1,9 @@
|
||||
# aicc-capsule build log
|
||||
# Fill this in as you go. Keep it short — one line per milestone.
|
||||
|
||||
- [ ] 2026-08: context pack created (AGENTS.md, docs, scripts)
|
||||
- [ ] bridge skeleton: world + capsule + WebSocketServer
|
||||
- [ ] tools: proprioception, vision, move, turn, look_at, interact (+ echo/boom/bump for conformance)
|
||||
- [ ] conformance 9/9 green
|
||||
- [ ] agent demo loop (LLM drives capsule to beacon, interacts)
|
||||
- [ ] README in testbed/ with run instructions
|
||||
- [x] 2026-08: context pack created (AGENTS.md, docs, scripts)
|
||||
- [x] 2026-08: bridge skeleton: world (room + capsule physics + beacon) + headless raycaster + WebSocketServer
|
||||
- [x] 2026-08: tools: proprioception, vision, depth, hear, world_query, move, turn, look_at, interact (+ echo/boom/bump for conformance)
|
||||
- [x] 2026-08: conformance 9/9 green against the testbed bridge
|
||||
- [x] 2026-08: pytest suite (33 tests: world physics, renderer, bridge tools, protocol)
|
||||
- [x] 2026-08: agent demo loop: LLM (OpenAI-compatible, ollama default) + scripted fallback + auto handoff; E2E verified over WebSocket
|
||||
- [x] 2026-08: README in testbed/ with run instructions, setup.sh deps, run_bridge.sh
|
||||
|
||||
+6
-2
@@ -8,7 +8,11 @@ source testbed/.venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
# aicc-py SDK from sibling checkout
|
||||
pip install -e "$HOME/Desktop/aicc-py"
|
||||
# Engine and demo deps go here (e.g. ursina, panda3d, pillow, websockets)
|
||||
# pip install ...
|
||||
# Engine + demo deps: Pillow (headless raycaster renderer), websockets (transport),
|
||||
# openai (LLM demo client), pytest (test suite)
|
||||
pip install pillow websockets openai pytest pytest-asyncio
|
||||
|
||||
echo "Setup done. Activate with: source testbed/.venv/bin/activate"
|
||||
echo "Run the bridge: scripts/run_bridge.sh"
|
||||
echo "Run the demo: testbed/.venv/bin/python -m testbed.demo --agent auto"
|
||||
echo "Conformance: testbed/.venv/bin/python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios"
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# aicc-capsule testbed
|
||||
|
||||
Reference implementation of the [AICC Protocol](https://github.com/emil28092005/AICC-Protocol)
|
||||
as a 3D room with an AI-controlled capsule. An agent (any LLM with tool
|
||||
calling, or the bundled scripted agent) connects over WebSocket, perceives the
|
||||
room through sensors, walks to a glowing beacon, and activates it — every step
|
||||
over the protocol, no engine hooks.
|
||||
|
||||
```
|
||||
agent (LLM) <--AICC over WebSocket--> RoomBridge <--> Room (world + renderer)
|
||||
(testbed/demo.py) (testbed/bridge.py) (testbed/room/)
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Purpose |
|
||||
|-------------------------|--------------------------------------------------------------|
|
||||
| `room/world.py` | World state: 16x16 room, capsule physics, crates, beacon, audio. Single source of truth. |
|
||||
| `room/render.py` | Headless first-person raycaster (Pillow): honest frames from world state. |
|
||||
| `bridge.py` | `RoomBridge(Bridge)`: registers all tools, emits events. |
|
||||
| `server.py` entry | `python -m testbed.bridge` — WebSocket server. |
|
||||
| `conformance.py` | Runs the 9 core conformance scenarios against this bridge. |
|
||||
| `demo.py` | Agent demo: LLM driver (OpenAI-compatible) or scripted. |
|
||||
| `tests/` | pytest suite (world, renderer, bridge, protocol). |
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
scripts/setup.sh # venv + aicc-py + pillow/websockets/openai
|
||||
source testbed/.venv/bin/activate
|
||||
```
|
||||
|
||||
The testbed needs the `aicc` SDK installed from `~/Desktop/aicc-py` (the setup
|
||||
script does `pip install -e`).
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# 1. Start the bridge (headless)
|
||||
scripts/run_bridge.sh # ws://127.0.0.1:8765
|
||||
|
||||
# 2. Run the demo — default `auto` tries the LLM, then hands off to the
|
||||
# scripted agent so the run always completes
|
||||
python -m testbed.demo --agent auto
|
||||
|
||||
# scripted only (deterministic, no LLM needed)
|
||||
python -m testbed.demo --agent scripted
|
||||
|
||||
# LLM only (any OpenAI-compatible endpoint; ollama by default)
|
||||
python -m testbed.demo --agent llm --model gemma4:e2b
|
||||
python -m testbed.demo --agent llm \
|
||||
--base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY
|
||||
```
|
||||
|
||||
The demo prints a full transcript of tool calls/results to stdout and saves the
|
||||
capsule's final first-person frame to `demo_final_frame.png`.
|
||||
|
||||
## Conformance
|
||||
|
||||
```bash
|
||||
python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios
|
||||
# [PASS] core-01..core-09 -> 9/9 scenarios passed
|
||||
```
|
||||
|
||||
The suite runs against the *same* bridge class used by the server and demo.
|
||||
|
||||
## Tools
|
||||
|
||||
Registered in the manifest (sensors first, then actuators):
|
||||
|
||||
| id | class | purpose |
|
||||
|-----------------|-----------|----------------------------------------------------|
|
||||
| `proprioception`| sensor | position, rotation, velocity, health, tick |
|
||||
| `vision` | sensor | first-person RGB frame as base64 PNG (160x120) |
|
||||
| `depth` | sensor | aligned depth map (40x30, meters) |
|
||||
| `hear` | sensor | audio since last call: beacon hum, collision thuds |
|
||||
| `world_query` | sensor | room bounds, obstacle layout, beacon position |
|
||||
| `move` | actuator | move forward N meters, collision-aware |
|
||||
| `turn` | actuator | rotate yaw/pitch |
|
||||
| `look_at` | actuator | aim camera at a named target (`beacon`) |
|
||||
| `interact` | actuator | activate the beacon within reach |
|
||||
| `echo`/`boom`/`bump` | — | conformance tools (design doc requirement) |
|
||||
|
||||
World data flows only through sensors: the manifest carries session metadata
|
||||
and tool schemas, never world state (protocol §7, single source of truth).
|
||||
|
||||
## Tick model
|
||||
|
||||
`tick_mode` is `event`: the world advances one tick per tool call, so every
|
||||
observation and event shares a monotonic `tick`. Collision events
|
||||
(`topic: collision`, payload `{other, normal, impulse}`) are pushed
|
||||
asynchronously when `move` hits a wall or crate.
|
||||
|
||||
## Demo agents
|
||||
|
||||
- **LLM agent** (`--agent llm`): generic tool-use loop — manifest tools are
|
||||
converted to OpenAI function schemas; every response is executed via
|
||||
`AICCClient.call_tool` and fed back as a `tool` message. Vision frames are
|
||||
decoded into a coarse color grid so text-only models can navigate. The loop
|
||||
keeps a compact `CURRENT STATE` note (agent-side working memory, protocol
|
||||
§9) and gently corrects a model that drifts: nudge after text-only replies,
|
||||
re-aim corrections when the capsule moves away or faces the wrong way,
|
||||
collision guidance. Works with any OpenAI-compatible endpoint (ollama,
|
||||
vLLM, OpenAI, ...). Model quality varies — a capable model completes on its
|
||||
own; a weak local model may hand off (see `auto`).
|
||||
- **Scripted agent** (`--agent scripted`): deterministic bug-algorithm robot —
|
||||
sensor-driven steering toward the beacon with detour-on-collision. No LLM,
|
||||
always completes. Used as the reference/fallback.
|
||||
- **Auto** (`--agent auto`): tries the LLM (bounded steps), then hands off to
|
||||
the scripted agent so a demo run always ends with an activated beacon.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
python -m pytest -q # 33 tests: physics, renderer, tools, protocol
|
||||
```
|
||||
Reference in New Issue
Block a user