aicc-capsule/testbed/README.md

aicc-capsule testbed

Reference implementation of the AICC Protocol as a 3D room with an AI-controlled capsule. An agent (any LLM with tool calling, or the bundled scripted agent) connects over WebSocket, perceives the room through sensors, walks to a glowing beacon, and activates it — every step over the protocol, no engine hooks.

        agent (LLM)  <--AICC over WebSocket-->  RoomBridge  <-->  Room (world + renderer)
       (testbed/demo.py)                        (testbed/bridge.py)   (testbed/room/)

Layout

Path	Purpose
room/world.py	World state: 16x16 room, capsule physics, crates, beacon, audio. Single source of truth.
room/render.py	Headless first-person raycaster (Pillow): honest frames from world state.
room/mapview.py	Top-down map drawn from sensor data (shared by recorder + live viewer).
bridge.py	RoomBridge(Bridge): registers all tools, emits events.
server.py entry	python -m testbed.bridge — WebSocket server.
conformance.py	Runs the 9 core conformance scenarios against this bridge.
live.py	Real-time browser viewer (itself an AICC client).
chat.py	Interactive chat: natural language -> tool calls.
llm_agent.py	Shared LLM driver (controller + autonomous loop).
demo.py	Agent demo: LLM driver (OpenAI-compatible) or scripted.
tests/	pytest suite (world, renderer, bridge, protocol).

Setup

scripts/setup.sh                 # venv + aicc-py + pillow/websockets/openai

The testbed needs the aicc SDK installed from ~/Desktop/aicc-py (the setup script does pip install -e). The launcher scripts below use the venv interpreter directly, so python does not need to be on your PATH. To run commands by hand instead, activate the venv first: source testbed/.venv/bin/activate.

Run

# 1. Start the bridge (headless; keep it running in its own terminal)
scripts/run_bridge.sh            # ws://127.0.0.1:8765

# 2. Run the demo — default `auto` tries the LLM, then hands off to the
#    scripted agent so the run always completes
scripts/run_demo.sh --agent auto

#   scripted only (deterministic, no LLM needed)
scripts/run_demo.sh --agent scripted

#   LLM only (any OpenAI-compatible endpoint; ollama by default)
scripts/run_demo.sh --agent llm --model gemma4:e2b
scripts/run_demo.sh --agent llm \
    --base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY

If the bridge is already running, run_bridge.sh will say so (the port is taken); stop the old one with fuser -k 8765/tcp or Ctrl-C in its terminal.

The demo prints a full transcript of tool calls/results to stdout and saves the capsule's final first-person frame to demo_final_frame.png.

Search challenge

A small orange triangle is painted on the back side of one of the crates (random per run, always on a face hidden from the spawn corner). The agent must explore the room, look at the crate faces, spot the triangle with vision, and report it (the bridge verifies the report by distance):

scripts/run_chat.sh --provider polza --search
# autonomous exploration, real vision, smooth turns, cruise gliding;
# success = report verified

Search missions run in --free mode: no re-aim corrections, no collision hints — the model plans its own exploration. --free also applies to beacon missions if you want fewer guardrails. look_at now accepts crates too (crate_red/crate_blue/crate_olive). Verified: gpt-5.6-luna explored the room with smooth moves/turns and reported the triangle at 0.5 m.

Smooth & proactive movement

The capsule no longer teleports or stops-and-thinks:

• Smooth: move(forward, duration) animates the displacement over duration seconds on the bridge, so the live viewer shows genuine gliding. The scripted agent and auto-cruise use it by default; the LLM can too.
• Proactive (--cruise N, meters per think): while the model is generating a response, the capsule keeps gliding forward (low-level controller pattern, like a real robot). Collisions stop the drift safely and are reported to the model, which re-plans. Verified: gpt-5.6-luna completed the beacon mission with cruise on, bumping into and avoiding crates while "thinking".

Providers

Provider presets resolve endpoint + API key + default model:

export POLZA_API_KEY="..."        # from https://polza.ai/dashboard/api-keys
scripts/run_chat.sh --provider polza            # openai/gpt-5.6-luna, vision ON
scripts/run_demo.sh --provider polza --agent llm
scripts/run_demo.sh --provider openai --model gpt-4o-mini
# local ollama stays the default (no flags needed)

# anything else OpenAI-compatible:
scripts/run_chat.sh --base-url https://... --model ... --api-key ...

Keys are read from the environment (POLZA_API_KEY, OPENAI_API_KEY), never stored in the repo. Multimodal models (gpt-5.x, gemma, qwen-vl, ...) get real camera frames automatically; --vision/--no-vision override.

Interactive chat mode

Talk to the capsule's brain in natural language (any language):

scripts/run_chat.sh --model gemma4:e2b    # small + fast
scripts/run_chat.sh --model gemma4:12b    # bigger gemma, slower (~20 s/turn)

It starts the bridge, the live viewer and a chat REPL — open http://127.0.0.1:8000 to watch the capsule while you type. The model translates your words into tool calls:

you> иди к маяку            → look_at + move step by step (auto-continue)
you> повернись налево       → turn(-90)
you> осмотрись              → vision + description
you> активируй маяк         → interact (when close)
you> /mission               → autonomous goal: reach & activate the beacon,
                             keeps trying until done (retries + corrections)
you> /mission дойди до маяка
you> /status  /stop         → mission progress / cancel
you> /state  /look  /map  /models  /model gemma4:12b  /steps N  /help  /exit

Missions run in the background while the REPL stays usable — watch the capsule on http://127.0.0.1:8000 as it works. Start one directly: python -m testbed.chat --mission [--mission-steps 50] [--mission-retries 3]. The mission keeps trying (corrections when it drifts, nudges when it stalls, fresh attempts on failure) until the goal is achieved; /stop cancels it.

Each turn's transcript is printed; the current frame lands in chat_frame.png and the sensor-built map in chat_map.png. Any OpenAI-compatible endpoint works: python -m testbed.chat --base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY. --auto-steps N controls how many tool steps the model may chain per request (0 = one action per turn).

Real-time perception

The model sees continuously, not only when it asks:

• --look-every N attaches a fresh camera frame every N steps/turns (default 3, 0 disables). Frames go in as images for multimodal models, as digests for text-only ones, and old frames are trimmed (max 6) to keep the context bounded.
• On a collision the current frame is attached immediately, so the model sees what it bumped into.
• The live viewer (http://127.0.0.1:8000) shows the same frames in the browser.

Real vision

Vision is the primary channel: when the model is multimodal, vision tool results attach the actual camera frame as an image to the conversation — the model sees the crate, the wall, the glowing beacon. The color-grid digest is only the fallback for text-only models:

• multimodal model (auto-detected for local ollama via /api/show; otherwise --vision): frame as image, digest off
• text-only model: digest on automatically; --digest forces the digest to be included even alongside images, --no-digest disables it everywhere

Verified locally: gemma4:12b sees frames and completed a beacon mission on its first attempt; gemma4:e2b describes what it sees (crates, beacon) and orients itself from the image.

Real-time mode

Watch the capsule drive live in your browser:

scripts/run_live.sh --agent scripted

This starts the bridge, a viewer server, and the demo; open http://127.0.0.1:8000 while the agent acts. The page shows the first-person frame (vision) and a top-down map (world_query + proprioception) updating a few times per second, with the capsule's path, heading, and distance to the beacon.

The viewer (testbed/live.py) is itself a plain AICC client — it sees the world only through the protocol sensors, so it works against any bridge, not just this one. You can also run it standalone:

testbed/.venv/bin/python -m testbed.live            # then run the demo in another terminal

Visual mode

scripts/run_demo.sh --agent scripted --frames-dir frames

Saves, for every step, the first-person frame (step_NNN_view.png) and a top-down map of the room with the capsule's path (step_NNN_map.png), then writes a demo.gif animation and a demo_summary.png (final map + last view). The map is rebuilt purely from sensor data (world_query, proprioception, vision) — the same view the agent itself has. Generated examples are committed at the repo root (demo.gif, demo_summary.png).

Conformance

testbed/.venv/bin/python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios
# [PASS] core-01..core-09  ->  9/9 scenarios passed

The suite runs against the same bridge class used by the server and demo.

Tools

Registered in the manifest (sensors first, then actuators):

id	class	purpose
proprioception	sensor	position, rotation, velocity, health, tick
vision	sensor	first-person RGB frame as base64 PNG (160x120)
depth	sensor	aligned depth map (40x30, meters)
hear	sensor	audio since last call: beacon hum, collision thuds
world_query	sensor	room bounds, obstacle layout, beacon position
move	actuator	move forward N meters, collision-aware
turn	actuator	rotate yaw/pitch
look_at	actuator	aim camera at a named target (beacon)
interact	actuator	activate the beacon within reach
echo/boom/bump	—	conformance tools (design doc requirement)

World data flows only through sensors: the manifest carries session metadata and tool schemas, never world state (protocol §7, single source of truth).

Tick model

tick_mode is event: the world advances one tick per tool call, so every observation and event shares a monotonic tick. Collision events (topic: collision, payload {other, normal, impulse}) are pushed asynchronously when move hits a wall or crate.

Demo agents

• LLM agent (--agent llm): generic tool-use loop — manifest tools are converted to OpenAI function schemas; every response is executed via AICCClient.call_tool and fed back as a tool message. Vision frames are decoded into a coarse color grid so text-only models can navigate. The loop keeps a compact CURRENT STATE note (agent-side working memory, protocol §9) and gently corrects a model that drifts: nudge after text-only replies, re-aim corrections when the capsule moves away or faces the wrong way, collision guidance. Works with any OpenAI-compatible endpoint (ollama, vLLM, OpenAI, ...). Model quality varies — a capable model completes on its own; a weak local model may hand off (see auto).
• Scripted agent (--agent scripted): deterministic bug-algorithm robot — sensor-driven steering toward the beacon with detour-on-collision. No LLM, always completes. Used as the reference/fallback.
• Auto (--agent auto): tries the LLM (bounded steps), then hands off to the scripted agent so a demo run always ends with an activated beacon.

Tests

testbed/.venv/bin/python -m pytest -q   # 33 tests: physics, renderer, tools, protocol