# aicc-capsule testbed

Reference implementation of the [AICC Protocol](https://github.com/emil28092005/AICC-Protocol)
as a 3D room with an AI-controlled capsule. An agent (any LLM with tool
calling, or the bundled scripted agent) connects over WebSocket, perceives the
room through sensors, walks to a glowing beacon, and activates it — every step
over the protocol, no engine hooks.

```
        agent (LLM)  <--AICC over WebSocket-->  RoomBridge  <-->  Room (world + renderer)
       (testbed/demo.py)                        (testbed/bridge.py)   (testbed/room/)
```

## Layout

| Path                    | Purpose                                                      |
|-------------------------|--------------------------------------------------------------|
| `room/world.py`         | World state: 16x16 room, capsule physics, crates, beacon, audio. Single source of truth. |
| `room/render.py`        | Headless first-person raycaster (Pillow): honest frames from world state. |
| `room/mapview.py`       | Top-down map drawn from sensor data (shared by recorder + live viewer). |
| `bridge.py`             | `RoomBridge(Bridge)`: registers all tools, emits events.     |
| `server.py` entry       | `python -m testbed.bridge` — WebSocket server.               |
| `conformance.py`        | Runs the 9 core conformance scenarios against this bridge.   |
| `live.py`               | Real-time browser viewer (itself an AICC client).            |
| `chat.py`               | Interactive chat: natural language -> tool calls.            |
| `llm_agent.py`          | Shared LLM driver (controller + autonomous loop).            |
| `demo.py`               | Agent demo: LLM driver (OpenAI-compatible) or scripted.      |
| `tests/`                | pytest suite (world, renderer, bridge, protocol).            |

## Setup

```bash
scripts/setup.sh                 # venv + aicc-py + pillow/websockets/openai
```

The testbed needs the `aicc` SDK installed from `~/Desktop/aicc-py` (the setup
script does `pip install -e`). The launcher scripts below use the venv
interpreter directly, so `python` does not need to be on your PATH. To run
commands by hand instead, activate the venv first:
`source testbed/.venv/bin/activate`.

## Run

```bash
# 1. Start the bridge (headless; keep it running in its own terminal)
scripts/run_bridge.sh            # ws://127.0.0.1:8765

# 2. Run the demo — default `auto` tries the LLM, then hands off to the
#    scripted agent so the run always completes
scripts/run_demo.sh --agent auto

#   scripted only (deterministic, no LLM needed)
scripts/run_demo.sh --agent scripted

#   LLM only (any OpenAI-compatible endpoint; ollama by default)
scripts/run_demo.sh --agent llm --model gemma4:e2b
scripts/run_demo.sh --agent llm \
    --base-url https://api.openai.com/v1 --model gpt-4o-mini --api-key $OPENAI_API_KEY
```

If the bridge is already running, `run_bridge.sh` will say so (the port is
taken); stop the old one with `fuser -k 8765/tcp` or Ctrl-C in its terminal.

The demo prints a full transcript of tool calls/results to stdout and saves the
capsule's final first-person frame to `demo_final_frame.png`.

### Search challenge

A small orange triangle is painted on the **back side** of one of the crates
(random per run, always on a face hidden from the spawn corner). The agent must
explore the room, look at the crate faces, spot the triangle with `vision`,
and `report` it (the bridge verifies the report by distance):

```bash
scripts/run_chat.sh --provider polza --search
# autonomous exploration, real vision, smooth turns, cruise gliding;
# success = report verified
```

Search missions run in `--free` mode: no re-aim corrections, no collision
hints — the model plans its own exploration. `--free` also applies to beacon
missions if you want fewer guardrails. `look_at` now accepts crates too
(crate_red/crate_blue/crate_olive). Verified: gpt-5.6-luna explored the room
with smooth moves/turns and reported the triangle at 0.5 m.

### Smooth & proactive movement

The capsule no longer teleports or stops-and-thinks:

- **Smooth**: `move(forward, duration)` animates the displacement over
  `duration` seconds on the bridge, so the live viewer shows genuine gliding.
  The scripted agent and auto-cruise use it by default; the LLM can too.
- **Proactive** (`--cruise N`, meters per think): while the model is
  generating a response, the capsule keeps gliding forward (low-level
  controller pattern, like a real robot). Collisions stop the drift safely
  and are reported to the model, which re-plans. Verified: gpt-5.6-luna
  completed the beacon mission with cruise on, bumping into and avoiding
  crates while "thinking".

## Providers

Provider presets resolve endpoint + API key + default model:

```bash
export POLZA_API_KEY="..."        # from https://polza.ai/dashboard/api-keys
scripts/run_chat.sh --provider polza            # openai/gpt-5.6-luna, vision ON
scripts/run_demo.sh --provider polza --agent llm
scripts/run_demo.sh --provider openai --model gpt-4o-mini
# local ollama stays the default (no flags needed)

# anything else OpenAI-compatible:
scripts/run_chat.sh --base-url https://... --model ... --api-key ...
```

Keys are read from the environment (`POLZA_API_KEY`, `OPENAI_API_KEY`), never
stored in the repo. Multimodal models (gpt-5.x, gemma, qwen-vl, ...) get real
camera frames automatically; `--vision`/`--no-vision` override.

## Interactive chat mode

Talk to the capsule's brain in natural language (any language):

```bash
scripts/run_chat.sh --model gemma4:e2b    # small + fast
scripts/run_chat.sh --model gemma4:12b    # bigger gemma, slower (~20 s/turn)
```

It starts the bridge, the live viewer and a chat REPL — open
[http://127.0.0.1:8000](http://127.0.0.1:8000) to watch the capsule while you
type. The model translates your words into tool calls:

```
you> иди к маяку            → look_at + move step by step (auto-continue)
you> повернись налево       → turn(-90)
you> осмотрись              → vision + description
you> активируй маяк         → interact (when close)
you> /mission               → autonomous goal: reach & activate the beacon,
                             keeps trying until done (retries + corrections)
you> /mission дойди до маяка
you> /status  /stop         → mission progress / cancel
you> /state  /look  /map  /models  /model gemma4:12b  /steps N  /help  /exit
```

Missions run in the background while the REPL stays usable — watch the
capsule on http://127.0.0.1:8000 as it works. Start one directly:
`python -m testbed.chat --mission [--mission-steps 50] [--mission-retries 3]`.
The mission keeps trying (corrections when it drifts, nudges when it stalls,
fresh attempts on failure) until the goal is achieved; `/stop` cancels it.

Each turn's transcript is printed; the current frame lands in `chat_frame.png`
and the sensor-built map in `chat_map.png`. Any OpenAI-compatible endpoint
works: `python -m testbed.chat --base-url https://api.openai.com/v1
--model gpt-4o-mini --api-key $OPENAI_API_KEY`. `--auto-steps N` controls how
many tool steps the model may chain per request (0 = one action per turn).

### Real-time perception

The model sees continuously, not only when it asks:

- `--look-every N` attaches a fresh camera frame every N steps/turns (default 3,
  `0` disables). Frames go in as images for multimodal models, as digests for
  text-only ones, and old frames are trimmed (max 6) to keep the context
  bounded.
- On a collision the current frame is attached immediately, so the model sees
  what it bumped into.
- The live viewer (http://127.0.0.1:8000) shows the same frames in the browser.

### Real vision

Vision is the **primary channel**: when the model is multimodal, `vision`
tool results attach the actual camera frame as an image to the conversation —
the model sees the crate, the wall, the glowing beacon. The color-grid
digest is only the **fallback** for text-only models:

- multimodal model (auto-detected for local ollama via `/api/show`;
  otherwise `--vision`): frame as image, digest off
- text-only model: digest on automatically; `--digest` forces the digest to
  be included even alongside images, `--no-digest` disables it everywhere

Verified locally: gemma4:12b sees frames and completed a beacon mission on
its first attempt; gemma4:e2b describes what it sees (crates, beacon) and
orients itself from the image.

## Real-time mode

Watch the capsule drive live in your browser:

```bash
scripts/run_live.sh --agent scripted
```

This starts the bridge, a viewer server, and the demo; open
[http://127.0.0.1:8000](http://127.0.0.1:8000) while the agent acts. The page
shows the first-person frame (`vision`) and a top-down map (`world_query` +
`proprioception`) updating a few times per second, with the capsule's path,
heading, and distance to the beacon.

The viewer (`testbed/live.py`) is itself a plain AICC client — it sees the
world only through the protocol sensors, so it works against any bridge, not
just this one. You can also run it standalone:

```bash
testbed/.venv/bin/python -m testbed.live            # then run the demo in another terminal
```

## Visual mode

```bash
scripts/run_demo.sh --agent scripted --frames-dir frames
```

Saves, for every step, the first-person frame (`step_NNN_view.png`) and a
top-down map of the room with the capsule's path (`step_NNN_map.png`), then
writes a `demo.gif` animation and a `demo_summary.png` (final map + last
view). The map is rebuilt purely from sensor data (`world_query`,
`proprioception`, `vision`) — the same view the agent itself has. Generated
examples are committed at the repo root (`demo.gif`, `demo_summary.png`).

## Conformance

```bash
testbed/.venv/bin/python -m testbed.conformance ~/Desktop/aicc-spec/conformance/scenarios
# [PASS] core-01..core-09  ->  9/9 scenarios passed
```

The suite runs against the *same* bridge class used by the server and demo.

## Tools

Registered in the manifest (sensors first, then actuators):

| id              | class     | purpose                                            |
|-----------------|-----------|----------------------------------------------------|
| `proprioception`| sensor    | position, rotation, velocity, health, tick         |
| `vision`        | sensor    | first-person RGB frame as base64 PNG (160x120)     |
| `depth`         | sensor    | aligned depth map (40x30, meters)                  |
| `hear`          | sensor    | audio since last call: beacon hum, collision thuds |
| `world_query`   | sensor    | room bounds, obstacle layout, beacon position      |
| `move`          | actuator  | move forward N meters, collision-aware             |
| `turn`          | actuator  | rotate yaw/pitch                                   |
| `look_at`       | actuator  | aim camera at a named target (`beacon`)            |
| `interact`      | actuator  | activate the beacon within reach                   |
| `echo`/`boom`/`bump` | —    | conformance tools (design doc requirement)         |

World data flows only through sensors: the manifest carries session metadata
and tool schemas, never world state (protocol §7, single source of truth).

## Tick model

`tick_mode` is `event`: the world advances one tick per tool call, so every
observation and event shares a monotonic `tick`. Collision events
(`topic: collision`, payload `{other, normal, impulse}`) are pushed
asynchronously when `move` hits a wall or crate.

## Demo agents

- **LLM agent** (`--agent llm`): generic tool-use loop — manifest tools are
  converted to OpenAI function schemas; every response is executed via
  `AICCClient.call_tool` and fed back as a `tool` message. Vision frames are
  decoded into a coarse color grid so text-only models can navigate. The loop
  keeps a compact `CURRENT STATE` note (agent-side working memory, protocol
  §9) and gently corrects a model that drifts: nudge after text-only replies,
  re-aim corrections when the capsule moves away or faces the wrong way,
  collision guidance. Works with any OpenAI-compatible endpoint (ollama,
  vLLM, OpenAI, ...). Model quality varies — a capable model completes on its
  own; a weak local model may hand off (see `auto`).
- **Scripted agent** (`--agent scripted`): deterministic bug-algorithm robot —
  sensor-driven steering toward the beacon with detour-on-collision. No LLM,
  always completes. Used as the reference/fallback.
- **Auto** (`--agent auto`): tries the LLM (bounded steps), then hands off to
  the scripted agent so a demo run always ends with an activated beacon.

## Tests

```bash
testbed/.venv/bin/python -m pytest -q   # 33 tests: physics, renderer, tools, protocol
```