commit fb3b4935d9f4148e1bc57abded4d03351d97d8fa Author: Emil Shanaty Date: Sat Aug 8 03:31:42 2026 +0300 chore: context pack for aicc-capsule reference testbed - AGENTS.md: build instructions for AI coding agents - docs/: AICC core spec copy, JSON schema, testbed design notes - scripts/: setup.sh, BUILDLOG.md - README.md: overview and reading order diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..bc246e5 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,38 @@ +# AGENTS.md — aicc-capsule build instructions + +You are building the reference testbed for the AICC Protocol: a 3D room with an AI-controlled capsule that an LLM agent drives through protocol tools. This file tells you what to build and how. Read `docs/aicc-core.md` (the protocol spec) and `docs/testbed-design.md` before writing code. The Python SDK to build on is at `~/Desktop/aicc-py` (sibling checkout; use it directly, do not reinvent). + +## Mission + +Prove the AICC Protocol end-to-end: + +1. A 3D room with obstacles and one interactable object. +2. A capsule the agent can move (bridge-side physics + state). +3. A bridge implementing the protocol with tools: `proprioception`, `vision`, `move`, `turn`, `look_at`, `interact`. +4. A runnable demo: an agent (any LLM with tool calling) connects via WebSocket, perceives the room, walks to a target, interacts. + +## Constraints + +- **Use the aicc-py SDK** at `~/Desktop/aicc-py` (Bridge, AICCClient, WebSocketServer). Import it, do not copy its internals. If the SDK lacks something, extend the SDK repo, not the testbed. +- **Engine choice is yours** but must meet: Python-friendly, can render a first-person RGB frame for `vision` (or emit a fake-but-honest frame), simple physics for `move`/collision. Candidates: Panda3D, ursina, Godot (via Python bindings), or a minimal custom scene. A headless-friendly setup is strongly preferred so the demo runs without a display. +- **Protocol compliance is mandatory.** The bridge must pass the conformance scenarios: `python -m aicc.conformance ~/Desktop/aicc-spec/conformance/scenarios` (all 9 core scenarios green). +- Tools must have `description` (first line = tool purpose). Use snake_case ids. +- Follow the "single source of truth" principle: all world data only via sensors, manifest carries no world state. +- Keep it minimal. This is a reference, not a product. One room, one capsule, one interactable. No inventory, no combat, no save system. + +## Definition of done + +- [ ] `python -m aicc.conformance ...` passes 9/9 against the testbed bridge +- [ ] A demo script that: starts bridge, connects an agent loop, agent reaches a target and interacts, prints a transcript of tool calls to stdout +- [ ] `README.md` in testbed/ documenting how to run bridge + agent +- [ ] At least one commit per milestone (bridge skeleton, tools, demo loop) + +## Environment + +- Python 3.11+; venv in `testbed/.venv` +- aicc-py SDK available at `~/Desktop/aicc-py` — add it to PYTHONPATH or `pip install -e ~/Desktop/aicc-py` +- If you pick an engine, add its install command to `scripts/setup.sh` + +## Communication + +Report progress in short summaries. When done, print the final tool transcript example and conformance result. diff --git a/README.md b/README.md new file mode 100644 index 0000000..335e479 --- /dev/null +++ b/README.md @@ -0,0 +1,37 @@ +# aicc-capsule — Reference Testbed + +Reference implementation of the [AICC Protocol](https://github.com/emil28092005/AICC-Protocol) in a 3D environment: an AI-controlled capsule in a room. This is the "hello world" of the protocol — a working end-to-end vertical that proves the spec is implementable and gives other projects a template to copy. + +## Goal + +A 3D room with a capsule that an LLM agent controls through AICC sensor/actuator tools. The agent perceives the world via tools (`vision`, `proprioception`, `hear`), reasons, and acts (`move`, `turn`, `look_at`, `interact`). No pixel-clicking, no engine-specific hooks — pure protocol. + +## Status + +**Not started.** This directory is the context pack for building it. The spec (core.md) and JSON Schema are in `docs/`. The Python SDK (`aicc-py`) is a sibling project at `~/Desktop/aicc-py` — the testbed should build on top of it. + +## Layout + +``` +aicc-capsule/ + AGENTS.md # instructions for AI coding agents (opencode, claude, etc.) + README.md # this file + docs/ + aicc-core.md # AICC Protocol spec, v0.1 (copy from AICC-Protocol repo) + aicc.schema.json # JSON Schema for all AICC messages + testbed-design.md # design notes for the testbed itself (see below) + scripts/ # build/run/dev scripts + testbed/ # the testbed source (empty until built) +``` + +## Reading order for an agent + +1. `AGENTS.md` — what to build, constraints, definition of done +2. `docs/aicc-core.md` — the protocol contract (sections 1–5 are essential) +3. `docs/testbed-design.md` — how the testbed maps onto the protocol +4. `~/Desktop/aicc-py` — the SDK to build on (Bridge, AICCClient, transports) + +## Related repositories + +- Spec: https://github.com/emil28092005/AICC-Protocol +- Python SDK: https://github.com/emil28092005/aicc-py (local checkout: `~/Desktop/aicc-py`) diff --git a/docs/aicc-core.md b/docs/aicc-core.md new file mode 100644 index 0000000..8008115 --- /dev/null +++ b/docs/aicc-core.md @@ -0,0 +1,477 @@ +# AICC Protocol — Core Specification (Draft 0.1) + +## Overview + +AICC (AI-Controlled Character) is a transport-agnostic protocol +for connecting language models to interactive virtual environments. It defines +how an agent perceives a world, reasons about it, and acts within it through +a structured set of sensor, actuator, and generator tools. + +The protocol makes no assumptions about: +- The host engine (Godot, Unity, Unreal, custom simulators, robotics stacks) +- The reasoning model (any LLM with tool-use capability) +- The transport layer (HTTP, WebSocket, stdio, gRPC, in-process) +- The visual style (2D, 3D, text-based, abstract) + +It only specifies message formats, tool taxonomy, capability negotiation, and +causality semantics. + +## Design Goals + +1. **Engine-agnostic.** The same protocol speaks to a game character, a robot, + or a sandboxed LLM playroom. +2. **Model-agnostic.** Any LLM that supports tool-use / function-calling can + act as the reasoning layer. No prompt engineering baked into the wire + format. +3. **Transport-agnostic.** Wire format is JSON. Transport is pluggable. +4. **Capability-aware.** An environment advertises what an agent may do. The + agent never has to guess. +5. **Deterministic when needed.** Every tool call returns a structured + outcome with timestamps, source, and confidence. Replay is a first-class + concept. +6. **Extensible.** New sensors, actuators, and generators can be added + without breaking older agents, provided they follow the extension rules + in §11. +7. **Single source of truth.** All world data flows through sensor tools. + The manifest carries session metadata only — never world state. This + makes the contract honest and uniform across any environment. + +## 1. Model of the World + +An AICC interaction involves three actors: + +- **Agent** — the reasoning layer (typically an LLM). Holds goals, plans, + and short-term working memory. +- **Environment** — the host system that owns world state. Renders, ticks + physics, runs NPCs, persists state. +- **Bridge** — the thin adapter that translates between the protocol and + the environment's native APIs. + +``` ++----------------+ +-------------+ +------------------+ +| Agent | <--> | Bridge | <--> | Environment | +| (LLM + tools) | AICC | (adapter) | native| (engine/world) | ++----------------+ +-------------+ +------------------+ +``` + +The bridge is implementation-specific and not part of this spec. The agent +sees only the AICC contract. + +### Tick + +A **tick** is one discrete update of the environment. The cadence is defined +by the environment (e.g. 60 Hz physics, 10 Hz AI updates, event-driven). +Agents do not assume a fixed tick rate — they negotiate it via the session +manifest (§3). + +### Session + +A **session** is the lifetime of one agent connected to one environment. +Sessions are identified by a UUID assigned at handshake. Sessions are +stateful: tool calls within a session may accumulate history that influences +later calls (e.g. episodic memory). + +## 2. Message Format + +All messages are UTF-8 JSON. Every message carries a top-level `protocol` +field with the version string `"aicc/0.1"`. + +### Common envelope + +```json +{ + "protocol": "aicc/0.1", + "type": "...", + "session_id": "...", + "message_id": "..." +} +``` + +| Field | Type | Notes | +|---------------|--------|---------------------------------------------------| +| `protocol` | string | Must be `"aicc/0.1"` for this version. | +| `type` | string | One of the message types below. | +| `session_id` | string | UUID assigned by the bridge at handshake. | +| `message_id` | string | UUID for this message. Used for correlation. | + +### Message types + +| `type` | Direction | Purpose | +|---------------------|----------------------|--------------------------------------------| +| `session_init` | bridge -> agent | Open a new session, advertise manifest. | +| `session_resume` | agent -> bridge | Resume a session by `session_id`. | +| `session_close` | both | Graceful close. | +| `manifest_request` | agent -> bridge | Re-fetch the manifest. | +| `tool_call` | agent -> bridge | Invoke a tool. | +| `tool_result` | bridge -> agent | Return outcome of a tool call. | +| `event` | bridge -> agent | Asynchronous notification from environment.| +| `error` | bridge -> agent | Protocol-level or environment error. | +| `heartbeat` | both | Keep-alive ping with optional payload. | + +### Versioning + +The protocol string uses semantic versioning (`aicc/MAJOR.MINOR`). +- MAJOR bump: any breaking change to message format or tool schema. +- MINOR bump: additive changes only. Agents must accept messages from a + bridge with a higher MINOR (forward compatibility), ignoring unknown + optional fields. + +## 3. Session Manifest + +Sent in `session_init` and on `manifest_request`. Describes the environment +and what the agent may do. + +```json +{ + "type": "session_init", + "session_id": "8f3a...", + "tick_rate_hz": 10, + "tick_mode": "fixed", + "world": { + "name": "testbed_room_01", + "kind": "3d" + }, + "capabilities": { + "sensors": ["vision", "depth", "proprioception", "memory_query"], + "actuators": ["move", "turn", "look_at", "interact", "say"], + "generators": [] + }, + "tools": [ + { + "id": "vision", + "class": "sensor", + "description": "Return the current first-person RGB frame as base64 PNG.", + "input_schema": { "type": "object", "properties": {}, "additionalProperties": false }, + "output_schema": { "type": "object", "properties": { "png_b64": { "type": "string" }, "width": { "type": "integer" }, "height": { "type": "integer" }, "tick": { "type": "integer" } }, "required": ["png_b64", "width", "height", "tick"] } + } + ] +} +``` + +> **Note.** World state — including the agent's own position, rotation, +> inventory, and the world's bounds — is **never** included in the manifest. +> It is accessible only through the appropriate sensor tools. The manifest +> is a contract on the session, not a snapshot of the world. + +### Tick modes + +- `fixed` — environment advances in regular intervals. Tools see a coherent + snapshot. +- `event` — environment advances only on tool calls or external triggers. +- `hybrid` — fixed physics tick, but agents may request immediate evaluation + via a `tick_now` tool. + +## 4. Tool Taxonomy + +Three top-level classes. Every tool belongs to exactly one. + +| Class | Side effect | Returns | +|--------------|-------------|----------------------------------| +| `sensor` | none | Read-only observation of world. | +| `actuator` | yes | Action performed on the world. | +| `generator` | yes | New world content created. | + +A tool is identified by its `id`. IDs are scoped to the session. + +### Standard sensors (initial set) + +| ID | Returns | +|-----------------|---------------------------------------------------| +| `vision` | RGB frame from agent's camera. | +| `depth` | Depth map aligned to vision. | +| `hear` | Audio events since last call. | +| `smell` | Active scent zones near the agent. | +| `touch` | Surface contact info from last physics tick. | +| `proprioception`| Position, rotation, velocity, health, inventory. | +| `memory_query` | Episodic memory hits. | +| `inspect` | Detailed view of a target entity or location. | + +### Standard actuators (initial set) + +| ID | Effect | +|-------------|-----------------------------------------------------| +| `move` | Translate along agent-relative or world axes. | +| `turn` | Rotate yaw / pitch. | +| `look_at` | Orient camera toward a target. | +| `interact` | Use / activate / pick up / talk to an entity. | +| `say` | Emit speech (or text for dialog-only sessions). | +| `wait` | Skip ticks without action. | + +### Standard generators (initial set) + +| ID | Effect | +|----------------|---------------------------------------------------| +| `place_object` | Spawn a registered prefab at a location. | +| `modify_terrain`| Edit terrain heightmap / material. | +| `spawn_entity` | Create a registered entity at a location. | + +Generators require the `can_modify_world` capability and a per-tool rate +limit. They are optional in the manifest. + +## 5. Tool Invocation + +### Request (`tool_call`) + +```json +{ + "type": "tool_call", + "message_id": "...", + "call_id": "tc_001", + "tool": "vision", + "input": {} +} +``` + +| Field | Type | Notes | +|--------------|--------|------------------------------------------------| +| `call_id` | string | Agent-assigned. Echoed in `tool_result`. | +| `tool` | string | Tool id from the manifest. | +| `input` | object | Must validate against the tool's input schema. | + +### Response (`tool_result`) + +```json +{ + "type": "tool_result", + "message_id": "...", + "call_id": "tc_001", + "ok": true, + "output": { "...": "..." }, + "meta": { + "tick": 142, + "latency_ms": 12, + "source": "render_thread" + } +} +``` + +| Field | Type | Notes | +|---------------|---------|---------------------------------------------| +| `call_id` | string | Echoed from the request. | +| `ok` | boolean | False on tool-level failure. | +| `output` | object | Conforms to the tool's output schema. | +| `error` | object | Present only when `ok` is false. | +| `meta` | object | Optional environment metadata. | + +### Failure shape + +```json +{ + "ok": false, + "error": { + "code": "tool_unavailable", + "message": "Generator 'spawn_entity' requires capability 'can_spawn_entities'.", + "retryable": false + } +} +``` + +Standard error codes are listed in §10. + +## 6. Events + +Asynchronous notifications from the environment. The agent may subscribe to a +subset at session init. + +```json +{ + "type": "event", + "topic": "collision", + "payload": { + "other": "wall_segment_03", + "normal": { "x": -1, "y": 0, "z": 0 }, + "impulse": 4.2 + }, + "meta": { "tick": 142 } +} +``` + +Reserved topics: + +- `tick` — fired on every environment tick (only if subscribed). +- `collision` — physical contact. +- `audio` — sound event outside the agent's request cycle. +- `state_change` — entity added, removed, or substantially changed. +- `agent_message` — incoming inter-agent communication in multi-agent sessions. + +## 7. Capabilities + +Capabilities are advertised in the manifest. They are coarse-grained +permissions, not per-entity ACLs. Fine-grained permissions are an extension +(see §11). + +### Model class declaration + +The bridge declares which model class the agent runs on. This lets +environments tune tick rates, sensor cadence, and event batching to match +the agent's expected response time. + +```json +{ + "agent_model": { + "class": "edge_small | edge_medium | cloud_medium | cloud_large", + "expected_first_token_ms": 800, + "expected_full_response_ms": 2500 + } +} +``` + +| Class | Typical target | First-token budget | +|----------------|---------------------------------------------|--------------------| +| `edge_small` | E2B / phone-class, on-device | 100-400 ms | +| `edge_medium` | E4B / 12B on consumer GPU | 300-800 ms | +| `cloud_medium` | 12B-26B MoE via hosted inference | 500-1500 ms | +| `cloud_large` | 31B+ via hosted inference, with reasoning | 1500-3000 ms | + +These are advisory. The bridge uses them to inform tick-rate choice and +sensor buffering strategy. They do not constrain the agent — the agent +may exceed or beat the budget — but environments should design their +default cadence so that a conformant agent of the declared class can +participate in real-time interaction. + +### Reserved capabilities + +- `can_observe_world` — read any sensor. +- `can_move_self` — invoke movement actuators. +- `can_modify_world` — invoke generators. +- `can_spawn_entities` — generators that create NPCs / objects. +- `can_modify_terrain` — generators that change terrain. +- `can_communicate` — invoke `say` and `agent_message` events. +- `can_persist_state` — write to long-term storage. + +An agent must not call a tool that requires an absent capability. Bridges +should reject such calls with `tool_unavailable`. + +## 8. Time and Causality + +### Tick clock + +Every message that carries a `meta.tick` field references the same monotonic +counter maintained by the environment. The agent uses this to reason about +ordering of observations. + +### Eventual consistency for events + +`event` messages may arrive between `tool_result` messages. The agent MUST +process events in `meta.tick` order; out-of-order arrival is the bridge's +responsibility to prevent or flag. + +### Heartbeat + +For long sessions, the bridge may send `heartbeat` every N seconds. Agents +may respond with `heartbeat` of their own. This is also the vehicle for +pushing manifest updates (e.g. capability changes). + +## 9. Memory and State + +Memory is split: + +- **World state** — owned by the environment, accessed **only** via sensors. + The manifest carries no world state. +- **Agent memory** — owned by the agent. Not transmitted over AICC unless + via the optional `memory_query` sensor, which is environment-side episodic + memory tagged with session events. + +The protocol does not prescribe how an LLM maintains its context window. +That's the agent's concern. The protocol only guarantees that any state +observable via a sensor is reproducible from the session. + +### Single source of truth + +Every piece of world data has exactly one way to reach the agent: through +a sensor tool. The agent's own position, the world's bounds, the contents +of nearby containers, the time of day — all of these are observable only +by calling the appropriate sensor. The manifest never duplicates or +caches this data. This eliminates a class of bugs where handshake data +drifts from real-time sensors, and keeps the contract uniform across +single-agent and multi-agent sessions. + +## 10. Errors + +Standard error codes returned in `tool_result.error` or top-level `error` +messages: + +| Code | When | +|----------------------|-----------------------------------------------------| +| `protocol_mismatch` | Wire version incompatible. | +| `session_expired` | Session id unknown or closed. | +| `tool_unknown` | Tool id not in manifest. | +| `tool_unavailable` | Capability missing or rate limit hit. | +| `invalid_input` | Input fails schema validation. | +| `execution_failed` | Tool ran but failed at the environment level. | +| `timeout` | Tool exceeded environment-defined timeout. | +| `internal_error` | Unspecified bridge failure. | + +All errors include `retryable: boolean` to indicate whether the agent may +safely retry the same call. + +## 11. Extension Rules + +New tools and capabilities MAY be added in a MINOR version. Additions MUST: + +1. Use a new `id` not previously registered. +2. Be advertised in the manifest. +3. Define input and output schemas as JSON Schema. +4. Use only documented error codes. + +Breaking changes (renaming, removing, changing semantics of existing tools, +changing required fields) MUST trigger a MAJOR bump. + +Agents SHOULD ignore unknown fields in `meta` and unknown optional fields in +`output`. Bridges SHOULD accept tool calls with extra fields by ignoring them +(unless `strict_input: true` is set on the tool in the manifest). + +## 12. Security and Trust + +- The bridge authenticates the agent (out of band of this spec). +- The agent authenticates the bridge (out of band of this spec). +- Tools that may be expensive or destructive MUST advertise rate limits in + the manifest under `tool.limits`, e.g. `{"calls_per_minute": 60}`. +- All transport MUST be encrypted when crossing untrusted networks. AICC + itself does not mandate TLS — that is a transport concern. + +## 13. Conformance + +A bridge is AICC-conformant for version `aicc/0.1` if it: + +1. Emits and accepts the message types in §2. +2. Validates all `tool_call` inputs against the manifest. +3. Returns errors using the codes in §10. +4. Preserves event ordering per §8. +5. Passes the conformance scenarios in `conformance/scenarios/`. + +An agent is conformant if it: + +1. Never invokes tools outside its advertised capabilities. +2. Echoes `call_id` correctly on result correlation. +3. Handles all standard error codes. +4. Respects tick ordering for events. + +## Appendix A: Example — Walking Around a Room + +1. Bridge -> Agent: `session_init` with vision, depth, proprioception, + move, turn, look_at. +2. Agent -> Bridge: `tool_call` `vision`. +3. Bridge -> Agent: `tool_result` with PNG. +4. Agent reasons, decides to walk forward. +5. Agent -> Bridge: `tool_call` `move` `{ "forward": 2.0 }`. +6. Bridge -> Agent: `tool_result` with new proprioception snapshot. +7. Bridge -> Agent: `event` `collision` when wall hit. +8. Agent -> Bridge: `tool_call` `turn` `{ "yaw": -45 }` to redirect. +9. Agent -> Bridge: `tool_call` `move` `{ "forward": 1.5 }`. +10. Loop continues. + +## Appendix B: Open Questions (to resolve in 0.2) + +- How are multi-agent sessions coordinated? Sub-sessions or shared session? +- How are assets (prefabs, models, audio) referenced — by id, URL, or + inline binary? +- Should `vision` support streaming chunks, or is single-frame enough? +- Standard for episodic memory schema (events vs summaries vs both)? +- Should `inspect` allow arbitrary queries, or a fixed set of detail + modes (`color`, `bbox`, `semantic`, `full`)? +- Resolved in 0.1.1: model class declaration (see §7). + +--- + +Status: **draft**. Not yet published. Internal review only. diff --git a/docs/aicc.schema.json b/docs/aicc.schema.json new file mode 100644 index 0000000..67831aa --- /dev/null +++ b/docs/aicc.schema.json @@ -0,0 +1,369 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://aicc.dev/schemas/0.1/aicc.schema.json", + "title": "AICC Protocol", + "description": "Schema for the AI-Controlled Character Protocol (AICC) version 0.1. Defines the message envelope, session manifest, tool definitions, and all message types.", + "type": "object", + "oneOf": [ + { "$ref": "#/$defs/sessionInit" }, + { "$ref": "#/$defs/sessionResume" }, + { "$ref": "#/$defs/sessionClose" }, + { "$ref": "#/$defs/manifestRequest" }, + { "$ref": "#/$defs/toolCall" }, + { "$ref": "#/$defs/toolResult" }, + { "$ref": "#/$defs/event" }, + { "$ref": "#/$defs/errorMessage" }, + { "$ref": "#/$defs/heartbeat" } + ], + "$defs": { + "protocolString": { + "type": "string", + "const": "aicc/0.1", + "description": "Protocol version string. Must be exactly 'aicc/0.1' for this version." + }, + "uuid": { + "type": "string", + "format": "uuid", + "pattern": "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$" + }, + "sessionId": { + "type": "string", + "format": "uuid", + "description": "UUID assigned by the bridge at handshake." + }, + "messageId": { + "type": "string", + "format": "uuid", + "description": "UUID for this message. Used for correlation." + }, + "callId": { + "type": "string", + "minLength": 1, + "description": "Agent-assigned id. Echoed in tool_result for correlation." + }, + "envelope": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { + "type": "string", + "enum": [ + "session_init", + "session_resume", + "session_close", + "manifest_request", + "tool_call", + "tool_result", + "event", + "error", + "heartbeat" + ] + }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" } + }, + "additionalProperties": false + }, + "tickMode": { + "type": "string", + "enum": ["fixed", "event", "hybrid"], + "description": "How the environment advances over time. fixed: regular intervals. event: only on tool calls or triggers. hybrid: fixed physics but agents may request immediate evaluation." + }, + "worldKind": { + "type": "string", + "enum": ["2d", "3d", "text", "abstract"], + "description": "Visual / spatial style of the world. Does not constrain tool implementations." + }, + "modelClass": { + "type": "string", + "enum": ["edge_small", "edge_medium", "cloud_medium", "cloud_large"], + "description": "Declared class of the agent's reasoning model. Advisory only — used by the bridge to tune tick rate and sensor cadence." + }, + "toolClass": { + "type": "string", + "enum": ["sensor", "actuator", "generator"] + }, + "errorCode": { + "type": "string", + "enum": [ + "protocol_mismatch", + "session_expired", + "tool_unknown", + "tool_unavailable", + "invalid_input", + "execution_failed", + "timeout", + "internal_error" + ] + }, + "error": { + "type": "object", + "required": ["code", "message", "retryable"], + "properties": { + "code": { "$ref": "#/$defs/errorCode" }, + "message": { "type": "string", "minLength": 1 }, + "retryable": { "type": "boolean" } + }, + "additionalProperties": false + }, + "meta": { + "type": "object", + "description": "Optional environment-supplied metadata for any message that reports state.", + "properties": { + "tick": { "type": "integer", "minimum": 0 }, + "latency_ms": { "type": "integer", "minimum": 0 }, + "source": { "type": "string" } + }, + "additionalProperties": true + }, + "tool": { + "type": "object", + "required": ["id", "class", "description", "input_schema", "output_schema"], + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z][a-z0-9_]*$", + "description": "Lowercase snake_case identifier, unique within the session." + }, + "class": { "$ref": "#/$defs/toolClass" }, + "description": { + "type": "string", + "minLength": 1, + "description": "Human-readable description surfaced to the agent. Should make the tool's purpose and any non-obvious side effects clear." + }, + "input_schema": { + "type": "object", + "description": "JSON Schema describing the tool's input object." + }, + "output_schema": { + "type": "object", + "description": "JSON Schema describing the tool's output object." + }, + "limits": { + "type": "object", + "description": "Rate limits and quotas. Enforced by the bridge.", + "properties": { + "calls_per_minute": { "type": "integer", "minimum": 1 }, + "calls_per_session": { "type": "integer", "minimum": 1 } + }, + "additionalProperties": false + }, + "requires_capability": { + "type": "string", + "description": "Capability the agent must hold to invoke this tool." + }, + "strict_input": { + "type": "boolean", + "default": false, + "description": "If true, the bridge rejects tool calls with input fields not declared in input_schema. Default false (extra fields ignored)." + } + }, + "additionalProperties": false + }, + "capabilities": { + "type": "object", + "required": ["sensors", "actuators", "generators"], + "properties": { + "sensors": { + "type": "array", + "items": { "type": "string" }, + "description": "Reserved capability names enabled for sensors." + }, + "actuators": { + "type": "array", + "items": { "type": "string" }, + "description": "Reserved capability names enabled for actuators." + }, + "generators": { + "type": "array", + "items": { "type": "string" }, + "description": "Reserved capability names enabled for generators." + } + }, + "additionalProperties": false + }, + "modelDecl": { + "type": "object", + "required": ["class"], + "properties": { + "class": { "$ref": "#/$defs/modelClass" }, + "expected_first_token_ms": { + "type": "integer", + "minimum": 0, + "description": "Advisory budget for time-to-first-token." + }, + "expected_full_response_ms": { + "type": "integer", + "minimum": 0, + "description": "Advisory budget for a full tool-call decision cycle." + } + }, + "additionalProperties": false + }, + "sessionInit": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id", "tick_rate_hz", "tick_mode", "world", "capabilities", "tools", "agent_model"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "session_init" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "tick_rate_hz": { + "type": "number", + "exclusiveMinimum": 0, + "description": "Environment's nominal tick rate in Hertz. Informational; bridges may exceed or undershoot." + }, + "tick_mode": { "$ref": "#/$defs/tickMode" }, + "world": { + "type": "object", + "required": ["name", "kind"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "kind": { "$ref": "#/$defs/worldKind" } + }, + "additionalProperties": false + }, + "agent_model": { "$ref": "#/$defs/modelDecl" }, + "capabilities": { "$ref": "#/$defs/capabilities" }, + "tools": { + "type": "array", + "items": { "$ref": "#/$defs/tool" }, + "minItems": 0, + "description": "Full catalog of tools available in this session." + }, + "event_subscriptions": { + "type": "array", + "items": { "type": "string" }, + "description": "Optional. Topics the agent is subscribed to. Empty or omitted means only events emitted as direct responses are sent." + } + }, + "additionalProperties": false + }, + "sessionResume": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "session_resume" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" } + }, + "additionalProperties": false + }, + "sessionClose": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "session_close" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "reason": { "type": "string" } + }, + "additionalProperties": false + }, + "manifestRequest": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "manifest_request" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" } + }, + "additionalProperties": false + }, + "toolCall": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id", "call_id", "tool", "input"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "tool_call" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "call_id": { "$ref": "#/$defs/callId" }, + "tool": { + "type": "string", + "pattern": "^[a-z][a-z0-9_]*$", + "description": "Tool id from the manifest." + }, + "input": { + "type": "object", + "description": "Must validate against the tool's input_schema." + } + }, + "additionalProperties": false + }, + "toolResult": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id", "call_id", "ok"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "tool_result" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "call_id": { "$ref": "#/$defs/callId" }, + "ok": { "type": "boolean" }, + "output": { + "type": "object", + "description": "Tool output. Must conform to the tool's output_schema when ok is true. Absent or null when ok is false." + }, + "error": { + "$ref": "#/$defs/error", + "description": "Present only when ok is false." + }, + "meta": { "$ref": "#/$defs/meta" } + }, + "additionalProperties": false + }, + "event": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id", "topic", "payload"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "event" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "topic": { + "type": "string", + "enum": ["tick", "collision", "audio", "state_change", "agent_message"], + "description": "Event topic. Reserved topics are listed; bridges may define additional topics but agents may not rely on them." + }, + "payload": { + "type": "object", + "description": "Topic-specific payload. Bridges MAY extend with additional fields; agents SHOULD ignore unknown fields." + }, + "meta": { "$ref": "#/$defs/meta" } + }, + "additionalProperties": false + }, + "errorMessage": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id", "error"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "error" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "error": { "$ref": "#/$defs/error" } + }, + "additionalProperties": false + }, + "heartbeat": { + "type": "object", + "required": ["protocol", "type", "session_id", "message_id"], + "properties": { + "protocol": { "$ref": "#/$defs/protocolString" }, + "type": { "const": "heartbeat" }, + "session_id": { "$ref": "#/$defs/sessionId" }, + "message_id": { "$ref": "#/$defs/messageId" }, + "manifest_update": { + "type": "object", + "description": "If present, replaces the current manifest (e.g. capability changes). Same shape as session_init minus session_id and message_id." + } + }, + "additionalProperties": false + } + } +} diff --git a/docs/testbed-design.md b/docs/testbed-design.md new file mode 100644 index 0000000..0ea4cec --- /dev/null +++ b/docs/testbed-design.md @@ -0,0 +1,57 @@ +# Testbed design notes + +How the aicc-capsule testbed maps onto the AICC Protocol. Read `aicc-core.md` for the protocol itself; this file is testbed-specific. + +## World + +- One room: a floor, four walls, 2–3 obstacles (boxes), one interactable (a glowing beacon). +- Coordinate system: right-handed, Y-up. Room ~16×16 units. +- The capsule starts at a fixed corner; the beacon sits in the opposite area. + +## The capsule + +- A cylinder/capsule body with a heading (yaw) and a camera (pitch). +- Physics: simple — position, velocity, collision against walls/obstacles. AABB or capsule-vs-box is enough. No gravity needed (or trivial gravity). +- `move(forward)` pushes along heading; `turn(yaw)` rotates; collisions stop movement. + +## Tools (all must exist in the bridge) + +| id | class | purpose | returns | +|----|-------|---------|---------| +| `proprioception` | sensor | agent's own state | position, rotation, velocity, health | +| `vision` | sensor | first-person RGB frame | base64 PNG + width/height/tick | +| `hear` | sensor | audio events since last call | list of {kind, direction, intensity} | +| `move` | actuator | translate along heading | new position | +| `turn` | actuator | rotate yaw/pitch | new rotation | +| `look_at` | actuator | orient camera at a target | new rotation | +| `interact` | actuator | use the beacon | result message | + +Optional: `depth` (depth map), `world_query` (room bounds). If you add tools beyond the list, document them in the bridge manifest via descriptions. + +## Vision + +The most important sensor. Render the capsule's view to an image and return it as base64 PNG. Resolution small (e.g. 160×120) to keep latency and tokens down. If the engine can't render, fall back to a canvas-drawn approximation (raycast floor + box silhouettes) — but it must reflect actual world state, not a placeholder. + +## Events + +- `collision` event with payload `{other, normal, impulse}` when the capsule hits something. +- Use `bridge.emit_event(...)` (available in aicc-py) from tool handlers. +- The agent can subscribe to `tick` for a periodic heartbeat if useful. + +## Agent loop (demo) + +1. `AICCClient(WebSocketClientTransport("ws://localhost:8765"))` +2. `handshake()` → manifest +3. Loop: call `vision` + `proprioception`, feed to LLM with tool schemas, execute returned tool calls, repeat until `interact` succeeds. +4. Print every tool call and result to stdout (transcript). + +The demo should work with any tool-calling LLM. Provide a generic loop that takes a model function; include one example wired to a local/cheap model (ollama or similar) and note in README how to swap providers. + +## Conformance + +The bridge must pass all 9 core scenarios. Note: the conformance reference bridge registers tools `echo`, `boom`, `bump` in addition to the world tools — register those three on the testbed bridge too (trivial: echo returns input; boom raises; bump emits a collision event) so the scenario suite runs green against the same bridge instance used in the demo. + +## Non-goals + +- No networking beyond WebSocket. No multi-agent. No persistence. No rendering window (headless preferred; a window is optional debug aid). +- No engine-specific protocol extensions. If the engine needs something extra, it goes in the manifest as an extra tool, not a protocol change. diff --git a/scripts/BUILDLOG.md b/scripts/BUILDLOG.md new file mode 100644 index 0000000..6d08ade --- /dev/null +++ b/scripts/BUILDLOG.md @@ -0,0 +1,9 @@ +# aicc-capsule build log +# Fill this in as you go. Keep it short — one line per milestone. + +- [ ] 2026-08: context pack created (AGENTS.md, docs, scripts) +- [ ] bridge skeleton: world + capsule + WebSocketServer +- [ ] tools: proprioception, vision, move, turn, look_at, interact (+ echo/boom/bump for conformance) +- [ ] conformance 9/9 green +- [ ] agent demo loop (LLM drives capsule to beacon, interacts) +- [ ] README in testbed/ with run instructions diff --git a/scripts/setup.sh b/scripts/setup.sh new file mode 100755 index 0000000..be7efc7 --- /dev/null +++ b/scripts/setup.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Setup script for aicc-capsule: create venv, install deps. +set -euo pipefail +cd "$(dirname "$0")/.." + +python3 -m venv testbed/.venv +source testbed/.venv/bin/activate +pip install --upgrade pip +# aicc-py SDK from sibling checkout +pip install -e "$HOME/Desktop/aicc-py" +# Engine and demo deps go here (e.g. ursina, panda3d, pillow, websockets) +# pip install ... + +echo "Setup done. Activate with: source testbed/.venv/bin/activate"