AICC-Protocol/core.md

# AICC Protocol — Core Specification (Draft 0.1)

## Overview

AICC (AI-Controlled Character) is a transport-agnostic protocol
for connecting language models to interactive virtual environments. It defines
how an agent perceives a world, reasons about it, and acts within it through
a structured set of sensor, actuator, and generator tools.

The protocol makes no assumptions about:
- The host engine (Godot, Unity, Unreal, custom simulators, robotics stacks)
- The reasoning model (any LLM with tool-use capability)
- The transport layer (HTTP, WebSocket, stdio, gRPC, in-process)
- The visual style (2D, 3D, text-based, abstract)

It only specifies message formats, tool taxonomy, capability negotiation, and
causality semantics.

## Design Goals

1. **Engine-agnostic.** The same protocol speaks to a game character, a robot,
   or a sandboxed LLM playroom.
2. **Model-agnostic.** Any LLM that supports tool-use / function-calling can
   act as the reasoning layer. No prompt engineering baked into the wire
   format.
3. **Transport-agnostic.** Wire format is JSON. Transport is pluggable.
4. **Capability-aware.** An environment advertises what an agent may do. The
   agent never has to guess.
5. **Deterministic when needed.** Every tool call returns a structured
   outcome with timestamps, source, and confidence. Replay is a first-class
   concept.
6. **Extensible.** New sensors, actuators, and generators can be added
   without breaking older agents, provided they follow the extension rules
   in §11.
7. **Single source of truth.** All world data flows through sensor tools.
   The manifest carries session metadata only — never world state. This
   makes the contract honest and uniform across any environment.

## 1. Model of the World

An AICC interaction involves three actors:

- **Agent** — the reasoning layer (typically an LLM). Holds goals, plans,
  and short-term working memory.
- **Environment** — the host system that owns world state. Renders, ticks
  physics, runs NPCs, persists state.
- **Bridge** — the thin adapter that translates between the protocol and
  the environment's native APIs.

```
+----------------+        +-------------+        +------------------+
|     Agent      |  <-->  |   Bridge    |  <-->  |   Environment    |
| (LLM + tools)  | AICC   | (adapter)   | native| (engine/world)   |
+----------------+        +-------------+        +------------------+
```

The bridge is implementation-specific and not part of this spec. The agent
sees only the AICC contract.

### Tick

A **tick** is one discrete update of the environment. The cadence is defined
by the environment (e.g. 60 Hz physics, 10 Hz AI updates, event-driven).
Agents do not assume a fixed tick rate — they negotiate it via the session
manifest (§3).

### Session

A **session** is the lifetime of one agent connected to one environment.
Sessions are identified by a UUID assigned at handshake. Sessions are
stateful: tool calls within a session may accumulate history that influences
later calls (e.g. episodic memory).

## 2. Message Format

All messages are UTF-8 JSON. Every message carries a top-level `protocol`
field with the version string `"aicc/0.1"`.

### Common envelope

```json
{
  "protocol": "aicc/0.1",
  "type": "...",
  "session_id": "...",
  "message_id": "..."
}
```

| Field         | Type   | Notes                                             |
|---------------|--------|---------------------------------------------------|
| `protocol`    | string | Must be `"aicc/0.1"` for this version.            |
| `type`        | string | One of the message types below.                   |
| `session_id`  | string | UUID assigned by the bridge at handshake.         |
| `message_id`  | string | UUID for this message. Used for correlation.      |

### Message types

| `type`              | Direction            | Purpose                                    |
|---------------------|----------------------|--------------------------------------------|
| `session_init`      | bridge -> agent      | Open a new session, advertise manifest.    |
| `session_resume`    | agent -> bridge      | Resume a session by `session_id`.          |
| `session_close`     | both                 | Graceful close.                            |
| `manifest_request`  | agent -> bridge      | Re-fetch the manifest.                     |
| `tool_call`         | agent -> bridge      | Invoke a tool.                             |
| `tool_result`       | bridge -> agent      | Return outcome of a tool call.             |
| `event`             | bridge -> agent      | Asynchronous notification from environment.|
| `error`             | bridge -> agent      | Protocol-level or environment error.       |
| `heartbeat`         | both                 | Keep-alive ping with optional payload.     |

### Versioning

The protocol string uses semantic versioning (`aicc/MAJOR.MINOR`).
- MAJOR bump: any breaking change to message format or tool schema.
- MINOR bump: additive changes only. Agents must accept messages from a
  bridge with a higher MINOR (forward compatibility), ignoring unknown
  optional fields.

## 3. Session Manifest

Sent in `session_init` and on `manifest_request`. Describes the environment
and what the agent may do.

```json
{
  "type": "session_init",
  "session_id": "8f3a...",
  "tick_rate_hz": 10,
  "tick_mode": "fixed",
  "world": {
    "name": "testbed_room_01",
    "kind": "3d"
  },
  "capabilities": {
    "sensors":    ["vision", "depth", "proprioception", "memory_query"],
    "actuators":  ["move", "turn", "look_at", "interact", "say"],
    "generators": []
  },
  "tools": [
    {
      "id": "vision",
      "class": "sensor",
      "description": "Return the current first-person RGB frame as base64 PNG.",
      "input_schema":  { "type": "object", "properties": {}, "additionalProperties": false },
      "output_schema": { "type": "object", "properties": { "png_b64": { "type": "string" }, "width": { "type": "integer" }, "height": { "type": "integer" }, "tick": { "type": "integer" } }, "required": ["png_b64", "width", "height", "tick"] }
    }
  ]
}
```

> **Note.** World state — including the agent's own position, rotation,
> inventory, and the world's bounds — is **never** included in the manifest.
> It is accessible only through the appropriate sensor tools. The manifest
> is a contract on the session, not a snapshot of the world.

### Tick modes

- `fixed` — environment advances in regular intervals. Tools see a coherent
  snapshot.
- `event` — environment advances only on tool calls or external triggers.
- `hybrid` — fixed physics tick, but agents may request immediate evaluation
  via a `tick_now` tool.

## 4. Tool Taxonomy

Three top-level classes. Every tool belongs to exactly one.

| Class        | Side effect | Returns                          |
|--------------|-------------|----------------------------------|
| `sensor`     | none        | Read-only observation of world.  |
| `actuator`   | yes         | Action performed on the world.   |
| `generator`  | yes         | New world content created.       |

A tool is identified by its `id`. IDs are scoped to the session.

### Standard sensors (initial set)

| ID              | Returns                                           |
|-----------------|---------------------------------------------------|
| `vision`        | RGB frame from agent's camera.                    |
| `depth`         | Depth map aligned to vision.                      |
| `hear`          | Audio events since last call.                     |
| `smell`         | Active scent zones near the agent.                |
| `touch`         | Surface contact info from last physics tick.      |
| `proprioception`| Position, rotation, velocity, health, inventory.  |
| `memory_query`  | Episodic memory hits.                             |
| `inspect`       | Detailed view of a target entity or location.     |

### Standard actuators (initial set)

| ID          | Effect                                              |
|-------------|-----------------------------------------------------|
| `move`      | Translate along agent-relative or world axes.       |
| `turn`      | Rotate yaw / pitch.                                 |
| `look_at`   | Orient camera toward a target.                      |
| `interact`  | Use / activate / pick up / talk to an entity.       |
| `say`       | Emit speech (or text for dialog-only sessions).     |
| `wait`      | Skip ticks without action.                          |

### Standard generators (initial set)

| ID             | Effect                                            |
|----------------|---------------------------------------------------|
| `place_object` | Spawn a registered prefab at a location.          |
| `modify_terrain`| Edit terrain heightmap / material.               |
| `spawn_entity` | Create a registered entity at a location.         |

Generators require the `can_modify_world` capability and a per-tool rate
limit. They are optional in the manifest.

## 5. Tool Invocation

### Request (`tool_call`)

```json
{
  "type": "tool_call",
  "message_id": "...",
  "call_id": "tc_001",
  "tool": "vision",
  "input": {}
}
```

| Field        | Type   | Notes                                          |
|--------------|--------|------------------------------------------------|
| `call_id`    | string | Agent-assigned. Echoed in `tool_result`.      |
| `tool`       | string | Tool id from the manifest.                     |
| `input`      | object | Must validate against the tool's input schema. |

### Response (`tool_result`)

```json
{
  "type": "tool_result",
  "message_id": "...",
  "call_id": "tc_001",
  "ok": true,
  "output": { "...": "..." },
  "meta": {
    "tick": 142,
    "latency_ms": 12,
    "source": "render_thread"
  }
}
```

| Field         | Type    | Notes                                       |
|---------------|---------|---------------------------------------------|
| `call_id`     | string  | Echoed from the request.                    |
| `ok`          | boolean | False on tool-level failure.                |
| `output`      | object  | Conforms to the tool's output schema.       |
| `error`       | object  | Present only when `ok` is false.            |
| `meta`        | object  | Optional environment metadata.              |

### Failure shape

```json
{
  "ok": false,
  "error": {
    "code": "tool_unavailable",
    "message": "Generator 'spawn_entity' requires capability 'can_spawn_entities'.",
    "retryable": false
  }
}
```

Standard error codes are listed in §10.

## 6. Events

Asynchronous notifications from the environment. The agent may subscribe to a
subset at session init.

```json
{
  "type": "event",
  "topic": "collision",
  "payload": {
    "other": "wall_segment_03",
    "normal": { "x": -1, "y": 0, "z": 0 },
    "impulse": 4.2
  },
  "meta": { "tick": 142 }
}
```

Reserved topics:

- `tick` — fired on every environment tick (only if subscribed).
- `collision` — physical contact.
- `audio` — sound event outside the agent's request cycle.
- `state_change` — entity added, removed, or substantially changed.
- `agent_message` — incoming inter-agent communication in multi-agent sessions.

## 7. Capabilities

Capabilities are advertised in the manifest. They are coarse-grained
permissions, not per-entity ACLs. Fine-grained permissions are an extension
(see §11).

### Model class declaration

The bridge declares which model class the agent runs on. This lets
environments tune tick rates, sensor cadence, and event batching to match
the agent's expected response time.

```json
{
  "agent_model": {
    "class": "edge_small | edge_medium | cloud_medium | cloud_large",
    "expected_first_token_ms": 800,
    "expected_full_response_ms": 2500
  }
}
```

| Class          | Typical target                              | First-token budget |
|----------------|---------------------------------------------|--------------------|
| `edge_small`   | E2B / phone-class, on-device                | 100-400 ms         |
| `edge_medium`  | E4B / 12B on consumer GPU                   | 300-800 ms         |
| `cloud_medium` | 12B-26B MoE via hosted inference            | 500-1500 ms        |
| `cloud_large`  | 31B+ via hosted inference, with reasoning   | 1500-3000 ms       |

These are advisory. The bridge uses them to inform tick-rate choice and
sensor buffering strategy. They do not constrain the agent — the agent
may exceed or beat the budget — but environments should design their
default cadence so that a conformant agent of the declared class can
participate in real-time interaction.

### Reserved capabilities

- `can_observe_world` — read any sensor.
- `can_move_self` — invoke movement actuators.
- `can_modify_world` — invoke generators.
- `can_spawn_entities` — generators that create NPCs / objects.
- `can_modify_terrain` — generators that change terrain.
- `can_communicate` — invoke `say` and `agent_message` events.
- `can_persist_state` — write to long-term storage.

An agent must not call a tool that requires an absent capability. Bridges
should reject such calls with `tool_unavailable`.

## 8. Time and Causality

### Tick clock

Every message that carries a `meta.tick` field references the same monotonic
counter maintained by the environment. The agent uses this to reason about
ordering of observations.

### Eventual consistency for events

`event` messages may arrive between `tool_result` messages. The agent MUST
process events in `meta.tick` order; out-of-order arrival is the bridge's
responsibility to prevent or flag.

### Heartbeat

For long sessions, the bridge may send `heartbeat` every N seconds. Agents
may respond with `heartbeat` of their own. This is also the vehicle for
pushing manifest updates (e.g. capability changes).

## 9. Memory and State

Memory is split:

- **World state** — owned by the environment, accessed **only** via sensors.
  The manifest carries no world state.
- **Agent memory** — owned by the agent. Not transmitted over AICC unless
  via the optional `memory_query` sensor, which is environment-side episodic
  memory tagged with session events.

The protocol does not prescribe how an LLM maintains its context window.
That's the agent's concern. The protocol only guarantees that any state
observable via a sensor is reproducible from the session.

### Single source of truth

Every piece of world data has exactly one way to reach the agent: through
a sensor tool. The agent's own position, the world's bounds, the contents
of nearby containers, the time of day — all of these are observable only
by calling the appropriate sensor. The manifest never duplicates or
caches this data. This eliminates a class of bugs where handshake data
drifts from real-time sensors, and keeps the contract uniform across
single-agent and multi-agent sessions.

## 10. Errors

Standard error codes returned in `tool_result.error` or top-level `error`
messages:

| Code                 | When                                                |
|----------------------|-----------------------------------------------------|
| `protocol_mismatch`  | Wire version incompatible.                          |
| `session_expired`    | Session id unknown or closed.                        |
| `tool_unknown`       | Tool id not in manifest.                             |
| `tool_unavailable`   | Capability missing or rate limit hit.                |
| `invalid_input`      | Input fails schema validation.                       |
| `execution_failed`   | Tool ran but failed at the environment level.        |
| `timeout`            | Tool exceeded environment-defined timeout.           |
| `internal_error`     | Unspecified bridge failure.                          |

All errors include `retryable: boolean` to indicate whether the agent may
safely retry the same call.

## 11. Extension Rules

New tools and capabilities MAY be added in a MINOR version. Additions MUST:

1. Use a new `id` not previously registered.
2. Be advertised in the manifest.
3. Define input and output schemas as JSON Schema.
4. Use only documented error codes.

Breaking changes (renaming, removing, changing semantics of existing tools,
changing required fields) MUST trigger a MAJOR bump.

Agents SHOULD ignore unknown fields in `meta` and unknown optional fields in
`output`. Bridges SHOULD accept tool calls with extra fields by ignoring them
(unless `strict_input: true` is set on the tool in the manifest).

## 12. Security and Trust

- The bridge authenticates the agent (out of band of this spec).
- The agent authenticates the bridge (out of band of this spec).
- Tools that may be expensive or destructive MUST advertise rate limits in
  the manifest under `tool.limits`, e.g. `{"calls_per_minute": 60}`.
- All transport MUST be encrypted when crossing untrusted networks. AICC
  itself does not mandate TLS — that is a transport concern.

## 13. Conformance

A bridge is AICC-conformant for version `aicc/0.1` if it:

1. Emits and accepts the message types in §2.
2. Validates all `tool_call` inputs against the manifest.
3. Returns errors using the codes in §10.
4. Preserves event ordering per §8.
5. Passes the conformance scenarios in `conformance/scenarios/`.

An agent is conformant if it:

1. Never invokes tools outside its advertised capabilities.
2. Echoes `call_id` correctly on result correlation.
3. Handles all standard error codes.
4. Respects tick ordering for events.

## Appendix A: Example — Walking Around a Room

1. Bridge -> Agent: `session_init` with vision, depth, proprioception,
   move, turn, look_at.
2. Agent -> Bridge: `tool_call` `vision`.
3. Bridge -> Agent: `tool_result` with PNG.
4. Agent reasons, decides to walk forward.
5. Agent -> Bridge: `tool_call` `move` `{ "forward": 2.0 }`.
6. Bridge -> Agent: `tool_result` with new proprioception snapshot.
7. Bridge -> Agent: `event` `collision` when wall hit.
8. Agent -> Bridge: `tool_call` `turn` `{ "yaw": -45 }` to redirect.
9. Agent -> Bridge: `tool_call` `move` `{ "forward": 1.5 }`.
10. Loop continues.

## Appendix B: Open Questions (to resolve in 0.2)

- How are multi-agent sessions coordinated? Sub-sessions or shared session?
- How are assets (prefabs, models, audio) referenced — by id, URL, or
  inline binary?
- Should `vision` support streaming chunks, or is single-frame enough?
- Standard for episodic memory schema (events vs summaries vs both)?
- Should `inspect` allow arbitrary queries, or a fixed set of detail
  modes (`color`, `bbox`, `semantic`, `full`)?
- Resolved in 0.1.1: model class declaration (see §7).

---

Status: **draft**. Not yet published. Internal review only.