Emil Shanaty
fa5ef35bff
Declarative conformance scenarios for aicc/0.1 covering: handshake, manifest_request, tool call ok/unknown/execution-failed, protocol mismatch, heartbeat, session close, event delivery. Plus runner documentation (conformance/README.md).
75 lines
2.7 KiB
Markdown
75 lines
2.7 KiB
Markdown
# AICC Conformance Scenarios
|
|
|
|
This directory contains declarative conformance scenarios for the AICC Protocol (aicc/0.1).
|
|
|
|
Each JSON file describes a sequence of messages exchanged with a bridge and
|
|
the expected responses. Any bridge that passes all scenarios in this
|
|
directory satisfies the conformance criteria in §13 of `../core.md`.
|
|
|
|
## Scenario format
|
|
|
|
```json
|
|
{
|
|
"id": "core-01-handshake",
|
|
"title": "Human readable title",
|
|
"category": "core",
|
|
"version": "aicc/0.1",
|
|
"steps": [
|
|
{ "send": { "type": "tool_call", "tool": "echo", "input": {} } },
|
|
{ "expect": { "type": "tool_result", "ok": true, "call_id_echo": true } }
|
|
]
|
|
}
|
|
```
|
|
|
|
### Steps
|
|
|
|
| Step kind | Semantics |
|
|
|-----------|-----------|
|
|
| `send` | Send a message to the bridge. `session_id` and `message_id` are filled in by the runner if absent. |
|
|
| `expect` | Wait for the next message and assert it matches the expectation. |
|
|
|
|
### Expectation matching
|
|
|
|
Expectations are matched **subset-wise**: every field present in the
|
|
expectation must match the received message; extra fields in the received
|
|
message are allowed.
|
|
|
|
Special matchers:
|
|
|
|
| Field | Meaning |
|
|
|-------|---------|
|
|
| `call_id_echo: true` | Received `call_id` must equal the `call_id` of the last `send`. |
|
|
| `session_id_echo: true` | Received `session_id` must equal the session's id. |
|
|
| `{"$kind": "uuid"}` | Field must look like a UUID. |
|
|
| `{"$kind": "integer"}` / `{"$kind": "array"}` | Field must be of that JSON type. |
|
|
| `{"$gt": N}` | Field must be numerically greater than N. |
|
|
| `{"$enum": [...]}` | Field must be one of the listed values. |
|
|
| `{"$required": [...]}` | Object field must contain these keys. |
|
|
|
|
## Running
|
|
|
|
Reference runner: `aicc-py` (see https://github.com/emil28092005/aicc-py).
|
|
Run `python -m aicc.conformance <scenarios-dir>` or the pytest suite.
|
|
|
|
## Scenario inventory
|
|
|
|
| ID | Checks |
|
|
|----|--------|
|
|
| core-01-handshake | Valid session_init on connect |
|
|
| core-02-manifest-request | manifest_request returns manifest |
|
|
| core-03-tool-call-ok | Valid tool call succeeds, call_id echoed |
|
|
| core-04-tool-unknown | Unknown tool → tool_unknown |
|
|
| core-05-tool-execution-failed | Throwing tool → execution_failed |
|
|
| core-06-protocol-mismatch | Wrong protocol → protocol_mismatch |
|
|
| core-07-heartbeat | Heartbeat acknowledged |
|
|
| core-08-session-close | Close ends session, then session_expired |
|
|
| core-09-event-delivery | Environment events delivered |
|
|
|
|
## Adding scenarios
|
|
|
|
1. Create `N-<name>.json` following the format above.
|
|
2. Add a row to the inventory table.
|
|
3. Run the runner against the reference bridge in aicc-py.
|
|
4. Scenarios requiring tools not in the reference bridge must document
|
|
the requirement in `description`.
|