vision: images are the primary channel for multimodal models (digest off by default); digest is the text-only fallback; --digest/--no-digest to override
This commit is contained in:
+11
-9
@@ -103,17 +103,19 @@ many tool steps the model may chain per request (0 = one action per turn).
|
||||
|
||||
### Real vision
|
||||
|
||||
By default the model gets frames as a color-grid digest (works for any
|
||||
text-only model). If the model can actually see images (gemma3/gemma4,
|
||||
qwen2.5-vl, gpt-4o-mini, ...), pass the frames as real images:
|
||||
Vision is the **primary channel**: when the model is multimodal, `vision`
|
||||
tool results attach the actual camera frame as an image to the conversation —
|
||||
the model sees the crate, the wall, the glowing beacon. The color-grid
|
||||
digest is only the **fallback** for text-only models:
|
||||
|
||||
- auto-detected for local ollama (`/api/show` capabilities) — nothing to do
|
||||
- otherwise force it: `--vision` (or `--no-vision` to disable)
|
||||
- multimodal model (auto-detected for local ollama via `/api/show`;
|
||||
otherwise `--vision`): frame as image, digest off
|
||||
- text-only model: digest on automatically; `--digest` forces the digest to
|
||||
be included even alongside images, `--no-digest` disables it everywhere
|
||||
|
||||
With vision ON, `vision` tool results attach the actual camera frame as an
|
||||
image to the conversation — the model sees the crate, the wall, the glowing
|
||||
beacon, and orients itself. Verified locally: gemma4:12b completed a beacon
|
||||
mission on its first attempt using look_at/move navigation.
|
||||
Verified locally: gemma4:12b sees frames and completed a beacon mission on
|
||||
its first attempt; gemma4:e2b describes what it sees (crates, beacon) and
|
||||
orients itself from the image.
|
||||
|
||||
## Real-time mode
|
||||
|
||||
|
||||
Reference in New Issue
Block a user