shacraft-core/docs/WORLD_STREAMING.md

# Procedural worlds and movement diagnostics

The server creates `overworld` on startup if that name is unused. Open
`http://localhost:4000/?world=overworld`, or select it in the world menu.
Existing worlds, packages, edits and arenas remain available. An existing world
named `overworld` is never automatically converted.

## World dimensions and generation

- Playable horizontal coordinates: **−29,999,872 through +29,999,872**.
- Building layers: **Y = −64 through 319**, inclusive: **384 layers**.
- Default seed: `20260914`; generator version: `1`.
- Original deterministic terrain: continents, hills, mountains, rivers,
  oceans, forests, plains, desert areas, snow, trees, caves, tunnels, aquifers,
  deep stone, ores, low lava pockets and a bedrock bottom.
- A spawn clearing and a narrow ravine near X=30 give access to the surface and
  underground. Fluids are generated blocks; they do not yet simulate flow.

The height range follows the modern Java Overworld convention described in
[Mojang's Caves & Cliffs Part II announcement](https://www.minecraft.net/fr-ca/article/caves---cliffs--part-ii-out-today-java).
Terrain generation is independently authored. Matching Minecraft seed output,
all vanilla biomes, structures and gameplay systems is outside this implementation.

The version and seed are durable in `server.sqlite3`. Generation depends on
absolute integer coordinates, so request order and section boundaries do not
change trees, caves or terrain. Keep generator version 1 stable for saved worlds;
future algorithms need a new version and an explicit migration strategy.

Player simulation, action rays and spawn coordinates use double precision.
Meshes store section-local float positions; draw matrices and dynamic actors use
an origin near the camera. Small geometry and movement steps remain representable
near the horizontal border. Arena/entity administration retains its earlier
configuration limits where documented.

## CPU work, streaming and storage

The server uses **two CPU generation threads** with **32 pending jobs** at most.
The simulation thread owns storage and gameplay. Requests are shared between
players and ordered by distance, with extra weight on vertical distance. Old
view requests are dropped; completed jobs outside current interests are discarded.
Control reads retain bounded, temporary interests so remote reads can complete
without a nearby player. Generation does not require a GPU.

**Game menu → Render distance** selects a radius from 2 to 64 chunks
(32–1,024 blocks around the player), default 3. Each player chooses their own
radius; changes apply without reconnecting and persist locally. The current
client negotiates an extra loaded chunk ring beyond that visible radius. Its
default data window is **9×9×24 sections**, or **144×384×144 blocks / 1,944 sections**.
At radius 8 it is **19×19×24**, or **304×384×304 blocks / 8,664 sections**.
At radius 64 it is **131×131×24**, or **2,096×384×2,096 blocks / 411,864 sections**,
including the one-chunk preload ring. The queue covers this entire window.
It is sorted once when the view changes; each tick examines a bounded lookahead
of 256 pending sections plus collision neighborhoods. Up to 128 sections are
sent per 50 ms interval, including at most 32 nonuniform sections, with a
256 KiB estimated section-payload budget, backpressure and time budgets.

Horizontal streaming has one chunk of hysteresis: crossing a single chunk
boundary does not move the loading anchor or unload its opposite edge. The
anchor moves after a two-chunk displacement. The extra ring preserves coverage
of the selected radius between anchor changes. Generation, delivery and meshing
prioritize nearby sections; overlapping cells and meshes remain in place.
Expansion retains overlap; explicit reduction unloads excess sections. Distance
fog completely blends the visible edge into the sky before the buffer is
unloaded. Horizontal culling and the projection far plane follow the radius.

The current client negotiates `full_height_v1`: every loaded column spans
**Y=−64 through 319**, all 24 vertical sections. Climbing and flying above the
world ceiling retain ground and roof sections. Missing interior terrain still
blocks prediction; space outside the world height is known void. Clients that
do not negotiate this feature retain the legacy 80-block sliding window.
Distant-terrain LOD is not implemented; these bounds and local benchmarks do
not establish parity with Minecraft's renderer.

Already delivered distant sections do not need to remain in the server cache;
only collision neighborhoods and unsent client sections trigger regeneration.
Requests from multiple players still deduplicate, including a new player
requesting terrain previously sent to someone else. Natural sections live in a bounded LRU: at most 8,192 decoded sections, or
128 MiB of voxel payload. This is separate from the core's encoded-section
cache, SQLite, metadata, meshes and total process RSS. Visiting more terrain does
not allocate a permanent world-sized array. Before editing a natural section,
the server materializes its immutable baseline in `generated_sections` in
`worlds.sqlite3`; the existing revision/idempotency journal then stores the edit.
An explicit air deletion survives cache eviction and restart. Undo/reset use
the saved baseline. World templates inherit the generator configuration and
pin existing edited sections. Natural materialization does not advance the
edit revision. Standalone core exports contain stored sections; exporting the
entire procedural world is not supported. Back up both SQLite databases.

The browser stores nonuniform cells in `Uint32Array` sections: **30.38 MiB** for a dense
1,944-section default voxel map, or **135.38 MiB** at radius 8, excluding worker
copies, lighting and GPU geometry. Uniform received sections use four bytes
instead of 16 KiB, retain known-air readiness, and expand on edit. Empty sections
need no mesh job. Lighting has a separate allocation cap of 7,393,280 cells;
it does not scale with the maximum stream volume. With the local mesh worker,
the shared actor field is limited to 112×384×112 blocks around the view center.
Palette/RLE avoids JSON objects and position strings per block.

A dedicated terrain-mesh worker builds local light, AO, culling and geometry
without waiting for whole-view lighting. A light tile covers 2×2 chunk columns
plus an 18-block horizontal halo, at most **68×384×68 / 1,775,616 cells**. The full
active vertical range preserves skylight under tall roofs. An eight-entry LRU
shares these fields between nearby and vertically stacked sections. Spatial
invalidation and per-section tickets reject obsolete work without discarding
nearby results on every distant section batch. Moving a distant view edge keeps
an interior tile's light and mesh valid if its sampling bounds are unchanged.

A separate persistent worker computes a bounded local light field for actor and quick-edit
sampling, starting after 225 ms without a new sync. Its completion does not gate
terrain publication. Large views still need time to finish distant meshes.
Uploads are staged in 64 KiB steps targeting 2 ms per frame; old meshes remain
until their replacements are ready. Driver calls are not preemptible.

Unknown sections are distinct from received air. Both authoritative simulation
and prediction wait for collision data instead of falling through missing
terrain. The server resets the input epoch on a terrain wait, holds the body,
and resumes when its neighborhood is available. Teleports also reset prediction.

Skylight includes full-height column occlusion data. A cave remains dark when
its roof lies above the streamed vertical window. Editing a roof updates its
column data. The background uses the camera's sky exposure to avoid bright sky
appearing through unloaded underground terrain. Full lateral lighting outside
the active window is not simulated.

## Section protocol v2

Join with `features:["chunk_stream_v2","movement_prediction_v1","view_buffer_v1","full_height_v1"]` and optional
`view_distance:2..64` (default 3). A server advertising `view_distance_v1` accepts
`{"type":"view_distance","chunks":64}` while connected. Noninteger or out-of-range
values are rejected before mutation; changes are limited to once per 500 ms.
The server responds with a normal `view` transition containing `view_distance`
and the new `total_sections`, preserving position, motion epoch and revision.
Negotiating `view_buffer_v1` adds the extra loaded ring and horizontal hysteresis;
`stream_radius` reports the loaded radius, while `view_distance` remains the
selected visible radius. Older v2 clients retain the unbuffered behavior: 245
sections by default and 1,445 at radius 8.
`full_height_v1` fixes the vertical bounds to the world height and adds
`full_height:true` to view/welcome/snapshot messages. It also enables batches
of up to 128 sections; older clients receive at most 32.
Welcome/snapshot messages also expose `max_view_distance:64`. The initial
`welcome` or explicit `snapshot` supplies the world, revision, motion, generation,
`view_center`, inclusive `view_min`/`view_max`, `total_sections`, terrain settings
and world bounds. Its `blocks` array is empty; sections follow incrementally.
Generation numbers increase for each new view or resync.

A `view` message updates bounds and lists departing section coordinates in
`unload`. Retained sections keep their current data. A `sections` message contains
at most 128 complete sections (32 for legacy clients), each as:

```json
{"section":[0,4,0],"palette":[0,123],"runs":[256,1,3840,0]}
```

Runs are `(count, paletteIndex)` pairs totaling exactly 4096 cells. Cell order is
`x + 16*z + 256*y`. Section coordinates are integers; world coordinates are
section coordinates multiplied by 16. Even empty sections are sent explicitly.
Messages include `world`, `revision`, `generation`, material definitions and
`columns:[{column:[sectionX,sectionZ],heights:[256 values]}]`.

Batches run at most once per 50 ms per client. The server pauses section sends
above 512 KiB of queued output; existing queue and timeout limits still apply.
Packing has a soft 10 ms work budget. Clients validate the complete batch before
mutation, ignore obsolete generations and resync on revision gaps. Ordinary
block edits advance revisions; view changes never do.

Old clients retain v1/full-snapshot behavior in nonprocedural worlds. A client
without v2 is rejected from a procedural world with an instruction to refresh.

## Diagnostics

Open **F3 → Movement diagnostics**. Choose manual recording, automatic flight,
automatic running, or flight with turns, then start a 30-second recording.
Automatic scenarios use ordinary authoritative input, climb obstacles during
flight, and need no pointer lock. Recording starts after terrain/meshes settle.
Stop or Esc cancels automatic movement. Hiding the tab stops recording and marks
the result `interrupted: "tab-hidden"`.

The panel stays open while playing or recording; clicking the world restores control, and F3 or the close button hides it. Its controls pause manual input while focused. The panel shows a rolling frame/CPU chart and a summary. **Download JSON** exports
bounded frame samples, network/stream events and server samples. It measures
frame p50/p95/p99/max, frames over 25/50 ms, main-thread physics/update and draw
submission time, long tasks, position corrections, terrain waits, distance,
view changes, batches and mesh resets. Draw CPU duration is not GPU duration.
The canvas DOM exposes the summary as `data-dynamics-result`, plus current
section counts, coordinates, readiness and streaming timings for browser checks.
`data-first-terrain-ms` measures first nonempty mesh publication after the world
reset. `data-near-meshes` / `data-near-loaded` count completed and received
sections in the camera's 3×3×3 neighborhood, including completed empty sections.
`data-terrain-meshing`, `data-local-light-cells` and `data-local-light-ms` expose
the independent geometry path and its local lighting work.

Authenticated Control API methods for repeatable diagnosis:

```json
{"method":"diagnostics.snapshot","params":{}}
{"method":"terrain.inspect","params":{"world":"overworld","position":[1000000,0,-1000000]}}
{"method":"player.teleport","params":{"id":"player-id","position":[1000000.125,180,-1000000.875],"flying":true}}
{"method":"world.create","params":{"world":"another-overworld","terrain":{"seed":42,"version":1}}}
```

`diagnostics.snapshot` reports generation queues/cache, simulation timings,
positions, velocity, input backlog, waits and per-player stream state, including
`view_distance` and `stream_radius`. Teleport
is an administrative API, not a client movement permission. `/api/metrics`
adds generation/stream counters and the most recent 600 tick durations.
`world.read`, `world.edit`, build commit and `camera.capture` may return
`terrain pending` for cold sections; retry the same read or idempotent operation
after a short interval. No blocking generation is performed for a cold read.

## Local verification, 2026-09-14

A release server and the real browser ran against an isolated copy of the local
data, using the Pixel32 Study package. Automatic flight traversed **495.08 m in
30.003 s**, with **39 view changes and 78 section batches**. Across 4,317 frames,
frame p95 was **7.0 ms**, p99 **7.1 ms**, maximum **13.8 ms**; no frame exceeded
25 ms. The run reported no terrain-wait frames, position correction or full mesh
reset. Section application p95 was 5.6 ms, maximum 7.3 ms. At the last server
sample, tick work was 0.53 ms and RSS was 78.5 MiB. Three entering sections were
still in transit at recording end; all 245 completed after stopping.

These measurements are one local workload, not a comparison with Minecraft or
a guarantee for other devices, multiplayer loads or long sessions. The local
summary is saved in ignored `artifacts/large-world-flight.json`.

Automated checks cover generation determinism, negative/far coordinates, deep
content, skyline accuracy, durable mining/undo/restart, remote reads without
players, bounded batches, overlap retention, malformed/stale data, known air,
shared physics and section-local geometry precision. Browser checks also cover
an open ravine at Y=10, a closed cave at Y=−29, and coordinates near ±29,999,000.

## Render distance and live edit verification, 2026-09-15

These measurements predate the additional loaded ring described above.

In an isolated copy of the Overworld, the real browser expanded from radius 3
to 8 and received all 1445 sections (23,674,880 raw voxel bytes). Shrinking to
radius 2 retained 125 sections (2,048,000 bytes), without a full snapshot, mesh
reset, revision change or player displacement. Reloading restored the selected
radius from local storage. The server regression also checks per-player
isolation, invalid values, overlap retention and the world-border margin.

At radius 8, confirmed-to-visible-section publication took 121.9 ms for a break
and 120.6 ms for placement in two browser samples. Including server response,
they took 149.4 ms and 137.2 ms. Local packet preparation was 0.6–0.9 ms. The
regular whole-view light calculation in the same session took about 4.7 seconds;
the independent edit worker published geometry before it finished. These are
local observations at 1280×720 with shadows and about 1.27 million triangles,
not frame-time or network guarantees. Propagated light can still catch up later
on large views.

F3 remained open on canvas clicks, block edits and recording start; explicit F3
toggled it closed and open. The full verification script passed (126 Rust and
171 JavaScript tests, storage crash checks and all three network suites); an
additional staged edit-upload regression also passed afterwards.

## Nearby mesh priority and buffered streaming verification, 2026-09-15

In the real browser at radius 8 with the buffered 1,805-section window, first
nonempty mesh publication took **354 ms and 388 ms** in two startup samples.
All **27 nearest sections** had completed while only **1,167 of 1,805** sections
had arrived and whole-view lighting was still building. Subsequent local-light
cache tiling improves reuse; those startup samples were taken before that tuning.
A final reload with the tiled cache and a warm server published its first mesh
in **189 ms**, completed all 1,805 sections and logged no browser warnings/errors.

A 30-second automatic flight at 1280×720 with 2048² shadows traversed **497.12 m**.
Across 3,746 frames, frame p95 was **13.9 ms**, p99 **20.9 ms**, maximum **48.6 ms**.
There were no frames above 50 ms, terrain-wait frames or full mesh resets during
the recording. It recorded 39 view changes and 147 batches. Stream application
p95 was 9.4 ms. These are local workload observations, not cross-device or
Minecraft comparison results.

Server regressions exercise repeated crossings of a single chunk boundary,
two-chunk anchor changes, coverage of the selected radius and departing rows
outside that radius. Client regressions compare local-light mesh bytes with the
whole-view result, including roofs, water, glass and negative coordinates; they
also check distant-update retention, bounded caches, stale jobs, unload/re-entry
tickets and unchanged interior tiles during view shifts. The verification script
passed 127 Rust tests, storage crash checks and all three network suites. The
final client suite passed 180 JavaScript tests.