Expand voxel gameplay, lighting, full-height streaming and world imports

Add shared Rust/WASM physics, worker meshing and diagnostics, 64-chunk full-height streaming, atlas texture support, and baseline world import. Document the current implementation and include the supplied in-game lobby screenshot.

docs/WORLD_STREAMING.md

@@ -0,0 +1,278 @@
+# Procedural worlds and movement diagnostics
+
+The server creates `overworld` on startup if that name is unused. Open
+`http://localhost:4000/?world=overworld`, or select it in the world menu.
+Existing worlds, packages, edits and arenas remain available. An existing world
+named `overworld` is never automatically converted.
+
+## World dimensions and generation
+
+- Playable horizontal coordinates: **−29,999,872 through +29,999,872**.
+- Building layers: **Y = −64 through 319**, inclusive: **384 layers**.
+- Default seed: `20260914`; generator version: `1`.
+- Original deterministic terrain: continents, hills, mountains, rivers,
+  oceans, forests, plains, desert areas, snow, trees, caves, tunnels, aquifers,
+  deep stone, ores, low lava pockets and a bedrock bottom.
+- A spawn clearing and a narrow ravine near X=30 give access to the surface and
+  underground. Fluids are generated blocks; they do not yet simulate flow.
+
+The height range follows the modern Java Overworld convention described in
+[Mojang's Caves & Cliffs Part II announcement](https://www.minecraft.net/fr-ca/article/caves---cliffs--part-ii-out-today-java).
+Terrain generation is independently authored. Matching Minecraft seed output,
+all vanilla biomes, structures and gameplay systems is outside this implementation.
+
+The version and seed are durable in `server.sqlite3`. Generation depends on
+absolute integer coordinates, so request order and section boundaries do not
+change trees, caves or terrain. Keep generator version 1 stable for saved worlds;
+future algorithms need a new version and an explicit migration strategy.
+
+Player simulation, action rays and spawn coordinates use double precision.
+Meshes store section-local float positions; draw matrices and dynamic actors use
+an origin near the camera. Small geometry and movement steps remain representable
+near the horizontal border. Arena/entity administration retains its earlier
+configuration limits where documented.
+
+## CPU work, streaming and storage
+
+The server uses **two CPU generation threads** with **32 pending jobs** at most.
+The simulation thread owns storage and gameplay. Requests are shared between
+players and ordered by distance, with extra weight on vertical distance. Old
+view requests are dropped; completed jobs outside current interests are discarded.
+Control reads retain bounded, temporary interests so remote reads can complete
+without a nearby player. Generation does not require a GPU.
+
+**Game menu → Render distance** selects a radius from 2 to 64 chunks
+(32–1,024 blocks around the player), default 3. Each player chooses their own
+radius; changes apply without reconnecting and persist locally. The current
+client negotiates an extra loaded chunk ring beyond that visible radius. Its
+default data window is **9×9×24 sections**, or **144×384×144 blocks / 1,944 sections**.
+At radius 8 it is **19×19×24**, or **304×384×304 blocks / 8,664 sections**.
+At radius 64 it is **131×131×24**, or **2,096×384×2,096 blocks / 411,864 sections**,
+including the one-chunk preload ring. The queue covers this entire window.
+It is sorted once when the view changes; each tick examines a bounded lookahead
+of 256 pending sections plus collision neighborhoods. Up to 128 sections are
+sent per 50 ms interval, including at most 32 nonuniform sections, with a
+256 KiB estimated section-payload budget, backpressure and time budgets.
+
+Horizontal streaming has one chunk of hysteresis: crossing a single chunk
+boundary does not move the loading anchor or unload its opposite edge. The
+anchor moves after a two-chunk displacement. The extra ring preserves coverage
+of the selected radius between anchor changes. Generation, delivery and meshing
+prioritize nearby sections; overlapping cells and meshes remain in place.
+Expansion retains overlap; explicit reduction unloads excess sections. Distance
+fog completely blends the visible edge into the sky before the buffer is
+unloaded. Horizontal culling and the projection far plane follow the radius.
+
+The current client negotiates `full_height_v1`: every loaded column spans
+**Y=−64 through 319**, all 24 vertical sections. Climbing and flying above the
+world ceiling retain ground and roof sections. Missing interior terrain still
+blocks prediction; space outside the world height is known void. Clients that
+do not negotiate this feature retain the legacy 80-block sliding window.
+Distant-terrain LOD is not implemented; these bounds and local benchmarks do
+not establish parity with Minecraft's renderer.
+
+Already delivered distant sections do not need to remain in the server cache;
+only collision neighborhoods and unsent client sections trigger regeneration.
+Requests from multiple players still deduplicate, including a new player
+requesting terrain previously sent to someone else. Natural sections live in a bounded LRU: at most 8,192 decoded sections, or
+128 MiB of voxel payload. This is separate from the core's encoded-section
+cache, SQLite, metadata, meshes and total process RSS. Visiting more terrain does
+not allocate a permanent world-sized array. Before editing a natural section,
+the server materializes its immutable baseline in `generated_sections` in
+`worlds.sqlite3`; the existing revision/idempotency journal then stores the edit.
+An explicit air deletion survives cache eviction and restart. Undo/reset use
+the saved baseline. World templates inherit the generator configuration and
+pin existing edited sections. Natural materialization does not advance the
+edit revision. Standalone core exports contain stored sections; exporting the
+entire procedural world is not supported. Back up both SQLite databases.
+
+The browser stores nonuniform cells in `Uint32Array` sections: **30.38 MiB** for a dense
+1,944-section default voxel map, or **135.38 MiB** at radius 8, excluding worker
+copies, lighting and GPU geometry. Uniform received sections use four bytes
+instead of 16 KiB, retain known-air readiness, and expand on edit. Empty sections
+need no mesh job. Lighting has a separate allocation cap of 7,393,280 cells;
+it does not scale with the maximum stream volume. With the local mesh worker,
+the shared actor field is limited to 112×384×112 blocks around the view center.
+Palette/RLE avoids JSON objects and position strings per block.
+
+A dedicated terrain-mesh worker builds local light, AO, culling and geometry
+without waiting for whole-view lighting. A light tile covers 2×2 chunk columns
+plus an 18-block horizontal halo, at most **68×384×68 / 1,775,616 cells**. The full
+active vertical range preserves skylight under tall roofs. An eight-entry LRU
+shares these fields between nearby and vertically stacked sections. Spatial
+invalidation and per-section tickets reject obsolete work without discarding
+nearby results on every distant section batch. Moving a distant view edge keeps
+an interior tile's light and mesh valid if its sampling bounds are unchanged.
+
+A separate persistent worker computes a bounded local light field for actor and quick-edit
+sampling, starting after 225 ms without a new sync. Its completion does not gate
+terrain publication. Large views still need time to finish distant meshes.
+Uploads are staged in 64 KiB steps targeting 2 ms per frame; old meshes remain
+until their replacements are ready. Driver calls are not preemptible.
+
+Unknown sections are distinct from received air. Both authoritative simulation
+and prediction wait for collision data instead of falling through missing
+terrain. The server resets the input epoch on a terrain wait, holds the body,
+and resumes when its neighborhood is available. Teleports also reset prediction.
+
+Skylight includes full-height column occlusion data. A cave remains dark when
+its roof lies above the streamed vertical window. Editing a roof updates its
+column data. The background uses the camera's sky exposure to avoid bright sky
+appearing through unloaded underground terrain. Full lateral lighting outside
+the active window is not simulated.
+
+## Section protocol v2
+
+Join with `features:["chunk_stream_v2","movement_prediction_v1","view_buffer_v1","full_height_v1"]` and optional
+`view_distance:2..64` (default 3). A server advertising `view_distance_v1` accepts
+`{"type":"view_distance","chunks":64}` while connected. Noninteger or out-of-range
+values are rejected before mutation; changes are limited to once per 500 ms.
+The server responds with a normal `view` transition containing `view_distance`
+and the new `total_sections`, preserving position, motion epoch and revision.
+Negotiating `view_buffer_v1` adds the extra loaded ring and horizontal hysteresis;
+`stream_radius` reports the loaded radius, while `view_distance` remains the
+selected visible radius. Older v2 clients retain the unbuffered behavior: 245
+sections by default and 1,445 at radius 8.
+`full_height_v1` fixes the vertical bounds to the world height and adds
+`full_height:true` to view/welcome/snapshot messages. It also enables batches
+of up to 128 sections; older clients receive at most 32.
+Welcome/snapshot messages also expose `max_view_distance:64`. The initial
+`welcome` or explicit `snapshot` supplies the world, revision, motion, generation,
+`view_center`, inclusive `view_min`/`view_max`, `total_sections`, terrain settings
+and world bounds. Its `blocks` array is empty; sections follow incrementally.
+Generation numbers increase for each new view or resync.
+
+A `view` message updates bounds and lists departing section coordinates in
+`unload`. Retained sections keep their current data. A `sections` message contains
+at most 128 complete sections (32 for legacy clients), each as:
+
+```json
+{"section":[0,4,0],"palette":[0,123],"runs":[256,1,3840,0]}
+```
+
+Runs are `(count, paletteIndex)` pairs totaling exactly 4096 cells. Cell order is
+`x + 16*z + 256*y`. Section coordinates are integers; world coordinates are
+section coordinates multiplied by 16. Even empty sections are sent explicitly.
+Messages include `world`, `revision`, `generation`, material definitions and
+`columns:[{column:[sectionX,sectionZ],heights:[256 values]}]`.
+
+Batches run at most once per 50 ms per client. The server pauses section sends
+above 512 KiB of queued output; existing queue and timeout limits still apply.
+Packing has a soft 10 ms work budget. Clients validate the complete batch before
+mutation, ignore obsolete generations and resync on revision gaps. Ordinary
+block edits advance revisions; view changes never do.
+
+Old clients retain v1/full-snapshot behavior in nonprocedural worlds. A client
+without v2 is rejected from a procedural world with an instruction to refresh.
+
+## Diagnostics
+
+Open **F3 → Movement diagnostics**. Choose manual recording, automatic flight,
+automatic running, or flight with turns, then start a 30-second recording.
+Automatic scenarios use ordinary authoritative input, climb obstacles during
+flight, and need no pointer lock. Recording starts after terrain/meshes settle.
+Stop or Esc cancels automatic movement. Hiding the tab stops recording and marks
+the result `interrupted: "tab-hidden"`.
+
+The panel stays open while playing or recording; clicking the world restores control, and F3 or the close button hides it. Its controls pause manual input while focused. The panel shows a rolling frame/CPU chart and a summary. **Download JSON** exports
+bounded frame samples, network/stream events and server samples. It measures
+frame p50/p95/p99/max, frames over 25/50 ms, main-thread physics/update and draw
+submission time, long tasks, position corrections, terrain waits, distance,
+view changes, batches and mesh resets. Draw CPU duration is not GPU duration.
+The canvas DOM exposes the summary as `data-dynamics-result`, plus current
+section counts, coordinates, readiness and streaming timings for browser checks.
+`data-first-terrain-ms` measures first nonempty mesh publication after the world
+reset. `data-near-meshes` / `data-near-loaded` count completed and received
+sections in the camera's 3×3×3 neighborhood, including completed empty sections.
+`data-terrain-meshing`, `data-local-light-cells` and `data-local-light-ms` expose
+the independent geometry path and its local lighting work.
+
+Authenticated Control API methods for repeatable diagnosis:
+
+```json
+{"method":"diagnostics.snapshot","params":{}}
+{"method":"terrain.inspect","params":{"world":"overworld","position":[1000000,0,-1000000]}}
+{"method":"player.teleport","params":{"id":"player-id","position":[1000000.125,180,-1000000.875],"flying":true}}
+{"method":"world.create","params":{"world":"another-overworld","terrain":{"seed":42,"version":1}}}
+```
+
+`diagnostics.snapshot` reports generation queues/cache, simulation timings,
+positions, velocity, input backlog, waits and per-player stream state, including
+`view_distance` and `stream_radius`. Teleport
+is an administrative API, not a client movement permission. `/api/metrics`
+adds generation/stream counters and the most recent 600 tick durations.
+`world.read`, `world.edit`, build commit and `camera.capture` may return
+`terrain pending` for cold sections; retry the same read or idempotent operation
+after a short interval. No blocking generation is performed for a cold read.
+
+## Local verification, 2026-09-14
+
+A release server and the real browser ran against an isolated copy of the local
+data, using the Pixel32 Study package. Automatic flight traversed **495.08 m in
+30.003 s**, with **39 view changes and 78 section batches**. Across 4,317 frames,
+frame p95 was **7.0 ms**, p99 **7.1 ms**, maximum **13.8 ms**; no frame exceeded
+25 ms. The run reported no terrain-wait frames, position correction or full mesh
+reset. Section application p95 was 5.6 ms, maximum 7.3 ms. At the last server
+sample, tick work was 0.53 ms and RSS was 78.5 MiB. Three entering sections were
+still in transit at recording end; all 245 completed after stopping.
+
+These measurements are one local workload, not a comparison with Minecraft or
+a guarantee for other devices, multiplayer loads or long sessions. The local
+summary is saved in ignored `artifacts/large-world-flight.json`.
+
+Automated checks cover generation determinism, negative/far coordinates, deep
+content, skyline accuracy, durable mining/undo/restart, remote reads without
+players, bounded batches, overlap retention, malformed/stale data, known air,
+shared physics and section-local geometry precision. Browser checks also cover
+an open ravine at Y=10, a closed cave at Y=−29, and coordinates near ±29,999,000.
+
+## Render distance and live edit verification, 2026-09-15
+
+These measurements predate the additional loaded ring described above.
+
+In an isolated copy of the Overworld, the real browser expanded from radius 3
+to 8 and received all 1445 sections (23,674,880 raw voxel bytes). Shrinking to
+radius 2 retained 125 sections (2,048,000 bytes), without a full snapshot, mesh
+reset, revision change or player displacement. Reloading restored the selected
+radius from local storage. The server regression also checks per-player
+isolation, invalid values, overlap retention and the world-border margin.
+
+At radius 8, confirmed-to-visible-section publication took 121.9 ms for a break
+and 120.6 ms for placement in two browser samples. Including server response,
+they took 149.4 ms and 137.2 ms. Local packet preparation was 0.6–0.9 ms. The
+regular whole-view light calculation in the same session took about 4.7 seconds;
+the independent edit worker published geometry before it finished. These are
+local observations at 1280×720 with shadows and about 1.27 million triangles,
+not frame-time or network guarantees. Propagated light can still catch up later
+on large views.
+
+F3 remained open on canvas clicks, block edits and recording start; explicit F3
+toggled it closed and open. The full verification script passed (126 Rust and
+171 JavaScript tests, storage crash checks and all three network suites); an
+additional staged edit-upload regression also passed afterwards.
+
+## Nearby mesh priority and buffered streaming verification, 2026-09-15
+
+In the real browser at radius 8 with the buffered 1,805-section window, first
+nonempty mesh publication took **354 ms and 388 ms** in two startup samples.
+All **27 nearest sections** had completed while only **1,167 of 1,805** sections
+had arrived and whole-view lighting was still building. Subsequent local-light
+cache tiling improves reuse; those startup samples were taken before that tuning.
+A final reload with the tiled cache and a warm server published its first mesh
+in **189 ms**, completed all 1,805 sections and logged no browser warnings/errors.
+
+A 30-second automatic flight at 1280×720 with 2048² shadows traversed **497.12 m**.
+Across 3,746 frames, frame p95 was **13.9 ms**, p99 **20.9 ms**, maximum **48.6 ms**.
+There were no frames above 50 ms, terrain-wait frames or full mesh resets during
+the recording. It recorded 39 view changes and 147 batches. Stream application
+p95 was 9.4 ms. These are local workload observations, not cross-device or
+Minecraft comparison results.
+
+Server regressions exercise repeated crossings of a single chunk boundary,
+two-chunk anchor changes, coverage of the selected radius and departing rows
+outside that radius. Client regressions compare local-light mesh bytes with the
+whole-view result, including roofs, water, glass and negative coordinates; they
+also check distant-update retention, bounded caches, stale jobs, unload/re-entry
+tickets and unchanged interior tiles during view shifts. The verification script
+passed 127 Rust tests, storage crash checks and all three network suites. The
+final client suite passed 180 JavaScript tests.