shacraft-core/docs/MEMORY_AND_STORAGE.md

Shacraft Core memory, storage, and recovery

Status: implementation plan incorporating decisions D008–D009. This file defines invariants and verification criteria; it does not claim that all mechanisms listed here have already been implemented. The public API is in CONTRACT.md, and accepted decisions are in DECISIONS.md. Additional proposals are listed separately at the end.

1. Primary goal and scope of guarantees

The core's primary goal is to bound server RAM when many independent matches use identical maps. Unchanged blocks in a shared template are stored once. A match owns only its changes. Inactive areas and operation history remain on disk.

Do not promise a percentage saving relative to Paper or a specific number of players per gigabyte in advance. Savings depend on block diversity, the number of simultaneously active sections, how much maps change, entities, client visibility, and workload. Success is demonstrated by reproducible measurements alongside the implementation's functional limitations.

The process budget includes more than blocks: the section cache, temporary decoded sections, storage indexes, block registry, world metadata, history, network queues, players, and entities. Bounding one cache does not bound all RAM.

2. Addressing and section format

The storage unit is a 16 × 16 × 16 section, or 4,096 cells. World coordinates remain signed i32 values as specified by the contract. Use div_euclid(16) and rem_euclid(16) on each axis: block -1 belongs to section -1 at local coordinate 15. The cell index is x + 16*z + 256*y. The format version fixes this order.

In the first stage, a section chooses a compact uniform/palette encoding:

• Uniform(BlockId): all 4,096 blocks are identical; there is no index array.
• Paletted: unique global BlockId values and densely packed palette indices. For P > 1, each cell requires ceil(log2(P)) bits.

A future Dense extension may store 4,096 global u32 values when the palette and its indices take more space. This requires an explicit encoding version/tag and is not yet part of the accepted minimum contract.

Without headers or alignment, a palette occupies 4*P + 4096*ceil(log2(P))/8 bytes. That is 520 bytes for two states, 2,112 for 16, and 5,120 for 256. With 4,096 unique states it reaches 22,528 bytes, making a direct 16,384-byte array smaller. This calculates payload size, not RSS or Rust object size.

Reading one block should not require decoding the entire section. An edit uses a temporary array of 4,096 values, then selects the encoding again. Editing must remove unused states from the palette. Disk compression can be added after measurement; it does not replace cache limits.

The decoder validates the version, lengths, palette size, range of every index, and existence of global BlockId values. A corrupt section returns an error identifying the section; silently substituting air is prohibited. The encoding has a fixed byte order and integrity checks. A checksum detects corruption but does not protect against deliberate tampering.

3. Shared templates and independent worlds

A template is an immutable snapshot of a world, not a reference to its current mutable state. Internal world, snapshot, and section IDs are distinct from user-facing names. WorldInfo.template may display the source name, but internal data must retain the exact snapshot_id and source revision.

A world holds a reference to its base snapshot and its own section override table. A read looks for a world override, then a section in the snapshot, and otherwise returns air. An explicit all-air section override is required: absence of an override means inheritance, so removing all template blocks cannot be represented by a missing entry.

The first edit to a template section creates a new section owned by the world. The template section itself never changes. Identical references to immutable sections can share one cached object. Restoring a section to the exact template content allows its override to be removed after checking equality.

Practical MVP design:

1. Immutable sections are stored on disk by ID.
2. A snapshot contains an on-disk index mapping section coordinate → section ID.
3. A world contains a snapshot_id, its current revision, and an on-disk index of its own overrides.
4. create_world(template=source) takes a snapshot from exactly one committed source revision. A previously created snapshot may be reused if the source has not changed.
5. If no snapshot exists for that revision, the index of effective sections is copied incrementally by reference within a consistent transaction. Blocks and section contents are not copied into RAM or duplicated on disk.

Creating a snapshot this way can take time and space proportional to the number of section references. The first fork must not be promised as O(1). A more complex persistent index with shared pages can be added later if measurements show a need. Do not replace this design with an unbounded chain of parent worlds: deep chains slow reads and complicate garbage collection.

Isolation check: modify the source after a fork, modify one of two child worlds, restart the process, and verify that all three states differ exactly as expected. The snapshot remains available even if the source is reset or later deleted.

4. Bounding memory

cache_sections limits the number of immutable sections retained in the cache. Opening with zero is explicitly rejected, as accepted in CONTRACT. The first implementation uses an LRU of decoded sections: an array of 4,096 u32 values has a known 16,384-byte payload, with objects and the LRU index counted separately. Storing palettes directly in the cache is a later optimization; compact disk encoding alone does not shrink a decoded cache.

Also track cache bytes: account for the section buffer, palette, and entry metadata, and report these separately from payload size. The cache key is the immutable section ID. A cache keyed only by world name and coordinates can easily retain stale data after reset.

Temporary transaction buffers must not enter an unbounded dirty cache. Sections are processed in bounded batches, with completed writes passed to the disk transaction mechanism. A Vec<BlockChange> received through the API already occupies RAM, so streaming internal writes does not remove the need for request size limits.

Operation history, deduplication records, and section indexes must not be loaded in full at startup. Use an on-disk SQL index with a bounded SQLite page cache. Configure and measure the page and temporary-data budgets. Choosing a library does not by itself prove that a budget is respected.

Metadata is not free. The contract's registry() -> &[String] implies an in-memory registry; list_worlds() creates a complete list. Define and document explicit MVP limits on world count, state count, and string length. Exceeding those boundaries will require a paginated API.

Accepted limits:

• At most 32,768 unique positions in one edit operation.
• read_region covers at most 262,144 cells, as specified by the contract; compute the product with checked wide arithmetic before allocating memory.
• A world name matches [A-Za-z0-9_-]{1,64}; coordinates on each axis fall within [-30000000,30000000].
• The server additionally bounds JSON size before deserialization, operation frequency, network queues, and concurrent requests.
• Maximum entity count and change queue size are defined separately from the block cache.

For the not-yet-finalized operation_id limit, 1 to 128 UTF-8 bytes is proposed. Exact registry and other queue limits are defined and documented in the implementation. Sizes must be tested with a real workload. The read_region limit applies to the volume of the region, not just the number of nonempty blocks returned.

5. On-disk history and atomicity

The minimum logical storage entities are the block registry, worlds, snapshots, snapshot-to-section references, world section overrides, immutable sections, operations, and operation changes. History stores the previous and new state of each affected cell, operation type, revisions, ID, and a fingerprint of the normalized request. History supports undo and idempotent retries; an ordinary server message log does not replace it.

Deduplication is stored on disk with the unique key (world_id, operation_id). Do not retain all operation IDs in a HashMap. Finding the operation to undo and subsequent cell changes requires disk indexes, rather than scanning the entire log for every undo.

One successful mutation atomically commits:

1. New sections and changes to world references.
2. The new revision of the accepted operation, including a no-op.
3. The operation record, its fingerprint, and the exact result to return on replay.
4. History data required for safe undo.

The client receives a success acknowledgment after all four parts are durably committed. A write failure must not leave a new revision with old blocks, or changed blocks without a deduplication record. Allocating a new BlockId is also durable: reopening the store does not renumber the registry, and air always has ID 0.

Decision D008 selects SQLite with WAL and synchronous=FULL; a custom WAL is not implemented. Sections retain an original, versioned binary format inside the transactional store. Verify that the connection settings actually take effect. Writing to an OS buffer is not a durable commit. Durability guarantees assume a healthy filesystem and device that correctly honor synchronization requests.

A storage path permits one writer. &mut WorldStore protects only a particular Rust object: a second process or independently opened object is rejected by an exclusive advisory file lock. The lock remains held for the lifetime of WorldStore; the existence of a lock file alone does not mean that the lock is held. A second open must not silently create two independent views of the metadata.

6. Revisions and idempotency

A revision is a monotonically increasing state number for one world. Although the API uses u64, storage is limited to the nonnegative range of SQLite i64: a next revision above i64::MAX returns an explicit error. Wrapping to zero is prohibited. Reset does not return the revision to zero or allow an old request to accidentally pass validation against a new state.

Processing order for edit and undo:

1. Validate sizes and ID formats, then normalize the request. Rejecting duplicate positions in changes is recommended to avoid dependence on duplicate ordering.
2. Look up (world_id, operation_id) in the durable table. If the fingerprint matches, return the stored EditResult with replayed=true. This happens before comparing against the current revision: a normal retry after a lost response must work.
3. If the ID exists with different content, return an idempotency conflict and change nothing.
4. For a new operation, check expected_revision in the same transaction as the mutation. If it differs, return a conflict with the current revision.
5. Validate all positions and BlockId values, apply and durably commit the result, then send the response and change event.

The fingerprint includes the method and all meaningful arguments, including expected_revision; the order of unique positions is canonicalized. An ID's uniqueness is scoped to the stable internal world ID, not a name that might later be reused.

Accepted first-stage semantics: every newly accepted successful edit increments the revision, even when changed=0. Such an operation also establishes a boundary for conservative undo and stores a durable deduplication result. An exact replay does not increment the revision.

A replay returns the original operation's revision, which may be lower than the current revision. The client must not roll back its current revision based on that response. An error after commit but before response delivery leaves the caller uncertain; a safe retry uses the same ID and arguments.

Automatically deleting history changes the deduplication guarantee. Until an explicit retention policy exists for IDs and operations, history remains durable. Its retention period must not be shortened silently; RAM is bounded by disk storage, not by forgetting previously acknowledged requests.

7. Safe undo

Undo is a new atomic operation with its own revision and operation_id; it does not rewrite the old log. Its target must belong to the same world and contain changes that were actually applied.

The accepted conservative MVP rule (D009) allows undo only when the current world revision matches the target edit's resulting revision. A later edit, even to a different cell, blocks undo. Comparing the current BlockId with after alone is insufficient: the sequence “stone → air → stone” restores the same block but represents someone else's later work. Selective undo of nonoverlapping changes is a later extension requiring separate on-disk indexes of change provenance.

A conflict in even one cell rejects the entire undo with a conflict description. Partial undo is not the default behavior. A new undo of an already undone operation receives a conflict; an exact replay of the same undo returns the stored result. Undoing an undo may be added separately after defining its semantics; the MVP must explicitly report that it is unsupported.

Reset establishes an undo history barrier: undo of pre-reset operations is rejected. A monotonically increasing world epoch, recorded alongside operations, is a useful representation. Deduplication records from previous epochs remain: replaying an old operation returns its old result without applying it again.

8. Fork, reset, and lifecycle

A fork pins a snapshot of a specific source revision in one consistent operation. Do not use the sequence “read revision → read unprotected sections → create world”: the source can change between steps. Source history does not become child-world history; a new world's revision may start at zero while retaining a reference to the snapshot revision.

Reset returns a world to its pinned template, or to air if it has no template. It atomically removes the world's overrides, increments the revision and epoch, records the reset operation, and invalidates the relevant derived caches. Snapshots pinned by other worlds do not change.

On reset, the server must notify clients that a new snapshot/resynchronization is required. An event with empty changes alone will not remove blocks already displayed. The server must also coordinate player relocation, entities, and arena state; the block core cannot make those decisions for the game layer.

Sections and snapshots may be released only after checking durable references from worlds, snapshots, and required history. Reference traversal and deletion proceed in batches. A snapshot must not be deleted merely because its source name no longer exists. Until verified garbage collection is implemented, retaining unreachable data on disk and reporting its size in metrics is safer.

9. Failures and recovery

After opening a store, validate the format version and metadata consistency, and complete recovery before serving requests. An incomplete transaction is invisible. A completed, acknowledged operation survives a process crash.

SQLite performs WAL recovery. The application does not rewrite or truncate it itself. Database integrity errors, unknown section format versions, and incorrect checksums require explicit errors and preservation of files for diagnosis. Do not “repair” these errors by deleting data or creating an empty world.

SQLite performs checkpointing so that a crash leaves a consistent state. The application does not delete WAL/SHM files manually. flush() returns synchronization/checkpoint errors without hiding them; its precise guarantees must be compatible with commit-before-ack rather than replacing it.

Required verification scenarios:

• Process termination before a write, during a write, and after commit but before the response.
• Replaying the operation after each such failure.
• Exhausted disk space and write/synchronization failures.
• A truncated tail and corruption in the middle of the log.
• Crashes during snapshot, reset, block registration, and checkpoint.
• Two opens of the same path; negative and boundary coordinates.
• Reusing an ID for a different request, revision conflicts, ABA undo, and undo after reset.
• Rejection of a zero-sized cache; eviction with a limit of 1 and with a small ordinary limit, followed by reopening.

A process-termination test verifies process-crash recovery, but does not faithfully simulate power loss or hardware disk-cache behavior. State these boundaries alongside the results.

10. Metrics and comparison with Paper

stats() should return stable named fields with units. A useful minimum set is cache_sections, cache_limit_sections, cache_payload_bytes, cache_hits, cache_misses, cache_evictions, transient_peak_bytes, world_count, snapshot_count, registry_states, history_operations, storage_bytes, wal_bytes, dirty_overlay_sections, commit_latency_ms, and recovery_duration_ms. A byte counter must state whether it measures actual allocation or estimated payload.

Measure process RSS/PSS, peak RAM, cgroup memory where available, CPU, disk reads/writes, and tick and administrative-operation latency externally. OS file-cache memory must not automatically be claimed as “saved” or added to RSS without explaining the methodology.

Load scenarios:

1. The same template with 1, 10, and 100 independent, unchanged worlds.
2. The same worlds with an identical fixed number of edited sections, then with 1%, 10%, and 100% edited.
3. Sequential and random movement of the active area across a map larger than the cache.
4. Sustained editing with growing on-disk history and a constant active set of sections.
5. Alternating forks/resets and restarts, checking content after each stage.

Record the seed, map, number of worlds, players/bots, view distance, entity set, warmup duration, and measurement duration. First compare variants of the project's own core: direct arrays versus palettes, copying versus a shared template, and different cache limits. This helps attribute an effect to a specific decision.

When comparing with Paper, record exact server, Minecraft, Java, and Rust versions; JVM settings; hardware; OS; plugins; how worlds were created/copied; and an identical player scenario. Report identical-block storage and complete gameplay scenarios separately. If Shacraft does not perform lighting, AI, redstone, generation, or other functions present in the Paper scenario, list those differences explicitly: that measurement does not demonstrate superiority at equal functionality.

Publish the original commands and raw results, the median and spread across multiple runs, and latency and I/O alongside RAM. Reducing memory at the cost of unacceptable disk access or latency is a measured tradeoff, not automatically a success.

11. Contract review before subsequent stages

Some original comments have already been incorporated into CONTRACT.md and DECISIONS D008–D009; the others concern future networking stages:

• Clarify snapshot/revision in template semantics; adding them to WorldInfo would be useful.
• Update the control HTTP method world.reset to require expected_revision and operation_id, as already accepted in the Rust API. Define idempotency and the source revision for create/fork retries.
• Specify no-op rules and exact string/registry limits; this does not override other accepted invariants.
• Introduce machine-readable error codes: unknown world/block, revision conflict, ID conflict, undo conflict, exceeded limit, corrupt store, and write failure.
• Clarify reset/resync events, a consistent welcome snapshot, and delivery of changes after its revision; otherwise, the client can miss a change between the snapshot and subscription.
• Explicitly validate finite numbers, coordinate bounds, message size, request frequency, interaction reach, and per-world permissions for server inputs. The control token must not appear in public responses or logs.
• manifest_hash confirms only the declared resource version. It does not prove that the client is unmodified; gameplay validation remains the server's responsibility.

Implementation order: compact sections and their tests → durable atomic operations and recovery → revisions/deduplication/undo → snapshot/fork/reset → bounded cache and metrics → load measurements. Each stage preserves correctness and recovery first, then adds optimization.

Durability references

SQLite WAL and PRAGMA synchronous describe WAL synchronization on every commit in FULL mode. This is the selected configuration; the correctness of our schema and recovery still needs separate verification.