zhila/README.md

ZHILA

Sound is terrain. A sampling instrument where material is cut in a quarry, laid as stones in a nave, and played by a hand or an agent through the same verbs.

This repository is P2 complete: the engine, the wire, the descent, the quarry, a persistent project, recording, manual faces, undo across restarts, a native host, an 8-bar arrangement of four sections, an offline WAV render, and the choir — an in-app LLM agent on the same verbs. A found or recorded sound is cut into sixteen faces, laid on a vein, arranged across A–D, heard as an eight-bar loop, rendered to a 16.5-second WAV, and moved by a performer that hears the levels and asks the hand to approve its edits. The visual demo in reference/demo remains a design reference, not the codebase. The roadmap is in docs/PLAN.md.

What is real

• Rust engine. cpal output at 48 kHz, five veins, a sixteen-step phrase, a fixed voice pool, per-vein levels and gains, a bar clock. Procedural voices for empty pattern rows; real sample voices for rows with a face. No Web Audio.
• Material. Import through the local wire; symphonia decodes WAV/FLAC/MP3/OGG; onset analysis finds seams and cuts sixteen faces with RMS, rank and waveform peaks; the archive keeps the mono f32 samples. The sample rate is preserved and the engine handles the playback ratio.
• Project. A portable folder: project.db (SQLite) plus assets/. On daemon start it loads the cores, pattern, tempo, gains and journal; every project edit autosaves. The default is ~/.local/share/zhila/descent-04; use --project PATH for another. The project name is shown in the top rail.
• Undo/redo. Tempo, gains, pattern steps/faces and face cuts push a small history; Ctrl/⌘+Z and Shift+Ctrl/⌘+Z, or the rail buttons. The bounded history and redo stacks are saved in the project (up to 64 edits, with every core's cuts) and restored on load; the journal still records every edit.
• Recording. record in the quarry opens the microphone, captures mono PCM through an AudioWorklet, encodes a 16-bit WAV in the surface and sends it through the same import path. The take becomes a core with sixteen faces, seal recorded, and is autosaved in the project. Stops at 30 s; no monitoring.
• Quarry editing. Drag any of the seventeen seams on the waveform: the inner seams recut the sixteen faces, the outer seams trim the playable range. The core recomputes RMS and rank from the samples; the engine hears the new ranges on the next trigger; the project stores the cuts; the edit is undoable.
• Quarry. Import, core strip, zoomed waveform with face boundaries, sixteen-face wall, face audition, seal and stats; assign a face to a pattern row; edit the sixteen steps; strike a face; all against the live session.
• Protocol v1. JSON control frames, a 31-byte binary meter frame, and a binary import path (an import header followed by the audio bytes). Snapshot, results, events, token and origin guard.
• Session + journal. Validation, revisions, a journal for project edits (tempo, gains, pattern faces and steps); play, stop and strikes are performance, not history.
• Surface. A Svelte 5 descent whose terrain, eye, city, road, stones and playhead react to the five levels and the master; a real wire log; engine facts; cilia; a master signal trace; vein gains; transport; keyboard.
• Native host. The Tauri app starts the same host inside its own process: engine, session, SQLite project, assets and loopback wire. The built window is self-contained; the browser path uses the standalone daemon. One process owns 127.0.0.1:4317 at a time.
• Arrangement. Four 2-bar sections (A–D) make an 8-bar loop. The engine plays the sections in order; the descent draws all eight bars with section seams and letters; the quarry edits the active section, selects, copies and clears. The sections, the active section and their edits are persisted and undoable; the pattern rows and faces stay per section.
• Render. The render button (or the render command) renders the full 8-bar arrangement offline with the same voices, gains, sample faces and master as the live engine, then writes a 32-bit float stereo WAV to <project>/renders/. The rendered event carries the path and duration; no device is opened for the render.
• Choir. An in-app LLM performer on the same command verbs. It receives the project, the sections and the live levels, speaks in short concrete moves, performs safe verbs immediately and turns project edits into proposals the hand approves or rejects. The choir room carries the conversation, the proposals and the section/level readout. Without an API key it runs a deterministic mock; set ZHILA_AGENT_KEY (or DEEPSEEK_API_KEY) and optionally ZHILA_AGENT_URL/ZHILA_AGENT_MODEL for the live provider.
• Verification. Rust unit tests, a live wire test, a material import test, a browser quarry test and the P0 regression smoke, plus svelte-check and a production build.

The Tauri 2 app is built and running with its own engine: the native window loads the embedded surface and the host starts the engine, session, project and wire in the app process. It is built with Tauri 2.12.5 / CLI 2.12.4 against WebKitGTK 2.54.2, GTK 3.24.52 and dbus 1.16.2. The binary is app/src-tauri/target/release/zhila-app (14 MB); installers are a later step.

Quick start

Requirements: Rust (tested on 1.99), Node 22, ALSA. The daemon uses the default output device; without one it reports offline and the surface still connects.

./scripts/dev.sh

Open http://127.0.0.1:5173. The daemon listens on ws://127.0.0.1:4317 with the development token dev.

Or the two halves separately:

cargo run -p zhila-daemon -- --port 4317
npm --prefix ui run dev

Try the material loop

1. Open quarry in the top rail.
2. take a core and choose a WAV (a short field recording works; the fixture fixtures/clicks-01.wav is one).
3. The file is decoded, the seams are found and sixteen faces are cut. Or press record, speak/play into the microphone, press stop; the take arrives the same way. Click a face, then lay on bass.
4. space plays the loop; the bass row now triggers the sample while the other veins keep their procedural voices. Edit steps in the pattern; each edit goes to the engine and the journal.
5. Go back to nave to see the pattern on the road and the levels in the room.

Tauri

npm --prefix ui run build
cd app
./node_modules/.bin/tauri build --no-bundle
./src-tauri/target/release/zhila-app

The built window embeds the surface and starts the host itself; it does not need the daemon. The host owns 127.0.0.1:4317, loads the default project, and stops when the app exits. For browser development use the standalone daemon (cargo run -p zhila-daemon -- --port 4317) and stop the native app, because both bind the same port. tauri dev uses the Vite server at 127.0.0.1:5173. Rebuild the native app after any UI change; the binary embeds ui/dist at compile time. The recording and cutting UI are in the shared surface; WebKitGTK microphone permission still needs a manual check on this machine.

Controls: space play/stop · s strike the selected face in the quarry or the selected vein in the nave · 1–5 select a vein (nave) or a pattern row (quarry) · click a face to select, double-click to strike · tempo − / +.

The daemon prints the audio device, rate, buffer and session id. Use --no-audio for UI/protocol tests, --port to change the port, and ZHILA_TOKEN or --token to change the dev token. The surface reads VITE_ZHILA_WS and VITE_ZHILA_TOKEN at build time.

Layout

crates/zhila-protocol   wire types, binary meter frame, cores/faces, tests
crates/zhila-core       session, journal, material snapshot, engine trait
crates/zhila-engine     cpal stream, procedural + sample voices, levels
crates/zhila-material   decode, onset analysis, sixteen faces, peaks
crates/zhila-project    SQLite project, assets, load/save, undo persistence
crates/zhila-daemon     WebSocket daemon, auth, binary import, autosave
ui                      Svelte 5 surface: descent + quarry
fixtures                a small click track for the smoke tests
app                     Tauri 2 app with the embedded host
docs                    plan, protocol, P0/P1 notes
reference/demo          the earlier design demo

Tests

./scripts/check.sh                    # fmt, clippy, cargo tests, svelte-check, build
node scripts/wire-smoke.mjs           # daemon running
node scripts/material-smoke.mjs       # daemon running: import, face, play
node scripts/persistence-smoke.mjs    # spawned daemon: import/edit/cut/undo/restart/history
npm --prefix ui run smoke             # daemon + Vite: P0 regression, screenshots
npm --prefix ui run quarry-smoke      # daemon + Vite: import, assign, play, screenshots
npm --prefix ui run project-smoke     # daemon + Vite: project, undo, import, screenshots
npm --prefix ui run record-smoke      # daemon + Vite: fake microphone take, faces, seal
npm --prefix ui run faces-smoke       # daemon + Vite: drag a seam, trim, save, screenshot
npm --prefix ui run arrangement-smoke # daemon + Vite: sections, edit B, 8-bar descent
node scripts/render-smoke.mjs        # daemon running: render, parse the WAV
npm --prefix ui run render-ui-smoke  # daemon + Vite: render button, wire, screenshot
node scripts/agent-smoke.mjs         # daemon running: ask, proposal, approve, tempo
npm --prefix ui run agent-ui-smoke   # daemon + Vite: choir ask/approve, screenshot
cd app && ./node_modules/.bin/tauri build --no-bundle   # native app with the embedded host
npm --prefix ui run restored-smoke    # after restarting the daemon on the saved project

npm --prefix ui run smoke writes preview-p0*.png; quarry-smoke writes preview-p1-*.png.

P2 limits (in progress, not hidden)

• The arrangement is exactly four 2-bar sections; no section length, reorder, insert or per-section mute yet.
• Render is 48 kHz stereo 32-bit float with a 1.5 s tail; no stems, normalisation or per-section export.
• The choir is in-app only; no MCP server yet. Without a key it is the deterministic mock. The provider path is OpenAI-compatible, one tool round per ask, and the conversation lives in daemon memory (not the project).
• The agent's proposals are applied with Actor::Agent and saved; the chat is not persisted across host restarts.

P1 limits (kept honest)

• Recording is mono, 16-bit, requested 48 kHz, up to 30 s, no monitoring and no input device choice. WebKitGTK/Tauri microphone permission is untested; the browser path is verified with Chromium's fake capture.
• Face editing is seventeen contiguous cuts: no zoom, fades or per-face loop points. The trim is the outer pair of cuts; the source bytes are untouched.
• Undo history is a bounded snapshot stack stored as JSON; imports are still not undoable, and the journal is an audit trail rather than the undo source.
• The native app uses the development token and a fixed port; the browser and the app cannot share 4317 at the same time.
• The arrangement is still one two-bar loop on five veins; no render, no 8-bar timeline, no agent yet.
• One fixed two-bar phrase and five veins; pattern rows are veins, one face per row, no per-step face or pitch.
• The Tauri binary currently embeds an older surface build; rebuild it after UI changes (npm --prefix ui run build, then tauri build). The engine remains the separate daemon.
• Formats depend on symphonia; WAV is the tested path. AIFF, multi-channel files and very long files are not specifically handled.
• No agent/MCP in this phase.
• The Tauri shell now builds and runs; the engine is still the separate daemon (embedding is next).

P2 is complete: the arrangement, the offline WAV render and the choir. P2a, P2b and P2c are in this tree. The plan is in docs/PLAN.md; the phase notes are in docs/P2.md. The next phase is P3: the arrangement across scenes, MCP for external agents, and the deeper mix. The plan is in docs/PLAN.md; the phase notes are in docs/P1.md.