diff --git a/README.md b/README.md index d703ba4..4c21d9a 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,111 @@ # Soundgen -8-bit sound synthesizer in Rust for game audio assets, with LLM integration via MCP. +8-bit sound synthesizer in Rust with AI-powered sound generation via MCP + human feedback loop. + +## Why Soundgen? + +Most free 8-bit sound tools are either closed-source, hard to automate, or don't integrate with AI workflows. Soundgen fixes this: + +- **LLM generates sounds** by calling MCP tools — no manual parameter tweaking +- **You rate the results** in the GUI — the AI learns from your feedback +- **The AI improves** — next time it generates a similar sound, it references your top-rated examples +- **Export to WAV** — drop the files straight into your game + +``` +┌──────┐ generate_batch ┌─────────┐ rate 1-5★ ┌──────────┐ +│ AI │ ────────────────→ │ Sounds │ ──────────→ │ You │ +└──────┘ └─────────┘ └──────────┘ + ▲ uses top-rated examples as reference │ + │ │ + └──────────────── ┌──────────────┐ ◄───────────────────┘ + │ Feedback DB │ + └──────────────┘ +``` ## Features -- **6 voice types**: pulse (NES duty cycles), triangle, noise (LFSR), DPCM samples, wavetable (Game Boy wave), FM (2-operator) -- **Effects**: ADSR envelope, frequency sweep (linear/exponential), biquad filter (lowpass/highpass), vibrato -- **JSON-first**: every sound is a `SoundSpec` JSON object — LLMs generate JSON, CLI/MCP render to WAV -- **Presets as data**: 13 built-in presets in `presets/{sfx,ui,ambient}/` — extend without recompilation -- **MCP server**: LLMs can call `list_presets`, `generate_sfx`, `render_sound` as tools -- **Sequencer**: pattern-based song playback (JSON format) -- **GUI**: egui editor with virtual keyboard, preset browser, channel controls, sequencer +- **Training pipeline**: AI generates → you rate → AI learns (SQLite feedback database) +- **6 voice types**: pulse (NES duty cycles), triangle, noise (LFSR), DPCM, wavetable, FM +- **Effects**: ADSR envelope, frequency sweep, biquad filter, vibrato (LFO) +- **JSON-first**: every sound is a `SoundSpec` JSON — LLMs generate JSON naturally +- **MCP server**: 5 tools for LLM integration (`list_presets`, `generate_sfx`, `render_sound`, `generate_batch`, `get_reference_sounds`) +- **GUI**: egui editor with virtual keyboard, preset browser, channel controls, training tab +- **13 built-in presets**: SFX, UI, ambient — extend with JSON files, no recompilation +- **Sequencer**: pattern-based song playback (JSON) - **Runtime library**: NES-authentic nonlinear DAC + SoundBank for game embedding -- **No allocations in audio hot path**: `tick() -> f32`, `&mut self` +- **No allocations in audio hot path**: `tick() -> f32` ## Quick Start ```bash -# List available presets +# CLI — list presets cargo run --bin soundgen -- list-presets -# Generate a sound from a preset -cargo run --bin soundgen -- gen jump --out assets/jump.wav +# Generate from preset +cargo run --bin soundgen -- gen jump --out jump.wav -# Generate with parameter override -cargo run --bin soundgen -- gen explosion --out assets/explosion.wav --param volume=0.95 - -# Render from a custom JSON spec +# Render custom spec cargo run --bin soundgen -- render presets/sfx/laser.json --out laser.wav -# Render a song (sequencer) +# Render a song cargo run --bin soundgen -- render-song song.json --out music.wav # Launch the GUI editor cargo run -p soundgen-gui ``` -## MCP Server (for LLM integration) +## Training Pipeline (AI + Human Feedback) -Run the MCP server on stdio: +### 1. Connect the MCP server ```bash -cargo run -p soundgen-mcp -- --presets-dir presets +cargo run -p soundgen-mcp -- --presets-dir presets --db feedback.db ``` -Configure in Claude Desktop / MCP client: +Add to your MCP client config (Claude Desktop, OpenCode, etc.): ```json { "mcpServers": { "soundgen": { "command": "/path/to/soundgen-mcp", - "args": ["--presets-dir", "/path/to/presets"] + "args": ["--presets-dir", "/path/to/presets", "--db", "/path/to/feedback.db"] } } } ``` -LLM workflow: -1. `list_presets` → see available sounds -2. `generate_sfx { preset: "jump", out_path: "assets/jump.wav" }` → WAV created -3. `render_sound { spec: {...}, out_path: "assets/custom.wav" }` → custom sound +### 2. AI generates sounds + +The AI calls MCP tools to create sounds: + +| Tool | What it does | +|---|---| +| `list_presets` | List available built-in presets | +| `generate_sfx` | Generate WAV from a named preset | +| `render_sound` | Generate WAV from a custom SoundSpec JSON — returns reference examples from your top-rated sounds | +| `generate_batch` | Generate multiple sounds, store in feedback DB (unrated) | +| `get_reference_sounds` | Search DB for highly-rated similar sounds | + +### 3. You rate them + +Open the GUI → **Training** tab → play each sound, rate 1-5 stars, type feedback: +- "too loud" → AI lowers volume next time +- "pitch зачем-то возрастает" → AI fixes the sweep +- "Идеально!" → AI uses this as a reference for future sounds + +### 4. AI improves + +When the AI calls `render_sound` with `name: "explosion"`, the MCP server searches the feedback DB for similar highly-rated sounds and returns their specs as reference examples. The AI sees what worked and adjusts its approach. + +### 5. Export for fine-tuning (optional) + +Once you have 100+ rated sounds, export them as a JSONL dataset for LoRA fine-tuning: + +```bash +# In GUI: Training tab → "Export Dataset" +# Creates feedback_dataset.jsonl with all 4★+ sounds +``` ## SoundSpec JSON Format @@ -75,6 +120,7 @@ LLM workflow: "duty": 50, "frequency": { "start": 200, "end": 800, "curve": "exponential" }, "envelope": { "attack": 0.01, "decay": 0.15, "sustain": 0.0, "release": 0.14 }, + "vibrato": { "rate": 8, "depth": 200 }, "volume": 0.7 } ] @@ -82,36 +128,20 @@ LLM workflow: ``` Channel types: `pulse`, `triangle`, `noise` - -## Song JSON Format (Sequencer) - -```json -{ - "bpm": 120, - "rows_per_beat": 4, - "tracks": [ - { "type": "pulse", "duty": 50, "volume": 0.4 } - ], - "patterns": [ - { "rows": [ { "notes": [{"frequency": 440}] }, {"notes": [null]} ] } - ], - "pattern_order": [0] -} -``` +Optional fields: `filter` (lowpass/highpass + cutoff sweep), `vibrato` (LFO frequency modulation) ## Architecture -Cargo workspace: - | Crate | Purpose | |---|---| | `soundgen-core` | Synthesis engine (generators, effects, mixer). No I/O. | | `soundgen-fmt` | `SoundSpec` JSON schema + `PresetRegistry` | -| `soundgen-io` | WAV writer (`hound`) + playback (subprocess) | +| `soundgen-io` | WAV writer (`hound`) + playback | | `soundgen-seq` | Sequencer: patterns, songs | | `soundgen-cli` | `gen`, `render`, `render-song`, `list-presets` | -| `soundgen-mcp` | MCP server for LLM tool-use | -| `soundgen-gui` | egui GUI editor | +| `soundgen-mcp` | MCP server: 5 tools for LLM integration | +| `soundgen-gui` | egui GUI: editor + training tab | +| `soundgen-feedback` | SQLite feedback database + similarity search | | `soundgen-runtime` | NES-authentic DAC + `SoundBank` for game embedding | ## Runtime Library (for game integration)