- 'Why Soundgen?' section with AI feedback loop diagram - Training pipeline section: 5-step guide (connect MCP, AI generates, rate, AI improves, export) - All 5 MCP tools documented in table - Updated architecture table with soundgen-feedback crate - SoundSpec example now includes vibrato field
This commit is contained in:
@@ -1,66 +1,111 @@
|
|||||||
# Soundgen
|
# Soundgen
|
||||||
|
|
||||||
8-bit sound synthesizer in Rust for game audio assets, with LLM integration via MCP.
|
8-bit sound synthesizer in Rust with AI-powered sound generation via MCP + human feedback loop.
|
||||||
|
|
||||||
|
## Why Soundgen?
|
||||||
|
|
||||||
|
Most free 8-bit sound tools are either closed-source, hard to automate, or don't integrate with AI workflows. Soundgen fixes this:
|
||||||
|
|
||||||
|
- **LLM generates sounds** by calling MCP tools — no manual parameter tweaking
|
||||||
|
- **You rate the results** in the GUI — the AI learns from your feedback
|
||||||
|
- **The AI improves** — next time it generates a similar sound, it references your top-rated examples
|
||||||
|
- **Export to WAV** — drop the files straight into your game
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────┐ generate_batch ┌─────────┐ rate 1-5★ ┌──────────┐
|
||||||
|
│ AI │ ────────────────→ │ Sounds │ ──────────→ │ You │
|
||||||
|
└──────┘ └─────────┘ └──────────┘
|
||||||
|
▲ uses top-rated examples as reference │
|
||||||
|
│ │
|
||||||
|
└──────────────── ┌──────────────┐ ◄───────────────────┘
|
||||||
|
│ Feedback DB │
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **6 voice types**: pulse (NES duty cycles), triangle, noise (LFSR), DPCM samples, wavetable (Game Boy wave), FM (2-operator)
|
- **Training pipeline**: AI generates → you rate → AI learns (SQLite feedback database)
|
||||||
- **Effects**: ADSR envelope, frequency sweep (linear/exponential), biquad filter (lowpass/highpass), vibrato
|
- **6 voice types**: pulse (NES duty cycles), triangle, noise (LFSR), DPCM, wavetable, FM
|
||||||
- **JSON-first**: every sound is a `SoundSpec` JSON object — LLMs generate JSON, CLI/MCP render to WAV
|
- **Effects**: ADSR envelope, frequency sweep, biquad filter, vibrato (LFO)
|
||||||
- **Presets as data**: 13 built-in presets in `presets/{sfx,ui,ambient}/` — extend without recompilation
|
- **JSON-first**: every sound is a `SoundSpec` JSON — LLMs generate JSON naturally
|
||||||
- **MCP server**: LLMs can call `list_presets`, `generate_sfx`, `render_sound` as tools
|
- **MCP server**: 5 tools for LLM integration (`list_presets`, `generate_sfx`, `render_sound`, `generate_batch`, `get_reference_sounds`)
|
||||||
- **Sequencer**: pattern-based song playback (JSON format)
|
- **GUI**: egui editor with virtual keyboard, preset browser, channel controls, training tab
|
||||||
- **GUI**: egui editor with virtual keyboard, preset browser, channel controls, sequencer
|
- **13 built-in presets**: SFX, UI, ambient — extend with JSON files, no recompilation
|
||||||
|
- **Sequencer**: pattern-based song playback (JSON)
|
||||||
- **Runtime library**: NES-authentic nonlinear DAC + SoundBank for game embedding
|
- **Runtime library**: NES-authentic nonlinear DAC + SoundBank for game embedding
|
||||||
- **No allocations in audio hot path**: `tick() -> f32`, `&mut self`
|
- **No allocations in audio hot path**: `tick() -> f32`
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# List available presets
|
# CLI — list presets
|
||||||
cargo run --bin soundgen -- list-presets
|
cargo run --bin soundgen -- list-presets
|
||||||
|
|
||||||
# Generate a sound from a preset
|
# Generate from preset
|
||||||
cargo run --bin soundgen -- gen jump --out assets/jump.wav
|
cargo run --bin soundgen -- gen jump --out jump.wav
|
||||||
|
|
||||||
# Generate with parameter override
|
# Render custom spec
|
||||||
cargo run --bin soundgen -- gen explosion --out assets/explosion.wav --param volume=0.95
|
|
||||||
|
|
||||||
# Render from a custom JSON spec
|
|
||||||
cargo run --bin soundgen -- render presets/sfx/laser.json --out laser.wav
|
cargo run --bin soundgen -- render presets/sfx/laser.json --out laser.wav
|
||||||
|
|
||||||
# Render a song (sequencer)
|
# Render a song
|
||||||
cargo run --bin soundgen -- render-song song.json --out music.wav
|
cargo run --bin soundgen -- render-song song.json --out music.wav
|
||||||
|
|
||||||
# Launch the GUI editor
|
# Launch the GUI editor
|
||||||
cargo run -p soundgen-gui
|
cargo run -p soundgen-gui
|
||||||
```
|
```
|
||||||
|
|
||||||
## MCP Server (for LLM integration)
|
## Training Pipeline (AI + Human Feedback)
|
||||||
|
|
||||||
Run the MCP server on stdio:
|
### 1. Connect the MCP server
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cargo run -p soundgen-mcp -- --presets-dir presets
|
cargo run -p soundgen-mcp -- --presets-dir presets --db feedback.db
|
||||||
```
|
```
|
||||||
|
|
||||||
Configure in Claude Desktop / MCP client:
|
Add to your MCP client config (Claude Desktop, OpenCode, etc.):
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"mcpServers": {
|
"mcpServers": {
|
||||||
"soundgen": {
|
"soundgen": {
|
||||||
"command": "/path/to/soundgen-mcp",
|
"command": "/path/to/soundgen-mcp",
|
||||||
"args": ["--presets-dir", "/path/to/presets"]
|
"args": ["--presets-dir", "/path/to/presets", "--db", "/path/to/feedback.db"]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
LLM workflow:
|
### 2. AI generates sounds
|
||||||
1. `list_presets` → see available sounds
|
|
||||||
2. `generate_sfx { preset: "jump", out_path: "assets/jump.wav" }` → WAV created
|
The AI calls MCP tools to create sounds:
|
||||||
3. `render_sound { spec: {...}, out_path: "assets/custom.wav" }` → custom sound
|
|
||||||
|
| Tool | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `list_presets` | List available built-in presets |
|
||||||
|
| `generate_sfx` | Generate WAV from a named preset |
|
||||||
|
| `render_sound` | Generate WAV from a custom SoundSpec JSON — returns reference examples from your top-rated sounds |
|
||||||
|
| `generate_batch` | Generate multiple sounds, store in feedback DB (unrated) |
|
||||||
|
| `get_reference_sounds` | Search DB for highly-rated similar sounds |
|
||||||
|
|
||||||
|
### 3. You rate them
|
||||||
|
|
||||||
|
Open the GUI → **Training** tab → play each sound, rate 1-5 stars, type feedback:
|
||||||
|
- "too loud" → AI lowers volume next time
|
||||||
|
- "pitch зачем-то возрастает" → AI fixes the sweep
|
||||||
|
- "Идеально!" → AI uses this as a reference for future sounds
|
||||||
|
|
||||||
|
### 4. AI improves
|
||||||
|
|
||||||
|
When the AI calls `render_sound` with `name: "explosion"`, the MCP server searches the feedback DB for similar highly-rated sounds and returns their specs as reference examples. The AI sees what worked and adjusts its approach.
|
||||||
|
|
||||||
|
### 5. Export for fine-tuning (optional)
|
||||||
|
|
||||||
|
Once you have 100+ rated sounds, export them as a JSONL dataset for LoRA fine-tuning:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In GUI: Training tab → "Export Dataset"
|
||||||
|
# Creates feedback_dataset.jsonl with all 4★+ sounds
|
||||||
|
```
|
||||||
|
|
||||||
## SoundSpec JSON Format
|
## SoundSpec JSON Format
|
||||||
|
|
||||||
@@ -75,6 +120,7 @@ LLM workflow:
|
|||||||
"duty": 50,
|
"duty": 50,
|
||||||
"frequency": { "start": 200, "end": 800, "curve": "exponential" },
|
"frequency": { "start": 200, "end": 800, "curve": "exponential" },
|
||||||
"envelope": { "attack": 0.01, "decay": 0.15, "sustain": 0.0, "release": 0.14 },
|
"envelope": { "attack": 0.01, "decay": 0.15, "sustain": 0.0, "release": 0.14 },
|
||||||
|
"vibrato": { "rate": 8, "depth": 200 },
|
||||||
"volume": 0.7
|
"volume": 0.7
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
@@ -82,36 +128,20 @@ LLM workflow:
|
|||||||
```
|
```
|
||||||
|
|
||||||
Channel types: `pulse`, `triangle`, `noise`
|
Channel types: `pulse`, `triangle`, `noise`
|
||||||
|
Optional fields: `filter` (lowpass/highpass + cutoff sweep), `vibrato` (LFO frequency modulation)
|
||||||
## Song JSON Format (Sequencer)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"bpm": 120,
|
|
||||||
"rows_per_beat": 4,
|
|
||||||
"tracks": [
|
|
||||||
{ "type": "pulse", "duty": 50, "volume": 0.4 }
|
|
||||||
],
|
|
||||||
"patterns": [
|
|
||||||
{ "rows": [ { "notes": [{"frequency": 440}] }, {"notes": [null]} ] }
|
|
||||||
],
|
|
||||||
"pattern_order": [0]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
Cargo workspace:
|
|
||||||
|
|
||||||
| Crate | Purpose |
|
| Crate | Purpose |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `soundgen-core` | Synthesis engine (generators, effects, mixer). No I/O. |
|
| `soundgen-core` | Synthesis engine (generators, effects, mixer). No I/O. |
|
||||||
| `soundgen-fmt` | `SoundSpec` JSON schema + `PresetRegistry` |
|
| `soundgen-fmt` | `SoundSpec` JSON schema + `PresetRegistry` |
|
||||||
| `soundgen-io` | WAV writer (`hound`) + playback (subprocess) |
|
| `soundgen-io` | WAV writer (`hound`) + playback |
|
||||||
| `soundgen-seq` | Sequencer: patterns, songs |
|
| `soundgen-seq` | Sequencer: patterns, songs |
|
||||||
| `soundgen-cli` | `gen`, `render`, `render-song`, `list-presets` |
|
| `soundgen-cli` | `gen`, `render`, `render-song`, `list-presets` |
|
||||||
| `soundgen-mcp` | MCP server for LLM tool-use |
|
| `soundgen-mcp` | MCP server: 5 tools for LLM integration |
|
||||||
| `soundgen-gui` | egui GUI editor |
|
| `soundgen-gui` | egui GUI: editor + training tab |
|
||||||
|
| `soundgen-feedback` | SQLite feedback database + similarity search |
|
||||||
| `soundgen-runtime` | NES-authentic DAC + `SoundBank` for game embedding |
|
| `soundgen-runtime` | NES-authentic DAC + `SoundBank` for game embedding |
|
||||||
|
|
||||||
## Runtime Library (for game integration)
|
## Runtime Library (for game integration)
|
||||||
|
|||||||
Reference in New Issue
Block a user