Initial commit.

DESIGN.md

@@ -0,0 +1,324 @@
+# memba — Design Document
+
+> **One-liner:** Persistent memory layer for local SSM-based LLMs. Save, load, and share model *understanding* — not weights, not chat history, but accumulated intelligence.
+
+---
+
+## 1. Problem Statement
+
+### 1.1 The Stateless Nature of Local LLMs
+
+Every local inference tool (Ollama, llama.cpp, LM Studio) treats each invocation as a blank slate. When you close the terminal or reboot the laptop, the model forgets everything it has "read." There is no concept of a **session** that survives process death.
+
+For Transformer-based models (Llama, GPT, Qwen), this is a technical necessity: their "memory" is a KV-cache that grows linearly with context length (15 GB for 128K tokens). Saving it is impractical.
+
+For SSM-based models (Mamba, Falcon-Mamba, Zamba), this is an **infrastructure gap**. Their memory is a fixed-size recurrent state (~64 MB for 7B parameters). It *could* be serialized, yet no production tool exposes this capability.
+
+### 1.2 Consequences
+
+- **Re-reading cost:** Analysts must re-ingest 500-page documents every morning.
+- **Knowledge fragmentation:** A research team cannot hand off a model's accumulated understanding to a colleague.
+- **Air-gapped paralysis:** In offline environments, there is no way to checkpoint progress.
+- **Privacy trade-offs:** To preserve context, users paste entire documents into cloud APIs (OpenAI, Perplexity), leaking confidential data.
+
+---
+
+## 2. Vision
+
+**memba** closes this gap by making SSM state a first-class citizen: a portable, inspectable (via generation), and shareable asset.
+
+We do not compete with ChatGPT on model quality. We compete on **autonomy**: offline-first, privacy-preserving, and stateful edge intelligence.
+
+---
+
+## 3. Core Concepts
+
+### 3.1 What is "State"?
+
+In the context of SSM models, "state" is the recurrent hidden representation that the model maintains while reading text. It is:
+
+- **Fixed-size:** ~64 MB for Falcon-Mamba-7B, regardless of whether the model read 1 page or 10,000 pages.
+- **Lossy:** It encodes *understanding*, not verbatim text. You cannot reconstruct the source document from state alone.
+- **Opaque:** It is a binary blob of tensors. Human interpretation requires asking the model to verbalize its contents.
+- **Portable:** The same state file works on CPU (laptop) and GPU (server), because `llama.cpp` abstracts the backend.
+
+### 3.2 State vs. KV-Cache
+
+| Property | Transformer KV-Cache | SSM State |
+|----------|-------------------|-----------|
+| Size | 15–20 GB @ 128K | ~64 MB |
+| Growth | O(n) linear | O(1) constant |
+| Saveable | Impractical | Trivial |
+| Cross-device | Impossible | Seamless |
+| Content | Exact token history | Compressed understanding |
+
+### 3.3 State vs. Vector Database
+
+| Property | Vector DB (RAG) | SSM State |
+|----------|----------------|-----------|
+| Stores | Original text chunks | No original text |
+| Retrieval | Exact citation via similarity search | No random access |
+| Synthesis | Retrieves facts; does not *connect* them | Encodes cross-domain connections |
+| Privacy | High risk (raw text on disk) | Low risk (lossy, non-reconstructible) |
+| Use case | Lookup | Deep, accumulated reasoning |
+
+**Conclusion:** State and Vector DB are **complementary**, not competitive. An optimal system uses state for *synthesis* and a local encrypted index for *citation*.
+
+---
+
+## 4. Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  memba Ecosystem                                            │
+│                                                             │
+│  ┌──────────────┐      ┌──────────────┐                  │
+│  │   C++ Core   │      │  Python SDK  │                  │
+│  │  libmemba    │◄────►│   memba      │                  │
+│  │  (.so/.dll)  │      │   (pip)      │                  │
+│  └──────┬───────┘      └──────┬───────┘                  │
+│         │                       │                          │
+│         └──────────┬──────────┘                          │
+│                    │                                       │
+│              ┌─────┴─────┐                               │
+│              │ llama.cpp │  ← git submodule               │
+│              │ (backend) │                                 │
+│              └─────┬─────┘                               │
+│                    │                                       │
+│         ┌──────────┼──────────┐                          │
+│         │          │          │                          │
+│      ┌──┴──┐   ┌───┴───┐  ┌──┴──┐                       │
+│      │ CPU │   │ CUDA  │  │Metal│                       │
+│      └──┬──┘   └───┬───┘  └──┬──┘                       │
+│         │          │          │                          │
+│         └──────────┴──────────┘                          │
+│                    │                                       │
+│              ┌─────┴─────┐                               │
+│              │  State    │  ← ~/.memba/states/            │
+│              │  Files    │     *.bin (portable)           │
+│              └───────────┘                                 │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### 4.1 Layer Responsibilities
+
+**llama.cpp (Submodule):**
+- Provides inference engine, GGUF loading, SSM ops, backend abstraction.
+- memba does **not** fork or patch it unless absolutely necessary.
+
+**C++ Core (`libmemba`):**
+- Thin wrapper around `llama_copy_state_data()` / `llama_set_state_data()`.
+- Adds file header (magic, version, model_id, CRC32).
+- Zero knowledge of model internals; treats state as opaque blob.
+
+**Python SDK (`memba`):**
+- `Session` class: high-level UX with auto-save/load.
+- CLI: `memba save`, `memba load`, `memba list`, `memba rm`.
+- Optional integrations (LangGraph, etc.) via examples.
+
+---
+
+## 5. Feature Specification
+
+### 5.1 MVP (Weeks 1–2)
+
+**Goal:** A working CLI and Python API that proves state persistence across process restarts.
+
+- [ ] **State I/O**
+  - `memba_state_save()` — serialize `llama_context` to `.bin` file.
+  - `memba_state_load()` — restore context from `.bin` file.
+  - Header validation: magic (`MEMB`), version, `model_id` (SHA256 of first 1KB of GGUF), `n_ctx`, CRC32.
+  - Error handling: return codes + descriptive messages.
+
+- [ ] **Session Management**
+  - `~/.memba/states/` as default storage directory.
+  - `Session` class with `session_id`, auto-save on `chat()`, explicit `save()` / `load()`.
+  - `memba list` — table of sessions (name, model, size, modified date).
+  - `memba rm <session>` — delete state file.
+
+- [ ] **Backend Agnostic**
+  - `n_gpu_layers` parameter passed through to `llama.cpp`.
+  - Same state file loads on CPU-only laptop and GPU server.
+  - No backend-specific code in memba core.
+
+- [ ] **Privacy-First Default**
+  - `mode="state-only"` by default.
+  - No source text stored anywhere.
+  - State file is opaque binary; cannot be reverse-engineered to reconstruct documents.
+
+- [ ] **Examples**
+  - `01_basic_save_load.py` — demonstrate save/load cycle.
+  - `02_chat_session.py` — REPL with persistent memory.
+
+### 5.2 Phase 2 (Months 2–3)
+
+**Goal:** Team collaboration and advanced persistence.
+
+- [ ] **Cloud Sync (Opt-in)**
+  - `memba push <session>` → S3 / MinIO / WebDAV.
+  - `memba pull <session>` → download to local machine.
+  - User brings their own storage credentials (no memba-hosted cloud).
+
+- [ ] **Encryption at Rest**
+  - AES-256 encryption for state files.
+  - Key derived from user passphrase or hardware TPM / Secure Enclave.
+  - Encrypted state files are still portable (decrypt on load).
+
+- [ ] **Versioning / Checkpointing**
+  - `memba checkpoint <session>` — immutable snapshot.
+  - `memba rollback <session> --to <checkpoint_id>`.
+  - Git-like history for model understanding.
+
+- [ ] **Team Sharing**
+  - `memba share <session> --team <id>` — generate presigned URL or LAN broadcast.
+  - Access control: read-only or read-write state sharing.
+
+- [ ] **Hybrid Mode (State + Local Index)**
+  - Optional integration with Meilisearch / ChromaDB for exact citation.
+  - Encrypted index storage.
+  - Explicit opt-in with privacy warning.
+
+### 5.3 Future (6+ months)
+
+**Goal:** Ecosystem and enterprise readiness.
+
+- [ ] **Dataset Generation (Enterprise Add-on)**
+  - Use stateful model as oracle to generate synthetic training data.
+  - Export to JSONL / HuggingFace datasets format.
+  - Use case: distilling domain expertise into smaller Transformer models.
+
+- [ ] **State Merge / Diff**
+  - `memba merge session_a session_b` — combine two accumulated understandings.
+  - Conceptual: weighted averaging or concatenation of state sequences (research-heavy).
+
+- [ ] **Hierarchical States**
+  - `core_state.bin` — system-wide architecture.
+  - `service_auth.bin`, `service_payment.bin` — module-specific deep dives.
+  - On query, load relevant substates to stay within memory limits.
+
+- [ ] **Observability**
+  - `memba inspect <session>` — high-level report of what the model "knows" (via probing prompts).
+  - State size analytics, compression ratios, coverage heatmaps.
+
+---
+
+## 6. User Stories
+
+### 6.1 OSINT Analyst (Air-gapped)
+
+> *"I work in a facility with no internet. I spend a week reading 10,000 pages of intelligence reports through a local Mamba model. On Friday, I save the state to a USB drive. Monday, my colleague loads it on her machine and continues the analysis without re-reading anything."*
+
+**Value:** Knowledge handoff in zero-trust environments.
+
+### 6.2 Legal Due Diligence
+
+> *"I need to analyze a 500-page M&A contract. I cannot upload it to ChatGPT because of confidentiality. I load the contract into a local Falcon-Mamba model, save the state, and ask questions throughout the week. The state is 64 MB; I can email it to my partner for a second opinion without emailing the contract itself."*
+
+**Value:** Privacy-preserving collaboration on sensitive documents.
+
+### 6.3 SRE / DevOps
+
+> *"Our microservices platform has 200 repos. I scan them once into a state file. New engineers load this 'architecture state' on day one and ask questions like 'Why does the payment service talk directly to Postgres instead of going through the queue?' The model answers from accumulated understanding, not from grepping code."*
+
+**Value:** Accelerated onboarding and architectural preservation.
+
+### 6.4 Academic Researcher
+
+> *"I am writing a systematic review. I feed 2,000 papers into the model over a month, saving state after each batch. The model helps me see connections between papers I never would have noticed through keyword search alone."*
+
+**Value:** Cross-domain synthesis at scale.
+
+---
+
+## 7. Privacy & Security Model
+
+### 7.1 Threat Model
+
+| Threat | Mitigation |
+|--------|------------|
+| **State file stolen** | State is lossy and non-reconstructible. Attacker cannot recover source documents. |
+| **State file corrupted** | CRC32 + version header validation on load. Refuse to load damaged state. |
+| **Wrong model loaded** | `model_id` header (SHA256 of GGUF). Prevents loading state into incompatible architecture. |
+| **Memory dump during inference** | Source text exists only in RAM during active reading. No persistent storage of raw text in state-only mode. |
+| **Cloud sync interception** | Phase 2: AES-256 encryption before upload. User controls keys. |
+
+### 7.2 Privacy Modes
+
+| Mode | Source Text Stored | Use Case |
+|------|-------------------|----------|
+| **State-only (default)** | ❌ No | Maximum privacy, air-gapped |
+| **Hybrid (opt-in)** | ⚠️ Encrypted local index | When exact citation is required |
+| **Cloud sync (opt-in)** | ❌ No (state encrypted) | Team collaboration |
+
+---
+
+## 8. Competitive Analysis
+
+| Competitor | What they do | Why memba is different |
+------------|--------------|------------------------|
+| **Ollama** | Local LLM runner | Stateless. No session persistence. |
+| **LM Studio** | GUI for local LLM | Stateless. No session persistence. |
+| **llama.cpp (main)** | Inference engine | Stateless by design. No save/load UX. |
+| **BitMamba** | POC state serialization | CLI-only, 200 lines of C, no Python API, no product. |
+| **Perplexity / OpenAI DR** | Cloud research agents | Cloud-only, no persistence, privacy risk. |
+| **ChromaDB / Pinecone** | Vector databases | Store raw text; no accumulated reasoning. |
+
+**memba's unique position:** The only tool that treats **model understanding** as a portable, persistent, privacy-safe asset.
+
+---
+
+## 9. Roadmap
+
+| Phase | Timeline | Deliverable | Success Metric |
+|-------|----------|-------------|----------------|
+| **MVP** | Weeks 1–2 | C++ core + Python SDK + CLI | 100 GitHub stars, 5 pilot users |
+| **Stability** | Month 2 | Tests, CI/CD, macOS support | All tests green, 500 stars |
+| **Collaboration** | Month 3 | Cloud sync, encryption, sharing | 10 teams using shared states |
+| **Enterprise** | Month 6 | Hybrid mode, dataset generation, support | First paid enterprise contract |
+| **Ecosystem** | Month 12 | Hierarchical states, merge, observability | 5K stars, plugin ecosystem |
+
+---
+
+## 10. Open Questions & Risks
+
+### 10.1 Technical Risks
+
+- **llama.cpp API drift:** If `llama_copy_state_data()` changes signature or internal layout, state files break. Mitigation: version header + pinned submodule commits.
+- **SSM model availability:** Falcon-Mamba is the only viable 7B SSM with open weights. If the ecosystem shifts away from SSM, memba's addressable market shrinks. Mitigation: monitor Jamba, Zamba, Granite adoption.
+- **State degradation:** Very long sequences (>100K tokens) may cause SSM state to "forget" early content. Mitigation: hierarchical states (future feature).
+
+### 10.2 Product Risks
+
+- **Adoption curve:** Users may not understand why they need persistent state until they try it. Mitigation: compelling demo videos ("close laptop, open tomorrow, model remembers").
+- **Competition from upstream:** llama.cpp could add `--save-state` natively. Mitigation: build ecosystem (cloud sync, encryption, team features) faster than upstream moves.
+
+### 10.3 Ethical Risks
+
+- **Synthetic data hallucinations:** If users generate datasets from state, hallucinations in the state propagate to training data. Mitigation: clear documentation that synthetic data requires human review.
+
+---
+
+## 11. Glossary
+
+| Term | Definition |
+|------|------------|
+| **SSM** | State Space Model. A sequence modeling architecture (e.g., Mamba) with fixed-size recurrent state. |
+| **State** | In this document, always refers to the SSM recurrent hidden state, not model weights or KV-cache. |
+| **KV-Cache** | Transformer memory mechanism. Grows with sequence length. Not supported by memba. |
+| **GGUF** | File format for quantized LLM weights, used by llama.cpp. |
+| **State-only mode** | memba operating without any vector database or text storage. Only the 64 MB state file persists. |
+| **Opaque blob** | Binary data whose internal structure is not parsed by memba. llama.cpp manages the format. |
+
+---
+
+## 12. Contributing & Philosophy
+
+- **Open core:** All MVP features are MIT-licensed and free.
+- **No telemetry:** memba never phones home. Sync is explicit, user-controlled, and uses the user's own storage.
+- **Unix philosophy:** Do one thing well. memba manages state; it is not an IDE, not a search engine, not a cloud service.
+
+---
+
+*Document version: 0.1*  
+*Last updated: 2026-05-16*  
+*Status: Draft for MVP kickoff*