Files

T

2026-05-16 07:01:57 +03:00

16 KiB

Raw Blame History

memba — Design Document

One-liner: Persistent memory layer for local SSM-based LLMs. Save, load, and share model understanding — not weights, not chat history, but accumulated intelligence.

1. Problem Statement

1.1 The Stateless Nature of Local LLMs

Every local inference tool (Ollama, llama.cpp, LM Studio) treats each invocation as a blank slate. When you close the terminal or reboot the laptop, the model forgets everything it has "read." There is no concept of a session that survives process death.

For Transformer-based models (Llama, GPT, Qwen), this is a technical necessity: their "memory" is a KV-cache that grows linearly with context length (15 GB for 128K tokens). Saving it is impractical.

For SSM-based models (Mamba, Falcon-Mamba, Zamba), this is an infrastructure gap. Their memory is a fixed-size recurrent state (~64 MB for 7B parameters). It could be serialized, yet no production tool exposes this capability.

1.2 Consequences

Re-reading cost: Analysts must re-ingest 500-page documents every morning.
Knowledge fragmentation: A research team cannot hand off a model's accumulated understanding to a colleague.
Air-gapped paralysis: In offline environments, there is no way to checkpoint progress.
Privacy trade-offs: To preserve context, users paste entire documents into cloud APIs (OpenAI, Perplexity), leaking confidential data.

2. Vision

memba closes this gap by making SSM state a first-class citizen: a portable, inspectable (via generation), and shareable asset.

We do not compete with ChatGPT on model quality. We compete on autonomy: offline-first, privacy-preserving, and stateful edge intelligence.

3. Core Concepts

3.1 What is "State"?

In the context of SSM models, "state" is the recurrent hidden representation that the model maintains while reading text. It is:

Fixed-size: ~64 MB for Falcon-Mamba-7B, regardless of whether the model read 1 page or 10,000 pages.
Lossy: It encodes understanding, not verbatim text. You cannot reconstruct the source document from state alone.
Opaque: It is a binary blob of tensors. Human interpretation requires asking the model to verbalize its contents.
Portable: The same state file works on CPU (laptop) and GPU (server), because llama.cpp abstracts the backend.

3.2 State vs. KV-Cache

Property	Transformer KV-Cache	SSM State
Size	15–20 GB @ 128K	~64 MB
Growth	O(n) linear	O(1) constant
Saveable	Impractical	Trivial
Cross-device	Impossible	Seamless
Content	Exact token history	Compressed understanding

3.3 State vs. Vector Database

Property	Vector DB (RAG)	SSM State
Stores	Original text chunks	No original text
Retrieval	Exact citation via similarity search	No random access
Synthesis	Retrieves facts; does not connect them	Encodes cross-domain connections
Privacy	High risk (raw text on disk)	Low risk (lossy, non-reconstructible)
Use case	Lookup	Deep, accumulated reasoning

Conclusion: State and Vector DB are complementary, not competitive. An optimal system uses state for synthesis and a local encrypted index for citation.

4. Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│  memba Ecosystem                                            │
│                                                             │
│  ┌──────────────┐      ┌──────────────┐                  │
│  │   C++ Core   │      │  Python SDK  │                  │
│  │  libmemba    │◄────►│   memba      │                  │
│  │  (.so/.dll)  │      │   (pip)      │                  │
│  └──────┬───────┘      └──────┬───────┘                  │
│         │                       │                          │
│         └──────────┬──────────┘                          │
│                    │                                       │
│              ┌─────┴─────┐                               │
│              │ llama.cpp │  ← git submodule               │
│              │ (backend) │                                 │
│              └─────┬─────┘                               │
│                    │                                       │
│         ┌──────────┼──────────┐                          │
│         │          │          │                          │
│      ┌──┴──┐   ┌───┴───┐  ┌──┴──┐                       │
│      │ CPU │   │ CUDA  │  │Metal│                       │
│      └──┬──┘   └───┬───┘  └──┬──┘                       │
│         │          │          │                          │
│         └──────────┴──────────┘                          │
│                    │                                       │
│              ┌─────┴─────┐                               │
│              │  State    │  ← ~/.memba/states/            │
│              │  Files    │     *.bin (portable)           │
│              └───────────┘                                 │
└─────────────────────────────────────────────────────────────┘

4.1 Layer Responsibilities

llama.cpp (Submodule):

Provides inference engine, GGUF loading, SSM ops, backend abstraction.
memba does not fork or patch it unless absolutely necessary.

C++ Core (libmemba):

Thin wrapper around llama_copy_state_data() / llama_set_state_data().
Adds file header (magic, version, model_id, CRC32).
Zero knowledge of model internals; treats state as opaque blob.

Python SDK (memba):

Session class: high-level UX with auto-save/load.
CLI: memba save, memba load, memba list, memba rm.
Optional integrations (LangGraph, etc.) via examples.

5. Feature Specification

5.1 MVP (Weeks 1–2)

Goal: A working CLI and Python API that proves state persistence across process restarts.

State I/O
- memba_state_save() — serialize llama_context to .bin file.
- memba_state_load() — restore context from .bin file.
- Header validation: magic (MEMB), version, model_id (SHA256 of first 1KB of GGUF), n_ctx, CRC32.
- Error handling: return codes + descriptive messages.
Session Management
- ~/.memba/states/ as default storage directory.
- Session class with session_id, auto-save on chat(), explicit save() / load().
- memba list — table of sessions (name, model, size, modified date).
- memba rm <session> — delete state file.
Backend Agnostic
- n_gpu_layers parameter passed through to llama.cpp.
- Same state file loads on CPU-only laptop and GPU server.
- No backend-specific code in memba core.
Privacy-First Default
- mode="state-only" by default.
- No source text stored anywhere.
- State file is opaque binary; cannot be reverse-engineered to reconstruct documents.
Examples
- 01_basic_save_load.py — demonstrate save/load cycle.
- 02_chat_session.py — REPL with persistent memory.

5.2 Phase 2 (Months 2–3)

Goal: Team collaboration and advanced persistence.

Cloud Sync (Opt-in)
- memba push <session> → S3 / MinIO / WebDAV.
- memba pull <session> → download to local machine.
- User brings their own storage credentials (no memba-hosted cloud).
Encryption at Rest
- AES-256 encryption for state files.
- Key derived from user passphrase or hardware TPM / Secure Enclave.
- Encrypted state files are still portable (decrypt on load).
Versioning / Checkpointing
- memba checkpoint <session> — immutable snapshot.
- memba rollback <session> --to <checkpoint_id>.
- Git-like history for model understanding.
Team Sharing
- memba share <session> --team <id> — generate presigned URL or LAN broadcast.
- Access control: read-only or read-write state sharing.
Hybrid Mode (State + Local Index)
- Optional integration with Meilisearch / ChromaDB for exact citation.
- Encrypted index storage.
- Explicit opt-in with privacy warning.

5.3 Future (6+ months)

Goal: Ecosystem and enterprise readiness.

Dataset Generation (Enterprise Add-on)
- Use stateful model as oracle to generate synthetic training data.
- Export to JSONL / HuggingFace datasets format.
- Use case: distilling domain expertise into smaller Transformer models.
State Merge / Diff
- memba merge session_a session_b — combine two accumulated understandings.
- Conceptual: weighted averaging or concatenation of state sequences (research-heavy).
Hierarchical States
- core_state.bin — system-wide architecture.
- service_auth.bin, service_payment.bin — module-specific deep dives.
- On query, load relevant substates to stay within memory limits.
Observability
- memba inspect <session> — high-level report of what the model "knows" (via probing prompts).
- State size analytics, compression ratios, coverage heatmaps.

6. User Stories

6.1 OSINT Analyst (Air-gapped)

"I work in a facility with no internet. I spend a week reading 10,000 pages of intelligence reports through a local Mamba model. On Friday, I save the state to a USB drive. Monday, my colleague loads it on her machine and continues the analysis without re-reading anything."

Value: Knowledge handoff in zero-trust environments.

6.2 Legal Due Diligence

"I need to analyze a 500-page M&A contract. I cannot upload it to ChatGPT because of confidentiality. I load the contract into a local Falcon-Mamba model, save the state, and ask questions throughout the week. The state is 64 MB; I can email it to my partner for a second opinion without emailing the contract itself."

Value: Privacy-preserving collaboration on sensitive documents.

6.3 SRE / DevOps

"Our microservices platform has 200 repos. I scan them once into a state file. New engineers load this 'architecture state' on day one and ask questions like 'Why does the payment service talk directly to Postgres instead of going through the queue?' The model answers from accumulated understanding, not from grepping code."

Value: Accelerated onboarding and architectural preservation.

6.4 Academic Researcher

"I am writing a systematic review. I feed 2,000 papers into the model over a month, saving state after each batch. The model helps me see connections between papers I never would have noticed through keyword search alone."

Value: Cross-domain synthesis at scale.

7. Privacy & Security Model

7.1 Threat Model

Threat	Mitigation
State file stolen	State is lossy and non-reconstructible. Attacker cannot recover source documents.
State file corrupted	CRC32 + version header validation on load. Refuse to load damaged state.
Wrong model loaded	`model_id` header (SHA256 of GGUF). Prevents loading state into incompatible architecture.
Memory dump during inference	Source text exists only in RAM during active reading. No persistent storage of raw text in state-only mode.
Cloud sync interception	Phase 2: AES-256 encryption before upload. User controls keys.

7.2 Privacy Modes

Mode	Source Text Stored	Use Case
State-only (default)	❌ No	Maximum privacy, air-gapped
Hybrid (opt-in)	⚠️ Encrypted local index	When exact citation is required
Cloud sync (opt-in)	❌ No (state encrypted)	Team collaboration

8. Competitive Analysis

Competitor	What they do	Why memba is different
Ollama	Local LLM runner	Stateless. No session persistence.
LM Studio	GUI for local LLM	Stateless. No session persistence.
llama.cpp (main)	Inference engine	Stateless by design. No save/load UX.
BitMamba	POC state serialization	CLI-only, 200 lines of C, no Python API, no product.
Perplexity / OpenAI DR	Cloud research agents	Cloud-only, no persistence, privacy risk.
ChromaDB / Pinecone	Vector databases	Store raw text; no accumulated reasoning.

memba's unique position: The only tool that treats model understanding as a portable, persistent, privacy-safe asset.

9. Roadmap

Phase	Timeline	Deliverable	Success Metric
MVP	Weeks 1–2	C++ core + Python SDK + CLI	100 GitHub stars, 5 pilot users
Stability	Month 2	Tests, CI/CD, macOS support	All tests green, 500 stars
Collaboration	Month 3	Cloud sync, encryption, sharing	10 teams using shared states
Enterprise	Month 6	Hybrid mode, dataset generation, support	First paid enterprise contract
Ecosystem	Month 12	Hierarchical states, merge, observability	5K stars, plugin ecosystem

10. Open Questions & Risks

10.1 Technical Risks

llama.cpp API drift: If llama_copy_state_data() changes signature or internal layout, state files break. Mitigation: version header + pinned submodule commits.
SSM model availability: Falcon-Mamba is the only viable 7B SSM with open weights. If the ecosystem shifts away from SSM, memba's addressable market shrinks. Mitigation: monitor Jamba, Zamba, Granite adoption.
State degradation: Very long sequences (>100K tokens) may cause SSM state to "forget" early content. Mitigation: hierarchical states (future feature).

10.2 Product Risks

Adoption curve: Users may not understand why they need persistent state until they try it. Mitigation: compelling demo videos ("close laptop, open tomorrow, model remembers").
Competition from upstream: llama.cpp could add --save-state natively. Mitigation: build ecosystem (cloud sync, encryption, team features) faster than upstream moves.

10.3 Ethical Risks

Synthetic data hallucinations: If users generate datasets from state, hallucinations in the state propagate to training data. Mitigation: clear documentation that synthetic data requires human review.

11. Glossary

Term	Definition
SSM	State Space Model. A sequence modeling architecture (e.g., Mamba) with fixed-size recurrent state.
State	In this document, always refers to the SSM recurrent hidden state, not model weights or KV-cache.
KV-Cache	Transformer memory mechanism. Grows with sequence length. Not supported by memba.
GGUF	File format for quantized LLM weights, used by llama.cpp.
State-only mode	memba operating without any vector database or text storage. Only the 64 MB state file persists.
Opaque blob	Binary data whose internal structure is not parsed by memba. llama.cpp manages the format.

12. Contributing & Philosophy

Open core: All MVP features are MIT-licensed and free.
No telemetry: memba never phones home. Sync is explicit, user-controlled, and uses the user's own storage.
Unix philosophy: Do one thing well. memba manages state; it is not an IDE, not a search engine, not a cloud service.

Document version: 0.1
Last updated: 2026-05-16
Status: Draft for MVP kickoff

16 KiB Raw Blame History Unescape Escape