files cant't be seen by LLM, but everything else works.
This commit is contained in:
+8
-8
@@ -1,9 +1,9 @@
|
|||||||
# Strapi secrets - change these in production!
|
# OpenAI-compatible API configuration
|
||||||
JWT_SECRET=your-jwt-secret-here
|
OPENAI_API_URL=https://api.openai.com/v1
|
||||||
ADMIN_JWT_SECRET=your-admin-jwt-secret-here
|
OPENAI_API_KEY=your-api-key-here
|
||||||
APP_KEYS=your-app-key-1,your-app-key-2
|
|
||||||
API_TOKEN_SALT=your-api-token-salt
|
|
||||||
TRANSFER_TOKEN_SALT=your-transfer-token-salt
|
|
||||||
|
|
||||||
# Optional: Strapi API token for backend integration
|
# Default model
|
||||||
STRAPI_API_TOKEN=
|
DEFAULT_MODEL=gpt-4o-mini
|
||||||
|
|
||||||
|
# Backend JWT secret (change in production)
|
||||||
|
JWT_SECRET=your-jwt-secret-here
|
||||||
|
|||||||
@@ -3,5 +3,4 @@ dist
|
|||||||
.env
|
.env
|
||||||
*.log
|
*.log
|
||||||
data/
|
data/
|
||||||
strapi/
|
|
||||||
!.env.example
|
!.env.example
|
||||||
|
|||||||
@@ -1,122 +1,91 @@
|
|||||||
# AIUI Chat
|
|
||||||
|
|
||||||
AIUI Chat — это современный веб-интерфейс для общения с LLM (большими языковыми моделями), вдохновленный Open WebUI. Поддерживает любой OpenAI-compatible API (OpenAI, Ollama, LM Studio, vLLM и другие).
|
Modern, responsive chat interface inspired by Open WebUI. Supports OpenAI-compatible API with streaming responses, token analysis, and RAG knowledge bases.
|
||||||
|
|
||||||
## Особенности
|
## 🚀 Quick Start (Docker)
|
||||||
|
|
||||||
- Чат с поддержкой **streaming** ответов
|
|
||||||
- Управление **историей** диалогов
|
|
||||||
- Настройки API, модели, temperature, max tokens
|
|
||||||
- Загрузка файлов в чат
|
|
||||||
- **Базы знаний** с ChromaDB — загружайте документы и получайте ответы с учетом контекста
|
|
||||||
- Адаптивный дизайн (десктоп + мобильные)
|
|
||||||
- Темная тема в стиле modern tech
|
|
||||||
|
|
||||||
## Структура проекта
|
|
||||||
|
|
||||||
```
|
|
||||||
AIUI/
|
|
||||||
├── frontend/ # React + TypeScript + Tailwind CSS
|
|
||||||
│ ├── src/
|
|
||||||
│ │ ├── components/ # UI компоненты
|
|
||||||
│ │ │ ├── chat/ # Компоненты чата
|
|
||||||
│ │ │ ├── layout/ # Layout компоненты
|
|
||||||
│ │ │ └── knowledge/ # Компоненты баз знаний
|
|
||||||
│ │ ├── services/ # API сервисы
|
|
||||||
│ │ ├── stores/ # Zustand stores
|
|
||||||
│ │ ├── types/ # TypeScript типы
|
|
||||||
│ │ └── utils/ # Утилиты
|
|
||||||
│ └── package.json
|
|
||||||
├── backend/ # Express.js proxy сервер
|
|
||||||
│ └── src/
|
|
||||||
│ └── index.js # Основной сервер
|
|
||||||
└── package.json # Root package.json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Быстрый старт
|
|
||||||
|
|
||||||
### 1. Установка зависимостей
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env — set your API URL and key
|
||||||
|
docker compose up --build -d
|
||||||
|
# Open http://localhost
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🏗️ Development
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
npm install
|
npm install
|
||||||
|
npm run dev # port 3001
|
||||||
```
|
```
|
||||||
|
|
||||||
Это установит зависимости для root, frontend и backend.
|
### Frontend
|
||||||
|
```bash
|
||||||
|
cd frontend
|
||||||
|
npm install
|
||||||
|
npm run dev # port 5173 (with HMR)
|
||||||
|
```
|
||||||
|
|
||||||
### 2. Настройка API
|
## ✨ Features
|
||||||
|
|
||||||
Запустите frontend и backend, затем откройте настройки (иконка шестеренки) и укажите:
|
| Feature | Description |
|
||||||
|
|---------|-------------|
|
||||||
|
| 💬 Chat | Streaming messages with markdown & LaTeX |
|
||||||
|
| ⚙️ Settings | Model, system prompt, temperature, tokens |
|
||||||
|
| 📊 Token Analysis | Usage stats panel (tokens/sec, context, cost) |
|
||||||
|
| 📁 Attachments | Image upload + analysis |
|
||||||
|
| 📚 Knowledge Bases | RAG with multiple bases, file/folder upload |
|
||||||
|
| 🎨 Theme | Dark mode, accent colors |
|
||||||
|
| 📱 Responsive | Works on mobile and desktop |
|
||||||
|
|
||||||
- **API Base URL** — например `https://api.openai.com/v1` или `http://localhost:11434/v1` (для Ollama)
|
## 🏛️ Architecture
|
||||||
- **API Key** — ваш ключ
|
|
||||||
- **Model** — например `gpt-4`, `gpt-3.5-turbo`, `llama2` и т.д.
|
|
||||||
- **Temperature** и **Max Tokens** — по желанию
|
|
||||||
- **System Prompt** — системный промпт
|
|
||||||
- **Embedding Model** — для баз знаний (например `text-embedding-3-small`)
|
|
||||||
- **ChromaDB Directory** — путь к хранилищу ChromaDB
|
|
||||||
|
|
||||||
### 3. Запуск
|
```
|
||||||
|
┌──────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||||
|
│ Nginx (80) │────────▶│ Backend │────────▶│ OpenAI API │
|
||||||
|
│ SPA + /api/* │ │ (Node.js) │ │ (or compatible)│
|
||||||
|
└──────────────────┘ └──────────────┘ └─────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│ sqlite db │
|
||||||
|
│ (chats) │
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔧 Environment Variables
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `OPENAI_API_URL` | `https://api.openai.com/v1` | API endpoint |
|
||||||
|
| `OPENAI_API_KEY` | - | Your API key |
|
||||||
|
| `DEFAULT_MODEL` | `gpt-4o-mini` | Default model ID |
|
||||||
|
| `JWT_SECRET` | random | JWT secret |
|
||||||
|
| `NODE_ENV` | `development` | Environment |
|
||||||
|
|
||||||
|
## 🐳 Production
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Запуск backend (порт 3001)
|
docker compose -f docker-compose.yml up -d --build
|
||||||
npm run dev:backend
|
|
||||||
|
|
||||||
# В другом терминале — запуск frontend (порт 3000)
|
|
||||||
npm run dev:frontend
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Frontend запустится на `http://localhost:3000` и будет проксировать API-запросы на backend.
|
Nginx serves static frontend + proxies `/api` to backend.
|
||||||
|
|
||||||
### Production сборка
|
## 📂 Project Structure
|
||||||
|
|
||||||
```bash
|
```
|
||||||
# Сборка frontend
|
├── frontend/ # Vite + React + TypeScript + Tailwind
|
||||||
npm run build:frontend
|
│ ├── src/components/ # Chat, UI, Layout components
|
||||||
|
│ ├── src/stores/ # Zustand store
|
||||||
# Запуск backend (раздает собранный frontend)
|
│ ├── src/services/ # API client
|
||||||
npm run start:backend
|
│ └── Dockerfile
|
||||||
|
├── backend/ # Express + proxy + chat storage
|
||||||
|
│ ├── src/index.js # Main server
|
||||||
|
│ └── Dockerfile
|
||||||
|
├── nginx/ # Reverse proxy config
|
||||||
|
└── docker-compose.yml # Full stack
|
||||||
```
|
```
|
||||||
|
|
||||||
## Базы знаний (Knowledge Bases)
|
## 📜 License
|
||||||
|
|
||||||
Базы знаний позволяют загружать документы и использовать их как контекст для ответов LLM.
|
MIT
|
||||||
|
|
||||||
### Как использовать:
|
|
||||||
|
|
||||||
1. Откройте панель **Knowledge Bases** (иконка базы данных в сайдбаре)
|
|
||||||
2. Создайте новую базу знаний
|
|
||||||
3. Загрузите документы (`.txt`, `.md`, `.pdf`, `.docx`)
|
|
||||||
4. В сайдбаре выберите базу знаний для текущего диалога
|
|
||||||
5. Задавайте вопросы — LLM будет использовать контекст из документов
|
|
||||||
|
|
||||||
### Технические детали:
|
|
||||||
|
|
||||||
- Документы разбиваются на **чанки** по ~500 символов с перекрытием
|
|
||||||
- Для каждого чанка генерируется **embedding** через API
|
|
||||||
- Чанки хранятся в **ChromaDB** (векторная база данных)
|
|
||||||
- При отправке сообщения запрос эмбеддится и ищутся похожие чанки через semantic search
|
|
||||||
- Найденный контекст добавляется к системному промпту
|
|
||||||
|
|
||||||
## API Endpoints
|
|
||||||
|
|
||||||
Backend проксирует запросы к OpenAI-compatible API:
|
|
||||||
|
|
||||||
| Endpoint | Method | Описание |
|
|
||||||
|----------|--------|----------|
|
|
||||||
| `/api/v1/chat/completions` | POST | Streaming чат |
|
|
||||||
| `/api/v1/models` | POST | Список моделей |
|
|
||||||
| `/api/files/upload` | POST | Загрузка файлов |
|
|
||||||
| `/api/files` | GET | Список файлов |
|
|
||||||
| `/api/files/:id` | DELETE | Удаление файла |
|
|
||||||
| `/api/knowledge` | GET | Список баз знаний |
|
|
||||||
| `/api/knowledge` | POST | Создать базу знаний |
|
|
||||||
| `/api/knowledge/:id` | DELETE | Удалить базу знаний |
|
|
||||||
| `/api/knowledge/:id/files` | POST | Добавить файл в базу |
|
|
||||||
| `/api/knowledge/:id/files/:fileId` | DELETE | Удалить файл из базы |
|
|
||||||
| `/api/knowledge/:id/query` | POST | Поиск по базе знаний |
|
|
||||||
|
|
||||||
## Технологии
|
|
||||||
|
|
||||||
- **Frontend**: React 18, TypeScript, Tailwind CSS, Zustand, Lucide icons
|
|
||||||
- **Backend**: Node.js, Express.js, ChromaDB, Multer
|
|
||||||
- **Build**: Vite
|
|
||||||
|
|||||||
+165
-98
@@ -7,6 +7,9 @@ const path = require('path');
|
|||||||
|
|
||||||
const app = express();
|
const app = express();
|
||||||
const PORT = process.env.PORT || 3002;
|
const PORT = process.env.PORT || 3002;
|
||||||
|
const CHROMA_HOST = process.env.CHROMA_HOST || 'chromadb';
|
||||||
|
const CHROMA_PORT = process.env.CHROMA_PORT || '8000';
|
||||||
|
const CHROMA_URL = `http://${CHROMA_HOST}:${CHROMA_PORT}`;
|
||||||
|
|
||||||
// Middleware
|
// Middleware
|
||||||
app.use(cors());
|
app.use(cors());
|
||||||
@@ -26,21 +29,22 @@ const storage = multer.diskStorage({
|
|||||||
|
|
||||||
const upload = multer({ storage });
|
const upload = multer({ storage });
|
||||||
|
|
||||||
// In-memory storage for knowledge bases metadata
|
// In-memory storage
|
||||||
const knowledgeBases = new Map();
|
const knowledgeBases = new Map();
|
||||||
const knowledgeFiles = new Map();
|
const knowledgeFiles = new Map();
|
||||||
const uploadedFiles = new Map();
|
const uploadedFiles = new Map();
|
||||||
|
|
||||||
// Token usage tracking
|
|
||||||
const tokenUsageStats = new Map(); // conversationId -> aggregated token stats
|
|
||||||
|
|
||||||
// In-memory storage for token usage events
|
|
||||||
const tokenUsageLog = [];
|
const tokenUsageLog = [];
|
||||||
|
|
||||||
// Extract token usage from SSE response
|
// ===== CHROMA CLIENT =====
|
||||||
|
|
||||||
|
function getChromaClient() {
|
||||||
|
return new ChromaClient({ path: CHROMA_URL });
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===== TOKEN USAGE =====
|
||||||
|
|
||||||
function extractTokenUsage(fullResponse) {
|
function extractTokenUsage(fullResponse) {
|
||||||
try {
|
try {
|
||||||
// Parse the last SSE data block which contains usage info
|
|
||||||
const lines = fullResponse.split('\n');
|
const lines = fullResponse.split('\n');
|
||||||
for (let i = lines.length - 1; i >= 0; i--) {
|
for (let i = lines.length - 1; i >= 0; i--) {
|
||||||
const line = lines[i];
|
const line = lines[i];
|
||||||
@@ -57,7 +61,7 @@ function extractTokenUsage(fullResponse) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
// Not valid JSON, skip
|
// skip
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -67,13 +71,6 @@ function extractTokenUsage(fullResponse) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Global ChromaDB client (recreated per request when path changes)
|
|
||||||
function getChromaClient(persistDir = './data/chromadb') {
|
|
||||||
return new ChromaClient({
|
|
||||||
path: `file://${path.resolve(persistDir)}`,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
// ===== OPENAI PROXY =====
|
// ===== OPENAI PROXY =====
|
||||||
|
|
||||||
app.post('/api/v1/chat/completions', async (req, res) => {
|
app.post('/api/v1/chat/completions', async (req, res) => {
|
||||||
@@ -107,7 +104,6 @@ app.post('/api/v1/chat/completions', async (req, res) => {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stream the response
|
|
||||||
res.setHeader('Content-Type', 'text/event-stream');
|
res.setHeader('Content-Type', 'text/event-stream');
|
||||||
res.setHeader('Cache-Control', 'no-cache');
|
res.setHeader('Cache-Control', 'no-cache');
|
||||||
res.setHeader('Connection', 'keep-alive');
|
res.setHeader('Connection', 'keep-alive');
|
||||||
@@ -125,7 +121,6 @@ app.post('/api/v1/chat/completions', async (req, res) => {
|
|||||||
}
|
}
|
||||||
res.end();
|
res.end();
|
||||||
|
|
||||||
// Parse token usage from the final SSE message
|
|
||||||
const tokenUsage = extractTokenUsage(fullResponse);
|
const tokenUsage = extractTokenUsage(fullResponse);
|
||||||
if (tokenUsage) {
|
if (tokenUsage) {
|
||||||
const usageEntry = {
|
const usageEntry = {
|
||||||
@@ -137,13 +132,11 @@ app.post('/api/v1/chat/completions', async (req, res) => {
|
|||||||
totalTokens: tokenUsage.totalTokens,
|
totalTokens: tokenUsage.totalTokens,
|
||||||
};
|
};
|
||||||
tokenUsageLog.push(usageEntry);
|
tokenUsageLog.push(usageEntry);
|
||||||
console.log(`Token usage: ${usageEntry.totalTokens} tokens (${usageEntry.promptTokens} prompt, ${usageEntry.completionTokens} completion) for model ${model}`);
|
console.log(`Token usage: ${usageEntry.totalTokens} tokens for model ${model}`);
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Proxy error:', error);
|
console.error('Proxy error:', error);
|
||||||
res.status(500).json({
|
res.status(500).json({ error: { message: error.message || 'Internal server error' } });
|
||||||
error: { message: error.message || 'Internal server error' },
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -156,9 +149,7 @@ app.post('/api/v1/models', async (req, res) => {
|
|||||||
|
|
||||||
const url = `${apiUrl.replace(/\/$/, '')}/models`;
|
const url = `${apiUrl.replace(/\/$/, '')}/models`;
|
||||||
const response = await fetch(url, {
|
const response = await fetch(url, {
|
||||||
headers: {
|
headers: { 'Authorization': `Bearer ${apiKey}` },
|
||||||
'Authorization': `Bearer ${apiKey}`,
|
|
||||||
},
|
|
||||||
});
|
});
|
||||||
|
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
@@ -224,7 +215,6 @@ function chunkText(text, chunkSize = 500, overlap = 50) {
|
|||||||
chunks.push(text.slice(start, end));
|
chunks.push(text.slice(start, end));
|
||||||
start += chunkSize - overlap;
|
start += chunkSize - overlap;
|
||||||
if (start >= text.length) break;
|
if (start >= text.length) break;
|
||||||
// Don't create tiny chunks at the end
|
|
||||||
if (text.length - start < chunkSize * 0.3) {
|
if (text.length - start < chunkSize * 0.3) {
|
||||||
chunks[chunks.length - 1] = text.slice(chunks.length > 0 ? start - (chunkSize - overlap) : 0);
|
chunks[chunks.length - 1] = text.slice(chunks.length > 0 ? start - (chunkSize - overlap) : 0);
|
||||||
break;
|
break;
|
||||||
@@ -253,11 +243,58 @@ async function getEmbeddings(texts, apiUrl, apiKey, model) {
|
|||||||
return data.data.map((d) => d.embedding);
|
return data.data.map((d) => d.embedding);
|
||||||
}
|
}
|
||||||
|
|
||||||
app.get('/api/knowledge', async (req, res) => {
|
async function processFileToKnowledge(filePath, fileName, baseId, apiUrl, apiKey, embeddingModel) {
|
||||||
const { chromaDir } = req.query;
|
let text = '';
|
||||||
try {
|
try {
|
||||||
const client = getChromaClient(chromaDir || './data/chromadb');
|
text = fs.readFileSync(filePath, 'utf-8');
|
||||||
// Get all collections to list knowledge bases
|
} catch (e) {
|
||||||
|
console.warn(`Cannot read ${fileName}:`, e.message);
|
||||||
|
return { success: false, error: e.message };
|
||||||
|
}
|
||||||
|
|
||||||
|
const chunks = chunkText(text, 500, 50);
|
||||||
|
if (chunks.length === 0) return { success: false, error: 'Empty file' };
|
||||||
|
|
||||||
|
const embeddings = await getEmbeddings(chunks, apiUrl, apiKey, embeddingModel || 'text-embedding-3-small');
|
||||||
|
|
||||||
|
const client = getChromaClient();
|
||||||
|
const collection = await client.getCollection({ name: baseId });
|
||||||
|
|
||||||
|
const ids = chunks.map((_, i) => `${baseId}_chunk_${Date.now()}_${i}_${Math.random().toString(36).substring(2, 6)}`);
|
||||||
|
const metadatas = chunks.map((chunk, i) => ({
|
||||||
|
source: fileName,
|
||||||
|
chunkIndex: i,
|
||||||
|
fileId: fileName,
|
||||||
|
}));
|
||||||
|
|
||||||
|
await collection.add({ ids, embeddings, documents: chunks, metadatas });
|
||||||
|
|
||||||
|
const fileInfo = {
|
||||||
|
id: fileName,
|
||||||
|
originalName: fileName,
|
||||||
|
size: fs.statSync(filePath).size,
|
||||||
|
mimeType: 'text/plain',
|
||||||
|
chunkCount: chunks.length,
|
||||||
|
uploadedAt: Date.now(),
|
||||||
|
};
|
||||||
|
|
||||||
|
const files = knowledgeFiles.get(baseId) || [];
|
||||||
|
files.push(fileInfo);
|
||||||
|
knowledgeFiles.set(baseId, files);
|
||||||
|
|
||||||
|
const kb = knowledgeBases.get(baseId);
|
||||||
|
if (kb) {
|
||||||
|
kb.documentCount = (kb.documentCount || 0) + chunks.length;
|
||||||
|
kb.files = files;
|
||||||
|
knowledgeBases.set(baseId, kb);
|
||||||
|
}
|
||||||
|
|
||||||
|
return { success: true, fileInfo, chunks: chunks.length };
|
||||||
|
}
|
||||||
|
|
||||||
|
app.get('/api/knowledge', async (req, res) => {
|
||||||
|
try {
|
||||||
|
const client = getChromaClient();
|
||||||
const collections = await client.listCollections();
|
const collections = await client.listCollections();
|
||||||
const bases = [];
|
const bases = [];
|
||||||
|
|
||||||
@@ -290,16 +327,22 @@ app.post('/api/knowledge', async (req, res) => {
|
|||||||
const id = name.trim().toLowerCase().replace(/[^a-z0-9]/g, '_') + '_' + Date.now();
|
const id = name.trim().toLowerCase().replace(/[^a-z0-9]/g, '_') + '_' + Date.now();
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const client = getChromaClient(chromaDir || './data/chromadb');
|
const client = getChromaClient();
|
||||||
await client.getOrCreateCollection({
|
await client.getOrCreateCollection({
|
||||||
name: id,
|
name: id,
|
||||||
metadata: { name: name.trim(), description: description || '', createdAt: Date.now() },
|
metadata: {
|
||||||
|
name: name.trim(),
|
||||||
|
description: description || '',
|
||||||
|
chromaDir: chromaDir || '/data/chromadb',
|
||||||
|
createdAt: Date.now(),
|
||||||
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
const kb = {
|
const kb = {
|
||||||
id,
|
id,
|
||||||
name: name.trim(),
|
name: name.trim(),
|
||||||
description: description || '',
|
description: description || '',
|
||||||
|
chromaDir: chromaDir || '/data/chromadb',
|
||||||
documentCount: 0,
|
documentCount: 0,
|
||||||
files: [],
|
files: [],
|
||||||
};
|
};
|
||||||
@@ -315,17 +358,15 @@ app.post('/api/knowledge', async (req, res) => {
|
|||||||
|
|
||||||
app.delete('/api/knowledge/:id', async (req, res) => {
|
app.delete('/api/knowledge/:id', async (req, res) => {
|
||||||
const { id } = req.params;
|
const { id } = req.params;
|
||||||
const { chromaDir } = req.query;
|
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const client = getChromaClient(chromaDir || './data/chromadb');
|
const client = getChromaClient();
|
||||||
await client.deleteCollection({ name: id });
|
await client.deleteCollection({ name: id });
|
||||||
knowledgeBases.delete(id);
|
knowledgeBases.delete(id);
|
||||||
knowledgeFiles.delete(id);
|
knowledgeFiles.delete(id);
|
||||||
res.json({ success: true });
|
res.json({ success: true });
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Delete knowledge base error:', error);
|
console.error('Delete knowledge base error:', error);
|
||||||
// Even if Chroma throws, clean up our state
|
|
||||||
knowledgeBases.delete(id);
|
knowledgeBases.delete(id);
|
||||||
knowledgeFiles.delete(id);
|
knowledgeFiles.delete(id);
|
||||||
res.json({ success: true });
|
res.json({ success: true });
|
||||||
@@ -343,67 +384,97 @@ app.post('/api/knowledge/:id/files', upload.single('file'), async (req, res) =>
|
|||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Read file content
|
const result = await processFileToKnowledge(
|
||||||
let text = '';
|
file.path,
|
||||||
if (file.mimetype === 'application/pdf') {
|
file.originalname,
|
||||||
text = `[PDF file: ${file.originalname}]`; // Simplified; in production use pdf-parse
|
id,
|
||||||
} else {
|
apiUrl,
|
||||||
text = fs.readFileSync(file.path, 'utf-8');
|
apiKey,
|
||||||
}
|
embeddingModel
|
||||||
|
);
|
||||||
// Chunk the text
|
if (!result.success) return res.status(400).json({ error: result.error });
|
||||||
const chunks = chunkText(text, 500, 50);
|
res.json({ file: result.fileInfo, chunks: result.chunks });
|
||||||
if (chunks.length === 0) {
|
|
||||||
return res.status(400).json({ error: 'File is empty or could not be parsed' });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get embeddings
|
|
||||||
const embeddings = await getEmbeddings(chunks, apiUrl, apiKey, embeddingModel || 'text-embedding-3-small');
|
|
||||||
|
|
||||||
// Store in ChromaDB
|
|
||||||
const client = getChromaClient();
|
|
||||||
const collection = await client.getCollection({ name: id });
|
|
||||||
|
|
||||||
const ids = chunks.map((_, i) => `${id}_chunk_${Date.now()}_${i}`);
|
|
||||||
const metadatas = chunks.map((chunk, i) => ({
|
|
||||||
source: file.originalname,
|
|
||||||
chunkIndex: i,
|
|
||||||
fileId: file.filename,
|
|
||||||
}));
|
|
||||||
|
|
||||||
await collection.add({
|
|
||||||
ids,
|
|
||||||
embeddings,
|
|
||||||
documents: chunks,
|
|
||||||
metadatas,
|
|
||||||
});
|
|
||||||
|
|
||||||
// Update metadata
|
|
||||||
const fileInfo = {
|
|
||||||
id: file.filename,
|
|
||||||
originalName: file.originalname,
|
|
||||||
size: file.size,
|
|
||||||
chunkCount: chunks.length,
|
|
||||||
};
|
|
||||||
|
|
||||||
const files = knowledgeFiles.get(id) || [];
|
|
||||||
files.push(fileInfo);
|
|
||||||
knowledgeFiles.set(id, files);
|
|
||||||
|
|
||||||
const kb = knowledgeBases.get(id);
|
|
||||||
if (kb) {
|
|
||||||
kb.documentCount = (kb.documentCount || 0) + chunks.length;
|
|
||||||
kb.files = files;
|
|
||||||
knowledgeBases.set(id, kb);
|
|
||||||
}
|
|
||||||
|
|
||||||
res.json({ file: fileInfo, chunks: chunks.length });
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Upload to knowledge base error:', error);
|
console.error('Upload to knowledge base error:', error);
|
||||||
res.status(500).json({ error: error.message });
|
res.status(500).json({ error: error.message });
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ===== SCAN LOCAL FOLDER =====
|
||||||
|
|
||||||
|
const SKIP_DIRS_SCAN = /node_modules|\.git|\.next|\.vscode|\.idea|dist|build|__pycache__|\.cache|\.venv|venv|env|target|\.turbo/i;
|
||||||
|
|
||||||
|
function shouldSkipPath(p) {
|
||||||
|
const parts = p.split('/');
|
||||||
|
for (const part of parts) {
|
||||||
|
if (part.startsWith('.') || SKIP_DIRS_SCAN.test(part)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function getFilesRecursively(dirPath, files = []) {
|
||||||
|
if (!fs.existsSync(dirPath) || !fs.statSync(dirPath).isDirectory()) return files;
|
||||||
|
const entries = fs.readdirSync(dirPath);
|
||||||
|
for (const entry of entries) {
|
||||||
|
const fullPath = path.join(dirPath, entry);
|
||||||
|
const relPath = fullPath.replace(dirPath, '');
|
||||||
|
if (shouldSkipPath(relPath)) continue;
|
||||||
|
const stat = fs.statSync(fullPath);
|
||||||
|
if (stat.isDirectory()) {
|
||||||
|
getFilesRecursively(fullPath, files);
|
||||||
|
} else {
|
||||||
|
files.push(fullPath);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return files;
|
||||||
|
}
|
||||||
|
|
||||||
|
app.post('/api/knowledge/:id/scan', async (req, res) => {
|
||||||
|
const { id } = req.params;
|
||||||
|
const { folderPath, apiUrl, apiKey, embeddingModel } = req.body;
|
||||||
|
|
||||||
|
if (!folderPath || !fs.existsSync(folderPath)) {
|
||||||
|
return res.status(400).json({ error: 'Invalid or non-existent folder path' });
|
||||||
|
}
|
||||||
|
if (!apiUrl || !apiKey) {
|
||||||
|
return res.status(400).json({ error: 'API URL and Key are required for embeddings' });
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const allPaths = getFilesRecursively(folderPath);
|
||||||
|
const results = [];
|
||||||
|
let processed = 0;
|
||||||
|
const maxFiles = 50;
|
||||||
|
|
||||||
|
for (const filePath of allPaths.slice(0, maxFiles)) {
|
||||||
|
try {
|
||||||
|
const fileName = path.relative(folderPath, filePath);
|
||||||
|
const result = await processFileToKnowledge(
|
||||||
|
filePath,
|
||||||
|
fileName,
|
||||||
|
id,
|
||||||
|
apiUrl,
|
||||||
|
apiKey,
|
||||||
|
embeddingModel
|
||||||
|
);
|
||||||
|
if (result.success) {
|
||||||
|
results.push({ file: fileName, chunks: result.chunks });
|
||||||
|
} else {
|
||||||
|
results.push({ file: fileName, error: result.error });
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
results.push({ file: path.relative(folderPath, filePath), error: e.message });
|
||||||
|
}
|
||||||
|
processed++;
|
||||||
|
}
|
||||||
|
|
||||||
|
res.json({ processed, total: allPaths.length, results });
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Scan folder error:', error);
|
||||||
|
res.status(500).json({ error: error.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
app.delete('/api/knowledge/:baseId/files/:fileId', async (req, res) => {
|
app.delete('/api/knowledge/:baseId/files/:fileId', async (req, res) => {
|
||||||
const { baseId, fileId } = req.params;
|
const { baseId, fileId } = req.params;
|
||||||
|
|
||||||
@@ -411,7 +482,6 @@ app.delete('/api/knowledge/:baseId/files/:fileId', async (req, res) => {
|
|||||||
const client = getChromaClient();
|
const client = getChromaClient();
|
||||||
const collection = await client.getCollection({ name: baseId });
|
const collection = await client.getCollection({ name: baseId });
|
||||||
|
|
||||||
// Find all chunks with this fileId
|
|
||||||
const results = await collection.get({
|
const results = await collection.get({
|
||||||
where: { fileId: { $eq: fileId } },
|
where: { fileId: { $eq: fileId } },
|
||||||
});
|
});
|
||||||
@@ -420,7 +490,6 @@ app.delete('/api/knowledge/:baseId/files/:fileId', async (req, res) => {
|
|||||||
await collection.delete({ ids: results.ids });
|
await collection.delete({ ids: results.ids });
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update metadata
|
|
||||||
const files = (knowledgeFiles.get(baseId) || []).filter((f) => f.id !== fileId);
|
const files = (knowledgeFiles.get(baseId) || []).filter((f) => f.id !== fileId);
|
||||||
knowledgeFiles.set(baseId, files);
|
knowledgeFiles.set(baseId, files);
|
||||||
|
|
||||||
@@ -478,14 +547,13 @@ app.post('/api/knowledge/:id/query', async (req, res) => {
|
|||||||
|
|
||||||
// Health check
|
// Health check
|
||||||
app.get('/api/health', (req, res) => {
|
app.get('/api/health', (req, res) => {
|
||||||
res.json({ status: 'ok', timestamp: new Date().toISOString() });
|
res.json({ status: 'ok', timestamp: new Date().toISOString(), chromaUrl: CHROMA_URL });
|
||||||
});
|
});
|
||||||
|
|
||||||
// Token usage tracking endpoints
|
// Token usage tracking
|
||||||
app.get('/api/token-usage', (req, res) => {
|
app.get('/api/token-usage', (req, res) => {
|
||||||
try {
|
try {
|
||||||
const { period = 'all', conversationId } = req.query;
|
const { period = 'all', conversationId } = req.query;
|
||||||
|
|
||||||
let filtered = [...tokenUsageLog];
|
let filtered = [...tokenUsageLog];
|
||||||
|
|
||||||
if (conversationId) {
|
if (conversationId) {
|
||||||
@@ -506,7 +574,6 @@ app.get('/api/token-usage', (req, res) => {
|
|||||||
const completionTokens = filtered.reduce((sum, e) => sum + e.completionTokens, 0);
|
const completionTokens = filtered.reduce((sum, e) => sum + e.completionTokens, 0);
|
||||||
const totalTokens = filtered.reduce((sum, e) => sum + e.totalTokens, 0);
|
const totalTokens = filtered.reduce((sum, e) => sum + e.totalTokens, 0);
|
||||||
|
|
||||||
// Per-model breakdown
|
|
||||||
const modelBreakdown = {};
|
const modelBreakdown = {};
|
||||||
filtered.forEach((entry) => {
|
filtered.forEach((entry) => {
|
||||||
if (!modelBreakdown[entry.model]) {
|
if (!modelBreakdown[entry.model]) {
|
||||||
@@ -521,7 +588,7 @@ app.get('/api/token-usage', (req, res) => {
|
|||||||
res.json({
|
res.json({
|
||||||
summary: { promptTokens, completionTokens, totalTokens, requestCount: filtered.length },
|
summary: { promptTokens, completionTokens, totalTokens, requestCount: filtered.length },
|
||||||
modelBreakdown,
|
modelBreakdown,
|
||||||
history: filtered.slice(-100), // last 100 entries
|
history: filtered.slice(-100),
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Token usage error:', error);
|
console.error('Token usage error:', error);
|
||||||
@@ -531,11 +598,11 @@ app.get('/api/token-usage', (req, res) => {
|
|||||||
|
|
||||||
app.delete('/api/token-usage', (req, res) => {
|
app.delete('/api/token-usage', (req, res) => {
|
||||||
tokenUsageLog.length = 0;
|
tokenUsageLog.length = 0;
|
||||||
tokenUsageStats.clear();
|
|
||||||
res.json({ success: true });
|
res.json({ success: true });
|
||||||
});
|
});
|
||||||
|
|
||||||
app.listen(PORT, () => {
|
app.listen(PORT, () => {
|
||||||
console.log(`AIUI Backend running on port ${PORT}`);
|
console.log(`AIUI Backend running on port ${PORT}`);
|
||||||
|
console.log(`ChromaDB URL: ${CHROMA_URL}`);
|
||||||
console.log(`Upload directory: ${uploadDir}`);
|
console.log(`Upload directory: ${uploadDir}`);
|
||||||
});
|
});
|
||||||
|
|||||||
+20
-20
@@ -1,5 +1,3 @@
|
|||||||
version: "3.8"
|
|
||||||
|
|
||||||
services:
|
services:
|
||||||
# Nginx reverse proxy
|
# Nginx reverse proxy
|
||||||
nginx:
|
nginx:
|
||||||
@@ -11,7 +9,6 @@ services:
|
|||||||
depends_on:
|
depends_on:
|
||||||
- frontend
|
- frontend
|
||||||
- backend
|
- backend
|
||||||
- strapi
|
|
||||||
networks:
|
networks:
|
||||||
- aiui-network
|
- aiui-network
|
||||||
|
|
||||||
@@ -26,7 +23,7 @@ services:
|
|||||||
- aiui-network
|
- aiui-network
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
# Backend (Express + ChromaDB)
|
# Backend (Express + SQLite)
|
||||||
backend:
|
backend:
|
||||||
build:
|
build:
|
||||||
context: ./backend
|
context: ./backend
|
||||||
@@ -38,28 +35,31 @@ services:
|
|||||||
environment:
|
environment:
|
||||||
- NODE_ENV=production
|
- NODE_ENV=production
|
||||||
- PORT=3002
|
- PORT=3002
|
||||||
- STRAPI_URL=http://strapi:1337
|
- OPENAI_API_URL=${OPENAI_API_URL:-https://api.openai.com/v1}
|
||||||
- STRAPI_API_TOKEN=${STRAPI_API_TOKEN:-}
|
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
|
||||||
|
- DEFAULT_MODEL=${DEFAULT_MODEL:-gpt-4o-mini}
|
||||||
|
- JWT_SECRET=${JWT_SECRET:-randomsecret}
|
||||||
|
- CHROMA_HOST=chromadb
|
||||||
|
- CHROMA_PORT=8000
|
||||||
networks:
|
networks:
|
||||||
- aiui-network
|
- aiui-network
|
||||||
|
depends_on:
|
||||||
|
- chromadb
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
# Strapi CMS (SQLite)
|
# ChromaDB - vector database for RAG
|
||||||
strapi:
|
chromadb:
|
||||||
image: strapi/strapi:5
|
image: chromadb/chroma:latest
|
||||||
expose:
|
expose:
|
||||||
- "1337"
|
- "8000"
|
||||||
volumes:
|
volumes:
|
||||||
- ./data/strapi:/srv/app
|
- ./data/chromadb:/data/chromadb
|
||||||
environment:
|
environment:
|
||||||
- DATABASE_CLIENT=sqlite
|
- CHROMA_SERVER_HOST=0.0.0.0
|
||||||
- DATABASE_FILENAME=.tmp/data.db
|
- CHROMA_SERVER_HTTP_PORT=8000
|
||||||
- JWT_SECRET=${JWT_SECRET:-changemejwt}
|
- CHROMA_SERVER_CORS_ALLOW_ORIGINS=["*"]
|
||||||
- ADMIN_JWT_SECRET=${ADMIN_JWT_SECRET:-changemeadmin}
|
- IS_PERSISTENT=TRUE
|
||||||
- APP_KEYS=${APP_KEYS:-changeme1,changeme2}
|
- PERSIST_DIRECTORY=/data/chromadb
|
||||||
- API_TOKEN_SALT=${API_TOKEN_SALT:-changemesalt}
|
|
||||||
- TRANSFER_TOKEN_SALT=${TRANSFER_TOKEN_SALT:-changemetoken}
|
|
||||||
- NODE_ENV=production
|
|
||||||
networks:
|
networks:
|
||||||
- aiui-network
|
- aiui-network
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
@@ -70,4 +70,4 @@ networks:
|
|||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
backend-data:
|
backend-data:
|
||||||
strapi-data:
|
chromadb-data:
|
||||||
|
|||||||
+1
-9
@@ -16,7 +16,7 @@ server {
|
|||||||
|
|
||||||
# API proxy
|
# API proxy
|
||||||
location /api/ {
|
location /api/ {
|
||||||
proxy_pass http://backend:3002/;
|
proxy_pass http://backend:3002/api/;
|
||||||
proxy_http_version 1.1;
|
proxy_http_version 1.1;
|
||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
@@ -29,14 +29,6 @@ server {
|
|||||||
proxy_read_timeout 86400s;
|
proxy_read_timeout 86400s;
|
||||||
}
|
}
|
||||||
|
|
||||||
# Strapi admin/proxy (optional)
|
|
||||||
location /strapi/ {
|
|
||||||
proxy_pass http://strapi:1337/;
|
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
}
|
|
||||||
|
|
||||||
# SPA fallback
|
# SPA fallback
|
||||||
location / {
|
location / {
|
||||||
try_files $uri $uri/ /index.html;
|
try_files $uri $uri/ /index.html;
|
||||||
|
|||||||
@@ -25,30 +25,44 @@ export default function ChatArea({ conversationId }: ChatAreaProps) {
|
|||||||
messagesRef.current = conversation?.messages || [];
|
messagesRef.current = conversation?.messages || [];
|
||||||
|
|
||||||
const buildSystemPrompt = useCallback(
|
const buildSystemPrompt = useCallback(
|
||||||
async (basePrompt: string, kbId?: string | null): Promise<string> => {
|
async (basePrompt: string, kbIds?: string[]): Promise<string> => {
|
||||||
if (!kbId || !settings.apiUrl || !settings.apiKey) return basePrompt;
|
if (!kbIds || kbIds.length === 0 || !settings.apiUrl || !settings.apiKey) return basePrompt;
|
||||||
try {
|
try {
|
||||||
const query = messagesRef.current
|
const query = messagesRef.current
|
||||||
.filter((m) => m.role === 'user')
|
.filter((m) => m.role === 'user')
|
||||||
.slice(-1)[0]?.content || '';
|
.slice(-1)[0]?.content || '';
|
||||||
if (!query) return basePrompt;
|
if (!query) return basePrompt;
|
||||||
|
|
||||||
const { results } = await queryKnowledge(
|
const allResults: Array<{ text: string; score: number }> = [];
|
||||||
kbId,
|
|
||||||
query,
|
|
||||||
settings.apiUrl,
|
|
||||||
settings.apiKey,
|
|
||||||
settings.embeddingModel,
|
|
||||||
5
|
|
||||||
);
|
|
||||||
|
|
||||||
if (results.length === 0) return basePrompt;
|
for (const kbId of kbIds) {
|
||||||
|
try {
|
||||||
|
const { results } = await queryKnowledge(
|
||||||
|
kbId,
|
||||||
|
query,
|
||||||
|
settings.apiUrl,
|
||||||
|
settings.apiKey,
|
||||||
|
settings.embeddingModel,
|
||||||
|
5
|
||||||
|
);
|
||||||
|
allResults.push(...results);
|
||||||
|
} catch (e) {
|
||||||
|
console.warn(`Knowledge query failed for ${kbId}:`, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const context = results
|
// Sort by relevance (score descending) and deduplicate
|
||||||
|
const sorted = allResults
|
||||||
|
.sort((a, b) => b.score - a.score)
|
||||||
|
.slice(0, 15);
|
||||||
|
|
||||||
|
if (sorted.length === 0) return basePrompt;
|
||||||
|
|
||||||
|
const context = sorted
|
||||||
.map((r, i) => `[${i + 1}] ${r.text}`)
|
.map((r, i) => `[${i + 1}] ${r.text}`)
|
||||||
.join('\n\n');
|
.join('\n\n');
|
||||||
|
|
||||||
return `${basePrompt}\n\nRelevant context from knowledge base:\n${context}\n\nAnswer using the context above when relevant.`;
|
return `${basePrompt}\n\nRelevant context from knowledge bases:\n${context}\n\nAnswer using the context above when relevant.`;
|
||||||
} catch {
|
} catch {
|
||||||
return basePrompt;
|
return basePrompt;
|
||||||
}
|
}
|
||||||
@@ -94,7 +108,7 @@ export default function ChatArea({ conversationId }: ChatAreaProps) {
|
|||||||
try {
|
try {
|
||||||
const systemPrompt = await buildSystemPrompt(
|
const systemPrompt = await buildSystemPrompt(
|
||||||
settings.systemPrompt,
|
settings.systemPrompt,
|
||||||
conversation?.knowledgeBaseId
|
conversation?.knowledgeBaseIds
|
||||||
);
|
);
|
||||||
|
|
||||||
const stream = streamChatCompletion(
|
const stream = streamChatCompletion(
|
||||||
@@ -140,7 +154,7 @@ export default function ChatArea({ conversationId }: ChatAreaProps) {
|
|||||||
abortControllerRef.current = null;
|
abortControllerRef.current = null;
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
[conversationId, settings, addMessage, updateMessage, setLoading, buildSystemPrompt, conversation?.knowledgeBaseId]
|
[conversationId, settings, addMessage, updateMessage, setLoading, buildSystemPrompt, conversation?.knowledgeBaseIds]
|
||||||
);
|
);
|
||||||
|
|
||||||
const handleStop = useCallback(() => {
|
const handleStop = useCallback(() => {
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { useState, useCallback, useRef } from 'react';
|
import { useState, useCallback } from 'react';
|
||||||
import { Upload, X, FileText, FileImage, File } from 'lucide-react';
|
import { Upload, X, FileText, FileImage, File, FolderOpen } from 'lucide-react';
|
||||||
import type { UploadedFile } from '@/types';
|
import type { UploadedFile } from '@/types';
|
||||||
|
|
||||||
interface FileUploadProps {
|
interface FileUploadProps {
|
||||||
@@ -8,10 +8,23 @@ interface FileUploadProps {
|
|||||||
disabled?: boolean;
|
disabled?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const SKIP_DIRS = /node_modules|\.git|\.next|\.vscode|\.idea|dist|build|__pycache__|\.cache|\.venv|venv|env|target|\.turbo/i;
|
||||||
|
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
|
||||||
|
const MAX_FILES = 50;
|
||||||
|
|
||||||
|
function shouldSkip(file: File): boolean {
|
||||||
|
// Skip hidden directories by checking webkitRelativePath
|
||||||
|
const parts = file.webkitRelativePath?.split('/') || [];
|
||||||
|
for (const part of parts) {
|
||||||
|
if (part.startsWith('.') || SKIP_DIRS.test(part)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
export default function FileUpload({ files, onFilesChange, disabled }: FileUploadProps) {
|
export default function FileUpload({ files, onFilesChange, disabled }: FileUploadProps) {
|
||||||
const [isDragging, setIsDragging] = useState(false);
|
const [isDragging, setIsDragging] = useState(false);
|
||||||
const [isUploading, setIsUploading] = useState(false);
|
const [isUploading, setIsUploading] = useState(false);
|
||||||
const inputRef = useRef<HTMLInputElement>(null);
|
const [progress, setProgress] = useState<{ current: number; total: number } | null>(null);
|
||||||
|
|
||||||
const handleDragOver = useCallback((e: React.DragEvent) => {
|
const handleDragOver = useCallback((e: React.DragEvent) => {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
@@ -33,32 +46,78 @@ export default function FileUpload({ files, onFilesChange, disabled }: FileUploa
|
|||||||
await uploadFiles(droppedFiles);
|
await uploadFiles(droppedFiles);
|
||||||
}, [disabled]);
|
}, [disabled]);
|
||||||
|
|
||||||
const handleFileSelect = useCallback(async (e: React.ChangeEvent<HTMLInputElement>) => {
|
const openFilePicker = () => {
|
||||||
if (!e.target.files?.length) return;
|
const input = document.createElement('input');
|
||||||
const selectedFiles = Array.from(e.target.files);
|
input.type = 'file';
|
||||||
await uploadFiles(selectedFiles);
|
input.multiple = true;
|
||||||
e.target.value = '';
|
input.onchange = (e) => {
|
||||||
}, []);
|
const target = e.target as HTMLInputElement;
|
||||||
|
if (target.files?.length) {
|
||||||
|
uploadFiles(Array.from(target.files));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
input.click();
|
||||||
|
};
|
||||||
|
|
||||||
|
const openFolderPicker = () => {
|
||||||
|
const input = document.createElement('input');
|
||||||
|
input.type = 'file';
|
||||||
|
// @ts-ignore
|
||||||
|
input.webkitdirectory = true;
|
||||||
|
input.multiple = true;
|
||||||
|
input.onchange = (e) => {
|
||||||
|
const target = e.target as HTMLInputElement;
|
||||||
|
if (target.files?.length) {
|
||||||
|
const allFiles = Array.from(target.files);
|
||||||
|
const filtered = allFiles.filter((f) => !shouldSkip(f));
|
||||||
|
if (filtered.length > MAX_FILES) {
|
||||||
|
alert(`Too many files. Maximum ${MAX_FILES} allowed. Got ${filtered.length}`);
|
||||||
|
uploadFiles(filtered.slice(0, MAX_FILES));
|
||||||
|
} else {
|
||||||
|
uploadFiles(filtered);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
input.click();
|
||||||
|
};
|
||||||
|
|
||||||
const uploadFiles = async (fileList: File[]) => {
|
const uploadFiles = async (fileList: File[]) => {
|
||||||
|
if (fileList.length === 0) return;
|
||||||
setIsUploading(true);
|
setIsUploading(true);
|
||||||
|
setProgress({ current: 0, total: fileList.length });
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const formData = new FormData();
|
const result: UploadedFile[] = [];
|
||||||
fileList.forEach((f) => formData.append('files', f));
|
|
||||||
|
|
||||||
const response = await fetch('/api/files/upload', {
|
for (let i = 0; i < fileList.length; i++) {
|
||||||
method: 'POST',
|
const file = fileList[i];
|
||||||
body: formData,
|
setProgress({ current: i + 1, total: fileList.length });
|
||||||
});
|
|
||||||
|
|
||||||
if (!response.ok) throw new Error('Upload failed');
|
if (file.size > MAX_FILE_SIZE) {
|
||||||
const data = await response.json();
|
console.warn(`Skipping ${file.name}: ${(file.size / 1024 / 1024).toFixed(1)}MB > ${MAX_FILE_SIZE / 1024 / 1024}MB limit`);
|
||||||
onFilesChange([...files, ...data.files]);
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const formData = new FormData();
|
||||||
|
formData.append('files', file);
|
||||||
|
|
||||||
|
const response = await fetch('/api/files/upload', {
|
||||||
|
method: 'POST',
|
||||||
|
body: formData,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) throw new Error('Upload failed');
|
||||||
|
const data = await response.json();
|
||||||
|
result.push(...data.files);
|
||||||
|
}
|
||||||
|
|
||||||
|
onFilesChange([...files, ...result]);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error('Upload error:', err);
|
console.error('Upload error:', err);
|
||||||
alert('Failed to upload files');
|
alert('Failed to upload files');
|
||||||
} finally {
|
} finally {
|
||||||
setIsUploading(false);
|
setIsUploading(false);
|
||||||
|
setProgress(null);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -108,39 +167,50 @@ export default function FileUpload({ files, onFilesChange, disabled }: FileUploa
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{/* Buttons row */}
|
||||||
|
<div className="flex gap-2">
|
||||||
|
<button
|
||||||
|
onClick={openFilePicker}
|
||||||
|
disabled={disabled || isUploading}
|
||||||
|
className="flex-1 flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:text-[var(--color-text)] hover:bg-[var(--color-surface-hover)] transition-colors disabled:opacity-50"
|
||||||
|
>
|
||||||
|
<Upload size={14} />
|
||||||
|
<span>Select files</span>
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={openFolderPicker}
|
||||||
|
disabled={disabled || isUploading}
|
||||||
|
className="flex-1 flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:text-[var(--color-text)] hover:bg-[var(--color-surface-hover)] transition-colors disabled:opacity-50"
|
||||||
|
>
|
||||||
|
<FolderOpen size={14} />
|
||||||
|
<span>Select folder</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
{/* Drop zone */}
|
{/* Drop zone */}
|
||||||
<button
|
<div
|
||||||
onClick={() => inputRef.current?.click()}
|
|
||||||
onDragOver={handleDragOver}
|
onDragOver={handleDragOver}
|
||||||
onDragLeave={handleDragLeave}
|
onDragLeave={handleDragLeave}
|
||||||
onDrop={handleDrop}
|
onDrop={handleDrop}
|
||||||
disabled={disabled || isUploading}
|
|
||||||
className={`w-full flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed text-xs transition-all
|
className={`w-full flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed text-xs transition-all
|
||||||
${isDragging
|
${isDragging
|
||||||
? 'border-[var(--color-accent)] bg-[var(--color-accent)]/10 text-[var(--color-accent)]'
|
? 'border-[var(--color-accent)] bg-[var(--color-accent)]/10 text-[var(--color-accent)]'
|
||||||
: 'border-[var(--color-border)] text-[var(--color-text-muted)] hover:border-[var(--color-text-secondary)] hover:text-[var(--color-text-secondary)]'
|
: 'border-[var(--color-border)] text-[var(--color-text-muted)]'
|
||||||
}
|
}
|
||||||
${disabled ? 'opacity-50 cursor-not-allowed' : 'cursor-pointer'}
|
${disabled ? 'opacity-50' : ''}
|
||||||
`}
|
`}
|
||||||
>
|
>
|
||||||
{isUploading ? (
|
{isUploading && progress ? (
|
||||||
|
<span className="animate-pulse">Uploading {progress.current}/{progress.total}...</span>
|
||||||
|
) : isUploading ? (
|
||||||
<span className="animate-pulse">Uploading...</span>
|
<span className="animate-pulse">Uploading...</span>
|
||||||
) : (
|
) : (
|
||||||
<>
|
<>
|
||||||
<Upload size={14} />
|
<Upload size={14} />
|
||||||
<span>Drop files or click to upload</span>
|
<span>Or drop files here</span>
|
||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
</button>
|
</div>
|
||||||
|
|
||||||
<input
|
|
||||||
ref={inputRef}
|
|
||||||
type="file"
|
|
||||||
multiple
|
|
||||||
onChange={handleFileSelect}
|
|
||||||
className="hidden"
|
|
||||||
disabled={disabled || isUploading}
|
|
||||||
/>
|
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { useState, useEffect, useRef } from 'react';
|
import { useState, useEffect, useCallback } from 'react';
|
||||||
import { X, Plus, Trash2, Upload, BookOpen, FileText, Database, ChevronDown, ChevronUp, FolderOpen, AlertCircle } from 'lucide-react';
|
import { X, Plus, Trash2, Upload, BookOpen, FileText, Database, ChevronDown, ChevronUp, FolderOpen, AlertCircle, RefreshCw, FolderSearch } from 'lucide-react';
|
||||||
import { useChatStore } from '@/stores/chatStore';
|
import { useChatStore } from '@/stores/chatStore';
|
||||||
|
|
||||||
interface FileResult {
|
interface FileResult {
|
||||||
@@ -9,37 +9,54 @@ interface FileResult {
|
|||||||
chunkCount?: number;
|
chunkCount?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const SKIP_DIRS = /node_modules|\.git|\.next|\.vscode|\.idea|dist|build|__pycache__|\.cache|\.venv|venv|env|target|\.turbo/i;
|
||||||
|
const MAX_FILES = 50;
|
||||||
|
|
||||||
|
function shouldSkip(file: File): boolean {
|
||||||
|
const parts = file.webkitRelativePath?.split('/') || [];
|
||||||
|
for (const part of parts) {
|
||||||
|
if (part.startsWith('.') || SKIP_DIRS.test(part)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
export default function KnowledgePanel() {
|
export default function KnowledgePanel() {
|
||||||
const { settings, toggleKnowledge, isKnowledgeOpen, setKnowledgeBases, knowledgeBases, removeKnowledgeBase } = useChatStore();
|
const { settings, toggleKnowledge, isKnowledgeOpen, setKnowledgeBases, knowledgeBases, removeKnowledgeBase } = useChatStore();
|
||||||
const [newName, setNewName] = useState('');
|
const [newName, setNewName] = useState('');
|
||||||
const [newDesc, setNewDesc] = useState('');
|
const [newDesc, setNewDesc] = useState('');
|
||||||
|
const [newChromaDir, setNewChromaDir] = useState('/data/chromadb');
|
||||||
const [isCreating, setIsCreating] = useState(false);
|
const [isCreating, setIsCreating] = useState(false);
|
||||||
const [expandedBase, setExpandedBase] = useState<string | null>(null);
|
const [expandedBase, setExpandedBase] = useState<string | null>(null);
|
||||||
const [uploadingBase, setUploadingBase] = useState<string | null>(null);
|
const [uploadingBase, setUploadingBase] = useState<string | null>(null);
|
||||||
const [loading, setLoading] = useState(false);
|
const [loading, setLoading] = useState(false);
|
||||||
const [folderLoading, setFolderLoading] = useState(false);
|
const [folderLoading, setFolderLoading] = useState(false);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
const [isBackendReady, setIsBackendReady] = useState(true);
|
||||||
const folderInputRef = useRef<HTMLInputElement>(null);
|
|
||||||
|
|
||||||
useEffect(() => {
|
const loadBases = useCallback(async () => {
|
||||||
if (isKnowledgeOpen) loadBases();
|
|
||||||
}, [isKnowledgeOpen]);
|
|
||||||
|
|
||||||
const loadBases = async () => {
|
|
||||||
try {
|
try {
|
||||||
setError(null);
|
setError(null);
|
||||||
const res = await fetch('/api/knowledge');
|
const res = await fetch('/api/knowledge');
|
||||||
if (!res.ok) throw new Error('Failed to load');
|
if (!res.ok) {
|
||||||
|
if (res.status === 500) {
|
||||||
|
setIsBackendReady(false);
|
||||||
|
throw new Error('Knowledge service not available. Make sure ChromaDB is running.');
|
||||||
|
}
|
||||||
|
throw new Error('Failed to load');
|
||||||
|
}
|
||||||
|
setIsBackendReady(true);
|
||||||
const data = await res.json();
|
const data = await res.json();
|
||||||
setKnowledgeBases(data.bases || []);
|
setKnowledgeBases(data.bases || []);
|
||||||
} catch (err) {
|
} catch (err: any) {
|
||||||
console.error('Failed to load knowledge bases:', err);
|
console.error('Failed to load knowledge bases:', err);
|
||||||
setError('Could not connect to knowledge base service. Is the backend running?');
|
setError(err.message || 'Could not connect to knowledge base service.');
|
||||||
setKnowledgeBases([]);
|
setKnowledgeBases([]);
|
||||||
}
|
}
|
||||||
};
|
}, [setKnowledgeBases]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (isKnowledgeOpen) loadBases();
|
||||||
|
}, [isKnowledgeOpen, loadBases]);
|
||||||
|
|
||||||
const handleCreate = async () => {
|
const handleCreate = async () => {
|
||||||
if (!newName.trim()) return;
|
if (!newName.trim()) return;
|
||||||
@@ -48,13 +65,14 @@ export default function KnowledgePanel() {
|
|||||||
const res = await fetch('/api/knowledge', {
|
const res = await fetch('/api/knowledge', {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: { 'Content-Type': 'application/json' },
|
headers: { 'Content-Type': 'application/json' },
|
||||||
body: JSON.stringify({ name: newName.trim(), description: newDesc.trim() }),
|
body: JSON.stringify({ name: newName.trim(), description: newDesc.trim(), chromaDir: newChromaDir.trim() || '/data/chromadb' }),
|
||||||
});
|
});
|
||||||
const base = await res.json();
|
const base = await res.json();
|
||||||
if (base.error) throw new Error(base.error);
|
if (base.error) throw new Error(base.error);
|
||||||
setKnowledgeBases([...knowledgeBases, base]);
|
setKnowledgeBases([...knowledgeBases, base]);
|
||||||
setNewName('');
|
setNewName('');
|
||||||
setNewDesc('');
|
setNewDesc('');
|
||||||
|
setNewChromaDir('/data/chromadb');
|
||||||
setIsCreating(false);
|
setIsCreating(false);
|
||||||
} catch (err: any) {
|
} catch (err: any) {
|
||||||
setError(err.message || 'Failed to create knowledge base');
|
setError(err.message || 'Failed to create knowledge base');
|
||||||
@@ -63,6 +81,56 @@ export default function KnowledgePanel() {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const openCreateFolderPicker = () => {
|
||||||
|
const input = document.createElement('input');
|
||||||
|
input.type = 'file';
|
||||||
|
// @ts-ignore
|
||||||
|
input.webkitdirectory = true;
|
||||||
|
input.multiple = true;
|
||||||
|
input.onchange = (e) => {
|
||||||
|
const target = e.target as HTMLInputElement;
|
||||||
|
if (target.files?.length) {
|
||||||
|
const allFiles = Array.from(target.files).filter(f => !shouldSkip(f));
|
||||||
|
handleCreateFromFiles(allFiles);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
input.click();
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleCreateFromFiles = async (files: File[]) => {
|
||||||
|
if (!files.length) return;
|
||||||
|
setFolderLoading(true);
|
||||||
|
setError(null);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const folderName = files[0].webkitRelativePath?.split('/')[0] || 'New Knowledge Base';
|
||||||
|
const res = await fetch('/api/knowledge', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ name: folderName, description: `From folder with ${files.length} files`, chromaDir: '/data/chromadb' }),
|
||||||
|
});
|
||||||
|
const base = await res.json();
|
||||||
|
if (base.error) throw new Error(base.error);
|
||||||
|
|
||||||
|
for (const file of files.slice(0, MAX_FILES)) {
|
||||||
|
const form = new FormData();
|
||||||
|
form.append('file', file);
|
||||||
|
form.append('apiUrl', settings.apiUrl);
|
||||||
|
form.append('apiKey', settings.apiKey);
|
||||||
|
form.append('embeddingModel', settings.embeddingModel);
|
||||||
|
try {
|
||||||
|
await fetch(`/api/knowledge/${base.id}/files`, { method: 'POST', body: form });
|
||||||
|
} catch { /* ignore single failures */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
await loadBases();
|
||||||
|
} catch (err: any) {
|
||||||
|
setError(err.message || 'Folder upload failed');
|
||||||
|
} finally {
|
||||||
|
setFolderLoading(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
const handleDelete = async (id: string) => {
|
const handleDelete = async (id: string) => {
|
||||||
if (!confirm('Delete this knowledge base?')) return;
|
if (!confirm('Delete this knowledge base?')) return;
|
||||||
try {
|
try {
|
||||||
@@ -73,93 +141,72 @@ export default function KnowledgePanel() {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleUploadFile = async (baseId: string, e: React.ChangeEvent<HTMLInputElement>) => {
|
const openFilePicker = (baseId: string) => {
|
||||||
if (!e.target.files?.[0]) return;
|
const input = document.createElement('input');
|
||||||
const file = e.target.files[0];
|
input.type = 'file';
|
||||||
|
input.multiple = true;
|
||||||
|
input.onchange = (e) => {
|
||||||
|
const target = e.target as HTMLInputElement;
|
||||||
|
if (target.files?.length) {
|
||||||
|
handleUploadFiles(baseId, Array.from(target.files));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
input.click();
|
||||||
|
};
|
||||||
|
|
||||||
|
const openFolderPicker = (baseId: string) => {
|
||||||
|
const input = document.createElement('input');
|
||||||
|
input.type = 'file';
|
||||||
|
// @ts-ignore
|
||||||
|
input.webkitdirectory = true;
|
||||||
|
input.multiple = true;
|
||||||
|
input.onchange = (e) => {
|
||||||
|
const target = e.target as HTMLInputElement;
|
||||||
|
if (target.files?.length) {
|
||||||
|
const allFiles = Array.from(target.files).filter(f => !shouldSkip(f));
|
||||||
|
handleUploadFiles(baseId, allFiles);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
input.click();
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleUploadFiles = async (baseId: string, files: File[]) => {
|
||||||
|
if (!files.length) return;
|
||||||
setUploadingBase(baseId);
|
setUploadingBase(baseId);
|
||||||
setError(null);
|
setError(null);
|
||||||
|
|
||||||
try {
|
for (const file of files.slice(0, MAX_FILES)) {
|
||||||
const formData = new FormData();
|
const form = new FormData();
|
||||||
formData.append('file', file);
|
form.append('file', file);
|
||||||
formData.append('apiUrl', settings.apiUrl);
|
form.append('apiUrl', settings.apiUrl);
|
||||||
formData.append('apiKey', settings.apiKey);
|
form.append('apiKey', settings.apiKey);
|
||||||
formData.append('embeddingModel', settings.embeddingModel);
|
form.append('embeddingModel', settings.embeddingModel);
|
||||||
|
try {
|
||||||
const res = await fetch(`/api/knowledge/${baseId}/files`, {
|
await fetch(`/api/knowledge/${baseId}/files`, { method: 'POST', body: form });
|
||||||
method: 'POST',
|
} catch { /* ignore */ }
|
||||||
body: formData,
|
|
||||||
});
|
|
||||||
|
|
||||||
if (!res.ok) throw new Error('Upload failed');
|
|
||||||
const data = await res.json();
|
|
||||||
|
|
||||||
const updated = knowledgeBases.map((b) =>
|
|
||||||
b.id === baseId
|
|
||||||
? { ...b, files: [...b.files, data.file], documentCount: (b.documentCount || 0) + data.chunks }
|
|
||||||
: b
|
|
||||||
);
|
|
||||||
setKnowledgeBases(updated);
|
|
||||||
} catch (err) {
|
|
||||||
setError('Failed to upload file. Make sure API settings are configured.');
|
|
||||||
} finally {
|
|
||||||
setUploadingBase(null);
|
|
||||||
e.target.value = '';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
setUploadingBase(null);
|
||||||
|
await loadBases();
|
||||||
};
|
};
|
||||||
|
|
||||||
const handleUploadFolder = async (baseId: string, e: React.ChangeEvent<HTMLInputElement>) => {
|
const handleScanLocalFolder = async (baseId: string) => {
|
||||||
const files = e.target.files;
|
const folderPath = prompt('Enter absolute folder path on server to scan:');
|
||||||
if (!files || files.length === 0) return;
|
if (!folderPath) return;
|
||||||
|
setUploadingBase(baseId);
|
||||||
setFolderLoading(true);
|
|
||||||
setError(null);
|
setError(null);
|
||||||
|
|
||||||
// For browser folder upload, we can't get the actual folder path.
|
|
||||||
// Instead, upload all files from the file picker using webkitdirectory.
|
|
||||||
const formData = new FormData();
|
|
||||||
Array.from(files).forEach((f) => formData.append('files', f));
|
|
||||||
formData.append('apiUrl', settings.apiUrl);
|
|
||||||
formData.append('apiKey', settings.apiKey);
|
|
||||||
formData.append('embeddingModel', settings.embeddingModel);
|
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Use a batch upload approach — upload files one by one
|
const res = await fetch(`/api/knowledge/${baseId}/scan`, {
|
||||||
const uploaded: FileResult[] = [];
|
method: 'POST',
|
||||||
const errors: string[] = [];
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ folderPath, apiUrl: settings.apiUrl, apiKey: settings.apiKey, embeddingModel: settings.embeddingModel }),
|
||||||
for (const file of Array.from(files)) {
|
});
|
||||||
const singleForm = new FormData();
|
if (!res.ok) throw new Error('Scan failed');
|
||||||
singleForm.append('file', file);
|
await loadBases();
|
||||||
singleForm.append('apiUrl', settings.apiUrl);
|
} catch (err: any) {
|
||||||
singleForm.append('apiKey', settings.apiKey);
|
setError(err.message || 'Failed to scan folder');
|
||||||
singleForm.append('embeddingModel', settings.embeddingModel);
|
|
||||||
|
|
||||||
try {
|
|
||||||
const res = await fetch(`/api/knowledge/${baseId}/files`, {
|
|
||||||
method: 'POST',
|
|
||||||
body: singleForm,
|
|
||||||
});
|
|
||||||
if (!res.ok) throw new Error('Failed');
|
|
||||||
const data = await res.json();
|
|
||||||
uploaded.push(data.file);
|
|
||||||
} catch {
|
|
||||||
errors.push(file.name);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (uploaded.length > 0) {
|
|
||||||
await loadBases();
|
|
||||||
}
|
|
||||||
|
|
||||||
if (errors.length > 0) {
|
|
||||||
setError(`${errors.length} files failed to upload out of ${files.length}`);
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
setError('Folder upload failed');
|
|
||||||
} finally {
|
} finally {
|
||||||
setFolderLoading(false);
|
setUploadingBase(null);
|
||||||
e.target.value = '';
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -170,8 +217,8 @@ export default function KnowledgePanel() {
|
|||||||
b.id === baseId
|
b.id === baseId
|
||||||
? {
|
? {
|
||||||
...b,
|
...b,
|
||||||
files: b.files.filter((f) => f.id !== fileId),
|
files: b.files.filter((f: any) => f.id !== fileId),
|
||||||
documentCount: b.files.filter((f) => f.id !== fileId).reduce((s, f) => s + (f.chunkCount || 0), 0),
|
documentCount: b.files.filter((f: any) => f.id !== fileId).reduce((s: number, f: any) => s + (f.chunkCount || 0), 0),
|
||||||
}
|
}
|
||||||
: b
|
: b
|
||||||
);
|
);
|
||||||
@@ -193,17 +240,25 @@ export default function KnowledgePanel() {
|
|||||||
<Database size={20} className="text-[var(--color-accent)]" />
|
<Database size={20} className="text-[var(--color-accent)]" />
|
||||||
<h2 className="text-lg font-semibold text-[var(--color-text)]">Knowledge Bases</h2>
|
<h2 className="text-lg font-semibold text-[var(--color-text)]">Knowledge Bases</h2>
|
||||||
</div>
|
</div>
|
||||||
<button
|
<div className="flex items-center gap-1">
|
||||||
onClick={toggleKnowledge}
|
<button
|
||||||
className="p-2 rounded-lg hover:bg-[var(--color-surface-hover)] text-[var(--color-text-secondary)] hover:text-[var(--color-text)] transition-colors"
|
onClick={loadBases}
|
||||||
>
|
className="p-2 rounded-lg hover:bg-[var(--color-surface-hover)] text-[var(--color-text-secondary)] hover:text-[var(--color-text)] transition-colors"
|
||||||
<X size={20} />
|
title="Refresh"
|
||||||
</button>
|
>
|
||||||
|
<RefreshCw size={16} />
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
onClick={toggleKnowledge}
|
||||||
|
className="p-2 rounded-lg hover:bg-[var(--color-surface-hover)] text-[var(--color-text-secondary)] hover:text-[var(--color-text)] transition-colors"
|
||||||
|
>
|
||||||
|
<X size={20} />
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Content */}
|
{/* Content */}
|
||||||
<div className="flex-1 overflow-y-auto p-5 space-y-4">
|
<div className="flex-1 overflow-y-auto p-5 space-y-4">
|
||||||
{/* Error */}
|
|
||||||
{error && (
|
{error && (
|
||||||
<div className="flex items-center gap-2 px-4 py-3 rounded-lg bg-red-500/10 border border-red-500/20 text-red-400 text-sm">
|
<div className="flex items-center gap-2 px-4 py-3 rounded-lg bg-red-500/10 border border-red-500/20 text-red-400 text-sm">
|
||||||
<AlertCircle size={16} />
|
<AlertCircle size={16} />
|
||||||
@@ -214,16 +269,33 @@ export default function KnowledgePanel() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Create new */}
|
{!isCreating && isBackendReady && (
|
||||||
{!isCreating ? (
|
<div className="flex gap-2">
|
||||||
<button
|
<button
|
||||||
onClick={() => setIsCreating(true)}
|
onClick={() => setIsCreating(true)}
|
||||||
className="w-full flex items-center justify-center gap-2 px-4 py-3 rounded-xl border-2 border-dashed border-[var(--color-border)] text-[var(--color-text-muted)] hover:border-[var(--color-accent)] hover:text-[var(--color-accent)] transition-colors"
|
className="flex-1 flex items-center justify-center gap-2 px-4 py-3 rounded-xl border-2 border-dashed border-[var(--color-border)] text-[var(--color-text-muted)] hover:border-[var(--color-accent)] hover:text-[var(--color-accent)] transition-colors"
|
||||||
>
|
>
|
||||||
<Plus size={18} />
|
<Plus size={18} />
|
||||||
<span className="text-sm font-medium">Create Knowledge Base</span>
|
<span className="text-sm font-medium">Create empty base</span>
|
||||||
</button>
|
</button>
|
||||||
) : (
|
<button
|
||||||
|
onClick={openCreateFolderPicker}
|
||||||
|
disabled={folderLoading}
|
||||||
|
className="flex-1 flex items-center justify-center gap-2 px-4 py-3 rounded-xl border-2 border-dashed border-[var(--color-border)] text-[var(--color-text-muted)] hover:border-green-500 hover:text-green-500 transition-colors disabled:opacity-50"
|
||||||
|
>
|
||||||
|
{folderLoading ? (
|
||||||
|
<span className="text-sm font-medium animate-pulse">Processing...</span>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<FolderOpen size={18} />
|
||||||
|
<span className="text-sm font-medium">Create from folder</span>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{isCreating && (
|
||||||
<div className="space-y-3 p-4 rounded-xl bg-[var(--color-bg)] border border-[var(--color-border)]">
|
<div className="space-y-3 p-4 rounded-xl bg-[var(--color-bg)] border border-[var(--color-border)]">
|
||||||
<input
|
<input
|
||||||
type="text"
|
type="text"
|
||||||
@@ -239,6 +311,19 @@ export default function KnowledgePanel() {
|
|||||||
placeholder="Description (optional)"
|
placeholder="Description (optional)"
|
||||||
className="input-field"
|
className="input-field"
|
||||||
/>
|
/>
|
||||||
|
<div className="space-y-1">
|
||||||
|
<label className="text-xs text-[var(--color-text-muted)]">ChromaDB path</label>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
value={newChromaDir}
|
||||||
|
onChange={(e) => setNewChromaDir(e.target.value)}
|
||||||
|
placeholder="/data/chromadb"
|
||||||
|
className="input-field text-sm"
|
||||||
|
/>
|
||||||
|
<p className="text-[10px] text-[var(--color-text-muted)]">
|
||||||
|
Path inside the ChromaDB container where this base will be stored
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
<div className="flex gap-2">
|
<div className="flex gap-2">
|
||||||
<button
|
<button
|
||||||
onClick={handleCreate}
|
onClick={handleCreate}
|
||||||
@@ -248,7 +333,7 @@ export default function KnowledgePanel() {
|
|||||||
{loading ? 'Creating...' : 'Create'}
|
{loading ? 'Creating...' : 'Create'}
|
||||||
</button>
|
</button>
|
||||||
<button
|
<button
|
||||||
onClick={() => { setIsCreating(false); setNewName(''); setNewDesc(''); }}
|
onClick={() => { setIsCreating(false); setNewName(''); setNewDesc(''); setNewChromaDir('/data/chromadb'); }}
|
||||||
className="px-4 py-2 rounded-lg text-[var(--color-text-secondary)] hover:text-[var(--color-text)] hover:bg-[var(--color-surface-hover)] transition-colors"
|
className="px-4 py-2 rounded-lg text-[var(--color-text-secondary)] hover:text-[var(--color-text)] hover:bg-[var(--color-surface-hover)] transition-colors"
|
||||||
>
|
>
|
||||||
Cancel
|
Cancel
|
||||||
@@ -257,14 +342,27 @@ export default function KnowledgePanel() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* List */}
|
{!isBackendReady && (
|
||||||
{knowledgeBases.length === 0 ? (
|
<div className="text-center py-12 text-[var(--color-text-muted)]">
|
||||||
|
<AlertCircle size={40} className="mx-auto mb-3 opacity-40" />
|
||||||
|
<p className="text-sm">Knowledge base service unavailable</p>
|
||||||
|
<p className="text-xs mt-1">Make sure ChromaDB is running</p>
|
||||||
|
<button
|
||||||
|
onClick={loadBases}
|
||||||
|
className="mt-4 px-4 py-2 rounded-lg bg-[var(--color-accent)] text-white text-sm hover:bg-[var(--color-accent-hover)] transition-colors"
|
||||||
|
>
|
||||||
|
Retry
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{isBackendReady && knowledgeBases.length === 0 ? (
|
||||||
<div className="text-center py-12 text-[var(--color-text-muted)]">
|
<div className="text-center py-12 text-[var(--color-text-muted)]">
|
||||||
<BookOpen size={40} className="mx-auto mb-3 opacity-40" />
|
<BookOpen size={40} className="mx-auto mb-3 opacity-40" />
|
||||||
<p className="text-sm">No knowledge bases yet</p>
|
<p className="text-sm">No knowledge bases yet</p>
|
||||||
<p className="text-xs mt-1">Create one to start adding documents</p>
|
<p className="text-xs mt-1">Create one to start adding documents</p>
|
||||||
</div>
|
</div>
|
||||||
) : (
|
) : isBackendReady && (
|
||||||
<div className="space-y-3">
|
<div className="space-y-3">
|
||||||
{knowledgeBases.map((base) => (
|
{knowledgeBases.map((base) => (
|
||||||
<div
|
<div
|
||||||
@@ -287,6 +385,7 @@ export default function KnowledgePanel() {
|
|||||||
</p>
|
</p>
|
||||||
<p className="text-xs text-[var(--color-text-muted)]">
|
<p className="text-xs text-[var(--color-text-muted)]">
|
||||||
{base.documentCount || 0} chunks · {base.files?.length || 0} files
|
{base.documentCount || 0} chunks · {base.files?.length || 0} files
|
||||||
|
{base.chromaDir ? ` · ${base.chromaDir}` : ''}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</button>
|
</button>
|
||||||
@@ -303,11 +402,13 @@ export default function KnowledgePanel() {
|
|||||||
{base.description && (
|
{base.description && (
|
||||||
<p className="text-xs text-[var(--color-text-muted)]">{base.description}</p>
|
<p className="text-xs text-[var(--color-text-muted)]">{base.description}</p>
|
||||||
)}
|
)}
|
||||||
|
<p className="text-[10px] text-[var(--color-text-muted)] font-mono">
|
||||||
|
ChromaDB: {base.chromaDir || '/data/chromadb'}
|
||||||
|
</p>
|
||||||
|
|
||||||
{/* Files */}
|
|
||||||
{base.files && base.files.length > 0 && (
|
{base.files && base.files.length > 0 && (
|
||||||
<div className="space-y-1.5">
|
<div className="space-y-1.5">
|
||||||
{base.files.map((file) => (
|
{base.files.map((file: any) => (
|
||||||
<div
|
<div
|
||||||
key={file.id}
|
key={file.id}
|
||||||
className="flex items-center justify-between px-3 py-2 rounded-lg bg-[var(--color-surface)] text-xs"
|
className="flex items-center justify-between px-3 py-2 rounded-lg bg-[var(--color-surface)] text-xs"
|
||||||
@@ -332,47 +433,31 @@ export default function KnowledgePanel() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Upload buttons */}
|
<div className="flex gap-2 flex-wrap">
|
||||||
<div className="flex gap-2">
|
<button
|
||||||
<label className={`flex-1 flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:border-[var(--color-accent)] hover:text-[var(--color-accent)] transition-colors cursor-pointer ${uploadingBase === base.id ? 'opacity-50' : ''}`}>
|
onClick={() => openFilePicker(base.id)}
|
||||||
{uploadingBase === base.id ? (
|
disabled={uploadingBase === base.id}
|
||||||
<span className="animate-pulse">Processing...</span>
|
className="flex-1 min-w-[80px] flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:border-[var(--color-accent)] hover:text-[var(--color-accent)] transition-colors disabled:opacity-50"
|
||||||
) : (
|
>
|
||||||
<>
|
<Upload size={14} />
|
||||||
<Upload size={14} />
|
<span>Upload file</span>
|
||||||
<span>Upload file</span>
|
</button>
|
||||||
</>
|
<button
|
||||||
)}
|
onClick={() => openFolderPicker(base.id)}
|
||||||
<input
|
disabled={uploadingBase === base.id}
|
||||||
type="file"
|
className="flex-1 min-w-[80px] flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:border-green-500 hover:text-green-500 transition-colors disabled:opacity-50"
|
||||||
ref={fileInputRef}
|
>
|
||||||
accept=".txt,.md,.pdf,.docx,.json,.csv,.js,.ts,.html,.css"
|
<FolderOpen size={14} />
|
||||||
onChange={(e) => handleUploadFile(base.id, e)}
|
<span>Upload folder</span>
|
||||||
disabled={uploadingBase === base.id || folderLoading}
|
</button>
|
||||||
className="hidden"
|
<button
|
||||||
/>
|
onClick={() => handleScanLocalFolder(base.id)}
|
||||||
</label>
|
disabled={uploadingBase === base.id}
|
||||||
<label className={`flex-1 flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:border-green-500 hover:text-green-500 transition-colors cursor-pointer ${folderLoading ? 'opacity-50' : ''}`}>
|
className="flex-1 min-w-[80px] flex items-center justify-center gap-2 px-3 py-2 rounded-lg border border-dashed border-[var(--color-border)] text-xs text-[var(--color-text-muted)] hover:border-blue-500 hover:text-blue-500 transition-colors disabled:opacity-50"
|
||||||
{folderLoading ? (
|
>
|
||||||
<span className="animate-pulse">Scanning folder...</span>
|
<FolderSearch size={14} />
|
||||||
) : (
|
<span>Scan server</span>
|
||||||
<>
|
</button>
|
||||||
<FolderOpen size={14} />
|
|
||||||
<span>Upload folder</span>
|
|
||||||
</>
|
|
||||||
)}
|
|
||||||
<input
|
|
||||||
type="file"
|
|
||||||
ref={folderInputRef}
|
|
||||||
// @ts-ignore — webkitdirectory is non-standard but widely supported
|
|
||||||
webkitdirectory=""
|
|
||||||
directory=""
|
|
||||||
multiple
|
|
||||||
onChange={(e) => handleUploadFolder(base.id, e)}
|
|
||||||
disabled={uploadingBase === base.id || folderLoading}
|
|
||||||
className="hidden"
|
|
||||||
/>
|
|
||||||
</label>
|
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -41,6 +41,17 @@ export default function Sidebar() {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const toggleKnowledgeBase = (baseId: string) => {
|
||||||
|
if (!currentConversation) return;
|
||||||
|
const currentIds = currentConversation.knowledgeBaseIds || [];
|
||||||
|
const newIds = currentIds.includes(baseId)
|
||||||
|
? currentIds.filter((id) => id !== baseId)
|
||||||
|
: [...currentIds, baseId];
|
||||||
|
updateConversation(currentConversation.id, {
|
||||||
|
knowledgeBaseIds: newIds.length > 0 ? newIds : undefined,
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="flex flex-col h-full">
|
<div className="flex flex-col h-full">
|
||||||
{/* Header */}
|
{/* Header */}
|
||||||
@@ -118,29 +129,32 @@ export default function Sidebar() {
|
|||||||
No knowledge bases yet
|
No knowledge bases yet
|
||||||
</p>
|
</p>
|
||||||
) : (
|
) : (
|
||||||
<div className="space-y-1">
|
<div className="space-y-1 max-h-40 overflow-y-auto">
|
||||||
<select
|
{knowledgeBases.map((base) => (
|
||||||
value={currentConversation?.knowledgeBaseId || ''}
|
<label
|
||||||
onChange={(e) => {
|
key={base.id}
|
||||||
if (currentConversation) {
|
className={`flex items-center gap-2 px-2 py-1.5 rounded-lg cursor-pointer text-xs transition-colors ${
|
||||||
updateConversation(currentConversation.id, {
|
currentConversation?.knowledgeBaseIds?.includes(base.id)
|
||||||
knowledgeBaseId: e.target.value || undefined,
|
? 'text-[var(--color-accent)] bg-[var(--color-accent)]/10'
|
||||||
});
|
: 'text-[var(--color-text-secondary)] hover:bg-[var(--color-surface-hover)]'
|
||||||
}
|
}`}
|
||||||
}}
|
>
|
||||||
disabled={!currentConversation}
|
<input
|
||||||
className="w-full text-xs bg-[var(--color-bg)] border border-[var(--color-border)] rounded-lg px-2 py-1.5 text-[var(--color-text)] outline-none focus:ring-1 focus:ring-[var(--color-accent)] disabled:opacity-50"
|
type="checkbox"
|
||||||
>
|
checked={currentConversation?.knowledgeBaseIds?.includes(base.id) || false}
|
||||||
<option value="">No knowledge base</option>
|
onChange={() => toggleKnowledgeBase(base.id)}
|
||||||
{knowledgeBases.map((base) => (
|
disabled={!currentConversation}
|
||||||
<option key={base.id} value={base.id}>
|
className="rounded border-[var(--color-border)] text-[var(--color-accent)] focus:ring-[var(--color-accent)]"
|
||||||
{base.name}
|
/>
|
||||||
</option>
|
<span className="truncate flex-1">{base.name}</span>
|
||||||
))}
|
<span className="text-[var(--color-text-muted)] shrink-0">
|
||||||
</select>
|
{base.documentCount || 0} chunks
|
||||||
{currentConversation?.knowledgeBaseId && (
|
</span>
|
||||||
<p className="text-[10px] text-[var(--color-accent)] px-1">
|
</label>
|
||||||
Using: {knowledgeBases.find((b) => b.id === currentConversation.knowledgeBaseId)?.name}
|
))}
|
||||||
|
{currentConversation && (currentConversation.knowledgeBaseIds?.length || 0) > 0 && (
|
||||||
|
<p className="text-[10px] text-[var(--color-accent)] px-1 pt-0.5">
|
||||||
|
{currentConversation.knowledgeBaseIds?.length} base(s) active
|
||||||
</p>
|
</p>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ export interface Conversation {
|
|||||||
createdAt: number;
|
createdAt: number;
|
||||||
updatedAt: number;
|
updatedAt: number;
|
||||||
model?: string;
|
model?: string;
|
||||||
knowledgeBaseId?: string | null;
|
knowledgeBaseIds?: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ChatSettings {
|
export interface ChatSettings {
|
||||||
@@ -42,6 +42,7 @@ export interface KnowledgeBase {
|
|||||||
id: string;
|
id: string;
|
||||||
name: string;
|
name: string;
|
||||||
description: string;
|
description: string;
|
||||||
|
chromaDir: string;
|
||||||
files: KnowledgeFile[];
|
files: KnowledgeFile[];
|
||||||
documentCount: number;
|
documentCount: number;
|
||||||
createdAt: number;
|
createdAt: number;
|
||||||
|
|||||||
+1
-11
@@ -18,7 +18,7 @@ server {
|
|||||||
|
|
||||||
# Backend API
|
# Backend API
|
||||||
location /api/ {
|
location /api/ {
|
||||||
proxy_pass http://backend:3002/;
|
proxy_pass http://backend:3002/api/;
|
||||||
proxy_http_version 1.1;
|
proxy_http_version 1.1;
|
||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
@@ -30,14 +30,4 @@ server {
|
|||||||
proxy_cache off;
|
proxy_cache off;
|
||||||
proxy_read_timeout 86400s;
|
proxy_read_timeout 86400s;
|
||||||
}
|
}
|
||||||
|
|
||||||
# Strapi
|
|
||||||
location /strapi/ {
|
|
||||||
proxy_pass http://strapi:1337/;
|
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user