Compare commits

..
Author SHA1 Message Date
Emil 6b67326c3b Run claim-execute-upload loops concurrently in the worker agent
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 15:36:22 +03:00
Emil d57f8778ac Log the parallel search workload in the session goal 2026-08-03 15:20:10 +03:00
Emil 5b2d5b5f6e Add similarity-search-parallel: thread-pool scoring, byte-identical to similarity-search
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 15:12:18 +03:00
Emil b63441bb8c Revert "Warn about non-ASCII characters in text fields of the job form"
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
This reverts commit 91d049b4df.
2026-08-03 14:34:46 +03:00
Emil 91d049b4df Warn about non-ASCII characters in text fields of the job form 2026-08-03 14:34:12 +03:00
Emil 70c9e567ca Probe the busy port with a cancellable request
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 14:26:40 +03:00
Emil 81e854eb1d Reopen the already-running setup wizard instead of failing on a busy port 2026-08-03 14:26:17 +03:00
Emil 9a90d59ecf Complete the overnight plan: mark all items done in the session goal 2026-08-03 04:28:43 +03:00
Emil 2b6531fa7e Front the worker-key exchange through the coordinator in serve mode
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 04:24:52 +03:00
Emil ab922356e9 Apply migrations in the admin postgres integration tests regardless of run order 2026-08-03 04:16:37 +03:00
Emil fdea3a7152 Sync STATUS.md with the admin console, wizard, wheel pipeline and CTX-19/20 2026-08-03 04:15:00 +03:00
Emil c90c6b2ef6 Derive the package version from git tags via setuptools_scm 2026-08-03 04:14:23 +03:00
Emil ea0fb8c595 Silence the gosec warning for the wizard's own log path 2026-08-03 04:12:15 +03:00
Emil 357ed34714 Add worker stats to the wizard, artifact pruning and offline-worker removal to the admin console 2026-08-03 04:11:43 +03:00
Emil 7ea7c52325 Cover the admin read model and workload settings with postgres integration tests 2026-08-03 04:07:15 +03:00
Emil 8db1f9846c Restyle the remaining operator pages into the admin design system 2026-08-03 04:05:19 +03:00
Emil 7d1aacb19f Reject blank worker names and probe the real credential and venv in --check 2026-08-03 04:01:45 +03:00
Emil 0be114c8da Write the overnight plan into the session goal 2026-08-03 03:58:19 +03:00
Emil 6956ea6b1f Update the session goal with the final polish round 2026-08-03 03:49:41 +03:00
Emil 5790ae78a7 Create the worker directory before the wizard downloads the release wheel
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 03:46:10 +03:00
Emil 40ff688416 Probe the managed venv in the wizard preflight and document RDKit's X11 deps
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 03:23:28 +03:00
Emil 1ec4b2e60b Point the wizard's task runner at the venv python automatically
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 03:16:23 +03:00
Emil d309145290 Set the agent version before dispatching the setup wizard
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 03:05:55 +03:00
Emil 15dd9651a8 Ship the scimesh wheel in releases; wizard and serve install it version-locked
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / wheel (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 02:59:22 +03:00
Emil 30c441a7e9 Require an explicit scimesh source in the worker wizard (PyPI name is another project)
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 02:39:24 +03:00
Emil cb51172885 Let the worker wizard install the scimesh runtime into a managed venv
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 02:32:06 +03:00
Emil 029b26e6ae Explain the admin-required error and offer account switching on the login page
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 02:06:45 +03:00
Emil 12499d2d7b Remove the demo control room; /ui lands on the admin console
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 01:58:02 +03:00
Emil c4d88c7ffc Satisfy the linter gate: errors.Is, context-aware exec/listen, gosec nolints
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 01:47:53 +03:00
Emil 44247bd94e Give every wizard test its own port and disable keep-alive pooling
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 01:43:48 +03:00
Emil 7c1d0dc568 Tell non-admins why the console is off-limits instead of bouncing them to the dashboard
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 01:34:22 +03:00
Emil 82eaec4c55 Make the wizard supervisor portable (no signal probing) and bind tests to an ephemeral port
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-03 00:38:33 +03:00
Emil c301499bef Sync README and installer header with the auto-open flow 2026-08-03 00:33:50 +03:00
Emil 8b7094b0cb Open the admin console from serve --open and land login on the intended page 2026-08-03 00:32:47 +03:00
Emil bdbe0e3e39 Auto-open the coordinator UI and the worker wizard from the installers 2026-08-03 00:27:48 +03:00
Emil c11756c3d8 Document the admin console and the worker setup wizard 2026-08-03 00:24:06 +03:00
Emil 46645b8730 Add the worker setup wizard (setup, --config, --check) to worker-agent 2026-08-03 00:22:37 +03:00
Emil e923627ce0 Add admin console access and platform pages (trust, users, keys, workloads, settings) 2026-08-03 00:11:47 +03:00
Emil 7fb1059401 Add coordinator admin console foundation (system, jobs, metrics) 2026-08-02 23:51:26 +03:00
Emil 41cae546ff Approve UI mockups and write the full implementation plan 2026-08-02 23:30:23 +03:00
Emil f8ff0956b9 Move the worker setup UI into the worker-agent binary 2026-08-02 23:02:31 +03:00
Emil 7da9bf8c3a Plan the coordinator admin UI and worker setup wizard 2026-08-02 22:55:33 +03:00
Emil 69a2d59b23 Add the coordinator token subcommand for joining workers 2026-08-02 22:43:17 +03:00
Emil 45d5c2eaad Update the add-worker command and expose the coordinator public URL in serve 2026-08-02 22:32:54 +03:00
Emil 10a7e8df1d Document the standalone split setup and build a userservice binary 2026-08-02 22:29:05 +03:00
Emil 76746187eb Add a standalone worker installer and dynamic userservice port 2026-08-02 22:23:54 +03:00
Emil 9b0b9b208a Warn on a stale download in the installer
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
2026-08-02 22:07:04 +03:00
Emil a339ac853b Install the newest release including pre-releases by default 2026-08-02 22:03:10 +03:00
105 changed files with 8804 additions and 687 deletions
+33 -1
View File
@@ -54,8 +54,35 @@ jobs:
path: coordinator/dist/*
if-no-files-found: error
wheel:
runs-on: ubuntu-latest
steps:
# Full history: setuptools_scm derives the package version from the
# release tag (v1.1.0-alpha.16 -> 1.1.0a16), so the wheel is
# version-locked to the binaries of the same release.
- uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: build the scimesh wheel
run: |
python -m pip install --quiet build
python -m build --wheel --outdir dist
ls -la dist/
- uses: actions/upload-artifact@v4
with:
name: wheel
path: dist/*.whl
if-no-files-found: error
release:
needs: binaries
needs: [binaries, wheel]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -68,6 +95,11 @@ jobs:
pattern: binaries-*
merge-multiple: true
- uses: actions/download-artifact@v4
with:
name: wheel
path: artifacts
- name: checksums
working-directory: artifacts
run: sha256sum * > SHA256SUMS.txt
+69 -17
View File
@@ -1,22 +1,74 @@
# Session Goal
COMPLETED
выполни план на ночь (полный план ниже — автономное исполнение, пользователь спит)
выполни полностью намеченный сейчас план. автономно
## Ночная сессия — полный план (пользователь спит, 2026-08-03)
## Plan
1. Фаза 1 — SQLite-хранилище: `coordinator/internal/storage/sqlite` (все порты, TxManager, миграции, `SCIMESH_DB=sqlite|postgres`, тесты).
2. Фаза 2 — Встроенный userservice: перенос `users/internal/*` в `coordinator/internal/userservice/` (sqlite-хранилище), запуск на 127.0.0.1, BootstrapAdmin.
3. Фаза 3 — `coordinator serve` (data-dir, всё-в-одном, --workers N, --open) + subcommand `coordinator agent`.
4. Фаза 4 — Управляемый venv + install.sh/install.ps1 + ассеты релиза.
5. Документация: mkdocs, README, PLAN.md (CTX-17 done, CTX-18), STATUS.md.
6. Проверка: полный E2E без внешних сервисов + все тесты/lint/vet.
### 1. Баги (реальные, найденные в проде)
- [ ] **Пустое имя воркера**: регистрация принимает `name=""` (в БД пользователя было 15 таких). Фикс: валидация в `domain.NewWorker` (TrimSpace != "") + регрессионный тест. Частично начат — в `worker_test.go` сломан тест (`fixedTime` vs `testNow`), доделать.
- [ ] **`worker-agent --check` не видит venv**: проверяет только системный python3; после Install воркер работает через `~/.scimesh-worker/venv`. Выровнять с визардом (пробовать venv, если он есть).
## Progress
- ВСЕ ФАЗЫ ВЫПОЛНЕНЫ И ЗАПУШЕНЫ:
- 9883def — SQLite-бэкенд (SCIMESH_DB=sqlite|postgres, миграции, тесты).
- 1473bbe — встроенный userservice + serve/agent subcommands.
- c06d867 — install.sh/install.ps1 + make serve.
- 215325a — документация (README, mkdocs, PLAN CTX-17/CTX-18, STATUS).
- E2E «чистая машина»: `coordinator serve` → health/login (embedded userservice) → molwt-filter джоб через локального агента → результат byte-точный.
- Верификация: 208 pytest, pyright 0, 18 Go-пакетов ok, gofmt чист, vet чист, golangci 0 issues, postgres integration ok, все CI-раны success.
### 2. Визуальный долг — рестайлинг старых страниц в дизайн-систему админки (#0b0e13, карточки, pill-статусы, кнопки)
- [ ] `new-job.html` (форма запуска вычислений, UIElement-поля сохранить)
- [ ] `job.html` (детали джоба: шарды, артефакты, прогресс, JS-логику сохранить)
- [ ] `workloads.html`
- [ ] `add-worker.html`
- [ ] `profile.html`
- [ ] Браузерная проверка каждой страницы (playwright).
### 3. Технический долг (паритет движков, обещанный планом)
- [ ] **Postgres integration-тесты для admin-методов**: `SetTrust`, `WorkloadSettings` (Get/List/Set), `ListJobsPaginated`, метрики (`JobCountsByDay`, `TaskStats`, `ArtifactSizeByKind`, `DatabaseSizeBytes`) — сейчас покрыт только sqlite. (build tag integration, docker postgres как в CI.)
### 4. Фичи из плана (v2-задел)
- [ ] **Статистика воркера в визарде**: статусная страница — claimed/completed/failed/heartbeat, парсинг из `worker.log` (обещано в docs/ui-admin-worker-plan.md §4).
- [ ] **Admin: prune артефактов** (danger zone): удаление артефактов completed-джобов старше N дней (строки + blob-файлы), освобождает место; кнопка в Settings с подтверждением.
- [ ] **Admin: удаление offline-воркеров** (устаревшие, как 15 мусорных) — кнопкой в Workers вместо SQL.
### 5. Гигиена
- [ ] **`pyproject.toml` версия 0.1.0 → setuptools_scm** (версия из git-тегов), убрать sed-костыль из release.yml; локальный `pip install -e .` покажет правильную версию.
- [ ] Синхронизация STATUS.md / README с фактическим состоянием (админка, визард, wheel, авто-открытие, restyle).
### 6. Верификация и релиз
- [ ] Полный гейт: `go test -race ./...`, golangci-lint `--build-tags=integration`, pytest.
- [ ] Браузерные проверки: админка (все секции), визард (install+start), рестайленные страницы.
- [ ] E2E quorum-флоу (untrusted воркер через worker key → джоб требует quorum) в Docker, если останется время.
- [ ] Релиз `v1.1.0-alpha.16` (бинарники + wheel), проверка ассетов.
- [ ] Обновить этот файл: прогресс по пунктам, COMPLETED в конце.
## Plan (предыдущая задача — выполнена)
17. Docker E2E пайплайна «install как человек → serve → визард → воркер → джоб» — выполнено, см. Progress ниже.
## Progress (утро: similarity-search-parallel)
- ✅ Новый workload `similarity-search-parallel@1.0.0` (отдельная версия, как просил пользователь): подкласс `SimilaritySearchSDKWorkload` + параллельное ядро `search_parallel/core.py` — fingerprinting+скоринг шарда через `ThreadPoolExecutor` (параметр `threads`, default CPU count); `pool.map` сохраняет порядок строк, поэтому merge идентичен последовательному (`_HeapEntry`) и результат **байт-в-байт** равен `similarity-search` при любом числе потоков.
- ✅ Тесты: байт-в-байт vs эталон для threads 1/2/4 с намеренными связями (изомеры, дубликаты), executor-прогон, валидация параметров, регистрация манифеста; 213 pytest зелёные.
- ✅ Экспортирован в каталог координатора (5 ворклоадов), Go-тесты зелёные; release v1.1.0-alpha.19 (бинарники + wheel `scimesh-1.1.0a19`).
- ✅ Распределённый E2E в Docker: воркер с wheel a19, джоб similarity-search-parallel (threads=4) → completed → результат байт-в-байт = локальному эталону.
- ✅ Бинарники на машине пользователя обновлены до alpha.19 (для появления ворклоада в UI нужен рестарт serve + переустановка рантайма воркера).
- Примечание: в CPython потоки не ускоряют чистый RDKit-путь (GIL), но структура готова к ядрам, отпускающим GIL (numpy и т.п.); при желании можно добавить процесс-пул как отдельный workload в будущей версии протокола.
## Progress (ночная сессия)
-**П.1 Пустое имя воркера**: `domain.NewWorker` нормализует/отклоняет пустое имя + `TestNewWorkerRejectsBlankName` (починен `fixedTime``testNow`).
-**П.2 `--check`**: пробует managed venv (если установлен) + реальный пробинг учётки (exchange ключа / claim-пробa) — `CheckAuth` + тесты; на машине пользователя: `✓ auth: credential accepted`, venv python, scimesh installed.
-**П.3 Рестайлинг**: единый CSS-partial `ui-base.html` (дизайн-система админки), все 5 страниц (new-job, job, workloads, add-worker, profile) переведены, проверены в браузере без console-ошибок.
-**П.4 Postgres integration**: admin-методы (SetTrust, ListJobsPaginated, TaskCounts, byDay/byWorkload, TaskStats, ArtifactSize, DB size, WorkloadSettings) + `ensureMigrated` для порядка запуска; весь suite зелёный.
-**П.5 Статистика воркера в визарде**: лог `task claimed` в агента + парсинг registered/claimed/completed/failed → `/api/status.stats` + карточки в статусной странице + тест.
-**П.6 Prune артефактов**: `JobRepository.ListCompletedBefore/Delete` (sqlite+postgres+memstore), usecase `PruneArtifacts` (каскад + blob-файлы), `POST /ui/admin/api/prune`, кнопка в Settings, тесты (sqlite+usecase); E2E: 200, freed bytes.
-**П.7 Удаление offline-воркеров**: `WorkerRepository.Delete` + `Admin.RemoveWorker` (только offline) + `POST /ui/admin/api/workers/{id}/remove` + кнопка в Workers + тест; E2E: 204, строка удалена.
-**П.8 setuptools_scm**: `dynamic = ["version"]`, CI-джоба wheel без sed (fetch-depth 0); локальная проверка: wheel на теге = `scimesh-1.1.0a16-py3-none-any.whl` (совпадает с Go-нормализацией); релизный ассет подтверждён.
-**П.9 Docs**: STATUS.md синхронизирован (админка, визард, wheel, CTX-19/20 implemented).
-**П.10 (доп.) Баг в serve-режиме**: worker-key exchange был недоступен снаружи (userservice на loopback) — добавлен прокси `POST /worker-tokens/exchange` на координаторе, `PublicUserserviceURL=""` + fallback на origin в add-worker. Проверено E2E.
-**П.10 Quorum E2E (Docker)**: 2 untrusted-воркера с разными ключами (alice/bob) → джоб completed 3/3, в task_results по 2 голоса от разных владельцев с одинаковым sha256 → результат байт-в-байт = локальному эталону.
-**Финальный гейт**: `go test -race ./...` ✅, golangci-lint 0 issues ✅, pytest 208 ✅, postgres integration ✅, Windows кросс-сборка ✅.
-**Релиз v1.1.0-alpha.16** (бинарники + wheel `scimesh-1.1.0a16`), все воркфлоу success; бинарники на машине пользователя обновлены до alpha.16.
## Progress (прошлая работа — выполнено)
- ✅ Релизы alpha.12–15: фикс версии визарда, venv task_runner, preflight через venv, MkdirAll при скачивании wheel, кнопка Install в шаблоне.
- ✅ Docker E2E: координатор+воркер контейнеры, установка install.sh, визард (config→install→start), воркер online, джоб completed, результат байт-в-байт = локальному эталону.
- ✅ На машине пользователя: визард alpha.15, правильный токен, venv из wheel, воркер emil-pc online, 15 пустых воркеров вычищены из БД.
- ✅ Гейт: race + lint + pytest 208.
## Completion
COMPLETED — ночной план выполнен полностью (10 пунктов + 2 найденных бага, включая E2E quorum на релизном коде). Все гейты зелёные, релиз v1.1.0-alpha.16 опубликован.
## Completion (предыдущая задача)
COMPLETED — пайплайн доведён до рабочего состояния и проверен на релизных артефактах v1.1.0-alpha.14.
+45
View File
@@ -1056,6 +1056,51 @@ no PostgreSQL, no Docker, no Python setup.
---
### CTX-19 — Coordinator Admin UI
**Goal:** Replace the demo-level operator pages with a real admin console
(`/ui/admin`): system overview, job management with filters and pagination,
worker management (trust, capabilities), users/roles and worker keys,
workload enable/disable, settings (read-only), and storage metrics.
**Depends on:** CTX-11 (UI patterns, session/roles), CTX-15 (userservice
proxy), the sqlite/postgres storage pair.
**Status: implemented.** Admin console at `/ui/admin`: system/storage/health,
jobs (filter + pagination + owner resolution), workers with trust controls,
users & worker keys (userservice admin endpoints), workload enable/disable
(`workload_settings` migration in both engines, enforced at submit time and
hidden from the job form), metrics (7-day buckets, failure rate), and
settings with an audited worker-token reveal. All `/ui/admin/*` routes are
admin-only and backed by bounded read models; sqlite + postgres parity;
unit/permission/integration tests green.
### CTX-20 — Worker Setup UI
**Goal:** A local setup wizard embedded in the **worker-agent** binary
(`worker-agent setup`, browser on 127.0.0.1) that turns any machine into a
worker: coordinator URL + auth (serve token or worker key), work dir,
connection check, start/stop, and a live status page. Runs on machines that
have only the worker installed — no coordinator needed.
**Depends on:** the agent daemon; the worker-key exchange (CTX-15).
**Status: implemented.** `worker-agent setup` serves the local wizard on
127.0.0.1 (default port 12700): coordinator URL + token or worker key,
work dir and name, preflight check (coordinator/health, python3, scimesh),
config saved to `~/.scimesh-worker/config.json` (0600), start/stop of the
worker as a background process, and a live status page with the agent log.
The daemon also gained `--config <path>` (environment still wins) and
`--check`. End-to-end verified: the wizard started a real worker that
registered with a coordinator and appeared in the admin console.
**Acceptance criteria:** full plan in
[`docs/ui-admin-worker-plan.md`](docs/ui-admin-worker-plan.md) (section 4);
the agent gains `setup`, `--config <path>`, and `--check`; end-to-end: the
wizard starts a real worker that registers and completes a job.
---
## 10. Suggested assignment bundles
These bundles minimize overlap. Do not run tasks from the same bundle in
+25 -5
View File
@@ -49,19 +49,23 @@ checksums), the installer scripts, and the `coordinator` image on GHCR:
docker pull ghcr.io/emil28092005/SciMesh/coordinator:latest
```
For scientists: one command downloads the right binary and prints the start
instructions:
For scientists: one command downloads the right binary, starts it, and opens
the UI in the browser:
```bash
# Linux / macOS
# Linux / macOS — installs, starts and opens the admin console automatically
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash
coordinator serve --open
# Windows (PowerShell)
powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex"
coordinator serve --open
```
Set `SCIMESH_AUTO_START=0` to install without starting anything. The old demo
control room was removed: `/ui` is the admin console. A standalone
worker is installed the same way (`bash -s worker`, or
`SCIMESH_COMPONENT=worker` on Windows); its installer opens the local setup
wizard (`worker-agent setup`) in the browser automatically.
`coordinator serve` is the single-binary mode: it embeds SQLite (coordinator +
userservice databases), the userservice itself, and local worker agents
(`--workers N`, default 1). On first start it generates secrets and the admin
@@ -70,6 +74,18 @@ PostgreSQL, no Docker, no environment variables. The scientific runtime is a
managed venv (`~/.scimesh/venv`); point `SCIMESH_PIP_PACKAGE` at your scimesh
wheel to install it automatically.
The coordinator's UI is the **admin console** at `/ui/admin` — cluster health
and storage, paginated job table, worker fleet with trust controls, users and
worker keys, workload enable/disable, metrics and the worker token. The job
form (`/ui/jobs/new`), job detail pages and the workload library complete the
operator surface; `/ui` redirects to the console, `serve --open` lands on it,
and login returns you to the page you asked for. The **worker** binary (`worker-agent`) carries its own local setup
wizard for machines that run only a worker: `worker-agent setup` opens a
browser wizard at `127.0.0.1` that collects the coordinator URL and
credential, runs a preflight check, saves `~/.scimesh-worker/config.json` and
starts/stops the worker with a live log (see the
[standalone docs](mkdocs/standalone.md)).
Manual download and run of a release binary:
```bash
@@ -209,6 +225,10 @@ The coordinator and worker agent are Go modules under `coordinator/` and `users/
cd coordinator && make coordinator agent && go test ./...
```
On headless servers (no desktop environment), RDKit needs a few X11
libraries that desktops already ship: `sudo apt-get install -y libxrender1
libxext6 libxcursor1 libxfixes3 libxi6 libxrandr2`.
`make check` runs the full gate: vet, lint, race tests, the PostgreSQL
integration suite, and the two-worker end-to-end smoke script.
+27 -3
View File
@@ -1,7 +1,8 @@
# SciMesh Status
**Updated:** 2026-08-02
**Branch baseline:** `main`; this revision adds the single-binary platform.
**Updated:** 2026-08-03
**Branch baseline:** `main`; this revision adds the admin console, the worker
setup wizard, and release-shipped Python wheels.
## Current state
@@ -42,6 +43,27 @@ Users can create and revoke worker keys for self-service Worker Agent
enrollment. Untrusted workers require quorum agreement from distinct owners on
the complete result-artifact SHA-256 before a task is accepted.
**Admin console** (`/ui/admin`, admin role only): system/storage/health,
paginated jobs with owner resolution, worker fleet with trust controls and
offline-worker removal, users and worker keys, workload enable/disable
(persisted in both engines, enforced at submit time), 7-day metrics, and
settings with an audited worker-token reveal and artifact pruning for old
finished jobs.
**Worker setup wizard** (`worker-agent setup`, loopback `127.0.0.1`): browser
wizard that configures the coordinator URL and credential, runs a preflight
(coordinator reachability, real credential probe, python, scimesh), installs
the version-locked `scimesh` wheel from the release into a managed venv
(`~/.scimesh-worker/venv`), pins the venv task runner, and starts/stops the
worker with live log and parsed claim/completed/failed counters. The release
pipeline now also ships the Python wheel (`scimesh-<ver>-py3-none-any.whl`,
version derived from the git tag via setuptools_scm), and both `coordinator
serve` and the wizard install it automatically.
The full pipeline was verified end-to-end in containers on release artifacts:
installer → serve → wizard → registration → real similarity-search job →
byte-exact result vs the local reference.
## Milestone tracker
| CTX | Status | Notes |
@@ -57,7 +79,9 @@ the complete result-artifact SHA-256 before a task is accepted.
| CTX-08 Distributed similarity-search | Implemented | Planner/worker/reducer match the local reference byte-exactly. |
| CTX-09 Reducer and final-result API | Implemented | Atomic `reducing` claim, deterministic coordinator-side reducers (`top-k` and `ordered-concat`), sanitized failure, final artifact, `result_uri`. |
| CTX-10 Distributed similarity-graph | Not started | Local reference exists; the SDK-built local graph workload enforces the pair-coverage invariant. |
| CTX-11 Dashboard/operator view | Implemented | Protected live control room, workload library, workload-agnostic "New computation" form (SDK-declared `UIElement`s), MkDocs at `/ui/docs/`, final-result download. |
| CTX-11 Dashboard/operator view | Implemented | The demo control room was removed; `/ui` lands on the admin console. Job form/detail, workload library, add-machine and profile pages share the admin design system; MkDocs at `/ui/docs/`. |
| CTX-19 Coordinator Admin UI | Implemented | `/ui/admin` console (see above), bounded read models, admin-only routes, sqlite+postgres parity with integration tests. |
| CTX-20 Worker Setup UI | Implemented | `worker-agent setup` wizard with runtime installer, `--config`, `--check` (credential + venv probing), live status. |
| CTX-12 Reliability, security, CI | In progress | vet, gofmt, race tests, golangci-lint (0 issues), PostgreSQL integration, and smoke checks exist. |
| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, quorum; also embeddable (`coordinator serve`). |
| CTX-16 Workload SDK foundation | Implemented | Strict immutable manifests/plans/artifacts, digest/trust-pinned tasks, negotiation, verifier primitives, conformance harness. |
+42 -1
View File
@@ -30,16 +30,25 @@ func main() {
switch args[0] {
case "setup":
if err := runSetup(args[1:]); err != nil {
fmt.Fprintln(os.Stderr, "setup:", err)
os.Exit(1)
}
return
case "serve":
if err := runServe(args[1:]); err != nil {
fmt.Fprintln(os.Stderr, "serve:", err)
os.Exit(1)
}
return
case "agent":
if err := runAgent(args[1:]); err != nil {
fmt.Fprintln(os.Stderr, "agent:", err)
os.Exit(1)
}
return
case "token":
if err := runToken(args[1:]); err != nil {
fmt.Fprintln(os.Stderr, "token:", err)
os.Exit(1)
}
return
@@ -67,6 +76,8 @@ type storageDeps struct {
workerRepo usecase.WorkerRepository
artifactRepo usecase.ArtifactRepository
uiReadRepo usecase.UIReadRepository
adminReadRepo usecase.AdminReadRepository
settingsRepo usecase.WorkloadSettingsRepository
taskResultRepo usecase.TaskResultRepository
statsRepo interface {
Counts(ctx context.Context) (tasks, jobs, workers map[string]int, err error)
@@ -149,7 +160,7 @@ func runWithConfig(cfg infra.Config) error {
useCases := httptransport.UseCases{
RegisterWorker: usecase.NewRegisterWorker(workerRepo, clk),
CreateJob: usecase.NewCreateJob(jobRepo, taskRepo, tx, clk),
SubmitDataset: usecase.NewSubmitDataset(blobStore, artifactRepo, jobRepo, taskRepo, tx, clk, cfg.DefaultMaxAttempts, catalog),
SubmitDataset: usecase.NewSubmitDataset(blobStore, artifactRepo, jobRepo, taskRepo, tx, clk, cfg.DefaultMaxAttempts, catalog, deps.settingsRepo),
ClaimTask: usecase.NewClaimTask(taskRepo, jobRepo, workerRepo, tx, clk, cfg.LeaseDuration, catalog),
RenewLease: usecase.NewRenewLease(taskRepo, workerRepo, tx, clk, cfg.LeaseDuration),
CompleteTask: usecase.NewCompleteTask(taskRepo, jobRepo, artifactRepo, workerRepo, taskResultRepo, tx, clk, cfg.QuorumSize, catalog),
@@ -162,7 +173,23 @@ func runWithConfig(cfg infra.Config) error {
GetJobResult: usecase.NewGetJobResult(jobRepo, usecase.NewDownloadArtifact(artifactRepo, blobStore)),
GetTaskInput: usecase.NewGetTaskInput(taskRepo, artifactRepo, blobStore),
Dashboard: usecase.NewDashboard(uiReadRepo, catalog),
PruneArtifacts: usecase.NewPruneArtifacts(jobRepo, uiReadRepo, blobStore, clk),
PreviewArtifact: usecase.NewPreviewArtifact(uiReadRepo, blobStore),
Admin: usecase.NewAdmin(deps.adminReadRepo, uiReadRepo, workerRepo, deps.settingsRepo, catalog,
usecase.AdminNodeInfo{
Version: version,
StartedAt: clk.Now(),
Binary: executablePath(),
Addr: cfg.Addr,
DataDir: cfg.StorageDir,
DBEngine: cfg.DatabaseEngine,
PublicURL: cfg.PublicCoordinatorURL,
Userservice: cfg.UserserviceURL,
WorkerToken: func() string { return cfg.Token },
}, deps.ready, clk.Now).
WithAuditLog(log, func(ctx context.Context, action, detail string) {
log.Info("admin audit", "action", action, "detail", detail)
}),
}
// Background reapers are tracked so shutdown can wait for them. Without this
@@ -216,6 +243,16 @@ func runWithConfig(cfg infra.Config) error {
return err
}
// executablePath resolves the running binary for the admin console's node
// information, falling back to the invocation name.
func executablePath() string {
path, err := os.Executable()
if err != nil || path == "" {
return os.Args[0]
}
return path
}
// openSQLite opens the embedded database and builds the sqlite repositories.
func openSQLite(ctx context.Context, cfg infra.Config, log *slog.Logger) (*storageDeps, error) {
if err := os.MkdirAll(cfg.StorageDir, 0o750); err != nil {
@@ -233,6 +270,8 @@ func openSQLite(ctx context.Context, cfg infra.Config, log *slog.Logger) (*stora
workerRepo: sqlite.NewWorkerRepo(db),
artifactRepo: sqlite.NewArtifactRepo(db),
uiReadRepo: sqlite.NewUIReadRepo(db),
adminReadRepo: sqlite.NewAdminReadRepo(db),
settingsRepo: sqlite.NewWorkloadSettingsRepo(db),
taskResultRepo: sqlite.NewTaskResultRepo(db),
statsRepo: sqlite.NewStatsRepo(db),
ready: func(ctx context.Context) error { return db.PingContext(ctx) },
@@ -255,6 +294,8 @@ func openPostgres(ctx context.Context, cfg infra.Config, log *slog.Logger) (*sto
workerRepo: postgres.NewWorkerRepo(pool),
artifactRepo: postgres.NewArtifactRepo(pool),
uiReadRepo: postgres.NewUIReadRepo(pool),
adminReadRepo: postgres.NewAdminReadRepo(pool),
settingsRepo: postgres.NewWorkloadSettingsRepo(pool),
taskResultRepo: postgres.NewTaskResultRepo(pool),
statsRepo: postgres.NewStatsRepo(pool),
ready: func(ctx context.Context) error { return pool.Ping(ctx) },
+34 -13
View File
@@ -2,6 +2,9 @@ package main
import (
"context"
"github.com/emil28092005/SciMesh/coordinator/internal/agent"
"crypto/rand"
"encoding/hex"
"flag"
@@ -31,13 +34,14 @@ func runServe(args []string) error {
flags.PrintDefaults()
}
var (
dataDir = flags.String("data-dir", defaultDataDir(), "data directory (default: ~/.scimesh)")
addr = flags.String("addr", "127.0.0.1:8080", "listen address")
workers = flags.Int("workers", 1, "number of local worker agents to spawn")
open = flags.Bool("open", false, "open the UI in the browser")
docsDir = flags.String("docs-dir", "", "built MkDocs site directory to serve at /ui/docs/")
email = flags.String("admin-email", "admin@scimesh.local", "admin account email")
password = flags.String("admin-password", "", "admin password (generated on first run when empty)")
dataDir = flags.String("data-dir", defaultDataDir(), "data directory (default: ~/.scimesh)")
addr = flags.String("addr", "127.0.0.1:8080", "listen address")
workers = flags.Int("workers", 1, "number of local worker agents to spawn")
open = flags.Bool("open", false, "open the UI in the browser")
docsDir = flags.String("docs-dir", "", "built MkDocs site directory to serve at /ui/docs/")
email = flags.String("admin-email", "admin@scimesh.local", "admin account email")
password = flags.String("admin-password", "", "admin password (generated on first run when empty)")
publicURL = flags.String("public-url", "", "browser/worker-facing coordinator URL (default: http://<addr>)")
)
if err := flags.Parse(args); err != nil {
return err
@@ -100,6 +104,10 @@ func runServe(args []string) error {
defer stopAgents(agents)
// 6. The coordinator server itself.
coordinatorPublicURL := *publicURL
if coordinatorPublicURL == "" {
coordinatorPublicURL = "http://" + *addr
}
cfg := infra.Config{
Addr: *addr,
DatabaseEngine: "sqlite",
@@ -107,6 +115,7 @@ func runServe(args []string) error {
Token: workerToken,
JWTSecret: jwtSecret,
UserserviceURL: "http://" + usersAddr,
PublicCoordinatorURL: coordinatorPublicURL,
PublicUserserviceURL: "http://" + usersAddr,
LogLevel: "info",
StorageDir: filepath.Join(*dataDir, "artifacts"),
@@ -124,7 +133,7 @@ func runServe(args []string) error {
AutoMigrate: true,
}
if *open {
openBrowser("http://" + *addr + "/ui")
openBrowser("http://" + *addr + "/ui/admin")
}
// Print the login once the server is about to start.
@@ -253,13 +262,25 @@ func ensureRuntime(log *slog.Logger, dataDir, venvPython string) {
}
pip := filepath.Join(venvDir, binName("bin/pip"))
// The scimesh package is installed from an explicit source only: the PyPI
// name is not ours yet, so `pip install scimesh` would fetch a stranger's
// package. Operators publish a wheel or index via SCIMESH_PIP_PACKAGE.
// name belongs to an unrelated project, so `pip install scimesh` would
// fetch a stranger's package. Default: download the wheel attached to our
// own GitHub release for this binary version; SCIMESH_PIP_PACKAGE
// overrides with a custom wheel, checkout or index.
source := os.Getenv("SCIMESH_PIP_PACKAGE")
if source == "" {
log.Warn("scientific runtime venv created, but scimesh is not installed",
"hint", pip+" install <your scimesh wheel or index> (or set SCIMESH_PIP_PACKAGE)")
return
url, _, err := agent.ReleaseWheelURL(version)
if err != nil {
log.Warn("scientific runtime venv created, but scimesh is not installed",
"hint", "set SCIMESH_PIP_PACKAGE to your wheel or index")
return
}
downloaded, err := agent.DownloadWheel(context.Background(), url, venvDir)
if err != nil {
log.Warn("could not download the scimesh wheel for this release",
"err", err, "hint", "set SCIMESH_PIP_PACKAGE to your wheel or index")
return
}
source = downloaded
}
// #nosec G204,G702 -- pip and source are operator-configured paths.
install := exec.CommandContext(context.Background(), pip, "install", source)
+33
View File
@@ -0,0 +1,33 @@
package main
import (
"flag"
"fmt"
"os"
"path/filepath"
)
// runToken implements `coordinator token`: prints the worker auth token of a
// `coordinator serve` instance, so a scientist can join a worker without
// hunting through the data directory. The file itself is what serve created.
func runToken(args []string) error {
flags := flag.NewFlagSet("token", flag.ContinueOnError)
flags.Usage = func() {
_, _ = fmt.Fprintf(flags.Output(), "usage: coordinator token [options]\n")
_, _ = fmt.Fprintf(flags.Output(), "Prints the WORKER_AUTH_TOKEN of this coordinator's serve instance.\n\n")
flags.PrintDefaults()
}
var dataDir = flags.String("data-dir", defaultDataDir(), "data directory (default: ~/.scimesh)")
if err := flags.Parse(args); err != nil {
return err
}
if flags.NArg() > 0 {
return fmt.Errorf("token takes no positional arguments")
}
token, err := os.ReadFile(filepath.Join(*dataDir, "worker.token"))
if err != nil {
return fmt.Errorf("no worker token found in %s — start the coordinator with `coordinator serve` first", *dataDir)
}
fmt.Print(string(token))
return nil
}
+198 -4
View File
@@ -1,14 +1,27 @@
// Command worker-agent is the Go worker agent: a coordinator client that
// executes SDK workloads in a Python subprocess per claimed task.
// executes SDK workloads in a Python subprocess per claimed task. It also
// carries the local setup wizard (`worker-agent setup`) so a machine that
// installs only the worker can configure and start itself without a
// coordinator on site.
package main
import (
"context"
"errors"
"flag"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"os/exec"
"os/signal"
"strings"
"syscall"
"time"
"github.com/emil28092005/SciMesh/coordinator/internal/agent"
"github.com/emil28092005/SciMesh/coordinator/internal/agent/setupui"
)
// version is injected at build time (-ldflags "-X main.version=...") and
@@ -16,13 +29,56 @@ import (
var version = "dev"
func main() {
showVersion := flag.Bool("version", false, "print the build version and exit")
flag.Parse()
// The wizard and --check need the injected build version too (they resolve
// the release wheel matching this binary), so it is set before dispatch.
agent.Version = version
if len(os.Args) > 1 && os.Args[1] == "setup" {
os.Exit(runSetup(os.Args[2:]))
}
fs := flag.NewFlagSet("worker-agent", flag.ExitOnError)
showVersion := fs.Bool("version", false, "print the build version and exit")
configPath := fs.String("config", "", "path to a JSON config file (SCIMESH_WORKER_CONFIG overrides the default)")
checkMode := fs.Bool("check", false, "run the preflight check (coordinator + local runtime) and exit 0/1")
checkURL := fs.String("coordinator-url", "", "coordinator URL to probe in --check mode")
_ = fs.Parse(os.Args[1:])
if *showVersion {
fmt.Println("worker-agent " + version)
return
}
config, err := agent.LoadConfig()
if *checkMode {
url := *checkURL
if url == "" {
url = os.Getenv("COORDINATOR_URL")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if url == "" {
fmt.Println("check: no coordinator URL (pass --coordinator-url or set COORDINATOR_URL)")
os.Exit(1)
}
// Probe the managed venv when the wizard has installed it: workloads
// run with that interpreter, so checking the bare system python3
// would report a false negative.
checkPython, checkToken, checkKey, checkUsers := "", "", "", ""
if configPath := checkConfigPath(); configPath != "" {
checkPython = agent.VenvPython(configPath)
if config, err := agent.LoadConfigFile(configPath); err == nil {
checkToken, checkKey, checkUsers = config.Token, config.WorkerKey, config.UserserviceURL
}
}
report := agent.RunCheck(ctx, url, checkPython, checkToken, checkKey, checkUsers)
printCheck(report)
if !report.Coordinator.OK || !report.Python.OK || !report.Scimesh.OK {
os.Exit(1)
}
return
}
config, err := loadConfig(*configPath)
if err != nil {
slog.Error("invalid configuration", "error", err)
os.Exit(2)
@@ -42,3 +98,141 @@ func main() {
os.Exit(1)
}
}
// checkConfigPath resolves where the wizard's config would be, honouring
// SCIMESH_WORKER_CONFIG like the rest of the agent.
func checkConfigPath() string {
if env := os.Getenv("SCIMESH_WORKER_CONFIG"); env != "" {
return env
}
return agent.DefaultConfigPath()
}
// loadConfig prefers a --config file; environment variables override the file
// (see agent.ConfigFile.Config). Without a file, the plain environment path is
// used exactly as before.
func loadConfig(configPath string) (*agent.Config, error) {
if configPath != "" {
return agent.LoadConfigFile(configPath)
}
envPath := os.Getenv("SCIMESH_WORKER_CONFIG")
if envPath != "" {
if _, err := os.Stat(envPath); err == nil { //nolint:gosec // G703: path is the operator's own env var
return agent.LoadConfigFile(envPath)
}
}
return agent.LoadConfig()
}
func printCheck(report agent.CheckReport) {
fmt.Printf("worker-agent %s\n", report.Agent)
line := func(item agent.CheckItem) string {
mark := "✗"
if item.OK {
mark = "✓"
}
detail := item.Detail
if item.Latency > 0 {
detail = fmt.Sprintf("%s (%d ms)", detail, item.Latency)
}
return fmt.Sprintf(" %s %s: %s", mark, item.Name, detail)
}
fmt.Println(line(report.Coordinator))
fmt.Println(line(report.Auth))
fmt.Println(line(report.Python))
fmt.Println(line(report.Scimesh))
}
// runSetup serves the local setup wizard until interrupted. It binds the
// loopback interface only.
func runSetup(args []string) int {
fs := flag.NewFlagSet("worker-agent setup", flag.ContinueOnError)
port := fs.Int("port", 0, "listen port (default 12700)")
noOpen := fs.Bool("no-open", false, "do not open the browser automatically")
if err := fs.Parse(args); err != nil {
return 2
}
logger := slog.New(slog.NewTextHandler(os.Stderr, nil))
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
server := setupui.New(logger, setupui.Options{
Port: *port,
OpenBrowser: func(url string) {
if *noOpen {
return
}
openBrowser(url)
},
})
listener, err := server.Listen()
if err != nil {
// A wizard may already be running on this port (left open, or a
// second terminal). If it is ours, opening the browser is the
// friendlier outcome than failing the command.
if existing := wizardAlreadyRunning(*port); existing != "" {
logger.Info("the setup wizard is already running", "url", existing)
if !*noOpen {
openBrowser(existing)
}
return 0
}
logger.Error("setup wizard could not bind the loopback port", "err", err)
return 1
}
url := "http://" + listener.Addr().String()
logger.Info("SciMesh worker setup wizard", "url", url, "press-ctrl-c-to-stop", true)
server.OpenBrowser(url)
// Block until the signal arrives (never returns an error that matters: a
// cancelled context is the normal exit path).
err = server.Serve(ctx, listener)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
logger.Error("setup wizard stopped", "err", err)
return 1
}
return 0
}
// openBrowser points the user's default browser at the wizard. Best-effort:
// a missing browser must never fail the setup flow.
func openBrowser(url string) {
for _, candidate := range [][]string{
{"xdg-open", url},
{"open", url},
{"cmd", "/c", "start", url},
} {
binary, err := exec.LookPath(candidate[0])
if err != nil {
continue
}
//nolint:gosec // G204: candidates are our own fixed list; the url is a loopback literal
_ = exec.CommandContext(context.Background(), binary, candidate[1:]...).Start()
return
}
}
// wizardAlreadyRunning probes the requested loopback port and returns its URL
// when it serves the setup wizard page, or "" when it does not (another
// process, or nothing at all).
func wizardAlreadyRunning(port int) string {
url := fmt.Sprintf("http://127.0.0.1:%d/", port)
client := http.Client{Timeout: 2 * time.Second}
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, url, nil)
if err != nil {
return ""
}
resp, err := client.Do(req)
if err != nil {
return ""
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return ""
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<16))
if err != nil || !strings.Contains(string(body), "SciMesh Worker · Setup") {
return ""
}
return url
}
+213
View File
@@ -0,0 +1,213 @@
package agent
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os/exec"
"runtime"
"strings"
"time"
guuid "github.com/google/uuid"
)
// CheckItem is one line of the preflight report the setup wizard shows.
type CheckItem struct {
Name string `json:"name"`
OK bool `json:"ok"`
Detail string `json:"detail,omitempty"`
Latency int64 `json:"latency_ms,omitempty"`
}
// CheckReport is the full preflight result of `worker-agent --check` and of
// the wizard's test step.
type CheckReport struct {
Coordinator CheckItem `json:"coordinator"`
Auth CheckItem `json:"auth"`
Python CheckItem `json:"python"`
Scimesh CheckItem `json:"scimesh"`
Agent string `json:"agent_version"`
CoordinatorVersion string `json:"coordinator_version,omitempty"`
}
// checkHTTP runs one GET and reports reachability + latency, with a fallback
// detail message when the server answers without JSON.
func checkHTTP(ctx context.Context, url string, timeout time.Duration) (CheckItem, string) {
started := time.Now()
reqCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
req, err := http.NewRequestWithContext(reqCtx, http.MethodGet, url, nil)
if err != nil {
return CheckItem{Name: "coordinator", OK: false, Detail: "invalid URL"}, ""
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
detail := err.Error()
if strings.Contains(detail, "connection refused") {
detail = "no coordinator answering at this address"
}
return CheckItem{Name: "coordinator", OK: false, Detail: detail}, ""
}
defer func() { _ = resp.Body.Close() }()
version := ""
if resp.StatusCode == http.StatusOK {
var body struct {
Status string `json:"status"`
}
if err := json.NewDecoder(resp.Body).Decode(&body); err == nil && body.Status == "ok" {
return CheckItem{Name: "coordinator", OK: true, Latency: time.Since(started).Milliseconds()}, version
}
}
return CheckItem{Name: "coordinator", OK: false, Detail: fmt.Sprintf("HTTP %d", resp.StatusCode)}, version
}
// CheckCoordinator probes the coordinator's /health endpoint.
func CheckCoordinator(ctx context.Context, url string, timeout time.Duration) CheckReport {
report := CheckReport{Agent: Version}
item, _ := checkHTTP(ctx, strings.TrimRight(url, "/")+"/health", timeout)
report.Coordinator = item
report.Auth = CheckItem{Name: "auth", OK: true, Detail: "no token configured — will be checked at registration"}
return report
}
// CheckEnvironment verifies the local runtime against the python3 found on
// PATH.
func CheckEnvironment(ctx context.Context) CheckReport {
python, err := exec.LookPath("python3")
if err != nil {
return CheckReport{Agent: Version, Python: CheckItem{Name: "python", OK: false, Detail: "python3 not found on PATH"}}
}
return CheckEnvironmentWithPython(ctx, python)
}
// CheckEnvironmentWithPython verifies the local runtime against a specific
// interpreter — the wizard's managed venv python when the runtime installer
// has created one, so the preflight reflects what the worker will actually
// execute with.
func CheckEnvironmentWithPython(ctx context.Context, python string) CheckReport {
report := CheckReport{Agent: Version, Python: CheckItem{Name: "python", OK: true, Detail: python}}
//nolint:gosec // G204: python is a resolved interpreter path, the argument list is constant
cmd := exec.CommandContext(ctx, python, "-c", "import scimesh; print(scimesh.__version__ if hasattr(scimesh, '__version__') else 'installed')")
out, err := cmd.Output()
if err != nil {
// The worker executes workloads by spawning scimesh's task runner, so
// the package is a hard requirement, not an optimisation. The PyPI
// name belongs to a different project, so the wizard installs from
// SCIMESH_PIP_PACKAGE instead of suggesting a bare pip install.
report.Scimesh = CheckItem{Name: "scimesh", OK: false, Detail: "the worker runs workloads through scimesh — install it from your wheel or index (SCIMESH_PIP_PACKAGE)"}
return report
}
report.Scimesh = CheckItem{Name: "scimesh", OK: true, Detail: strings.TrimSpace(string(out))}
return report
}
// RunCheck combines the coordinator probe and the local environment probe; it
// is the body behind `worker-agent --check` and the wizard's test step. A
// non-empty python overrides the interpreter probed for the scimesh package
// (the managed venv after a runtime install).
// RunCheck combines the coordinator probe, a credential probe and the local
// environment probe; it is the body behind `worker-agent --check` and the
// wizard's test step. A non-empty python overrides the interpreter probed for
// the scimesh package (the managed venv after a runtime install).
func RunCheck(ctx context.Context, coordinatorURL, python, token, workerKey, userserviceURL string) CheckReport {
report := CheckCoordinator(ctx, coordinatorURL, 15*time.Second)
report.Auth = CheckAuth(ctx, coordinatorURL, token, workerKey, userserviceURL)
var env CheckReport
if python != "" {
env = CheckEnvironmentWithPython(ctx, python)
} else {
env = CheckEnvironment(ctx)
}
report.Python = env.Python
report.Scimesh = env.Scimesh
return report
}
// Version is the agent build version; main injects it via -ldflags and the
// setup wizard mirrors it into the report. "dev" marks a local build.
var Version = "dev"
// Platform is the host platform string shown on the wizard.
func Platform() string { return runtime.GOOS + "/" + runtime.GOARCH }
// CheckAuth verifies the configured credential against the coordinator
// without mutating anything: with a worker key it first exchanges it at the
// userservice for a short-lived JWT, then it probes /tasks/claim with a
// throwaway worker id and no capabilities. A 401 anywhere means the
// credential was rejected; any other status proves it was accepted.
func CheckAuth(ctx context.Context, url, token, workerKey, userserviceURL string) CheckItem {
item := CheckItem{Name: "auth"}
if token == "" && workerKey == "" {
item.OK = true
item.Detail = "no credential configured — will be checked at registration"
return item
}
client := &http.Client{Timeout: 30 * time.Second}
if workerKey != "" && userserviceURL != "" {
payload, _ := json.Marshal(map[string]string{"key": workerKey})
req, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(userserviceURL, "/")+"/worker-tokens/exchange", strings.NewReader(string(payload)))
if err != nil {
item.OK = false
item.Detail = "invalid userservice URL"
return item
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
item.OK = false
item.Detail = "userservice unreachable: " + err.Error()
return item
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode == http.StatusUnauthorized {
item.OK = false
item.Detail = "worker key rejected by the userservice"
return item
}
if resp.StatusCode != http.StatusOK {
item.OK = false
item.Detail = fmt.Sprintf("userservice exchange: HTTP %d", resp.StatusCode)
return item
}
var exchanged struct {
Token string `json:"token"`
}
if err := json.NewDecoder(resp.Body).Decode(&exchanged); err != nil || exchanged.Token == "" {
item.OK = false
item.Detail = "userservice exchange returned no token"
return item
}
token = exchanged.Token
}
if token == "" {
item.OK = false
item.Detail = "no usable credential after the key exchange"
return item
}
payload, _ := json.Marshal(map[string]any{"worker_id": guuid.NewString(), "capabilities": []string{}})
req, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(url, "/")+"/tasks/claim", strings.NewReader(string(payload)))
if err != nil {
item.OK = false
item.Detail = "invalid coordinator URL"
return item
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+token)
resp, err := client.Do(req)
if err != nil {
item.OK = false
item.Detail = "coordinator unreachable: " + err.Error()
return item
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode == http.StatusUnauthorized {
item.OK = false
item.Detail = "token rejected by the coordinator"
return item
}
item.OK = true
item.Detail = "credential accepted"
return item
}
+68
View File
@@ -0,0 +1,68 @@
package agent
import (
"context"
"net/http"
"net/http/httptest"
"testing"
)
func TestCheckAuthAcceptsToken(t *testing.T) {
stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/tasks/claim" && r.Header.Get("Authorization") == "Bearer good-token" {
w.WriteHeader(http.StatusNoContent)
return
}
w.WriteHeader(http.StatusUnauthorized)
}))
defer stub.Close()
item := CheckAuth(context.Background(), stub.URL, "good-token", "", "")
if !item.OK {
t.Errorf("good token: %+v", item)
}
item = CheckAuth(context.Background(), stub.URL, "bad-token", "", "")
if item.OK {
t.Error("bad token must fail")
}
}
func TestCheckAuthNoCredentialIsNotAFailure(t *testing.T) {
item := CheckAuth(context.Background(), "http://coord:8080", "", "", "")
if !item.OK {
t.Errorf("no credential must not fail the preflight: %+v", item)
}
}
func TestCheckAuthWorkerKeyExchange(t *testing.T) {
var exchanged bool
users := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/worker-tokens/exchange" {
t.Errorf("unexpected userservice path %q", r.URL.Path)
}
exchanged = true
_, _ = w.Write([]byte(`{"token":"jwt-after-exchange"}`))
}))
defer users.Close()
coord := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") == "Bearer jwt-after-exchange" {
w.WriteHeader(http.StatusNoContent)
return
}
w.WriteHeader(http.StatusUnauthorized)
}))
defer coord.Close()
item := CheckAuth(context.Background(), coord.URL, "", "smk_key", users.URL)
if !item.OK || !exchanged {
t.Errorf("key exchange flow: %+v exchanged=%v", item, exchanged)
}
rejected := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
}))
defer rejected.Close()
if item := CheckAuth(context.Background(), coord.URL, "", "smk_bad", rejected.URL); item.OK {
t.Error("rejected key must fail the preflight")
}
}
+17
View File
@@ -32,6 +32,10 @@ type Config struct {
TaskRunner []string // command + args; defaults to python -m scimesh.worker.task
MaxTasks int // 0 = unlimited
ExitWhenIdle bool
// Concurrency is how many claim→execute→upload loops run in parallel
// under one worker id: N shards processed concurrently on one machine,
// using the coordinator's own task pipeline as the parallel unit.
Concurrency int
}
func envList(name string) ([]string, error) {
@@ -145,9 +149,22 @@ func LoadConfig() (*Config, error) {
TaskRunner: runner,
MaxTasks: maxTasks,
ExitWhenIdle: os.Getenv("EXIT_WHEN_IDLE") == "1",
Concurrency: envInt("WORKER_CONCURRENCY", 1),
}, nil
}
func envInt(name string, fallback int) int {
raw := os.Getenv(name)
if raw == "" {
return fallback
}
parsed, err := strconv.Atoi(raw)
if err != nil || parsed < 1 {
return fallback
}
return parsed
}
func durationEnv(name string, fallback time.Duration) (time.Duration, error) {
raw := os.Getenv(name)
if raw == "" {
+156
View File
@@ -0,0 +1,156 @@
package agent
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// ConfigFile is the persisted worker configuration written by the setup
// wizard and read back by `worker-agent --config`. Environment variables
// still win: the file fills in what the environment left unset.
type ConfigFile struct {
CoordinatorURL string `json:"coordinator_url"`
Token string `json:"token,omitempty"`
WorkerKey string `json:"worker_key,omitempty"`
UserserviceURL string `json:"userservice_url,omitempty"`
WorkDir string `json:"work_dir"`
WorkerName string `json:"worker_name,omitempty"`
CPUCount int `json:"cpu_count"`
MemoryMB int `json:"memory_mb"`
Concurrency int `json:"concurrency,omitempty"`
TaskRunner []string `json:"task_runner,omitempty"`
}
// DefaultConfigPath is where the setup wizard stores the worker's
// configuration. SCIMESH_WORKER_CONFIG overrides it.
func DefaultConfigPath() string {
if raw := os.Getenv("SCIMESH_WORKER_CONFIG"); raw != "" {
return raw
}
home, err := os.UserHomeDir()
if err != nil || home == "" {
return filepath.Join(".", ".scimesh-worker", "config.json")
}
return filepath.Join(home, ".scimesh-worker", "config.json")
}
// LoadConfigFile reads and validates a persisted configuration. The file is
// created by the wizard with 0600 permissions, so no credential is exposed to
// other local users.
func LoadConfigFile(path string) (*Config, error) {
//nolint:gosec // G304: path is --config or SCIMESH_WORKER_CONFIG, operator-supplied
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read config file: %w", err)
}
var file ConfigFile
if err := json.Unmarshal(raw, &file); err != nil {
return nil, fmt.Errorf("parse config file: %w", err)
}
return file.Config()
}
// Config turns the file into the daemon configuration. Environment variables
// take precedence so operators can still override any value per-process.
func (f *ConfigFile) Config() (*Config, error) {
config := &Config{}
if env := os.Getenv("COORDINATOR_URL"); env != "" {
config.CoordinatorURL = env
} else {
config.CoordinatorURL = strings.TrimSpace(f.CoordinatorURL)
}
if config.CoordinatorURL == "" {
return nil, fmt.Errorf("coordinator_url is required")
}
if !strings.HasPrefix(config.CoordinatorURL, "http://") && !strings.HasPrefix(config.CoordinatorURL, "https://") {
return nil, fmt.Errorf("coordinator_url must be an absolute HTTP(S) URL")
}
if env := os.Getenv("WORKER_AUTH_TOKEN"); env != "" {
config.Token = env
} else {
config.Token = f.Token
}
if env := os.Getenv("WORKER_KEY"); env != "" {
config.WorkerKey = env
} else {
config.WorkerKey = f.WorkerKey
}
if env := os.Getenv("USERSERVICE_URL"); env != "" {
config.UserserviceURL = env
} else {
config.UserserviceURL = f.UserserviceURL
}
if env := os.Getenv("WORK_DIR"); env != "" {
config.WorkDir = env
} else if f.WorkDir != "" {
config.WorkDir = f.WorkDir
} else {
config.WorkDir = "./scimesh-agent-data"
}
if env := os.Getenv("WORKER_NAME"); env != "" {
config.WorkerName = env
} else {
config.WorkerName = f.WorkerName
}
config.CPUCount = f.CPUCount
if config.CPUCount < 1 {
config.CPUCount = 1
}
config.MemoryMB = f.MemoryMB
if config.MemoryMB < 0 {
config.MemoryMB = 0
}
config.Concurrency = f.Concurrency
if config.Concurrency < 1 {
config.Concurrency = 1
}
if len(f.TaskRunner) > 0 {
config.TaskRunner = f.TaskRunner
}
if len(config.TaskRunner) == 0 {
config.TaskRunner = []string{"python", "-m", "scimesh.worker.task"}
}
config.PollInterval = 2 * time.Second
config.RequestTimeout = 30 * time.Second
config.Heartbeat = 15 * time.Second
config.Capabilities = DefaultCapabilities()
return config, nil
}
// Save writes the configuration file, creating the parent directory and
// restricting permissions to the owner.
func SaveConfigFile(path string, file ConfigFile) error {
payload, err := json.MarshalIndent(file, "", " ")
if err != nil {
return err
}
payload = append(payload, '\n')
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return fmt.Errorf("create config directory: %w", err)
}
if err := os.WriteFile(path, payload, 0o600); err != nil {
return fmt.Errorf("write config file: %w", err)
}
return nil
}
// VenvPython returns the managed venv python next to the given config file,
// or "" when the runtime installer has not created one yet. Mirrors the
// wizard's probe so `worker-agent --check` and the preflight agree on what
// interpreter will actually execute workloads.
func VenvPython(configPath string) string {
dir := filepath.Dir(configPath)
for _, candidate := range []string{
filepath.Join(dir, "venv", "bin", "python"),
filepath.Join(dir, "venv", "Scripts", "python.exe"),
} {
if info, err := os.Stat(candidate); err == nil && !info.IsDir() {
return candidate
}
}
return ""
}
+48 -7
View File
@@ -37,15 +37,52 @@ func NewDaemon(config *Config, client *Client, runner *TaskRunner, log *slog.Log
return &Daemon{config: config, client: client, runner: runner, log: log}
}
// RunForever loops until interrupted, idle-exit, or max tasks.
// RunForever registers once, then runs the claim→execute→upload loop
// concurrently under one worker id. With Concurrency > 1, several shards are
// processed in parallel on this machine, using the coordinator's own task
// pipeline as the parallel unit.
func (d *Daemon) RunForever() error {
if !d.registered {
if err := d.register(); err != nil {
return err
}
}
workers := d.config.Concurrency
if workers < 1 {
workers = 1
}
if workers == 1 {
return d.loop()
}
d.log.Info("agent running concurrently", "loops", workers)
var wg sync.WaitGroup
errs := make(chan error, workers)
for i := 0; i < workers; i++ {
wg.Add(1)
go func(loop int) {
defer wg.Done()
if err := d.loop(); err != nil {
errs <- err
return
}
errs <- nil
}(i)
}
wg.Wait()
close(errs)
for err := range errs {
if err != nil {
return err
}
}
return nil
}
// loop is one claim→execute→upload cycle until interrupted, idle-exit, or
// the shared max-tasks budget is consumed.
func (d *Daemon) loop() error {
failures := 0
for {
if !d.registered {
if err := d.register(); err != nil {
return err
}
}
d.cleanupExpiredDirectories()
outcome, err := d.runOnce()
if err != nil {
@@ -63,8 +100,11 @@ func (d *Daemon) RunForever() error {
}
failures = 0
if outcome.Claimed && outcome.Completed {
d.mu.Lock()
d.completed++
if d.config.MaxTasks > 0 && d.completed >= d.config.MaxTasks {
done := d.config.MaxTasks > 0 && d.completed >= d.config.MaxTasks
d.mu.Unlock()
if done {
d.log.Info("max tasks reached")
return nil
}
@@ -158,6 +198,7 @@ func (d *Daemon) runOnce() (Outcome, error) {
if task == nil {
return Outcome{Claimed: false}, nil
}
d.log.Info("task claimed", "task_id", task.TaskID, "attempt", task.Attempt)
started := time.Now()
taskDir := filepath.Join(d.config.WorkDir, task.TaskID, fmt.Sprint(task.Attempt))
if err := os.MkdirAll(taskDir, 0o750); err != nil {
+79
View File
@@ -11,6 +11,7 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
@@ -55,6 +56,7 @@ type fakeCoordinator struct {
uploadSize int64
inputBytes []byte
conflict bool // 409 on heartbeat/upload/result
mu sync.Mutex
}
func newFakeCoordinator(t *testing.T, task map[string]any) *fakeCoordinator {
@@ -64,6 +66,8 @@ func newFakeCoordinator(t *testing.T, task map[string]any) *fakeCoordinator {
fake.uploadSize = int64(len(fake.inputBytes))
var server *httptest.Server
server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fake.mu.Lock()
defer fake.mu.Unlock()
switch {
case r.Method == http.MethodPost && r.URL.Path == "/workers/register":
writeJSON(w, http.StatusCreated, map[string]any{
@@ -294,3 +298,78 @@ func TestDaemonIdleClaimIsNotCompleted(t *testing.T) {
t.Fatalf("outcome = %+v", outcome)
}
}
func TestDaemonConcurrencyProcessesTasksInParallel(t *testing.T) {
t.Parallel()
marker := filepath.Join(t.TempDir(), "marker")
script := filepath.Join(t.TempDir(), "fake-runner.sh")
content := `#!/bin/sh
out=""
task_dir=""
while [ "$#" -gt 0 ]; do
case "$1" in
--output) out="$2"; shift 2;;
--task-dir) task_dir="$2"; shift 2;;
*) shift;;
esac
done
echo start >> ` + marker + `
sleep 1
echo end >> ` + marker + `
printf 'id,score\n1,1\n' > "$task_dir/result.csv"
printf '{"artifact_path":"%s/result.csv","content_type":"text/csv","metrics":{"rows":1}}' "$task_dir" > "$out"
exit 0
`
if err := os.WriteFile(script, []byte(content), 0o755); err != nil {
t.Fatal(err)
}
fake := newFakeCoordinator(t, validClaimedTaskPayload())
defer fake.close()
config := &Config{
CoordinatorURL: fake.server.URL,
WorkerName: "concurrent-worker",
WorkerID: "22222222-2222-4222-8222-222222222222",
WorkDir: t.TempDir(),
CPUCount: 1,
PollInterval: time.Millisecond,
RequestTimeout: 5 * time.Second,
Heartbeat: 15 * time.Second,
Capabilities: []string{"similarity-search"},
TaskRunner: []string{script},
MaxTasks: 3,
Concurrency: 3,
}
client := NewClient(fake.server.URL, &StaticToken{token: "test-token"}, 5*time.Second)
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
daemon := NewDaemon(config, client, NewTaskRunner(config.TaskRunner), logger)
if err := daemon.RunForever(); err != nil {
t.Fatalf("run: %v", err)
}
raw, err := os.ReadFile(marker)
if err != nil {
t.Fatalf("marker: %v", err)
}
starts := strings.Count(string(raw), "start\n")
ends := strings.Count(string(raw), "end\n")
if starts < 3 || ends < 3 {
t.Fatalf("marker: %d starts / %d ends, want at least 3/3", starts, ends)
}
// With three loops sleeping 1s each, the marker proves all three ran
// concurrently (three starts before the first end completes a 1s sleep).
lines := strings.Split(strings.TrimSpace(string(raw)), "\n")
concurrent := 0
running := 0
for _, line := range lines {
if line == "start" {
running++
if running > concurrent {
concurrent = running
}
} else {
running--
}
}
if concurrent < 3 {
t.Errorf("max concurrent executions = %d, want 3", concurrent)
}
}
@@ -0,0 +1,73 @@
package agent
import (
"context"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"time"
)
// ReleaseWheelURL returns the download URL of the scimesh wheel attached to
// the GitHub release that matches the given binary version (for example
// "1.1.0-alpha.10"), plus the wheel file name. The wheel is version-locked to
// the binary so a worker's catalog always matches its task runner.
func ReleaseWheelURL(version string) (string, string, error) {
if version == "" || version == "dev" {
return "", "", fmt.Errorf("no release wheel for build %q", version)
}
filename := fmt.Sprintf("scimesh-%s-py3-none-any.whl", NormalizePEP440(version))
return fmt.Sprintf("https://github.com/emil28092005/SciMesh/releases/download/v%s/%s", version, filename), filename, nil
}
// NormalizePEP440 turns our release tag suffixes into the PEP 440 form
// setuptools uses for wheel names: 1.1.0-alpha.10 -> 1.1.0a10,
// 1.1.0-beta.2 -> 1.1.0b2, 1.1.0-rc.1 -> 1.1.0rc1. Stable tags pass through.
func NormalizePEP440(version string) string {
for from, to := range map[string]string{"-alpha.": "a", "-beta.": "b", "-rc.": "rc"} {
version = strings.ReplaceAll(version, from, to)
}
return version
}
// DownloadWheel fetches the release wheel into dir (config directory of the
// wizard / serve data dir) and returns the local path. Best-effort download
// with a generous timeout: wheels can be several MB.
func DownloadWheel(ctx context.Context, url, dir string) (string, error) {
target := filepath.Join(dir, wheelNameFromURL(url))
if err := os.MkdirAll(dir, 0o700); err != nil {
return "", fmt.Errorf("download wheel: %w", err)
}
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", fmt.Errorf("download wheel: %w", err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("download wheel: HTTP %d", resp.StatusCode)
}
//nolint:gosec // G304: target is our own config dir + a fixed wheel name
out, err := os.Create(target)
if err != nil {
return "", fmt.Errorf("download wheel: %w", err)
}
defer func() { _ = out.Close() }()
if _, err := io.Copy(out, resp.Body); err != nil {
return "", fmt.Errorf("download wheel: %w", err)
}
return target, nil
}
// wheelNameFromURL extracts the trailing file name of a wheel URL.
func wheelNameFromURL(url string) string {
return url[strings.LastIndex(url, "/")+1:]
}
@@ -0,0 +1,81 @@
package agent
import (
"context"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
)
func TestNormalizePEP440(t *testing.T) {
cases := map[string]string{
"1.1.0": "1.1.0",
"1.1.0-alpha.10": "1.1.0a10",
"1.1.0-beta.2": "1.1.0b2",
"1.1.0-rc.1": "1.1.0rc1",
"1.0.0": "1.0.0",
}
for in, want := range cases {
if got := NormalizePEP440(in); got != want {
t.Errorf("NormalizePEP440(%q) = %q, want %q", in, got, want)
}
}
}
func TestReleaseWheelURL(t *testing.T) {
url, name, err := ReleaseWheelURL("1.1.0-alpha.10")
if err != nil {
t.Fatal(err)
}
wantURL := "https://github.com/emil28092005/SciMesh/releases/download/v1.1.0-alpha.10/scimesh-1.1.0a10-py3-none-any.whl"
if url != wantURL {
t.Errorf("url = %q, want %q", url, wantURL)
}
if name != "scimesh-1.1.0a10-py3-none-any.whl" {
t.Errorf("name = %q", name)
}
// A dev build has no release wheel.
if _, _, err := ReleaseWheelURL("dev"); err == nil {
t.Error("dev build must not resolve a release wheel")
}
if _, _, err := ReleaseWheelURL(""); err == nil {
t.Error("empty version must not resolve a release wheel")
}
}
func TestDownloadWheel(t *testing.T) {
payload := []byte("fake wheel bytes")
stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(payload)
}))
defer stub.Close()
dir := t.TempDir()
path, err := DownloadWheel(context.Background(), stub.URL+"/scimesh-1.1.0a10-py3-none-any.whl", dir)
if err != nil {
t.Fatal(err)
}
if !strings.HasSuffix(path, "scimesh-1.1.0a10-py3-none-any.whl") {
t.Errorf("path = %q", path)
}
got, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if string(got) != string(payload) {
t.Error("wheel bytes mismatch")
}
}
func TestDownloadWheelReportsHTTPErrors(t *testing.T) {
stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.NotFound(w, r)
}))
defer stub.Close()
if _, err := DownloadWheel(context.Background(), stub.URL+"/missing.whl", t.TempDir()); err == nil {
t.Error("404 must fail the download")
}
}
@@ -0,0 +1,630 @@
// Package setupui serves the local worker setup wizard: a small HTTP server
// bound to 127.0.0.1 that writes the worker's config file, runs preflight
// checks, and starts/stops the worker as a background process. It is part of
// the worker-agent binary so a machine that installs only a worker never needs
// a coordinator.
package setupui
import (
"context"
"embed"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"github.com/emil28092005/SciMesh/coordinator/internal/agent"
)
//go:embed template.html
var templateFS embed.FS
const (
defaultPort = 12700
pidFileName = "worker.pid"
logFileName = "worker.log"
)
// Supervisor starts and stops the worker process and tracks its pid. It is an
// interface so tests can substitute a fake.
type Supervisor interface {
// Start launches `worker-agent --config <path>` detached, writing output
// into the log file. Returns the child pid.
Start(configPath, logPath string) (int, error)
// Stop terminates the process recorded in the pid file.
Stop() error
// Pid returns the recorded child pid, or 0 when none is recorded.
Pid() int
// Alive reports whether the recorded child is still running.
Alive() bool
}
// PIDSupervisor is the real Supervisor: it spawns the running binary with
// --config and manages its pid file. Liveness comes from a Wait goroutine, so
// it works on every platform (no signal probing, which Windows lacks).
type PIDSupervisor struct {
mu sync.Mutex
pidPath string
proc *os.Process
done chan struct{} // closed when the spawned process exits; nil when not started
}
func NewPIDSupervisor(pidPath string) *PIDSupervisor { return &PIDSupervisor{pidPath: pidPath} }
func (s *PIDSupervisor) Pid() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.readPid()
}
func (s *PIDSupervisor) readPid() int {
raw, err := os.ReadFile(s.pidPath)
if err != nil {
return 0
}
pid, err := strconv.Atoi(strings.TrimSpace(string(raw)))
if err != nil || pid < 1 {
return 0
}
return pid
}
func (s *PIDSupervisor) Alive() bool {
s.mu.Lock()
defer s.mu.Unlock()
if s.done == nil {
return false
}
select {
case <-s.done:
return false
default:
return true
}
}
func (s *PIDSupervisor) Start(configPath, logPath string) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.done != nil {
select {
case <-s.done:
default:
return s.readPid(), fmt.Errorf("worker is already running (pid %d)", s.readPid())
}
}
exe, err := os.Executable()
if err != nil {
return 0, fmt.Errorf("resolve worker binary: %w", err)
}
//nolint:gosec // G304: logPath lives in the wizard's own config directory
logFile, err := os.OpenFile(logPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o600)
if err != nil {
return 0, fmt.Errorf("open worker log: %w", err)
}
defer func() { _ = logFile.Close() }()
//nolint:gosec // G204: exe is os.Executable, configPath is the wizard's own file;
// Background ctx: the child's lifecycle is managed by the supervisor, not the context
cmd := exec.CommandContext(context.Background(), exe, "--config", configPath)
cmd.Stdout = logFile
cmd.Stderr = logFile
cmd.Stdin = nil
if err := cmd.Start(); err != nil {
return 0, fmt.Errorf("start worker: %w", err)
}
// The child inherits our stdout/stderr descriptors pointing at the log
// file, so we can close our copy; the child keeps it open.
_ = logFile.Close()
s.proc = cmd.Process
s.done = make(chan struct{})
go func() { _ = cmd.Wait(); close(s.done) }()
if err := os.WriteFile(s.pidPath, []byte(strconv.Itoa(cmd.Process.Pid)+"\n"), 0o600); err != nil {
_ = cmd.Process.Kill()
return 0, fmt.Errorf("write pid file: %w", err)
}
return cmd.Process.Pid, nil
}
func (s *PIDSupervisor) Stop() error {
s.mu.Lock()
defer s.mu.Unlock()
if s.done == nil {
return nil
}
select {
case <-s.done:
s.done = nil
s.proc = nil
_ = os.Remove(s.pidPath)
return nil
default:
}
// Ask politely, then force. os.Interrupt terminates on Windows too.
_ = s.proc.Signal(os.Interrupt)
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
select {
case <-s.done:
s.done = nil
s.proc = nil
_ = os.Remove(s.pidPath)
return nil
default:
time.Sleep(100 * time.Millisecond)
}
}
_ = s.proc.Kill()
select {
case <-s.done:
case <-time.After(2 * time.Second):
}
s.done = nil
s.proc = nil
_ = os.Remove(s.pidPath)
return nil
}
// Server is the wizard HTTP server, bound to the loopback interface only.
type Server struct {
log *slog.Logger
cfgPath string
logPath string
dir string
sup Supervisor
openBrowser func(string)
port int
install func(ctx context.Context, venvPython, pkg string) error
downloadWheel func(ctx context.Context, url, dir string) (string, error)
}
// Options customises the wizard for tests and embedding.
type Options struct {
Port int
ConfigPath string
OpenBrowser func(url string)
Supervisor Supervisor
Dir string // directory for pid/log files; defaults to the config dir
// InstallScimesh overrides the pip step of the runtime installer (tests
// substitute a fake); nil uses the real pip inside the managed venv.
InstallScimesh func(ctx context.Context, venvPython, pkg string) error
// DownloadWheel overrides the release-wheel download (tests substitute a
// fake); nil downloads from the GitHub release matching the agent version.
DownloadWheel func(ctx context.Context, url, dir string) (string, error)
}
func New(log *slog.Logger, opts Options) *Server {
cfgPath := opts.ConfigPath
if cfgPath == "" {
cfgPath = agent.DefaultConfigPath()
}
dir := opts.Dir
if dir == "" {
dir = filepath.Dir(cfgPath)
}
sup := opts.Supervisor
if sup == nil {
sup = NewPIDSupervisor(filepath.Join(dir, pidFileName))
}
open := opts.OpenBrowser
if open == nil {
open = func(string) {}
}
port := opts.Port
if port == 0 {
port = defaultPort
}
install := opts.InstallScimesh
if install == nil {
install = installScimeshWithPip
}
downloadWheel := opts.DownloadWheel
if downloadWheel == nil {
downloadWheel = agent.DownloadWheel
}
return &Server{log: log, cfgPath: cfgPath, logPath: filepath.Join(dir, logFileName), dir: dir, sup: sup, openBrowser: open, port: port, install: install, downloadWheel: downloadWheel}
}
// Listen binds the loopback listener and returns it; Serve runs the server on
// it. Split so tests can inspect the actual ephemeral port.
func (s *Server) Listen() (net.Listener, error) {
return (&net.ListenConfig{}).Listen(context.Background(), "tcp", fmt.Sprintf("127.0.0.1:%d", s.port))
}
// OpenBrowser hands the wizard URL to the configured opener (default: no-op).
func (s *Server) OpenBrowser(url string) { s.openBrowser(url) }
// Serve runs the wizard until ctx is cancelled.
func (s *Server) Serve(ctx context.Context, listener net.Listener) error {
mux := http.NewServeMux()
mux.HandleFunc("GET /", s.handleIndex)
mux.HandleFunc("GET /api/status", s.handleStatus)
mux.HandleFunc("POST /api/config", s.handleSaveConfig)
mux.HandleFunc("POST /api/test", s.handleTest)
mux.HandleFunc("POST /api/runtime/install", s.handleInstallRuntime)
mux.HandleFunc("POST /api/start", s.handleStart)
mux.HandleFunc("POST /api/stop", s.handleStop)
mux.HandleFunc("GET /api/logs", s.handleLogs)
server := &http.Server{
Handler: mux,
ReadHeaderTimeout: 10 * time.Second,
}
go func() {
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
defer cancel()
_ = server.Shutdown(shutdownCtx)
}()
return server.Serve(listener)
}
func (s *Server) handleIndex(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.NotFound(w, r)
return
}
html, err := templateFS.ReadFile("template.html")
if err != nil {
http.Error(w, "template unavailable", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write(html)
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
// statusView is what the wizard needs to paint the running/stopped state.
type statusView struct {
ConfigPresent bool `json:"config_present"`
ConfigPath string `json:"config_path"`
LogPath string `json:"log_path"`
Running bool `json:"running"`
Pid int `json:"pid"`
WorkerName string `json:"worker_name,omitempty"`
Coordinator string `json:"coordinator,omitempty"`
WorkDir string `json:"work_dir,omitempty"`
TokenSet bool `json:"token_set"`
Stats WorkerStats `json:"stats"`
}
// WorkerStats is parsed from the worker log: the agent reports each claim,
// completion and failure as a structured line, so the wizard can show live
// counters without any coordinator access.
type WorkerStats struct {
Registered bool `json:"registered"`
Claimed int `json:"claimed"`
Completed int `json:"completed"`
Failed int `json:"failed"`
}
// ensureVenvTaskRunner rewrites the saved config so its task runner uses the
// managed venv python when one exists and the config does not already pin one.
func (s *Server) ensureVenvTaskRunner() {
raw, err := os.ReadFile(s.cfgPath)
if err != nil {
return
}
var file agent.ConfigFile
if json.Unmarshal(raw, &file) != nil || len(file.TaskRunner) > 0 {
return
}
if venv := s.venvPython(); venv != "" {
file.TaskRunner = []string{venv, "-m", "scimesh.worker.task"}
if payload, err := json.MarshalIndent(file, "", " "); err == nil {
_ = os.WriteFile(s.cfgPath, append(payload, '\n'), 0o600)
}
}
}
// parseWorkerStats counts the structured agent events in the worker log.
func parseWorkerStats(logPath string) WorkerStats {
var stats WorkerStats
//nolint:gosec // G304: logPath is the wizard's own log file next to the config
raw, err := os.ReadFile(logPath)
if err != nil {
return stats
}
for _, line := range strings.Split(string(raw), "\n") {
switch {
case strings.Contains(line, "msg=registered"):
stats.Registered = true
case strings.Contains(line, `msg="task claimed"`):
stats.Claimed++
case strings.Contains(line, `msg="task completed"`):
stats.Completed++
case strings.Contains(line, `msg="task failed"`):
stats.Failed++
}
}
return stats
}
func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) {
view := statusView{ConfigPath: s.cfgPath, LogPath: s.logPath, Running: s.sup.Alive(), Pid: s.sup.Pid(), Stats: parseWorkerStats(s.logPath)}
if raw, err := os.ReadFile(s.cfgPath); err == nil {
var file agent.ConfigFile
if json.Unmarshal(raw, &file) == nil {
view.ConfigPresent = true
view.WorkerName = file.WorkerName
view.Coordinator = file.CoordinatorURL
view.WorkDir = file.WorkDir
view.TokenSet = file.Token != "" || file.WorkerKey != ""
}
}
writeJSON(w, http.StatusOK, view)
}
type saveConfigRequest struct {
CoordinatorURL string `json:"coordinator_url"`
Token string `json:"token"`
WorkerKey string `json:"worker_key"`
UserserviceURL string `json:"userservice_url"`
WorkDir string `json:"work_dir"`
WorkerName string `json:"worker_name"`
CPUCount int `json:"cpu_count"`
MemoryMB int `json:"memory_mb"`
Concurrency int `json:"concurrency"`
TaskRunner []string `json:"task_runner"`
}
func (s *Server) handleSaveConfig(w http.ResponseWriter, r *http.Request) {
var req saveConfigRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&req); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON body"})
return
}
file := agent.ConfigFile{
CoordinatorURL: strings.TrimSpace(req.CoordinatorURL),
Token: req.Token,
WorkerKey: req.WorkerKey,
UserserviceURL: strings.TrimSpace(req.UserserviceURL),
WorkDir: strings.TrimSpace(req.WorkDir),
WorkerName: strings.TrimSpace(req.WorkerName),
CPUCount: req.CPUCount,
MemoryMB: req.MemoryMB,
Concurrency: req.Concurrency,
TaskRunner: req.TaskRunner,
}
if file.CoordinatorURL == "" {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "coordinator_url is required"})
return
}
if !strings.HasPrefix(file.CoordinatorURL, "http://") && !strings.HasPrefix(file.CoordinatorURL, "https://") {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "coordinator_url must be an absolute HTTP(S) URL"})
return
}
if file.WorkDir == "" {
file.WorkDir = "./scimesh-agent-data"
}
if file.WorkerName == "" {
if host, err := os.Hostname(); err == nil {
file.WorkerName = host
} else {
file.WorkerName = "worker"
}
}
if file.CPUCount < 1 {
file.CPUCount = 1
}
if file.Concurrency < 1 {
file.Concurrency = 1
}
// The wizard UI bakes the venv python into the runner after an install;
// an API-driven or scripted flow may not, so the server guarantees it:
// workloads execute through scimesh's task runner, which lives in the venv.
if len(file.TaskRunner) == 0 {
if venv := s.venvPython(); venv != "" {
file.TaskRunner = []string{venv, "-m", "scimesh.worker.task"}
}
}
if err := agent.SaveConfigFile(s.cfgPath, file); err != nil {
s.log.Error("save wizard config", "err", err)
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "could not write the config file"})
return
}
writeJSON(w, http.StatusOK, map[string]bool{"saved": true})
}
func (s *Server) handleTest(w http.ResponseWriter, r *http.Request) {
var req saveConfigRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&req); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON body"})
return
}
url := strings.TrimSpace(req.CoordinatorURL)
if url == "" {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "coordinator_url is required"})
return
}
// After the runtime installer created the venv, probe that interpreter:
// checking the bare system python3 would keep reporting scimesh as
// missing even though the worker would run with the venv.
report := agent.RunCheck(r.Context(), url, s.venvPython(), req.Token, req.WorkerKey, req.UserserviceURL)
writeJSON(w, http.StatusOK, report)
}
func (s *Server) handleStart(w http.ResponseWriter, r *http.Request) {
if _, err := os.Stat(s.cfgPath); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "no configuration saved yet"})
return
}
s.ensureVenvTaskRunner()
pid, err := s.sup.Start(s.cfgPath, s.logPath)
if err != nil {
writeJSON(w, http.StatusConflict, map[string]string{"error": err.Error()})
return
}
writeJSON(w, http.StatusOK, map[string]int{"pid": pid})
}
func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {
if err := s.sup.Stop(); err != nil {
writeJSON(w, http.StatusConflict, map[string]string{"error": err.Error()})
return
}
writeJSON(w, http.StatusOK, map[string]bool{"stopped": true})
}
func (s *Server) handleLogs(w http.ResponseWriter, r *http.Request) {
raw, err := os.ReadFile(s.logPath)
if err != nil {
writeJSON(w, http.StatusOK, map[string]string{"log": ""})
return
}
lines := strings.Split(strings.TrimRight(string(raw), "\n"), "\n")
tail := 200
if n, err := strconv.Atoi(r.URL.Query().Get("tail")); err == nil && n > 0 && n < 5000 {
tail = n
}
if len(lines) > tail {
lines = lines[len(lines)-tail:]
}
writeJSON(w, http.StatusOK, map[string]string{"log": strings.Join(lines, "\n")})
}
// ErrCanceled mirrors context.Canceled for callers that treat a cancelled
// wizard as a clean exit.
var ErrCanceled = errors.New("setup wizard cancelled")
type installRuntimeRequest struct {
// ScimeshPackage overrides where the scimesh wheel comes from: a local
// wheel/index path or the PyPI name. Defaults to SCIMESH_PIP_PACKAGE, then
// to the PyPI name.
ScimeshPackage string `json:"scimesh_package"`
}
type installRuntimeResponse struct {
OK bool `json:"ok"`
Python string `json:"python,omitempty"` // venv python to use as TASK_RUNNER[0]
Installed bool `json:"installed"`
}
// handleInstallRuntime creates a managed venv next to the worker config and
// installs the scimesh package into it, so the machine needs no manual pip
// step. The venv python path is returned for the wizard to bake into the
// task runner.
func (s *Server) handleInstallRuntime(w http.ResponseWriter, r *http.Request) {
// The wizard may run the install before any config was saved, so the
// worker directory (venv, wheel) may not exist yet.
if err := os.MkdirAll(s.dir, 0o700); err != nil {
writeJSON(w, http.StatusConflict, map[string]string{"error": "could not create the worker directory"})
return
}
var req installRuntimeRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&req); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON body"})
return
}
pkg := strings.TrimSpace(req.ScimeshPackage)
if pkg == "" {
pkg = os.Getenv("SCIMESH_PIP_PACKAGE")
}
if pkg == "" {
// No PyPI default on purpose: the PyPI name "scimesh" belongs to an
// unrelated project. Instead we ship the wheel in our own GitHub
// release, version-locked to this binary, and download it from there.
url, _, err := agent.ReleaseWheelURL(agent.Version)
if err != nil {
writeJSON(w, http.StatusConflict, map[string]string{
"error": "no scimesh source configured: set SCIMESH_PIP_PACKAGE to your wheel, checkout or index, then retry",
})
return
}
local, err := s.downloadWheel(r.Context(), url, s.dir)
if err != nil {
s.log.Error("download release wheel", "err", err, "url", url)
writeJSON(w, http.StatusConflict, map[string]string{
"error": "could not download the scimesh wheel for this release: " + err.Error() + ". Set SCIMESH_PIP_PACKAGE to your wheel, checkout or index and retry.",
})
return
}
pkg = local
}
python3, err := exec.LookPath("python3")
if err != nil {
writeJSON(w, http.StatusConflict, map[string]string{"error": "python3 is not installed on this machine"})
return
}
venvDir := filepath.Join(s.dir, "venv")
venvPython := filepath.Join(venvDir, "bin", "python")
if _, err := os.Stat(venvPython); err != nil {
// Windows layout: Scripts/python.exe.
if win := filepath.Join(venvDir, "Scripts", "python.exe"); stat(win) {
venvPython = win
}
}
if _, err := os.Stat(venvPython); err != nil {
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Minute)
defer cancel()
cmd := exec.CommandContext(ctx, python3, "-m", "venv", venvDir) //nolint:gosec // G204: python3 from LookPath, venvDir is our own dir
if out, err := cmd.CombinedOutput(); err != nil {
s.log.Error("create runtime venv", "err", err, "out", truncate(string(out), 500))
writeJSON(w, http.StatusConflict, map[string]string{"error": "could not create the python venv"})
return
}
}
if err := s.install(r.Context(), venvPython, pkg); err != nil {
s.log.Error("install scimesh runtime", "err", err)
writeJSON(w, http.StatusConflict, map[string]string{
"error": "pip install " + pkg + " failed: " + err.Error() +
". Set SCIMESH_PIP_PACKAGE to your scimesh wheel or index and retry.",
})
return
}
writeJSON(w, http.StatusOK, installRuntimeResponse{OK: true, Python: venvPython, Installed: true})
}
// venvPython returns the managed venv python when the runtime installer has
// created one, so the task runner can be pointed at it automatically.
func (s *Server) venvPython() string {
for _, candidate := range []string{
filepath.Join(s.dir, "venv", "bin", "python"),
filepath.Join(s.dir, "venv", "Scripts", "python.exe"),
} {
if _, err := os.Stat(candidate); err == nil {
return candidate
}
}
return ""
}
// installScimeshWithPip installs the package with the venv's own pip,
// streaming into the agent log so a long build is not silent.
func installScimeshWithPip(ctx context.Context, venvPython, pkg string) error {
pip := filepath.Join(filepath.Dir(venvPython), "pip")
if _, err := os.Stat(pip); err != nil {
pip += ".exe"
}
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
cmd := exec.CommandContext(ctx, pip, "install", pkg) //nolint:gosec // G204: pip from our venv, pkg is operator-set or a fixed default
return cmd.Run()
}
func stat(path string) bool {
_, err := os.Stat(path)
return err == nil
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n] + "…"
}
@@ -0,0 +1,499 @@
package setupui
import (
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/emil28092005/SciMesh/coordinator/internal/agent"
)
func testLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(io.Discard, nil))
}
func newTestServer(t *testing.T, sup Supervisor) (*Server, string) {
t.Helper()
return newTestServerWithInstall(t, sup, nil)
}
func newTestServerWithInstall(t *testing.T, sup Supervisor, install func(ctx context.Context, venvPython, pkg string) error) (*Server, string) {
t.Helper()
return newTestServerWithInstallAndWheel(t, sup, install, nil)
}
func newTestServerWithInstallAndWheel(t *testing.T, sup Supervisor, install func(ctx context.Context, venvPython, pkg string) error, wheel func(ctx context.Context, url, dir string) (string, error)) (*Server, string) {
t.Helper()
dir := t.TempDir()
server := New(testLogger(), Options{
// A distinct random port per test: Port 0 means "the default 12700" in
// the server, which would let the shared http.Client pool reuse a stale
// keep-alive connection across tests (EOF after a Shutdown).
Port: freePort(t),
ConfigPath: filepath.Join(dir, "config.json"),
Dir: dir,
Supervisor: sup,
OpenBrowser: func(string) {},
InstallScimesh: install,
DownloadWheel: wheel,
})
listener, err := server.Listen()
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = listener.Close() })
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
go func() { _ = server.Serve(ctx, listener) }()
return server, "http://" + listener.Addr().String()
}
type fakeSup struct {
mu sync.Mutex
started bool
stopped bool
pid int
alive bool
}
func (f *fakeSup) Start(configPath, logPath string) (int, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.started = true
f.alive = true
f.pid = 4242
return f.pid, nil
}
func (f *fakeSup) Stop() error {
f.mu.Lock()
defer f.mu.Unlock()
f.stopped = true
f.alive = false
return nil
}
func (f *fakeSup) Pid() int { f.mu.Lock(); defer f.mu.Unlock(); return f.pid }
func (f *fakeSup) Alive() bool {
f.mu.Lock()
defer f.mu.Unlock()
return f.alive
}
func postJSON(t *testing.T, base, path string, body any) (*httptest.ResponseRecorder, map[string]any) {
t.Helper()
req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, base+path, strings.NewReader(mustJSON(t, body)))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
// No keep-alive pooling: a pooled connection to a shut-down test server
// would surface as an EOF instead of a fresh dial.
client := http.Client{Transport: &http.Transport{DisableKeepAlives: true}}
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
rec := httptest.NewRecorder()
rec.Code = resp.StatusCode
data := map[string]any{}
_ = json.NewDecoder(resp.Body).Decode(&data)
return rec, data
}
// freePort reserves an ephemeral port and returns it. The listener is closed
// immediately; the tiny reuse window is acceptable for tests and each test
// gets a different port, so nothing can collide or share pooled connections.
func freePort(t *testing.T) int {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen(context.Background(), "tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
port := listener.Addr().(*net.TCPAddr).Port
_ = listener.Close()
return port
}
func mustJSON(t *testing.T, v any) string {
t.Helper()
raw, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
func TestWizardSavesConfigWithStrictPermissions(t *testing.T) {
sup := &fakeSup{}
server, base := newTestServer(t, sup)
rec, _ := postJSON(t, base, "/api/config", map[string]any{
"coordinator_url": "http://192.168.1.10:8080",
"token": "sm_live_secret",
"work_dir": "/home/emil/scimesh-worker",
"worker_name": "emil-laptop",
"cpu_count": 8,
"memory_mb": 16384,
})
if rec.Code != http.StatusOK {
t.Fatalf("save config: got %d, want 200", rec.Code)
}
info, err := os.Stat(server.cfgPath)
if err != nil {
t.Fatal(err)
}
if perm := info.Mode().Perm(); perm != 0o600 {
t.Errorf("config perms = %o, want 600", perm)
}
config, err := agent.LoadConfigFile(server.cfgPath)
if err != nil {
t.Fatal(err)
}
if config.CoordinatorURL != "http://192.168.1.10:8080" || config.Token != "sm_live_secret" || config.WorkDir != "/home/emil/scimesh-worker" || config.WorkerName != "emil-laptop" {
t.Errorf("config = %+v", config)
}
if config.CPUCount != 8 || config.MemoryMB != 16384 {
t.Errorf("resources: cpu=%d mem=%d", config.CPUCount, config.MemoryMB)
}
}
func TestWizardRejectsInvalidConfig(t *testing.T) {
sup := &fakeSup{}
_, base := newTestServer(t, sup)
for _, body := range []map[string]any{
{"coordinator_url": "", "token": "x"},
{"coordinator_url": "not-a-url", "token": "x"},
} {
rec, _ := postJSON(t, base, "/api/config", body)
if rec.Code != http.StatusBadRequest {
t.Errorf("body %v: got %d, want 400", body, rec.Code)
}
}
}
func TestWizardStartStopLifecycle(t *testing.T) {
sup := &fakeSup{}
_, base := newTestServer(t, sup)
// Starting without a saved config is rejected.
rec, _ := postJSON(t, base, "/api/start", map[string]any{})
if rec.Code != http.StatusBadRequest {
t.Errorf("start without config: got %d, want 400", rec.Code)
}
postJSON(t, base, "/api/config", map[string]any{
"coordinator_url": "http://127.0.0.1:8080", "token": "t", "work_dir": ".",
})
rec, data := postJSON(t, base, "/api/start", map[string]any{})
if rec.Code != http.StatusOK || int(data["pid"].(float64)) != 4242 {
t.Errorf("start: got %d %v, want 200 pid 4242", rec.Code, data)
}
if !sup.started {
t.Error("supervisor never started the worker")
}
// Status reflects the running state.
req, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, base+"/api/status", nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
var status map[string]any
_ = json.NewDecoder(resp.Body).Decode(&status)
if status["running"] != true || status["pid"] != float64(4242) {
t.Errorf("status = %v, want running pid 4242", status)
}
rec, _ = postJSON(t, base, "/api/stop", map[string]any{})
if rec.Code != http.StatusOK || !sup.stopped {
t.Errorf("stop: got %d stopped=%v, want 200/true", rec.Code, sup.stopped)
}
}
func TestWizardStatusPrefillsSavedConfig(t *testing.T) {
sup := &fakeSup{}
_, base := newTestServer(t, sup)
postJSON(t, base, "/api/config", map[string]any{
"coordinator_url": "http://10.0.0.5:8080", "worker_key": "smk_abc", "work_dir": "/w", "worker_name": "n1",
})
req, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, base+"/api/status", nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
var status struct {
ConfigPresent bool `json:"config_present"`
WorkerName string `json:"worker_name"`
Coordinator string `json:"coordinator"`
TokenSet bool `json:"token_set"`
}
_ = json.NewDecoder(resp.Body).Decode(&status)
if !status.ConfigPresent || status.WorkerName != "n1" || status.Coordinator != "http://10.0.0.5:8080" || !status.TokenSet {
t.Errorf("status = %+v", status)
}
// The secret must never appear in the status projection.
if strings.Contains(strings.ToLower(mustJSON(t, status)), "smk_abc") {
t.Error("status leaks the worker key")
}
}
func TestCheckCoordinatorReachable(t *testing.T) {
stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/health" {
_, _ = w.Write([]byte(`{"status":"ok"}`))
return
}
http.NotFound(w, r)
}))
defer stub.Close()
report := agent.CheckCoordinator(context.Background(), stub.URL, 5*time.Second)
if !report.Coordinator.OK {
t.Errorf("coordinator check = %+v, want ok", report.Coordinator)
}
}
func TestCheckCoordinatorUnreachable(t *testing.T) {
report := agent.CheckCoordinator(context.Background(), "http://127.0.0.1:1", 2*time.Second)
if report.Coordinator.OK {
t.Error("unreachable coordinator reported ok")
}
}
func TestConfigFileDefaultsAndEnvOverride(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.json")
file := agent.ConfigFile{
CoordinatorURL: "http://coord:8080",
Token: "file-token",
WorkDir: "/w",
CPUCount: 4,
}
if err := agent.SaveConfigFile(path, file); err != nil {
t.Fatal(err)
}
t.Setenv("COORDINATOR_URL", "http://env:9090")
t.Setenv("WORKER_AUTH_TOKEN", "")
config, err := agent.LoadConfigFile(path)
if err != nil {
t.Fatal(err)
}
if config.CoordinatorURL != "http://env:9090" {
t.Errorf("env must win: %s", config.CoordinatorURL)
}
if config.Token != "file-token" {
t.Errorf("token = %q, want the file value", config.Token)
}
if config.CPUCount != 4 {
t.Errorf("cpu = %d", config.CPUCount)
}
}
func TestInstallRuntimeCreatesVenvAndReportsPython(t *testing.T) {
var installedPkg string
sup := &fakeSup{}
_, base := newTestServerWithInstall(t, sup, func(ctx context.Context, venvPython, pkg string) error {
installedPkg = pkg
// Prove the venv python path really exists by creating a marker file
// where the real venv python would be.
_ = os.MkdirAll(filepath.Dir(venvPython), 0o755)
_ = os.WriteFile(venvPython, []byte("#!/bin/sh\nexit 0\n"), 0o755)
return nil
})
rec, data := postJSON(t, base, "/api/runtime/install", map[string]any{"scimesh_package": "/wheels/scimesh.whl"})
if rec.Code != http.StatusOK || data["ok"] != true {
t.Fatalf("install: got %d %v, want 200 ok", rec.Code, data)
}
if installedPkg != "/wheels/scimesh.whl" {
t.Errorf("package = %q, want the requested wheel", installedPkg)
}
if !strings.HasSuffix(data["python"].(string), "venv/bin/python") {
t.Errorf("python = %v, want the venv python", data["python"])
}
}
func TestInstallRuntimeRequiresASource(t *testing.T) {
sup := &fakeSup{}
_, base := newTestServerWithInstall(t, sup, func(ctx context.Context, venvPython, pkg string) error {
t.Fatal("install must not run without a package source")
return nil
})
// No source anywhere (SCIMESH_PIP_PACKAGE unset, request empty): 409 with
// guidance. The PyPI name is another project, so no silent fallback.
rec, data := postJSON(t, base, "/api/runtime/install", map[string]any{})
if rec.Code != http.StatusConflict {
t.Fatalf("install without source: got %d, want 409", rec.Code)
}
if !strings.Contains(data["error"].(string), "SCIMESH_PIP_PACKAGE") {
t.Errorf("error = %v, want a hint about SCIMESH_PIP_PACKAGE", data["error"])
}
}
func TestInstallRuntimeFailureIsExplained(t *testing.T) {
sup := &fakeSup{}
_, base := newTestServerWithInstall(t, sup, func(ctx context.Context, venvPython, pkg string) error {
return errors.New("no matching distribution found")
})
rec, data := postJSON(t, base, "/api/runtime/install", map[string]any{"scimesh_package": "/wheels/scimesh.whl"})
if rec.Code != http.StatusConflict {
t.Fatalf("install failure: got %d, want 409", rec.Code)
}
if !strings.Contains(data["error"].(string), "SCIMESH_PIP_PACKAGE") {
t.Errorf("error = %v, want a hint about SCIMESH_PIP_PACKAGE", data["error"])
}
}
func TestInstallRuntimeDownloadsReleaseWheelWhenNoSource(t *testing.T) {
oldVersion := agent.Version
agent.Version = "1.1.0-alpha.10"
t.Cleanup(func() { agent.Version = oldVersion })
var downloadedURL, installedPkg string
sup := &fakeSup{}
_, base := newTestServerWithInstallAndWheel(t, sup,
func(ctx context.Context, venvPython, pkg string) error {
installedPkg = pkg
return nil
},
func(ctx context.Context, url, dir string) (string, error) {
downloadedURL = url
return filepath.Join(dir, "scimesh-1.1.0a10-py3-none-any.whl"), nil
})
rec, data := postJSON(t, base, "/api/runtime/install", map[string]any{})
if rec.Code != http.StatusOK || data["ok"] != true {
t.Fatalf("install: got %d %v, want 200 ok", rec.Code, data)
}
if !strings.Contains(downloadedURL, "releases/download/v1.1.0-alpha.10/scimesh-1.1.0a10-py3-none-any.whl") {
t.Errorf("download url = %q, want the release wheel of this version", downloadedURL)
}
if !strings.HasSuffix(installedPkg, "scimesh-1.1.0a10-py3-none-any.whl") {
t.Errorf("pip received %q, want the downloaded wheel", installedPkg)
}
}
func TestInstallRuntimeWheelDownloadFailureIsExplained(t *testing.T) {
oldVersion := agent.Version
agent.Version = "1.1.0-alpha.10"
t.Cleanup(func() { agent.Version = oldVersion })
sup := &fakeSup{}
_, base := newTestServerWithInstallAndWheel(t, sup,
func(ctx context.Context, venvPython, pkg string) error { t.Fatal("pip must not run"); return nil },
func(ctx context.Context, url, dir string) (string, error) {
return "", errors.New("HTTP 404")
})
rec, data := postJSON(t, base, "/api/runtime/install", map[string]any{})
if rec.Code != http.StatusConflict {
t.Fatalf("got %d, want 409", rec.Code)
}
if !strings.Contains(data["error"].(string), "SCIMESH_PIP_PACKAGE") {
t.Errorf("error = %v, want a SCIMESH_PIP_PACKAGE hint", data["error"])
}
}
func TestStartPinsTheVenvTaskRunner(t *testing.T) {
sup := &fakeSup{}
server, base := newTestServer(t, sup)
postJSON(t, base, "/api/config", map[string]any{
"coordinator_url": "http://coord:8080", "token": "t", "work_dir": ".",
})
// Simulate the runtime installer: create the venv python marker.
venvPython := filepath.Join(server.dir, "venv", "bin", "python")
_ = os.MkdirAll(filepath.Dir(venvPython), 0o755)
_ = os.WriteFile(venvPython, []byte("#!/bin/sh\nexit 0\n"), 0o755)
rec, _ := postJSON(t, base, "/api/start", map[string]any{})
if rec.Code != http.StatusOK {
t.Fatalf("start: got %d, want 200", rec.Code)
}
config, err := agent.LoadConfigFile(server.cfgPath)
if err != nil {
t.Fatal(err)
}
if len(config.TaskRunner) != 3 || config.TaskRunner[0] != venvPython || config.TaskRunner[1] != "-m" || config.TaskRunner[2] != "scimesh.worker.task" {
t.Errorf("task runner = %v, want the venv python runner", config.TaskRunner)
}
}
func TestSaveConfigPinsVenvRunnerWhenPresent(t *testing.T) {
server, base := newTestServer(t, &fakeSup{})
venvPython := filepath.Join(server.dir, "venv", "bin", "python")
_ = os.MkdirAll(filepath.Dir(venvPython), 0o755)
_ = os.WriteFile(venvPython, []byte("#!/bin/sh\nexit 0\n"), 0o755)
rec, _ := postJSON(t, base, "/api/config", map[string]any{
"coordinator_url": "http://coord:8080", "token": "t", "work_dir": ".",
})
if rec.Code != http.StatusOK {
t.Fatalf("config: got %d", rec.Code)
}
config, err := agent.LoadConfigFile(server.cfgPath)
if err != nil {
t.Fatal(err)
}
if len(config.TaskRunner) != 3 || config.TaskRunner[0] != venvPython {
t.Errorf("task runner = %v, want the venv python", config.TaskRunner)
}
}
func TestTestProbesTheVenvPythonAfterInstall(t *testing.T) {
sup := &fakeSup{}
server, base := newTestServer(t, sup)
// The runtime installer leaves a venv python; make it a stub that reports
// a fake scimesh version so the preflight goes green through the venv.
venvPython := filepath.Join(server.dir, "venv", "bin", "python")
_ = os.MkdirAll(filepath.Dir(venvPython), 0o755)
_ = os.WriteFile(venvPython, []byte("#!/bin/sh\nif [ \"$1\" = \"-c\" ]; then echo 9.9.9-test; exit 0; fi\nexit 0\n"), 0o755)
req, _ := http.NewRequestWithContext(context.Background(), http.MethodPost, base+"/api/test", strings.NewReader(`{"coordinator_url":"http://127.0.0.1:1"}`))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
var report agent.CheckReport
if err := json.NewDecoder(resp.Body).Decode(&report); err != nil {
t.Fatal(err)
}
if !report.Scimesh.OK || report.Scimesh.Detail != "9.9.9-test" {
t.Errorf("scimesh check = %+v, want the venv interpreter reporting 9.9.9-test", report.Scimesh)
}
}
func TestParseWorkerStats(t *testing.T) {
logPath := filepath.Join(t.TempDir(), "worker.log")
content := `time=1 level=INFO msg=registered worker_id=w1
time=2 level=INFO msg="task claimed" task_id=t1 attempt=0
time=3 level=INFO msg="task completed" task_id=t1 elapsed_seconds=2
time=4 level=INFO msg="task claimed" task_id=t2 attempt=0
time=5 level=WARN msg="task failed" task_id=t2 error_code=X retryable=true
time=6 level=WARN msg="agent cycle failed" error="boom"
`
if err := os.WriteFile(logPath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
stats := parseWorkerStats(logPath)
if !stats.Registered || stats.Claimed != 2 || stats.Completed != 1 || stats.Failed != 1 {
t.Errorf("stats = %+v, want registered claimed=2 completed=1 failed=1", stats)
}
}
@@ -0,0 +1,338 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SciMesh Worker · Setup</title>
<style>
:root{--bg:#0b0e13;--panel:#11151d;--panel-2:#161c27;--border:#1f2634;--border-soft:#181f2b;--text:#e9ecf3;--text-2:#98a2b5;--text-3:#5d6879;--accent:#5b8cff;--accent-soft:#5b8cff1f;--accent-strong:#3f6fe0;--green:#3fce8a;--green-soft:#3fce8a1a;--amber:#e5b64f;--amber-soft:#e5b64f1a;--red:#f2647c;--red-soft:#f2647c1a;--mono:ui-monospace,"SF Mono",SFMono-Regular,Menlo,monospace;color-scheme:dark}
*{box-sizing:border-box;margin:0;padding:0}
body{background:radial-gradient(900px 500px at 50% -180px,#16233d66,transparent),var(--bg);color:var(--text);font:14px/1.55 Inter,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased;min-height:100vh}
button{font:inherit;color:inherit;background:none;border:0;cursor:pointer}
input{font:inherit;color:var(--text);background:var(--panel-2);border:1px solid var(--border);border-radius:9px;padding:10px 13px;width:100%;outline:none;transition:border-color .12s,box-shadow .12s}
input:focus{border-color:var(--accent);box-shadow:0 0 0 3px var(--accent-soft)}
input::placeholder{color:var(--text-3)}
code{font-family:var(--mono);font-size:.86em}
.shell{max-width:660px;margin:0 auto;padding:44px 22px 70px}
.brand{display:flex;align-items:center;justify-content:center;gap:11px;margin-bottom:8px}
.brand-mark{display:grid;place-items:center;width:34px;height:34px;border-radius:10px;background:linear-gradient(135deg,#5b8cff,#7c5cff);box-shadow:0 4px 16px #5b8cff40}
.brand-mark svg{width:18px;height:18px;stroke:#fff}
.brand-name{font-weight:700;font-size:16px;letter-spacing:-.01em}
.brand-name span{color:var(--text-3);font-weight:500}
.tagline{text-align:center;color:var(--text-3);font-size:12.5px;margin-bottom:34px}
.tagline code{color:var(--text-2)}
.steps{display:flex;align-items:center;justify-content:center;gap:0;margin-bottom:30px}
.step{display:flex;flex-direction:column;align-items:center;gap:7px;width:96px}
.step-dot{display:grid;place-items:center;width:30px;height:30px;border-radius:50%;border:1.5px solid var(--border);background:var(--panel);color:var(--text-3);font-size:12.5px;font-weight:700;transition:all .2s}
.step-label{font-size:11px;font-weight:600;color:var(--text-3);letter-spacing:.02em}
.step.active .step-dot{border-color:var(--accent);background:var(--accent-soft);color:var(--accent);box-shadow:0 0 0 4px #5b8cff14}
.step.active .step-label{color:var(--text)}
.step.done .step-dot{border-color:var(--green);background:var(--green-soft);color:var(--green)}
.step.done .step-label{color:var(--text-2)}
.step-line{flex:1;max-width:44px;height:1.5px;background:var(--border);margin:0 6px 22px;position:relative;overflow:hidden}
.step-line.done:after{content:"";position:absolute;inset:0;background:var(--green)}
.card{background:var(--panel);border:1px solid var(--border-soft);border-radius:15px;padding:26px 28px;box-shadow:0 24px 60px #0000004d}
.card h1{font-size:18px;font-weight:700;letter-spacing:-.02em;margin-bottom:4px}
.card .sub{color:var(--text-2);font-size:13px;margin-bottom:22px}
.field{margin-bottom:16px}
.field label{display:block;font-size:12px;font-weight:650;letter-spacing:.04em;text-transform:uppercase;color:var(--text-3);margin-bottom:7px}
.field .hint{margin-top:6px;font-size:12px;color:var(--text-3)}
.field .hint code{color:var(--text-2)}
.radio-grid{display:grid;grid-template-columns:1fr 1fr;gap:10px}
.radio-card{border:1px solid var(--border);border-radius:11px;padding:13px 14px;cursor:pointer;transition:all .13s;background:var(--panel-2)}
.radio-card:hover{border-color:#2a3446}
.radio-card.sel{border-color:var(--accent);background:var(--accent-soft);box-shadow:0 0 0 3px #5b8cff14}
.radio-card b{display:flex;align-items:center;gap:8px;font-size:13.5px}
.radio-card b svg{width:15px;height:15px;stroke:var(--accent)}
.radio-card p{margin-top:4px;font-size:12px;color:var(--text-2)}
.check-row{display:flex;align-items:center;gap:12px;padding:11px 14px;border:1px solid var(--border-soft);border-radius:10px;margin-bottom:9px;background:var(--panel-2)}
.check-ic{display:grid;place-items:center;width:24px;height:24px;border-radius:50%;flex:none}
.check-ic svg{width:13px;height:13px;stroke-width:2.6}
.check-ok{background:var(--green-soft)}.check-ok svg{stroke:var(--green)}
.check-bad{background:var(--red-soft)}.check-bad svg{stroke:var(--red)}
.check-wait{background:#ffffff10}.check-wait svg{stroke:var(--text-3)}
.check-row b{font-size:13.5px;font-weight:600}
.check-row span{display:block;font-size:12px;color:var(--text-3)}
.check-row .ms{margin-left:auto;font:11.5px var(--mono);color:var(--text-3)}
.actions{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-top:24px}
.btn{display:inline-flex;align-items:center;gap:8px;border-radius:9px;padding:10px 18px;font-weight:650;font-size:13.5px;border:1px solid transparent;transition:all .13s}
.btn svg{width:15px;height:15px;stroke:currentColor}
.btn-primary{background:var(--accent);color:#0a1222}
.btn-primary:hover{background:var(--accent-strong);color:#fff}
.btn-ghost{border-color:var(--border);color:var(--text-2);background:var(--panel-2)}
.btn-ghost:hover{color:var(--text);border-color:#2a3446}
.btn-danger{background:var(--red-soft);color:var(--red)}
.btn-lg{padding:12px 24px;font-size:14.5px;border-radius:10px}
.link{color:var(--text-3);font-size:13px}
.link:hover{color:var(--text)}
.status-head{display:flex;align-items:center;gap:14px;margin-bottom:22px}
.pulse{position:relative;width:12px;height:12px;border-radius:50%;background:var(--green);flex:none}
.pulse:after{content:"";position:absolute;inset:-5px;border-radius:50%;border:2px solid var(--green);opacity:.5;animation:ping 1.6s ease-out infinite}
@keyframes ping{from{transform:scale(.6);opacity:.7}to{transform:scale(1.4);opacity:0}}
.status-head h1{font-size:19px}
.status-head .sub{font-size:12.5px;color:var(--text-3)}
.stat-row{display:grid;grid-template-columns:repeat(4,1fr);gap:10px;margin-bottom:18px}
.stat{background:var(--panel-2);border:1px solid var(--border-soft);border-radius:11px;padding:12px 14px}
.stat b{display:block;font-size:20px;font-weight:700;letter-spacing:-.02em}
.stat span{font-size:11px;font-weight:600;letter-spacing:.05em;text-transform:uppercase;color:var(--text-3)}
.stat.bad b{color:var(--red)}
.logbox{background:#0a0d12;border:1px solid var(--border-soft);border-radius:11px;padding:14px 16px;font:12px/1.7 var(--mono);color:#8fa3bf;max-height:210px;overflow-y:auto;white-space:pre-wrap;word-break:break-word}
.meta-line{display:flex;gap:8px;flex-wrap:wrap;margin-bottom:16px}
.chip{display:inline-flex;align-items:center;gap:6px;border:1px solid var(--border);border-radius:99px;padding:4px 11px;font-size:12px;color:var(--text-2);background:var(--panel-2)}
.chip svg{width:12px;height:12px;stroke:var(--text-3)}
.wizard-page{display:none}.wizard-page.active{display:block;animation:fade .18s ease}
@keyframes fade{from{opacity:0;transform:translateY(5px)}to{opacity:1}}
.error-strip{background:var(--red-soft);border:1px solid #f2647c33;border-radius:9px;padding:9px 12px;font-size:12.5px;color:#ffb3c0;margin-bottom:14px}
.spinner{display:inline-block;width:13px;height:13px;border:2px solid var(--text-3);border-top-color:transparent;border-radius:50%;animation:spin .7s linear infinite;vertical-align:-2px;margin-right:7px}
@keyframes spin{to{transform:rotate(360deg)}}
.checks{padding-bottom:6px}
</style>
</head>
<body>
<div class="shell">
<div class="brand">
<div class="brand-mark"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round"><circle cx="6" cy="6" r="2.4"/><circle cx="18" cy="7" r="2.4"/><circle cx="12" cy="17" r="2.4"/><path d="M8 7.5l7.6-1M7 8.3l3.7 6.5M16.6 9.2l-3.2 5.6"/></svg></div>
<div class="brand-name">SciMesh <span>· Worker setup</span></div>
</div>
<p class="tagline">Local wizard served by <code>worker-agent setup</code> · <code>127.0.0.1</code></p>
<!-- ═══ WIZARD VIEW ═══ -->
<div id="view-wizard">
<div class="steps" id="steps">
<div class="step active" id="st1"><div class="step-dot">1</div><div class="step-label">Connect</div></div>
<div class="step-line" id="sl1"></div>
<div class="step" id="st2"><div class="step-dot">2</div><div class="step-label">Machine</div></div>
<div class="step-line" id="sl2"></div>
<div class="step" id="st3"><div class="step-dot">3</div><div class="step-label">Check</div></div>
<div class="step-line" id="sl3"></div>
<div class="step" id="st4"><div class="step-dot">4</div><div class="step-label">Run</div></div>
</div>
<div id="error-box"></div>
<!-- step 1: connect -->
<div class="wizard-page active" id="wp1">
<div class="card">
<h1>Connect to a coordinator</h1>
<p class="sub">The coordinator hands out work and collects results. Ask your cluster admin for its address.</p>
<div class="field"><label>Coordinator URL</label><input id="in-url" placeholder="http://192.168.1.10:8080"><p class="hint">For a served instance this is the address printed by <code>coordinator serve</code>.</p></div>
<div class="field"><label>Authentication</label>
<div class="radio-grid" id="auth-grid">
<div class="radio-card sel" data-mode="token"><b><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><rect x="4" y="10" width="16" height="10" rx="2"/><path d="M8 10V7a4 4 0 0 1 8 0v3"/></svg>Cluster token</b><p>Serve instances: one token for every worker.</p></div>
<div class="radio-card" data-mode="key"><b><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="8" cy="14" r="4"/><path d="M10.8 11.2L20 2M15 4l3 3"/></svg>Worker key</b><p>Shared clusters: a key tied to your account.</p></div>
</div>
</div>
<div class="field" id="token-field"><label>Token</label><input id="in-token" type="password" placeholder="paste the token from coordinator token"><p class="hint">The admin can copy it from <code>coordinator token</code> on the server.</p></div>
<div class="field" id="key-field" style="display:none"><label>Worker key + userservice URL</label><input id="in-key" type="password" placeholder="smk_…"><input id="in-users" placeholder="http://userservice-host:8081" style="margin-top:8px"><p class="hint">Create a key in the coordinator UI: Users → worker keys.</p></div>
<div class="actions"><span class="link" id="l1">Step 1 of 4</span><button class="btn btn-primary" id="b1">Continue →</button></div>
</div>
</div>
<!-- step 2: machine -->
<div class="wizard-page" id="wp2">
<div class="card">
<h1>This machine</h1>
<p class="sub">Where tasks run and how the machine appears in the cluster.</p>
<div class="field"><label>Worker name</label><input id="in-name" placeholder="auto-detected"><p class="hint">Shown in the coordinators worker list.</p></div>
<div class="field"><label>Work directory</label><input id="in-dir" placeholder="./scimesh-agent-data"><p class="hint">Datasets and shard results live here. ~1 GB free space recommended.</p></div>
<div class="field"><label>Compute resources advertised</label>
<div class="radio-grid">
<div class="radio-card sel" data-cpu="auto"><b>Auto</b><p>Detect the machines CPU count.</p></div>
<div class="radio-card" data-cpu="custom"><b>Custom…</b><p>Limit what this machine advertises.</p></div>
</div>
</div>
<div class="field" id="cpu-field" style="display:none"><label>CPU count</label><input id="in-cpu" type="number" min="1" value="1"></div>
<div class="field"><label>Concurrent task loops</label><input id="in-conc" type="number" min="1" max="64" value="1"><p class="hint">Process this many shards in parallel on this machine. Each loop runs its own task runner subprocess.</p></div>
<div class="actions"><button class="btn btn-ghost" id="b2b">← Back</button><button class="btn btn-primary" id="b2">Continue →</button></div>
</div>
</div>
<!-- step 3: preflight -->
<div class="wizard-page" id="wp3">
<div class="card">
<h1>Preflight check</h1>
<p class="sub">Making sure this machine can reach the coordinator and run SciMesh workloads.</p>
<div class="checks" id="checks"></div>
<div class="actions"><button class="btn btn-ghost" id="b3b">← Back</button><button class="btn btn-primary" id="b3" disabled>Continue anyway →</button></div>
</div>
</div>
<!-- step 4: run -->
<div class="wizard-page" id="wp4">
<div class="card" style="text-align:center;padding:40px 28px">
<div class="brand-mark" style="margin:0 auto 18px;width:46px;height:46px;border-radius:13px"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round" style="width:22px;height:22px"><path d="M6 4l14 8-14 8V4z"/></svg></div>
<h1 style="font-size:20px">Ready to join the cluster</h1>
<p class="sub" style="max-width:380px;margin:8px auto 26px">Configuration will be saved and the worker started as a background process.</p>
<div class="actions" style="justify-content:space-between;margin-top:30px"><button class="btn btn-ghost" id="b4b">← Back</button><button class="btn btn-primary btn-lg" id="b4">Start worker</button></div>
</div>
</div>
</div>
<!-- ═══ STATUS VIEW ═══ -->
<div id="view-status" style="display:none">
<div class="card">
<div class="status-head">
<div class="pulse" id="st-pulse"></div>
<div><h1 id="st-title">Worker is working</h1><div class="sub" id="st-sub"></div></div>
<button class="btn btn-danger" id="st-stop" style="margin-left:auto"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>Stop</button>
</div>
<div class="stat-row">
<div class="stat"><b id="st-claimed"></b><span>claimed</span></div>
<div class="stat"><b id="st-completed"></b><span>completed</span></div>
<div class="stat bad"><b id="st-failed"></b><span>failed</span></div>
<div class="stat"><b id="st-registered"></b><span>registered</span></div>
</div>
<div class="meta-line" id="st-meta"></div>
<div class="logbox" id="st-log"></div>
<div class="actions"><span class="link" id="st-cfg"></span><button class="btn btn-ghost" id="st-reconfig">Reconfigure…</button></div>
</div>
</div>
</div>
<script>
const $=id=>document.getElementById(id);
let state={mode:'token',cpu:'auto',venvPython:null};
let checksOk=false;
function err(msg){$('error-box').innerHTML=msg?'<div class="error-strip">'+msg+'</div>':''}
function goto(n){
['wp1','wp2','wp3','wp4'].forEach((id,i)=>$(id).classList.toggle('active',i===n-1));
for(let i=1;i<=4;i++){
const st=$('st'+i);
st.classList.toggle('done',i<n);st.classList.toggle('active',i===n);
st.querySelector('.step-dot').textContent=i<n?'':i;
if(i<4)$('sl'+i).classList.toggle('done',i<n);
}
err('');
}
document.querySelectorAll('#auth-grid .radio-card').forEach(c=>c.addEventListener('click',()=>{
document.querySelectorAll('#auth-grid .radio-card').forEach(x=>x.classList.remove('sel'));
c.classList.add('sel');state.mode=c.dataset.mode;
$('token-field').style.display=state.mode==='token'?'':'none';
$('key-field').style.display=state.mode==='key'?'':'none';
}));
document.querySelectorAll('#wp2 .radio-card').forEach(c=>c.addEventListener('click',()=>{
c.parentElement.querySelectorAll('.radio-card').forEach(x=>x.classList.remove('sel'));
c.classList.add('sel');state.cpu=c.dataset.cpu;
$('cpu-field').style.display=state.cpu==='custom'?'':'none';
}));
async function postJSON(path,body){
const r=await fetch(path,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(body)});
const data=await r.json().catch(()=>({}));
return {status:r.status,data};
}
function draftConfig(){
const cfg={
coordinator_url:$('in-url').value.trim(),
token:state.mode==='token'?$('in-token').value.trim():'',
worker_key:state.mode==='key'?$('in-key').value.trim():'',
userservice_url:state.mode==='key'?$('in-users').value.trim():'',
work_dir:$('in-dir').value.trim(),
worker_name:$('in-name').value.trim(),
cpu_count:state.cpu==='custom'?parseInt($('in-cpu').value||'1',10):0,
concurrency:parseInt($('in-conc').value||'1',10)
};
if(state.venvPython)cfg.task_runner=[state.venvPython,'-m','scimesh.worker.task'];
return cfg;
}
$('b1').onclick=()=>{
const c=draftConfig();
if(!c.coordinator_url){err('Enter the coordinator URL.');return}
if(state.mode==='token'&&!c.token){err('Enter the cluster token.');return}
if(state.mode==='key'&&!c.worker_key){err('Enter the worker key.');return}
goto(2);
};
$('b2b').onclick=()=>goto(1);
$('b2').onclick=()=>{goto(3);runChecks()};
$('b3b').onclick=()=>goto(2);
$('b3').onclick=()=>goto(4);
$('b4b').onclick=()=>goto(3);
$('b4').onclick=async()=>{
const c=draftConfig();
const r=await postJSON('/api/config',c);
if(r.status!==200){err('Could not save the configuration: '+(r.data.error||'unknown error'));return}
const s=await postJSON('/api/start',{});
if(s.status!==200){err('Could not start the worker: '+(s.data.error||'unknown error'));return}
showStatus();
};
const checkRow=(name,ok,detail,ms,action)=>
'<div class="check-row"><div class="check-ic '+(ok===null?'check-wait':ok?'check-ok':'check-bad')+'"><svg viewBox="0 0 24 24" fill="none" stroke-linecap="round">'+(ok===null?'<path d="M12 7v5l3 3"/>':ok?'<path d="M4 12l5 5L20 6"/>':'<path d="M6 6l12 12M18 6L6 18"/>')+'</svg></div><div><b>'+name+'</b><span>'+(detail||'')+'</span></div>'+(ms?'<span class="ms">'+ms+' ms</span>':'')+(action?action:'')+'</div>';
async function runChecks(){
const box=$('checks');
box.innerHTML=checkRow('Coordinator reachable','',null,null)+checkRow('Python 3','',null,null)+checkRow('scimesh package','',null,null);
const r=await postJSON('/api/test',draftConfig());
box.innerHTML='';
checksOk=true;
const items=[r.data.coordinator,r.data.python,r.data.scimesh];
for(const item of items){
if(item&&!item.ok)checksOk=false;
let action='';
if(item&&item.name==='scimesh'&&!item.ok){
action='<div style="margin-left:auto;display:flex;gap:8px;align-items:center"><input id="in-pkg" placeholder="wheel path / checkout / index URL (optional)" style="width:230px;padding:7px 10px;font-size:12px"><button class="btn btn-primary" style="padding:6px 12px;font-size:12px" id="install-scimesh">Install</button></div>';
}
box.insertAdjacentHTML('beforeend',checkRow(item?item.name:'?',item?item.ok:null,item?item.detail:'',item?item.latency_ms:null,action));
}
const btn=$('install-scimesh');
if(btn)btn.onclick=async()=>{
const src=$('in-pkg')?$('in-pkg').value.trim():'';
btn.disabled=true;btn.textContent='Installing…';
const ir=await postJSON('/api/runtime/install',{scimesh_package:src});
if(ir.status!==200){err('Could not install scimesh: '+(ir.data.error||'unknown error'));btn.disabled=false;btn.textContent='Install';return}
state.venvPython=ir.data.python;
runChecks();
};
$('b3').disabled=!checksOk;
}
async function showStatus(){
$('view-wizard').style.display='none';
$('view-status').style.display='block';
await refreshStatus();
setInterval(refreshStatus,2000);
}
async function refreshStatus(){
const r=await fetch('/api/status');
if(!r.ok)return;
const v=await r.json();
$('st-pulse').style.background=v.running?'var(--green)':'var(--text-3)';
$('st-pulse').style.animation=v.running?'':'none';
$('st-title').textContent=v.running?(v.worker_name||'Worker')+' is working':(v.worker_name||'Worker')+' is stopped';
$('st-sub').textContent='pid '+(v.pid||'—')+' · config '+(v.config_present?v.config_path:'not saved yet');
$('st-cfg').textContent='Configuration: '+(v.config_present?v.config_path:'—');
const stats=v.stats||{};
$('st-claimed').textContent=stats.claimed!=null?stats.claimed:'—';
$('st-completed').textContent=stats.completed!=null?stats.completed:'—';
$('st-failed').textContent=stats.failed!=null?stats.failed:'—';
$('st-registered').textContent=stats.registered?'yes':'no';
const meta=$('st-meta');
meta.innerHTML='';
if(v.coordinator){const c=document.createElement('span');c.className='chip';c.innerHTML='<svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="12" cy="12" r="9"/></svg>'+v.coordinator;meta.append(c)}
if(v.work_dir){const d=document.createElement('span');d.className='chip';d.innerHTML='<svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M4 6h16M4 12h16M4 18h16"/></svg>'+v.work_dir;meta.append(d)}
if(!v.token_set){const t=document.createElement('span');t.className='chip';t.style.color='var(--amber)';t.textContent='no credential set';meta.append(t)}
const logs=await fetch('/api/logs?tail=200');
const lv=await logs.json();
$('st-log').textContent=lv.log||'(no log yet — the worker writes here once started)';
}
$('st-stop').onclick=async()=>{await postJSON('/api/stop',{});refreshStatus()};
$('st-reconfig').onclick=()=>{
$('view-status').style.display='none';
$('view-wizard').style.display='block';
goto(1);
};
// Prefill from a saved configuration, then decide which view to show.
(async()=>{
const r=await fetch('/api/status');
const v=await r.json();
if(v.config_present){
$('in-url').value=v.coordinator||'';
$('in-dir').value=v.work_dir||'';
$('in-name').value=v.worker_name||'';
}
if(v.running)showStatus();
})();
</script>
</body>
</html>
+1
View File
@@ -18,4 +18,5 @@ var (
ErrResultConflict = errors.New("different result already recorded")
ErrInvalidInput = errors.New("invalid input")
ErrTaskNotLeased = errors.New("task is not currently leased")
ErrWorkloadDisabled = errors.New("workload is disabled")
)
+5
View File
@@ -1,6 +1,7 @@
package domain
import (
"strings"
"time"
"github.com/google/uuid"
@@ -52,6 +53,10 @@ func NewWorker(name string, capabilities []string, now time.Time) (*Worker, erro
if len(capabilities) == 0 {
return nil, ErrInvalidInput
}
name = strings.TrimSpace(name)
if name == "" {
return nil, ErrInvalidInput
}
return &Worker{
ID: uuid.New(),
Name: name,
@@ -29,3 +29,11 @@ func TestNewWorkerRejectsNoCapabilities(t *testing.T) {
t.Errorf("empty slice: err = %v, want ErrInvalidInput", err)
}
}
func TestNewWorkerRejectsBlankName(t *testing.T) {
for _, name := range []string{"", " ", "\t\n"} {
if _, err := NewWorker(name, []string{"similarity-search"}, testNow); !errors.Is(err, ErrInvalidInput) {
t.Errorf("name %q: got %v, want ErrInvalidInput", name, err)
}
}
}
+40
View File
@@ -314,6 +314,17 @@ func (r *WorkerRepo) MarkStaleOffline(ctx context.Context, cutoff time.Time) (in
return n, nil
}
func (r *WorkerRepo) SetTrust(ctx context.Context, id uuid.UUID, trust domain.WorkerTrust) error {
r.mu.Lock()
defer r.mu.Unlock()
w, ok := r.workers[id]
if !ok {
return domain.ErrWorkerNotFound
}
w.TrustLevel = trust
return nil
}
// --- ArtifactRepo --------------------------------------------------------
type ArtifactRepo struct {
@@ -450,3 +461,32 @@ func (r *TaskResultRepo) CountAgreeing(_ context.Context, taskID uuid.UUID, sha2
}
return n, nil
}
func (r *JobRepo) ListCompletedBefore(ctx context.Context, cutoff time.Time) ([]domain.Job, error) {
r.mu.Lock()
defer r.mu.Unlock()
var out []domain.Job
for _, j := range r.jobs {
if j.CompletedAt != nil && j.CompletedAt.Before(cutoff) {
out = append(out, *j)
}
}
return out, nil
}
func (r *JobRepo) Delete(ctx context.Context, id uuid.UUID) error {
r.mu.Lock()
defer r.mu.Unlock()
delete(r.jobs, id)
return nil
}
func (r *WorkerRepo) Delete(ctx context.Context, id uuid.UUID) error {
r.mu.Lock()
defer r.mu.Unlock()
if _, ok := r.workers[id]; !ok {
return domain.ErrWorkerNotFound
}
delete(r.workers, id)
return nil
}
@@ -0,0 +1,278 @@
//go:build integration
package postgres
import (
"context"
"errors"
"os"
"testing"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
)
// ensureMigrated applies the embedded schema first: the admin tests run
// before the dedicated migration test (file order) and need real tables.
func ensureMigrated(t *testing.T) {
t.Helper()
url := os.Getenv("TEST_DATABASE_URL")
if url == "" {
t.Skip("TEST_DATABASE_URL is not set")
}
if err := Migrate(context.Background(), url, nil); err != nil {
t.Fatalf("migrate: %v", err)
}
}
func TestAdminWorkerSetTrust(t *testing.T) {
ensureMigrated(t)
pool := testPool(t)
ctx := context.Background()
repo := NewWorkerRepo(pool)
w, err := domain.NewWorker("trust-lab", []string{"similarity-search"}, time.Now().UTC())
if err != nil {
t.Fatal(err)
}
if err := repo.Insert(ctx, w); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _, _ = pool.Exec(context.Background(), `DELETE FROM workers WHERE id = $1`, w.ID) })
if err := repo.SetTrust(ctx, w.ID, domain.WorkerUntrusted); err != nil {
t.Fatal(err)
}
got, err := repo.Get(ctx, w.ID)
if err != nil {
t.Fatal(err)
}
if got.TrustLevel != domain.WorkerUntrusted {
t.Errorf("trust = %q, want untrusted", got.TrustLevel)
}
if err := repo.SetTrust(ctx, uuid.New(), domain.WorkerTrusted); !errors.Is(err, domain.ErrWorkerNotFound) {
t.Errorf("unknown worker: got %v, want ErrWorkerNotFound", err)
}
}
func TestAdminListJobsPaginatedAndCounts(t *testing.T) {
ensureMigrated(t)
pool := testPool(t)
ctx := context.Background()
jobRepo := NewJobRepo(pool)
adminRepo := NewAdminReadRepo(pool)
jobs := make([]*domain.Job, 4)
for i := range jobs {
job, _ := seedJob(t, pool, 1)
jobs[i] = job
}
for _, j := range jobs {
t.Cleanup(func() { _, _ = pool.Exec(context.Background(), `DELETE FROM jobs WHERE id = $1`, j.ID) })
}
if err := jobRepo.UpdateStatus(ctx, jobs[0].ID, domain.JobCompleted, nil); err != nil {
t.Fatal(err)
}
if err := jobRepo.UpdateStatus(ctx, jobs[1].ID, domain.JobCompleted, nil); err != nil {
t.Fatal(err)
}
if err := jobRepo.UpdateStatus(ctx, jobs[2].ID, domain.JobRunning, nil); err != nil {
t.Fatal(err)
}
all, total, err := adminRepo.ListJobsPaginated(ctx, "", 100, 0)
if err != nil {
t.Fatal(err)
}
if total != 4 || len(all) != 4 {
t.Errorf("all: total=%d len=%d, want 4/4", total, len(all))
}
completed, total, err := adminRepo.ListJobsPaginated(ctx, "completed", 100, 0)
if err != nil {
t.Fatal(err)
}
if total != 2 || len(completed) != 2 {
t.Errorf("completed: total=%d len=%d, want 2/2", total, len(completed))
}
page, total, err := adminRepo.ListJobsPaginated(ctx, "", 2, 2)
if err != nil {
t.Fatal(err)
}
if total != 4 || len(page) != 2 {
t.Errorf("page: total=%d len=%d, want 4/2", total, len(page))
}
counts, err := adminRepo.CountJobsByStatus(ctx)
if err != nil {
t.Fatal(err)
}
if counts["completed"] != 2 || counts["running"] != 1 || counts["pending"] != 1 {
t.Errorf("counts = %v", counts)
}
}
func TestAdminTaskCountsByJobs(t *testing.T) {
ensureMigrated(t)
pool := testPool(t)
ctx := context.Background()
job, tasks := seedJob(t, pool, 3)
t.Cleanup(func() { _, _ = pool.Exec(context.Background(), `DELETE FROM jobs WHERE id = $1`, job.ID) })
resultArtifact := seedArtifact(t, pool, job.ID, &tasks[0].ID, domain.ArtifactPartialResult)
if _, err := pool.Exec(ctx, `UPDATE tasks SET status = 'completed', result_artifact_id = $1 WHERE id = $2`,
resultArtifact.ID, tasks[0].ID); err != nil {
t.Fatal(err)
}
if _, err := pool.Exec(ctx, `UPDATE tasks SET status = 'failed' WHERE id = $1`, tasks[1].ID); err != nil {
t.Fatal(err)
}
counts, err := NewAdminReadRepo(pool).TaskCountsByJobs(ctx, []uuid.UUID{job.ID})
if err != nil {
t.Fatal(err)
}
got := counts[job.ID]
if got["completed"] != 1 || got["failed"] != 1 || got["pending"] != 1 {
t.Errorf("task counts = %v, want completed=1 failed=1 pending=1", got)
}
}
func TestAdminJobCountsByDayAndWorkload(t *testing.T) {
ensureMigrated(t)
pool := testPool(t)
ctx := context.Background()
repo := NewAdminReadRepo(pool)
job, _ := seedJob(t, pool, 1)
t.Cleanup(func() { _, _ = pool.Exec(context.Background(), `DELETE FROM jobs WHERE id = $1`, job.ID) })
if _, err := pool.Exec(ctx, `UPDATE jobs SET created_at = now() - interval '2 days' WHERE id = $1`, job.ID); err != nil {
t.Fatal(err)
}
other, _ := seedJob(t, pool, 1)
t.Cleanup(func() { _, _ = pool.Exec(context.Background(), `DELETE FROM jobs WHERE id = $1`, other.ID) })
byDay, err := repo.JobCountsByDay(ctx, time.Now().UTC().Add(-6*24*time.Hour))
if err != nil {
t.Fatal(err)
}
today := time.Now().UTC().Format("2006-01-02")
twoDays := time.Now().UTC().Add(-48 * time.Hour).Format("2006-01-02")
if byDay[today] != 1 || byDay[twoDays] != 1 {
t.Errorf("by day = %v, want today=1 twoDays=1", byDay)
}
byWorkload, err := repo.JobCountsByWorkload(ctx)
if err != nil {
t.Fatal(err)
}
if byWorkload[job.Workload] < 2 {
t.Errorf("by workload = %v, want at least 2 for %s", byWorkload, job.Workload)
}
}
func TestAdminTaskStatsAndStorage(t *testing.T) {
ensureMigrated(t)
pool := testPool(t)
ctx := context.Background()
repo := NewAdminReadRepo(pool)
job, tasks := seedJob(t, pool, 2)
t.Cleanup(func() { _, _ = pool.Exec(context.Background(), `DELETE FROM jobs WHERE id = $1`, job.ID) })
start := time.Now().UTC().Add(-2 * time.Minute)
done := time.Now().UTC().Add(-90 * time.Second)
resultArtifact := seedArtifact(t, pool, job.ID, &tasks[0].ID, domain.ArtifactPartialResult)
if _, err := pool.Exec(ctx, `UPDATE tasks SET status = 'completed', result_artifact_id = $1, started_at = $2, completed_at = $3 WHERE id = $4`,
resultArtifact.ID, start, done, tasks[0].ID); err != nil {
t.Fatal(err)
}
if _, err := pool.Exec(ctx, `UPDATE tasks SET status = 'failed' WHERE id = $1`, tasks[1].ID); err != nil {
t.Fatal(err)
}
completed, failed, avg, err := repo.TaskStats(ctx)
if err != nil {
t.Fatal(err)
}
if completed < 1 || failed < 1 {
t.Errorf("stats = completed %d failed %d, want >= 1/1", completed, failed)
}
if avg < 29 || avg > 31 {
t.Errorf("avg duration = %.1fs, want ~30s", avg)
}
for _, kind := range []domain.ArtifactKind{domain.ArtifactInput, domain.ArtifactShard} {
seedArtifact(t, pool, job.ID, nil, kind)
}
sizes, err := repo.ArtifactSizeByKind(ctx)
if err != nil {
t.Fatal(err)
}
if sizes["input"] < 1 || sizes["shard"] < 1 {
t.Errorf("sizes = %v, want input/shard > 0", sizes)
}
dbBytes, err := repo.DatabaseSizeBytes(ctx)
if err != nil {
t.Fatal(err)
}
if dbBytes <= 0 {
t.Errorf("database size = %d, want > 0", dbBytes)
}
}
func TestWorkloadSettingsRepoRoundTrip(t *testing.T) {
ensureMigrated(t)
pool := testPool(t)
ctx := context.Background()
repo := NewWorkloadSettingsRepo(pool)
enabled, err := repo.GetEnabled(ctx, "similarity-search")
if err != nil {
t.Fatal(err)
}
if !enabled {
t.Error("workload without an override must be enabled")
}
now := time.Now().UTC()
if err := repo.SetEnabled(ctx, "similarity-search", false, now); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_, _ = pool.Exec(context.Background(), `DELETE FROM workload_settings WHERE workload = 'similarity-search'`)
})
enabled, err = repo.GetEnabled(ctx, "similarity-search")
if err != nil {
t.Fatal(err)
}
if enabled {
t.Error("workload must be disabled after the override")
}
if err := repo.SetEnabled(ctx, "similarity-search", true, now.Add(time.Hour)); err != nil {
t.Fatal(err)
}
enabled, err = repo.GetEnabled(ctx, "similarity-search")
if err != nil {
t.Fatal(err)
}
if !enabled {
t.Error("workload must be re-enabled after the upsert")
}
list, err := repo.List(ctx)
if err != nil {
t.Fatal(err)
}
found := false
for _, s := range list {
if s.Workload == "similarity-search" {
found = true
if !s.Enabled {
t.Error("list must reflect the re-enabled state")
}
}
}
if !found {
t.Error("override missing from the settings list")
}
}
@@ -0,0 +1,198 @@
package postgres
import (
"context"
"fmt"
"time"
sq "github.com/Masterminds/squirrel"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
// AdminReadRepo backs the coordinator admin console: paginated jobs, status
// counters, metrics buckets and storage figures. Read-only.
type AdminReadRepo struct{ pool *pgxpool.Pool }
func NewAdminReadRepo(pool *pgxpool.Pool) *AdminReadRepo { return &AdminReadRepo{pool: pool} }
var _ usecase.AdminReadRepository = (*AdminReadRepo)(nil)
func (r *AdminReadRepo) ListJobsPaginated(ctx context.Context, status string, limit, offset int) ([]domain.Job, int, error) {
if limit < 1 || limit > 100 || offset < 0 {
return nil, 0, domain.ErrInvalidInput
}
countQ := psql.Select("COUNT(*)").From("jobs")
listQ := psql.Select(jobColumns...).From("jobs")
if status != "" {
countQ = countQ.Where(sq.Eq{"status": status})
listQ = listQ.Where(sq.Eq{"status": status})
}
countSQL, args, err := countQ.ToSql()
if err != nil {
return nil, 0, err
}
var total int
if err := conn(ctx, r.pool).QueryRow(ctx, countSQL, args...).Scan(&total); err != nil {
return nil, 0, fmt.Errorf("count jobs: %w", err)
}
listSQL, args, err := listQ.OrderBy("created_at DESC", "id DESC").Limit(uint64(limit)).Offset(uint64(offset)).ToSql()
if err != nil {
return nil, 0, err
}
rows, err := conn(ctx, r.pool).Query(ctx, listSQL, args...)
if err != nil {
return nil, 0, fmt.Errorf("list jobs paginated: %w", err)
}
defer rows.Close()
jobs := make([]domain.Job, 0)
for rows.Next() {
var j domain.Job
var statusRaw string
if err := rows.Scan(
&j.ID, &j.Workload, &j.InputURI, &j.Parameters, &statusRaw, &j.CreatedAt, &j.CompletedAt,
&j.InputArtifactID, &j.ResultArtifactID, &j.ErrorCode, &j.ErrorMessage, &j.ReducerStartedAt,
&j.OwnerID,
); err != nil {
return nil, 0, err
}
j.Status = domain.JobStatus(statusRaw)
jobs = append(jobs, j)
}
return jobs, total, rows.Err()
}
func (r *AdminReadRepo) CountJobsByStatus(ctx context.Context) (map[string]int, error) {
rows, err := conn(ctx, r.pool).Query(ctx, "SELECT status, COUNT(*) FROM jobs GROUP BY status")
if err != nil {
return nil, fmt.Errorf("count jobs by status: %w", err)
}
defer rows.Close()
out := make(map[string]int)
for rows.Next() {
var status string
var count int
if err := rows.Scan(&status, &count); err != nil {
return nil, err
}
out[status] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) TaskCountsByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID]map[string]int, error) {
out := make(map[uuid.UUID]map[string]int, len(jobIDs))
if len(jobIDs) == 0 {
return out, nil
}
sql, args, err := psql.Select("job_id", "status", "COUNT(*)").From("tasks").
Where(sq.Eq{"job_id": jobIDs}).GroupBy("job_id", "status").ToSql()
if err != nil {
return nil, err
}
rows, err := conn(ctx, r.pool).Query(ctx, sql, args...)
if err != nil {
return nil, fmt.Errorf("task counts by jobs: %w", err)
}
defer rows.Close()
for rows.Next() {
var jobID uuid.UUID
var status string
var count int
if err := rows.Scan(&jobID, &status, &count); err != nil {
return nil, err
}
if out[jobID] == nil {
out[jobID] = make(map[string]int)
}
out[jobID][status] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) JobCountsByDay(ctx context.Context, since time.Time) (map[string]int, error) {
rows, err := conn(ctx, r.pool).Query(ctx,
"SELECT to_char(date_trunc('day', created_at AT TIME ZONE 'UTC'), 'YYYY-MM-DD') AS day, COUNT(*) FROM jobs WHERE created_at >= $1 GROUP BY 1",
since.UTC())
if err != nil {
return nil, fmt.Errorf("job counts by day: %w", err)
}
defer rows.Close()
out := make(map[string]int)
for rows.Next() {
var day string
var count int
if err := rows.Scan(&day, &count); err != nil {
return nil, err
}
out[day] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) JobCountsByWorkload(ctx context.Context) (map[string]int, error) {
rows, err := conn(ctx, r.pool).Query(ctx, "SELECT workload, COUNT(*) FROM jobs GROUP BY workload")
if err != nil {
return nil, fmt.Errorf("job counts by workload: %w", err)
}
defer rows.Close()
out := make(map[string]int)
for rows.Next() {
var workload string
var count int
if err := rows.Scan(&workload, &count); err != nil {
return nil, err
}
out[workload] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) TaskStats(ctx context.Context) (int64, int64, float64, error) {
var completed, failed int64
var avgSeconds *float64
err := conn(ctx, r.pool).QueryRow(ctx, `
SELECT
COALESCE(SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END), 0),
COALESCE(SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END), 0),
AVG(CASE WHEN status = 'completed' AND started_at IS NOT NULL
THEN EXTRACT(EPOCH FROM completed_at - started_at) END)::float8
FROM tasks`).Scan(&completed, &failed, &avgSeconds)
if err != nil {
return 0, 0, 0, fmt.Errorf("task stats: %w", err)
}
var avg float64
if avgSeconds != nil {
avg = *avgSeconds
}
return completed, failed, avg, nil
}
func (r *AdminReadRepo) ArtifactSizeByKind(ctx context.Context) (map[string]int64, error) {
rows, err := conn(ctx, r.pool).Query(ctx, "SELECT kind, COALESCE(SUM(size_bytes), 0) FROM artifacts GROUP BY kind")
if err != nil {
return nil, fmt.Errorf("artifact sizes: %w", err)
}
defer rows.Close()
out := make(map[string]int64)
for rows.Next() {
var kind string
var size int64
if err := rows.Scan(&kind, &size); err != nil {
return nil, err
}
out[kind] = size
}
return out, rows.Err()
}
func (r *AdminReadRepo) DatabaseSizeBytes(ctx context.Context) (int64, error) {
var size int64
if err := conn(ctx, r.pool).QueryRow(ctx, "SELECT pg_database_size(current_database())").Scan(&size); err != nil {
return 0, fmt.Errorf("database size: %w", err)
}
return size, nil
}
@@ -145,20 +145,20 @@ func TestUIReadRepoListsReducerFields(t *testing.T) {
if claimed, err := jobs.ClaimReduction(ctx, job.ID, time.Now().UTC()); err != nil || !claimed {
t.Fatalf("claim reduction = (%v, %v)", claimed, err)
}
listed, err := NewUIReadRepo(pool).ListJobs(ctx, nil, 20)
listed, _, err := NewAdminReadRepo(pool).ListJobsPaginated(ctx, "", 20, 0)
if err != nil {
t.Fatalf("list UI jobs: %v", err)
t.Fatalf("list admin jobs: %v", err)
}
for _, item := range listed {
if item.ID != job.ID {
continue
}
if item.Status != domain.JobReducing || item.ReducerStartedAt == nil {
t.Fatalf("UI reducer projection = %+v", item)
t.Fatalf("admin reducer projection = %+v", item)
}
return
}
t.Fatalf("seeded job %s is missing from UI list", job.ID)
t.Fatalf("seeded job %s is missing from the admin list", job.ID)
}
// A job must land whole or not at all: a half-created job leaves chunks no
@@ -3,6 +3,7 @@ package postgres
import (
"context"
"errors"
"fmt"
"time"
sq "github.com/Masterminds/squirrel"
@@ -153,3 +154,46 @@ func (r *JobRepo) UpdateStatus(ctx context.Context, id uuid.UUID,
}
return nil
}
// ListCompletedBefore returns jobs whose completion timestamp is older than
// the cutoff (completed and failed both count as finished).
func (r *JobRepo) ListCompletedBefore(ctx context.Context, cutoff time.Time) ([]domain.Job, error) {
sql, args, err := psql.Select(jobColumns...).From("jobs").
Where(sq.NotEq{"completed_at": nil}).
Where(sq.Lt{"completed_at": cutoff}).
OrderBy("completed_at ASC").
ToSql()
if err != nil {
return nil, err
}
rows, err := conn(ctx, r.pool).Query(ctx, sql, args...)
if err != nil {
return nil, fmt.Errorf("list completed jobs: %w", err)
}
defer rows.Close()
var jobs []domain.Job
for rows.Next() {
var j domain.Job
var status string
if err := rows.Scan(
&j.ID, &j.Workload, &j.InputURI, &j.Parameters, &status, &j.CreatedAt, &j.CompletedAt,
&j.InputArtifactID, &j.ResultArtifactID, &j.ErrorCode, &j.ErrorMessage, &j.ReducerStartedAt,
&j.OwnerID,
); err != nil {
return nil, err
}
j.Status = domain.JobStatus(status)
jobs = append(jobs, j)
}
return jobs, rows.Err()
}
// Delete removes the job row; tasks, artifacts and task_results cascade.
func (r *JobRepo) Delete(ctx context.Context, id uuid.UUID) error {
sql, args, err := psql.Delete("jobs").Where(sq.Eq{"id": id}).ToSql()
if err != nil {
return err
}
_, err = conn(ctx, r.pool).Exec(ctx, sql, args...)
return err
}
@@ -54,6 +54,8 @@ func expectedMigrationName(version int) string {
return "0012_worker_trust.up.sql"
case 13:
return "0013_task_results.up.sql"
case 14:
return "0014_workload_settings.up.sql"
default:
return ""
}
@@ -0,0 +1,5 @@
BEGIN;
DROP TABLE IF EXISTS workload_settings;
COMMIT;
@@ -0,0 +1,11 @@
BEGIN;
-- Per-workload enable/disable. Absence of a row means "enabled" (the catalog
-- default); a row only exists once an admin flipped a workload off or back on.
CREATE TABLE workload_settings (
workload text NOT NULL PRIMARY KEY,
enabled boolean NOT NULL,
updated_at timestamptz NOT NULL DEFAULT now()
);
COMMIT;
@@ -24,40 +24,6 @@ func (r *UIReadRepo) GetJob(ctx context.Context, id uuid.UUID) (*domain.Job, err
return job, err
}
func (r *UIReadRepo) ListJobs(ctx context.Context, owner *uuid.UUID, limit int) ([]domain.Job, error) {
if limit < 1 || limit > 100 {
return nil, domain.ErrInvalidInput
}
q := psql.Select(jobColumns...).From("jobs")
if owner != nil {
q = q.Where(sq.Eq{"owner_id": *owner})
}
sql, args, err := q.OrderBy("created_at DESC", "id DESC").Limit(uint64(limit)).ToSql()
if err != nil {
return nil, err
}
rows, err := conn(ctx, r.pool).Query(ctx, sql, args...)
if err != nil {
return nil, fmt.Errorf("list jobs: %w", err)
}
defer rows.Close()
jobs := make([]domain.Job, 0)
for rows.Next() {
var j domain.Job
var status string
if err := rows.Scan(
&j.ID, &j.Workload, &j.InputURI, &j.Parameters, &status, &j.CreatedAt, &j.CompletedAt,
&j.InputArtifactID, &j.ResultArtifactID, &j.ErrorCode, &j.ErrorMessage, &j.ReducerStartedAt,
&j.OwnerID,
); err != nil {
return nil, err
}
j.Status = domain.JobStatus(status)
jobs = append(jobs, j)
}
return jobs, rows.Err()
}
func (r *UIReadRepo) ListTasksByJob(ctx context.Context, jobID uuid.UUID) ([]domain.Task, error) {
sql, args, err := psql.Select(taskColumns...).From("tasks").Where(sq.Eq{"job_id": jobID}).OrderBy("chunk_index ASC").ToSql()
if err != nil {
@@ -79,68 +45,18 @@ func (r *UIReadRepo) ListTasksByJob(ctx context.Context, jobID uuid.UUID) ([]dom
return tasks, rows.Err()
}
func (r *UIReadRepo) ListTasksByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID][]domain.Task, error) {
out := make(map[uuid.UUID][]domain.Task, len(jobIDs))
if len(jobIDs) == 0 {
return out, nil
}
sql, args, err := psql.Select(taskColumns...).From("tasks").
Where(sq.Eq{"job_id": jobIDs}).OrderBy("job_id ASC", "chunk_index ASC").ToSql()
if err != nil {
return nil, err
}
rows, err := conn(ctx, r.pool).Query(ctx, sql, args...)
if err != nil {
return nil, fmt.Errorf("list tasks by jobs: %w", err)
}
defer rows.Close()
for rows.Next() {
task, err := scanTask(rows)
if err != nil {
return nil, err
}
out[task.JobID] = append(out[task.JobID], *task)
}
return out, rows.Err()
}
func (r *UIReadRepo) ListWorkers(ctx context.Context, limit int) ([]domain.Worker, error) {
if limit < 1 || limit > 100 {
return nil, domain.ErrInvalidInput
}
sql, args, err := psql.Select(workerColumns...).From("workers").OrderBy("last_heartbeat_at DESC", "id DESC").Limit(uint64(limit)).ToSql()
if err != nil {
return nil, err
}
rows, err := conn(ctx, r.pool).Query(ctx, sql, args...)
if err != nil {
return nil, fmt.Errorf("list workers: %w", err)
}
defer rows.Close()
workers := make([]domain.Worker, 0)
for rows.Next() {
worker, err := scanWorker(rows)
if err != nil {
return nil, err
}
workers = append(workers, *worker)
}
return workers, rows.Err()
}
func (r *UIReadRepo) ListWorkersByOwner(ctx context.Context, owner uuid.UUID, limit int) ([]domain.Worker, error) {
if limit < 1 || limit > 100 {
return nil, domain.ErrInvalidInput
}
sql, args, err := psql.Select(workerColumns...).From("workers").
Where(sq.Eq{"owner_id": owner}).
OrderBy("last_heartbeat_at DESC", "id DESC").Limit(uint64(limit)).ToSql()
if err != nil {
return nil, err
}
rows, err := conn(ctx, r.pool).Query(ctx, sql, args...)
if err != nil {
return nil, fmt.Errorf("list workers by owner: %w", err)
return nil, fmt.Errorf("list workers: %w", err)
}
defer rows.Close()
workers := make([]domain.Worker, 0)
@@ -91,6 +91,24 @@ func (r *WorkerRepo) MarkStaleOffline(ctx context.Context, cutoff time.Time) (in
return tag.RowsAffected(), nil
}
func (r *WorkerRepo) SetTrust(ctx context.Context, id uuid.UUID, trust domain.WorkerTrust) error {
sql, args, err := psql.Update("workers").
SetMap(map[string]any{"trust_level": string(trust), "updated_at": time.Now()}).
Where(sq.Eq{"id": id}).
ToSql()
if err != nil {
return err
}
tag, err := conn(ctx, r.pool).Exec(ctx, sql, args...)
if err != nil {
return fmt.Errorf("set worker trust: %w", err)
}
if tag.RowsAffected() == 0 {
return domain.ErrWorkerNotFound
}
return nil
}
func scanWorker(row pgx.Row) (*domain.Worker, error) {
var (
w domain.Worker
@@ -105,3 +123,19 @@ func scanWorker(row pgx.Row) (*domain.Worker, error) {
w.TrustLevel = domain.WorkerTrust(trust)
return &w, nil
}
// Delete removes a worker from the registry.
func (r *WorkerRepo) Delete(ctx context.Context, id uuid.UUID) error {
sql, args, err := psql.Delete("workers").Where(sq.Eq{"id": id}).ToSql()
if err != nil {
return err
}
tag, err := conn(ctx, r.pool).Exec(ctx, sql, args...)
if err != nil {
return fmt.Errorf("delete worker: %w", err)
}
if tag.RowsAffected() == 0 {
return domain.ErrWorkerNotFound
}
return nil
}
@@ -0,0 +1,67 @@
package postgres
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
// WorkloadSettingsRepo persists the per-workload enable/disable overrides.
// Absence of a row means the workload is enabled (the catalog default).
type WorkloadSettingsRepo struct{ pool *pgxpool.Pool }
func NewWorkloadSettingsRepo(pool *pgxpool.Pool) *WorkloadSettingsRepo {
return &WorkloadSettingsRepo{pool: pool}
}
var _ usecase.WorkloadSettingsRepository = (*WorkloadSettingsRepo)(nil)
func (r *WorkloadSettingsRepo) GetEnabled(ctx context.Context, workload string) (bool, error) {
var enabled bool
err := conn(ctx, r.pool).QueryRow(ctx,
"SELECT enabled FROM workload_settings WHERE workload = $1", workload).Scan(&enabled)
if err != nil && err.Error() == "no rows in result set" {
return true, nil // no override: catalog default enabled
}
if err != nil {
return false, fmt.Errorf("get workload setting: %w", err)
}
return enabled, nil
}
func (r *WorkloadSettingsRepo) List(ctx context.Context) ([]usecase.WorkloadSetting, error) {
rows, err := conn(ctx, r.pool).Query(ctx,
"SELECT workload, enabled, updated_at FROM workload_settings ORDER BY workload ASC")
if err != nil {
return nil, fmt.Errorf("list workload settings: %w", err)
}
defer rows.Close()
var out []usecase.WorkloadSetting
for rows.Next() {
var s usecase.WorkloadSetting
if err := rows.Scan(&s.Workload, &s.Enabled, &s.UpdatedAt); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
func (r *WorkloadSettingsRepo) SetEnabled(ctx context.Context, workload string, enabled bool, now time.Time) error {
sql, args, err := psql.Insert("workload_settings").
Columns("workload", "enabled", "updated_at").
Values(workload, enabled, now).
Suffix(`ON CONFLICT (workload) DO UPDATE SET enabled = EXCLUDED.enabled, updated_at = EXCLUDED.updated_at`).
ToSql()
if err != nil {
return err
}
if _, err := conn(ctx, r.pool).Exec(ctx, sql, args...); err != nil {
return fmt.Errorf("set workload setting: %w", err)
}
return nil
}
@@ -0,0 +1,134 @@
package sqlite
import (
"context"
"errors"
"testing"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
)
func TestWorkloadSettingsRepoRoundTrip(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
repo := NewWorkloadSettingsRepo(db)
// No override: enabled by default.
enabled, err := repo.GetEnabled(ctx, "similarity-search")
if err != nil {
t.Fatal(err)
}
if !enabled {
t.Error("workload without an override must be enabled")
}
now := time.Date(2026, 8, 2, 12, 0, 0, 0, time.UTC)
if err := repo.SetEnabled(ctx, "similarity-search", false, now); err != nil {
t.Fatal(err)
}
enabled, err = repo.GetEnabled(ctx, "similarity-search")
if err != nil {
t.Fatal(err)
}
if enabled {
t.Error("workload must be disabled after the override")
}
// Upsert flips it back and updates the timestamp.
later := now.Add(time.Hour)
if err := repo.SetEnabled(ctx, "similarity-search", true, later); err != nil {
t.Fatal(err)
}
enabled, err = repo.GetEnabled(ctx, "similarity-search")
if err != nil {
t.Fatal(err)
}
if !enabled {
t.Error("workload must be re-enabled after the upsert")
}
list, err := repo.List(ctx)
if err != nil {
t.Fatal(err)
}
if len(list) != 1 || list[0].Workload != "similarity-search" || !list[0].Enabled {
t.Errorf("list = %+v, want the single re-enabled override", list)
}
}
func TestWorkerSetTrust(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
repo := NewWorkerRepo(db)
worker, err := domain.NewWorker("lab-node", []string{"similarity-search"}, fixedTime())
if err != nil {
t.Fatal(err)
}
if err := repo.Insert(ctx, worker); err != nil {
t.Fatal(err)
}
if err := repo.SetTrust(ctx, worker.ID, domain.WorkerUntrusted); err != nil {
t.Fatal(err)
}
got, err := repo.Get(ctx, worker.ID)
if err != nil {
t.Fatal(err)
}
if got.TrustLevel != domain.WorkerUntrusted {
t.Errorf("trust = %q, want untrusted", got.TrustLevel)
}
if err := repo.SetTrust(ctx, worker.ID, domain.WorkerTrusted); err != nil {
t.Fatal(err)
}
if err := repo.SetTrust(ctx, uuid.New(), domain.WorkerTrusted); !errors.Is(err, domain.ErrWorkerNotFound) {
t.Errorf("unknown worker trust err = %v, want ErrWorkerNotFound", err)
}
}
func TestJobRepoListCompletedBeforeAndDelete(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
repo := NewJobRepo(db)
old := seedJob(t, db, 2)
oldTime := fixedTime().Add(-40 * 24 * time.Hour)
if err := repo.UpdateStatus(ctx, old.ID, domain.JobCompleted, &oldTime); err != nil {
t.Fatal(err)
}
fresh := seedJob(t, db, 2)
freshTime := fixedTime().Add(-2 * time.Hour)
if err := repo.UpdateStatus(ctx, fresh.ID, domain.JobCompleted, &freshTime); err != nil {
t.Fatal(err)
}
// The failing check constraint needs no result artifact for completed; the
// UpdateStatus path is fine, but tasks stay pending — irrelevant here.
list, err := repo.ListCompletedBefore(ctx, fixedTime().Add(-7*24*time.Hour))
if err != nil {
t.Fatal(err)
}
if len(list) != 1 || list[0].ID != old.ID {
t.Errorf("list = %d jobs, want only the old one", len(list))
}
if err := repo.Delete(ctx, old.ID); err != nil {
t.Fatal(err)
}
var n int
if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM jobs WHERE id = ?", old.ID.String()).Scan(&n); err != nil {
t.Fatal(err)
}
if n != 0 {
t.Error("job row must be gone after Delete")
}
// Tasks cascaded away with the job.
if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM tasks WHERE job_id = ?", old.ID.String()).Scan(&n); err != nil {
t.Fatal(err)
}
if n != 0 {
t.Error("tasks must cascade with the job")
}
}
@@ -0,0 +1,191 @@
package sqlite
import (
"context"
"database/sql"
"fmt"
"strings"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
// AdminReadRepo backs the coordinator admin console: paginated jobs, status
// counters, metrics buckets and storage figures. Read-only.
type AdminReadRepo struct{ db *sql.DB }
func NewAdminReadRepo(db *sql.DB) *AdminReadRepo { return &AdminReadRepo{db: db} }
var _ usecase.AdminReadRepository = (*AdminReadRepo)(nil)
func (r *AdminReadRepo) ListJobsPaginated(ctx context.Context, status string, limit, offset int) ([]domain.Job, int, error) {
if limit < 1 || limit > 100 || offset < 0 {
return nil, 0, domain.ErrInvalidInput
}
where := ""
args := []any{}
if status != "" {
where = " WHERE status = ?"
args = append(args, status)
}
var total int
if err := conn(ctx, r.db).QueryRowContext(ctx, "SELECT COUNT(*) FROM jobs"+where, args...).Scan(&total); err != nil {
return nil, 0, fmt.Errorf("count jobs: %w", err)
}
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT "+jobColumns+" FROM jobs"+where+" ORDER BY created_at DESC, id DESC LIMIT ? OFFSET ?",
append(args, limit, offset)...)
if err != nil {
return nil, 0, fmt.Errorf("list jobs paginated: %w", err)
}
defer func() { _ = rows.Close() }()
jobs := make([]domain.Job, 0)
for rows.Next() {
job, err := scanJob(rows)
if err != nil {
return nil, 0, err
}
jobs = append(jobs, *job)
}
return jobs, total, rows.Err()
}
func (r *AdminReadRepo) CountJobsByStatus(ctx context.Context) (map[string]int, error) {
rows, err := conn(ctx, r.db).QueryContext(ctx, "SELECT status, COUNT(*) FROM jobs GROUP BY status")
if err != nil {
return nil, fmt.Errorf("count jobs by status: %w", err)
}
defer func() { _ = rows.Close() }()
out := make(map[string]int)
for rows.Next() {
var status string
var count int
if err := rows.Scan(&status, &count); err != nil {
return nil, err
}
out[status] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) TaskCountsByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID]map[string]int, error) {
out := make(map[uuid.UUID]map[string]int, len(jobIDs))
if len(jobIDs) == 0 {
return out, nil
}
placeholders := make([]string, 0, len(jobIDs))
args := make([]any, 0, len(jobIDs))
for _, id := range jobIDs {
placeholders = append(placeholders, "?")
args = append(args, id.String())
}
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT job_id, status, COUNT(*) FROM tasks WHERE job_id IN ("+strings.Join(placeholders, ", ")+") GROUP BY job_id, status",
args...)
if err != nil {
return nil, fmt.Errorf("task counts by jobs: %w", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var jobRaw, status string
var count int
if err := rows.Scan(&jobRaw, &status, &count); err != nil {
return nil, err
}
jobID, err := uuid.Parse(jobRaw)
if err != nil {
return nil, fmt.Errorf("task counts: parse job id: %w", err)
}
if out[jobID] == nil {
out[jobID] = make(map[string]int)
}
out[jobID][status] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) JobCountsByDay(ctx context.Context, since time.Time) (map[string]int, error) {
// created_at is unix nanos; the bucket is the UTC calendar day.
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT strftime('%Y-%m-%d', created_at / 1000000000, 'unixepoch') AS day, COUNT(*) FROM jobs WHERE created_at >= ? GROUP BY day",
since.UTC().UnixNano())
if err != nil {
return nil, fmt.Errorf("job counts by day: %w", err)
}
defer func() { _ = rows.Close() }()
out := make(map[string]int)
for rows.Next() {
var day string
var count int
if err := rows.Scan(&day, &count); err != nil {
return nil, err
}
out[day] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) JobCountsByWorkload(ctx context.Context) (map[string]int, error) {
rows, err := conn(ctx, r.db).QueryContext(ctx, "SELECT workload, COUNT(*) FROM jobs GROUP BY workload")
if err != nil {
return nil, fmt.Errorf("job counts by workload: %w", err)
}
defer func() { _ = rows.Close() }()
out := make(map[string]int)
for rows.Next() {
var workload string
var count int
if err := rows.Scan(&workload, &count); err != nil {
return nil, err
}
out[workload] = count
}
return out, rows.Err()
}
func (r *AdminReadRepo) TaskStats(ctx context.Context) (int64, int64, float64, error) {
var completed, failed int64
var avgNanos sql.NullFloat64
err := conn(ctx, r.db).QueryRowContext(ctx, `
SELECT
COALESCE(SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END), 0),
COALESCE(SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END), 0),
AVG(CASE WHEN status = 'completed' AND started_at IS NOT NULL THEN completed_at - started_at END)
FROM tasks`).Scan(&completed, &failed, &avgNanos)
if err != nil {
return 0, 0, 0, fmt.Errorf("task stats: %w", err)
}
return completed, failed, avgNanos.Float64 / 1e9, nil
}
func (r *AdminReadRepo) ArtifactSizeByKind(ctx context.Context) (map[string]int64, error) {
rows, err := conn(ctx, r.db).QueryContext(ctx, "SELECT kind, COALESCE(SUM(size_bytes), 0) FROM artifacts GROUP BY kind")
if err != nil {
return nil, fmt.Errorf("artifact sizes: %w", err)
}
defer func() { _ = rows.Close() }()
out := make(map[string]int64)
for rows.Next() {
var kind string
var size int64
if err := rows.Scan(&kind, &size); err != nil {
return nil, err
}
out[kind] = size
}
return out, rows.Err()
}
func (r *AdminReadRepo) DatabaseSizeBytes(ctx context.Context) (int64, error) {
var pageCount, pageSize int64
if err := conn(ctx, r.db).QueryRowContext(ctx, "PRAGMA page_count").Scan(&pageCount); err != nil {
return 0, fmt.Errorf("page count: %w", err)
}
if err := conn(ctx, r.db).QueryRowContext(ctx, "PRAGMA page_size").Scan(&pageSize); err != nil {
return 0, fmt.Errorf("page size: %w", err)
}
return pageCount * pageSize, nil
}
@@ -0,0 +1,171 @@
package sqlite
import (
"context"
"testing"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
)
func TestAdminListJobsPaginatedAndCounts(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
jobRepo := NewJobRepo(db)
adminRepo := NewAdminReadRepo(db)
jobs := make([]*domain.Job, 5)
for i := range jobs {
jobs[i] = seedJob(t, db, 2)
}
// Two completed, two running, one pending.
if err := jobRepo.UpdateStatus(ctx, jobs[0].ID, domain.JobCompleted, nil); err != nil {
t.Fatal(err)
}
if err := jobRepo.UpdateStatus(ctx, jobs[1].ID, domain.JobCompleted, nil); err != nil {
t.Fatal(err)
}
if err := jobRepo.UpdateStatus(ctx, jobs[2].ID, domain.JobRunning, nil); err != nil {
t.Fatal(err)
}
if err := jobRepo.UpdateStatus(ctx, jobs[3].ID, domain.JobRunning, nil); err != nil {
t.Fatal(err)
}
all, total, err := adminRepo.ListJobsPaginated(ctx, "", 100, 0)
if err != nil {
t.Fatal(err)
}
if total != 5 || len(all) != 5 {
t.Errorf("all: total=%d len=%d, want 5/5", total, len(all))
}
completed, total, err := adminRepo.ListJobsPaginated(ctx, "completed", 100, 0)
if err != nil {
t.Fatal(err)
}
if total != 2 || len(completed) != 2 {
t.Errorf("completed: total=%d len=%d, want 2/2", total, len(completed))
}
page, total, err := adminRepo.ListJobsPaginated(ctx, "", 2, 2)
if err != nil {
t.Fatal(err)
}
if total != 5 || len(page) != 2 {
t.Errorf("page: total=%d len=%d, want 5/2", total, len(page))
}
counts, err := adminRepo.CountJobsByStatus(ctx)
if err != nil {
t.Fatal(err)
}
if counts["completed"] != 2 || counts["running"] != 2 || counts["pending"] != 1 {
t.Errorf("counts = %v, want completed=2 running=2 pending=1", counts)
}
}
func TestAdminTaskCountsByJobs(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
job := seedJob(t, db, 3)
if _, err := db.ExecContext(ctx, "UPDATE tasks SET status = 'completed', result_artifact_id = ? WHERE chunk_index = 0 AND job_id = ?", uuid.NewString(), job.ID.String()); err != nil {
t.Fatal(err)
}
if _, err := db.ExecContext(ctx, "UPDATE tasks SET status = 'failed' WHERE chunk_index = 1 AND job_id = ?", job.ID.String()); err != nil {
t.Fatal(err)
}
counts, err := NewAdminReadRepo(db).TaskCountsByJobs(ctx, []uuid.UUID{job.ID})
if err != nil {
t.Fatal(err)
}
got := counts[job.ID]
if got["completed"] != 1 || got["failed"] != 1 || got["pending"] != 1 {
t.Errorf("task counts = %v, want completed=1 failed=1 pending=1", got)
}
}
func TestAdminJobCountsByDay(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
job := seedJob(t, db, 1)
// Move the seed job to two days ago; create two more today.
old := fixedTime().Add(-48 * time.Hour)
if _, err := db.ExecContext(ctx, "UPDATE jobs SET created_at = ? WHERE id = ?", old.UnixNano(), job.ID.String()); err != nil {
t.Fatal(err)
}
seedJob(t, db, 1)
seedJob(t, db, 1)
repo := NewAdminReadRepo(db)
counts, err := repo.JobCountsByDay(ctx, fixedTime().Add(-6*24*time.Hour))
if err != nil {
t.Fatal(err)
}
today := fixedTime().UTC().Format("2006-01-02")
oldDay := old.UTC().Format("2006-01-02")
if counts[today] != 2 {
t.Errorf("today count = %d, want 2 (got %v)", counts[today], counts)
}
if counts[oldDay] != 1 {
t.Errorf("old day count = %d, want 1 (got %v)", counts[oldDay], counts)
}
}
func TestAdminTaskStatsAndStorage(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
repo := NewAdminReadRepo(db)
// One completed task with a known duration, one failed.
job := seedJob(t, db, 2)
start := fixedTime().Add(-2 * time.Minute)
done := fixedTime().Add(-90 * time.Second)
queries := []string{
"UPDATE tasks SET status='completed', result_artifact_id=?, started_at=?, completed_at=? WHERE job_id=? AND chunk_index=0",
"UPDATE tasks SET status='failed' WHERE job_id=? AND chunk_index=1",
}
for i, q := range queries {
args := []any{uuid.NewString(), start.UnixNano(), done.UnixNano(), job.ID.String()}
if i == 1 {
args = []any{job.ID.String()}
}
if _, err := db.ExecContext(ctx, q, args...); err != nil {
t.Fatal(err)
}
}
completed, failed, avg, err := repo.TaskStats(ctx)
if err != nil {
t.Fatal(err)
}
if completed != 1 || failed != 1 {
t.Errorf("stats = completed %d failed %d, want 1/1", completed, failed)
}
if avg < 29 || avg > 31 {
t.Errorf("avg duration = %.1fs, want ~30s", avg)
}
// Artifact sizes by kind.
for _, kind := range []string{"input", "shard", "final_result"} {
if _, err := db.ExecContext(ctx, "INSERT INTO artifacts (id, job_id, kind, filename, storage_key, content_type, size_bytes, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
uuid.NewString(), job.ID.String(), kind, kind+".csv", "key-"+kind, "text/csv", int64(len(kind)*1000), fixedTime().UnixNano()); err != nil {
t.Fatal(err)
}
}
sizes, err := repo.ArtifactSizeByKind(ctx)
if err != nil {
t.Fatal(err)
}
if sizes["input"] != 5000 || sizes["shard"] != 5000 || sizes["final_result"] != 12000 {
t.Errorf("sizes = %v", sizes)
}
dbBytes, err := repo.DatabaseSizeBytes(ctx)
if err != nil {
t.Fatal(err)
}
if dbBytes <= 0 {
t.Errorf("database size = %d, want > 0", dbBytes)
}
}
@@ -3,6 +3,7 @@ package sqlite
import (
"context"
"database/sql"
"fmt"
"time"
"github.com/google/uuid"
@@ -166,3 +167,30 @@ func nullableUUID(id *uuid.UUID) any {
}
return id.String()
}
// ListCompletedBefore returns jobs whose completion timestamp is older than
// the cutoff (completed and failed both count as finished).
func (r *JobRepo) ListCompletedBefore(ctx context.Context, cutoff time.Time) ([]domain.Job, error) {
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT "+jobColumns+" FROM jobs WHERE completed_at IS NOT NULL AND completed_at < ? ORDER BY completed_at ASC",
encodeTime(cutoff))
if err != nil {
return nil, fmt.Errorf("list completed jobs: %w", err)
}
defer func() { _ = rows.Close() }()
var jobs []domain.Job
for rows.Next() {
job, err := scanJob(rows)
if err != nil {
return nil, err
}
jobs = append(jobs, *job)
}
return jobs, rows.Err()
}
// Delete removes the job row; tasks, artifacts and task_results cascade.
func (r *JobRepo) Delete(ctx context.Context, id uuid.UUID) error {
_, err := conn(ctx, r.db).ExecContext(ctx, "DELETE FROM jobs WHERE id = ?", id.String())
return err
}
@@ -0,0 +1,8 @@
-- 0002: per-workload enable/disable. Absence of a row means "enabled" (the
-- catalog default); a row only exists once an admin flipped a workload off or
-- back on.
CREATE TABLE IF NOT EXISTS workload_settings (
workload TEXT NOT NULL PRIMARY KEY,
enabled INTEGER NOT NULL CHECK (enabled IN (0, 1)),
updated_at INTEGER NOT NULL
);
@@ -69,8 +69,8 @@ func TestMigrateIsIdempotent(t *testing.T) {
if err := db.QueryRowContext(ctx, "PRAGMA user_version").Scan(&version); err != nil {
t.Fatal(err)
}
if version != 1 {
t.Errorf("user_version = %d, want 1", version)
if version != 2 {
t.Errorf("user_version = %d, want 2", version)
}
}
@@ -4,7 +4,6 @@ import (
"context"
"database/sql"
"fmt"
"strings"
"github.com/google/uuid"
@@ -20,34 +19,6 @@ func (r *UIReadRepo) GetJob(ctx context.Context, id uuid.UUID) (*domain.Job, err
return NewJobRepo(r.db).Get(ctx, id)
}
func (r *UIReadRepo) ListJobs(ctx context.Context, owner *uuid.UUID, limit int) ([]domain.Job, error) {
if limit < 1 || limit > 100 {
return nil, domain.ErrInvalidInput
}
query := "SELECT " + jobColumns + " FROM jobs"
args := []any{}
if owner != nil {
query += " WHERE owner_id = ?"
args = append(args, owner.String())
}
query += " ORDER BY created_at DESC, id DESC LIMIT ?"
args = append(args, limit)
rows, err := conn(ctx, r.db).QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("list jobs: %w", err)
}
defer func() { _ = rows.Close() }()
jobs := make([]domain.Job, 0)
for rows.Next() {
job, err := scanJob(rows)
if err != nil {
return nil, err
}
jobs = append(jobs, *job)
}
return jobs, rows.Err()
}
func (r *UIReadRepo) ListTasksByJob(ctx context.Context, jobID uuid.UUID) ([]domain.Task, error) {
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT "+taskColumns+" FROM tasks WHERE job_id = ? ORDER BY chunk_index ASC", jobID.String())
@@ -66,50 +37,12 @@ func (r *UIReadRepo) ListTasksByJob(ctx context.Context, jobID uuid.UUID) ([]dom
return tasks, rows.Err()
}
func (r *UIReadRepo) ListTasksByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID][]domain.Task, error) {
out := make(map[uuid.UUID][]domain.Task, len(jobIDs))
if len(jobIDs) == 0 {
return out, nil
}
placeholders := make([]string, 0, len(jobIDs))
args := make([]any, 0, len(jobIDs))
for _, id := range jobIDs {
placeholders = append(placeholders, "?")
args = append(args, id.String())
}
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT "+taskColumns+" FROM tasks WHERE job_id IN ("+strings.Join(placeholders, ", ")+") ORDER BY job_id ASC, chunk_index ASC",
args...)
if err != nil {
return nil, fmt.Errorf("list tasks by jobs: %w", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
task, err := scanTask(rows)
if err != nil {
return nil, err
}
out[task.JobID] = append(out[task.JobID], *task)
}
return out, rows.Err()
}
func (r *UIReadRepo) ListWorkers(ctx context.Context, limit int) ([]domain.Worker, error) {
return r.listWorkers(ctx, "", nil, limit)
}
func (r *UIReadRepo) ListWorkersByOwner(ctx context.Context, owner uuid.UUID, limit int) ([]domain.Worker, error) {
return r.listWorkers(ctx, " WHERE owner_id = ?", []any{owner.String()}, limit)
}
func (r *UIReadRepo) listWorkers(ctx context.Context, clause string, args []any, limit int) ([]domain.Worker, error) {
if limit < 1 || limit > 100 {
return nil, domain.ErrInvalidInput
}
query := "SELECT " + workerColumns + " FROM workers" + clause +
" ORDER BY last_heartbeat_at DESC, id DESC LIMIT ?"
fullArgs := append(args, limit)
rows, err := conn(ctx, r.db).QueryContext(ctx, query, fullArgs...)
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT "+workerColumns+" FROM workers ORDER BY last_heartbeat_at DESC, id DESC LIMIT ?", limit)
if err != nil {
return nil, fmt.Errorf("list workers: %w", err)
}
@@ -90,3 +90,36 @@ func (r *WorkerRepo) MarkStaleOffline(ctx context.Context, cutoff time.Time) (in
}
return res.RowsAffected()
}
func (r *WorkerRepo) SetTrust(ctx context.Context, id uuid.UUID, trust domain.WorkerTrust) error {
res, err := conn(ctx, r.db).ExecContext(ctx,
"UPDATE workers SET trust_level = ?, updated_at = ? WHERE id = ?",
string(trust), encodeTime(time.Now()), id.String())
if err != nil {
return err
}
affected, err := res.RowsAffected()
if err != nil {
return err
}
if affected == 0 {
return domain.ErrWorkerNotFound
}
return nil
}
// Delete removes a worker from the registry.
func (r *WorkerRepo) Delete(ctx context.Context, id uuid.UUID) error {
res, err := conn(ctx, r.db).ExecContext(ctx, "DELETE FROM workers WHERE id = ?", id.String())
if err != nil {
return err
}
affected, err := res.RowsAffected()
if err != nil {
return err
}
if affected == 0 {
return domain.ErrWorkerNotFound
}
return nil
}
@@ -0,0 +1,75 @@
package sqlite
import (
"context"
"database/sql"
"fmt"
"time"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
// WorkloadSettingsRepo persists the per-workload enable/disable overrides.
// Absence of a row means the workload is enabled (the catalog default).
type WorkloadSettingsRepo struct{ db *sql.DB }
func NewWorkloadSettingsRepo(db *sql.DB) *WorkloadSettingsRepo { return &WorkloadSettingsRepo{db: db} }
var _ usecase.WorkloadSettingsRepository = (*WorkloadSettingsRepo)(nil)
func (r *WorkloadSettingsRepo) GetEnabled(ctx context.Context, workload string) (bool, error) {
var enabled int
err := conn(ctx, r.db).QueryRowContext(ctx,
"SELECT enabled FROM workload_settings WHERE workload = ?", workload).Scan(&enabled)
if err == sql.ErrNoRows {
return true, nil // no override: catalog default enabled
}
if err != nil {
return false, fmt.Errorf("get workload setting: %w", err)
}
return enabled == 1, nil
}
func (r *WorkloadSettingsRepo) List(ctx context.Context) ([]usecase.WorkloadSetting, error) {
rows, err := conn(ctx, r.db).QueryContext(ctx,
"SELECT workload, enabled, updated_at FROM workload_settings ORDER BY workload ASC")
if err != nil {
return nil, fmt.Errorf("list workload settings: %w", err)
}
defer func() { _ = rows.Close() }()
var out []usecase.WorkloadSetting
for rows.Next() {
var (
name string
enabled int
updatedAt int64
)
if err := rows.Scan(&name, &enabled, &updatedAt); err != nil {
return nil, err
}
out = append(out, usecase.WorkloadSetting{
Workload: name,
Enabled: enabled == 1,
UpdatedAt: decodeTime(updatedAt),
})
}
return out, rows.Err()
}
func (r *WorkloadSettingsRepo) SetEnabled(ctx context.Context, workload string, enabled bool, now time.Time) error {
_, err := conn(ctx, r.db).ExecContext(ctx, `
INSERT INTO workload_settings (workload, enabled, updated_at) VALUES (?, ?, ?)
ON CONFLICT (workload) DO UPDATE SET enabled = excluded.enabled, updated_at = excluded.updated_at`,
workload, boolInt(enabled), now.UnixNano())
if err != nil {
return fmt.Errorf("set workload setting: %w", err)
}
return nil
}
func boolInt(b bool) int {
if b {
return 1
}
return 0
}
@@ -42,7 +42,7 @@ func (s *Server) writeError(w http.ResponseWriter, r *http.Request, err error) {
status := http.StatusInternalServerError
switch {
case errors.Is(err, domain.ErrInvalidInput):
case errors.Is(err, domain.ErrInvalidInput), errors.Is(err, domain.ErrWorkloadDisabled):
status = http.StatusBadRequest
case errors.Is(err, domain.ErrJobNotFound), errors.Is(err, domain.ErrTaskNotFound),
errors.Is(err, domain.ErrWorkerNotFound), errors.Is(err, domain.ErrArtifactNotFound):
+24 -1
View File
@@ -35,6 +35,8 @@ type UseCases struct {
GetTaskInput *usecase.GetTaskInput
Dashboard *usecase.Dashboard
PreviewArtifact *usecase.PreviewArtifact
Admin *usecase.Admin
PruneArtifacts *usecase.PruneArtifacts
}
type Server struct {
@@ -130,6 +132,11 @@ func (s *Server) Handler(token string, uiToken ...string) http.Handler {
mux.HandleFunc("GET /health", s.handleHealth)
// Unauthenticated like /health, so a Prometheus scraper needs no credential.
mux.Handle("GET /metrics", s.metrics.Handler())
// Worker-key exchange is fronted by the coordinator when the userservice
// is embedded (serve mode): the key itself is the credential.
if s.userserviceURL != "" {
mux.HandleFunc("POST /worker-tokens/exchange", s.handleWorkerTokenExchangeProxy)
}
hasBasicAuth := len(uiToken) > 0 && uiToken[0] != ""
if s.uc.Dashboard != nil && (s.uiSessionMode() || hasBasicAuth) {
@@ -146,7 +153,6 @@ func (s *Server) Handler(token string, uiToken ...string) http.Handler {
{"GET /ui/docs", s.handleUIDocsIndex},
{"GET /ui/docs/{path...}", s.handleUIDocs},
{"GET /ui/jobs/{job_id}", s.handleUIJob},
{"GET /ui/api/overview", s.handleUIOverviewJSON},
{"GET /ui/api/jobs/{job_id}", s.handleUIJobJSON},
{"POST /ui/api/jobs/{job_id}/cancel", s.handleCancelJob},
{"POST /ui/api/jobs/upload", s.handleUploadDataset},
@@ -158,6 +164,7 @@ func (s *Server) Handler(token string, uiToken ...string) http.Handler {
// Public auth pages — reachable without a session so a user can log in.
ui.HandleFunc("GET /ui/login", s.handleUILoginForm)
ui.HandleFunc("POST /ui/login", s.handleUILogin)
ui.HandleFunc("GET /ui/logout-form", s.handleUILogoutForm)
ui.HandleFunc("GET /ui/register", s.handleUIRegisterForm)
ui.HandleFunc("POST /ui/register", s.handleUIRegister)
ui.HandleFunc("POST /ui/logout", s.handleUILogout)
@@ -177,6 +184,22 @@ func (s *Server) Handler(token string, uiToken ...string) http.Handler {
// Admin panel: session + admin role.
ui.Handle("GET /ui/admin", chain(http.HandlerFunc(s.handleUIAdmin), gate, requireAdmin))
ui.Handle("POST /ui/admin/user-action", chain(http.HandlerFunc(s.handleUIAdminUserAction), gate, requireAdmin))
// Admin console APIs: session + admin role, bounded read models.
ui.Handle("GET /ui/admin/api/system", chain(http.HandlerFunc(s.handleUIAdminSystemJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/jobs", chain(http.HandlerFunc(s.handleUIAdminJobsJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/metrics", chain(http.HandlerFunc(s.handleUIAdminMetricsJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/workers", chain(http.HandlerFunc(s.handleUIAdminWorkersJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/workers/{id}/trust", chain(http.HandlerFunc(s.handleUIAdminSetTrustJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/workers/{id}/remove", chain(http.HandlerFunc(s.handleUIAdminRemoveWorkerJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/users", chain(http.HandlerFunc(s.handleUIAdminUsersJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/users/{id}/role", chain(http.HandlerFunc(s.handleUIAdminSetUserRoleJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/worker-keys", chain(http.HandlerFunc(s.handleUIAdminWorkerKeysJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/worker-keys/{id}/revoke", chain(http.HandlerFunc(s.handleUIAdminRevokeKeyJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/workloads", chain(http.HandlerFunc(s.handleUIAdminWorkloadsJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/workloads/{name}/enabled", chain(http.HandlerFunc(s.handleUIAdminSetWorkloadEnabledJSON), gate, requireAdmin))
ui.Handle("GET /ui/admin/api/settings", chain(http.HandlerFunc(s.handleUIAdminSettingsJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/token/reveal", chain(http.HandlerFunc(s.handleUIAdminRevealTokenJSON), gate, requireAdmin))
ui.Handle("POST /ui/admin/api/prune", chain(http.HandlerFunc(s.handleUIAdminPruneJSON), gate, requireAdmin))
} else {
for _, rt := range app {
ui.HandleFunc(rt.pattern, rt.handler)
@@ -39,6 +39,7 @@ func newEnvWithUIToken(t *testing.T, ready func(context.Context) error, configur
work := memstore.NewWorkerRepo()
arts := memstore.NewArtifactRepo()
blobs := memstore.NewBlobStore()
settings := memstoreSettings{}
clk := memstore.NewClock(time.Date(2026, 7, 21, 12, 0, 0, 0, time.UTC))
tx := memstore.Tx{}
lease := 2 * time.Minute
@@ -47,7 +48,7 @@ func newEnvWithUIToken(t *testing.T, ready func(context.Context) error, configur
uc := coordhttp.UseCases{
RegisterWorker: usecase.NewRegisterWorker(work, clk),
CreateJob: usecase.NewCreateJob(jobs, tasks, tx, clk),
SubmitDataset: usecase.NewSubmitDataset(blobs, arts, jobs, tasks, tx, clk, 3, testCatalog()),
SubmitDataset: usecase.NewSubmitDataset(blobs, arts, jobs, tasks, tx, clk, 3, testCatalog(), settings),
ClaimTask: usecase.NewClaimTask(tasks, jobs, work, tx, clk, lease, testCatalog()),
RenewLease: usecase.NewRenewLease(tasks, work, tx, clk, lease),
CompleteTask: usecase.NewCompleteTask(tasks, jobs, arts, work, memstore.NewTaskResultRepo(), tx, clk, 2, testCatalog()),
@@ -144,42 +145,17 @@ func TestUIRequiresDistinctCredentialAndRendersDashboard(t *testing.T) {
req = request()
req.SetBasicAuth("operator", uiToken)
resp, err = http.DefaultClient.Do(req)
// Do not follow the redirect: we assert it, not its target.
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
resp, err = client.Do(req)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("UI status: %d", resp.StatusCode)
}
body, _ := io.ReadAll(resp.Body)
if !strings.Contains(string(body), "SciMesh control room") {
t.Errorf("dashboard body missing title")
}
}
func TestUIOverviewReturnsLiveSafeProjection(t *testing.T) {
e := newEnv(t, healthy)
code, _ := e.do(t, "POST", "/jobs", `{"workload":"w","input_uri":"s3://in","chunks":[{"chunk_index":0,"input_uri":"s3://c","input_sha256":"sha"}]}`)
if code != http.StatusCreated {
t.Fatalf("create job: %d", code)
}
req, _ := http.NewRequestWithContext(context.Background(), "GET", e.ts.URL+"/ui/api/overview", nil)
req.SetBasicAuth("operator", uiToken)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
var overview map[string]any
if err := json.NewDecoder(resp.Body).Decode(&overview); err != nil {
t.Fatal(err)
}
if resp.StatusCode != http.StatusOK || overview["active_jobs"].(float64) != 1 || overview["online_workers"].(float64) != 1 {
t.Fatalf("overview = (%d, %v)", resp.StatusCode, overview)
}
if _, leaked := overview["worker_auth_token"]; leaked {
t.Fatal("overview must not expose authentication configuration")
// In basic-auth mode (no userservice) /ui lands on the job form: the admin
// console exists only in session mode.
if resp.StatusCode != http.StatusSeeOther || resp.Header.Get("Location") != "/ui/jobs/new" {
t.Fatalf("UI status: %d -> %s, want 303 -> /ui/jobs/new", resp.StatusCode, resp.Header.Get("Location"))
}
}
@@ -739,3 +715,22 @@ func (e *env) uploadDataset(t *testing.T, workload string, rows int, tsv string)
}
func itoa(n int) string { return strconv.Itoa(n) }
// memstoreSettings is an in-memory WorkloadSettingsRepository for tests.
type memstoreSettings struct{ overrides map[string]bool }
func (m memstoreSettings) GetEnabled(ctx context.Context, workload string) (bool, error) {
if enabled, ok := m.overrides[workload]; ok {
return enabled, nil
}
return true, nil
}
func (m memstoreSettings) List(ctx context.Context) ([]usecase.WorkloadSetting, error) {
return nil, nil
}
func (m memstoreSettings) SetEnabled(ctx context.Context, workload string, enabled bool, now time.Time) error {
m.overrides[workload] = enabled
return nil
}
@@ -5,13 +5,11 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Add your machine · SciMesh</title>
<style>
:root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:980px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.back{text-decoration:none}.eyebrow{margin:28px 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3.25rem);letter-spacing:-.055em}.lead{max-width:720px;margin:10px 0 0;color:#aabed9;font-size:1.06rem}.layout{display:grid;grid-template-columns:1.45fr .8fr;gap:15px;margin-top:28px}.card,.aside,.notice{border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021}.card{padding:22px}.aside,.notice{padding:18px}.aside h2,.notice h2{margin:0;color:#f1f6ff;font-size:1rem}.aside p,.notice p{color:#9fb3cf}.aside ol{margin:13px 0 0;padding-left:20px;color:#aebfda}.aside li{margin:10px 0}label{display:block;margin:18px 0 5px;color:#eaf2ff;font-weight:750}input{width:100%;border:1px solid #42617f;border-radius:9px;padding:10px 11px;background:#0a1626;color:#e5efff;font:inherit}input:focus{outline:2px solid #5d97f5;outline-offset:1px}.hint{margin:5px 0;color:#92a9c6;font-size:.87rem}.button{display:inline-flex;margin-top:16px;border:0;border-radius:10px;padding:11px 15px;background:#67e3b8;color:#062018;font:inherit;font-weight:850;cursor:pointer}.button.secondary{background:#23344d;color:#dce8ff}.button:disabled{opacity:.6;cursor:wait}.command{margin-top:18px;border:1px solid #2c8c70;border-radius:11px;padding:14px;background:#0c2b2a;color:#a8f1d0}.command strong{color:#e6fff4}.command pre{margin:10px 0 0;padding:12px;overflow-x:auto;border-radius:8px;background:#061a19;color:#c8ffe8;font:.82rem/1.5 ui-monospace,SFMono-Regular,monospace;white-space:pre;word-break:normal}.keys{margin-top:14px;display:grid;gap:9px}.key{display:flex;align-items:center;justify-content:space-between;gap:12px;border:1px solid #294662;border-radius:11px;padding:12px 14px;background:#0a1626}.key .kn{color:#f3f7ff;font-weight:700}.key .kp{margin-top:3px;color:#8196b3;font-family:ui-monospace,SFMono-Regular,monospace;font-size:.76rem}.key .kd{color:#8fa6c3;font-size:.8rem}.revoke{border:1px solid #6a2a3a;border-radius:8px;padding:7px 11px;background:#2a1420;color:#ff9bad;font:inherit;font-weight:700;cursor:pointer}.empty{padding:20px;border:1px dashed #35516f;border-radius:12px;color:#9ab0cb;text-align:center}.error{margin:12px 0 0;color:#ffacba}.warn{color:#ffd08a}.hidden{display:none}.cap{display:inline-block;margin:4px 5px 0 0;border:1px solid #375978;border-radius:5px;padding:2px 6px;color:#b5d3f5;font:.76rem ui-monospace,SFMono-Regular,monospace}@media(max-width:720px){.layout{grid-template-columns:1fr}.page{padding:22px 14px}}
</style>
{{template "ui-styles"}}
</head>
<body data-coordinator="{{.CoordinatorURL}}" data-userservice="{{.UserserviceURL}}">
<main class="page">
<a class="back" href="/ui">← Back to control room</a><p class="eyebrow">Contribute compute</p><h1>Turn this computer into a worker</h1><p class="lead">Create a key, install the worker, and run one command. The worker binds to your account and pulls tasks whenever it is online.</p>
<a class="back" href="/ui/admin">← Back to control room</a><p class="eyebrow">Contribute compute</p><h1>Turn this computer into a worker</h1><p class="lead">Create a key, install the worker, and run one command. The worker binds to your account and pulls tasks whenever it is online.</p>
<div class="layout">
<section class="card">
<h2 style="margin:0 0 4px;color:#f1f6ff">Your worker keys</h2>
@@ -29,8 +27,12 @@
<h2>Set it up</h2>
<ol>
<li><strong>Create a key</strong><br>Use the form; copy the command it generates.</li>
<li><strong>Paste it in a terminal</strong><br>The command clones the project, sets up a Python environment, installs the worker, and starts it. The machine then appears under <a href="/ui">My machines</a>.</li>
<li><strong>Paste it in a terminal</strong><br>The command installs the worker, points it at this coordinator, and starts it. The machine then appears under <a href="/ui/admin">Workers</a>.</li>
</ol>
<h2 style="margin-top:24px">Single-binary mode?</h2>
<p>If the coordinator runs as <code>coordinator serve</code>, skip the key:
use the token in <code>~/.scimesh/worker.token</code> on the coordinator
machine as <code>WORKER_AUTH_TOKEN</code> instead.</p>
<h2 style="margin-top:24px">Will my results count?</h2>
<p>Your worker is <strong>untrusted</strong> by default: its results are cross-checked and accepted once a second independent worker computes the same answer (quorum), or once an admin marks your account <strong>verified</strong> — then your workers are trusted and results count immediately.</p>
<p><span class="cap">similarity-search</span> and other SDK workloads from the library run on volunteer workers.</p>
@@ -39,12 +41,12 @@
</main>
<script>
const coord=(document.body.dataset.coordinator||location.origin).replace(/\/+$/,'');
const users=(document.body.dataset.userservice||'').replace(/\/+$/,'');
const users=(document.body.dataset.userservice||coord).replace(/\/+$/,'');
const keysBox=document.querySelector('#keys'),cmdBox=document.querySelector('#command'),form=document.querySelector('#create'),nameInput=document.querySelector('#key-name'),createBtn=document.querySelector('#create-btn'),error=document.querySelector('#error');
const node=(tag,text,cls)=>{const n=document.createElement(tag);if(text!==undefined)n.textContent=text;if(cls)n.className=cls;return n};
const shq=s=>"'"+String(s).replace(/'/g,"'\\''")+"'";
const buildCommand=(key,name)=>['git clone https://github.com/emil28092005/SciMesh.git','cd SciMesh','python -m venv .venv','source .venv/bin/activate','pip install -e .','','SCIMESH_COORDINATOR_URL='+coord+' \\','SCIMESH_USERSERVICE_URL='+(users||'<your userservice URL>')+' \\','SCIMESH_WORKER_KEY='+key+' \\','scimesh-worker --worker-name '+shq(name||'my-machine')].join('\n');
const showCommand=(key,name)=>{cmdBox.replaceChildren();cmdBox.append(node('strong','Key created — copy it now, it is shown only once.'));const pre=node('pre',buildCommand(key,name));cmdBox.append(pre);const copy=node('button','Copy command','button secondary');copy.type='button';copy.addEventListener('click',()=>{navigator.clipboard&&navigator.clipboard.writeText(buildCommand(key,name)).then(()=>{copy.textContent='Copied ✓'},()=>{copy.textContent='Press Ctrl+C to copy'})});cmdBox.append(document.createElement('br'),copy);if(!users){cmdBox.append(node('p','Set SCIMESH_USERSERVICE_URL to a userservice URL your machine can reach.','warn'))}cmdBox.classList.remove('hidden')};
const buildCommand=(key,name)=>['# install the worker binary (or download worker-agent from the release page)','curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash -s worker','','export COORDINATOR_URL='+coord,'export USERSERVICE_URL='+(users||'<your userservice URL>')+' \\','export WORKER_KEY='+key,'export WORKER_NAME='+shq(name||'my-machine'),'export WORK_DIR=~/scimesh-worker','worker-agent'].join('\n');
const showCommand=(key,name)=>{cmdBox.replaceChildren();cmdBox.append(node('strong','Key created — copy it now, it is shown only once.'));const pre=node('pre',buildCommand(key,name));cmdBox.append(pre);const copy=node('button','Copy command','button secondary');copy.type='button';copy.addEventListener('click',()=>{navigator.clipboard&&navigator.clipboard.writeText(buildCommand(key,name)).then(()=>{copy.textContent='Copied ✓'},()=>{copy.textContent='Press Ctrl+C to copy'})});cmdBox.append(document.createElement('br'),copy);cmdBox.classList.remove('hidden')};
const revoke=async id=>{const r=await fetch('/ui/api/worker-keys/'+encodeURIComponent(id)+'/revoke',{method:'POST'});if(r.status===204||r.ok){loadKeys()}else{error.textContent='Could not revoke the key.'}};
const renderKeys=keys=>{keysBox.replaceChildren();if(!keys.length){keysBox.append(node('div','No keys yet. Create one above to connect a machine.','empty'));return}for(const k of keys){const row=node('div',undefined,'key'),left=node('div');left.append(node('div',k.name||'unnamed','kn'),node('div',k.prefix+'…','kp'),node('div','Created '+new Date(k.created_at).toLocaleString()+(k.last_used_at?' · last used '+new Date(k.last_used_at).toLocaleString():' · never used'),'kd'));const btn=node('button','Revoke','revoke');btn.type='button';btn.addEventListener('click',()=>revoke(k.id));row.append(left,btn);keysBox.append(row)}};
const loadKeys=async()=>{try{const r=await fetch('/ui/api/worker-keys',{headers:{Accept:'application/json'}});if(!r.ok)throw Error();const data=await r.json();renderKeys(data.worker_keys||[])}catch(_){keysBox.replaceChildren(node('div','Could not load your keys.','empty'))}};
@@ -2,45 +2,558 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Admin · SciMesh</title>
<style>
:root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:820px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.top{display:flex;justify-content:space-between;align-items:center;gap:12px}.eyebrow{margin:0;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:6px 0 0;color:#f4f8ff;font-size:clamp(1.8rem,4vw,2.6rem);letter-spacing:-.04em}.lead{max-width:640px;margin:10px 0 0;color:#aabed9}.card{margin-top:24px;border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021;padding:22px}.card h2{margin:0 0 4px;color:#f1f6ff;font-size:1.1rem}.card p{margin:0;color:#9fb3cf;font-size:.92rem}label{display:block;margin:16px 0 5px;color:#eaf2ff;font-weight:750}input{width:100%;border:1px solid #42617f;border-radius:9px;padding:10px 11px;background:#0a1626;color:#e5efff;font:inherit}input:focus{outline:2px solid #5d97f5;outline-offset:1px}.actions{display:flex;flex-wrap:wrap;gap:9px;margin-top:16px}.btn{border:0;border-radius:9px;padding:10px 14px;font:inherit;font-weight:800;cursor:pointer}.btn-primary{background:#67e3b8;color:#062018}.btn-muted{background:#23344d;color:#dce8ff}.notice{margin-top:16px;border-radius:10px;padding:11px 13px;font-weight:700}.ok{background:#123f34;color:#76efb5}.err{background:#552334;color:#ff9bad}.muted{color:#8ba2c2}.hint{margin-top:4px;color:#92a9c6;font-size:.85rem}</style>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SciMesh · Coordinator Admin</title>
<style>
:root{--bg:#0b0e13;--panel:#11151d;--panel-2:#161c27;--border:#1f2634;--border-soft:#181f2b;--text:#e9ecf3;--text-2:#98a2b5;--text-3:#5d6879;--accent:#5b8cff;--accent-soft:#5b8cff1f;--accent-strong:#3f6fe0;--green:#3fce8a;--green-soft:#3fce8a1a;--amber:#e5b64f;--amber-soft:#e5b64f1a;--red:#f2647c;--red-soft:#f2647c1a;--mono:ui-monospace,"SF Mono",SFMono-Regular,Menlo,monospace;color-scheme:dark}
*{box-sizing:border-box;margin:0;padding:0}
body{background:var(--bg);color:var(--text);font:14px/1.55 Inter,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
button{font:inherit;color:inherit;background:none;border:0;cursor:pointer}
input,select{font:inherit;color:var(--text);background:var(--panel-2);border:1px solid var(--border);border-radius:8px;padding:8px 11px;outline:none}
input:focus,select:focus{border-color:var(--accent);box-shadow:0 0 0 3px var(--accent-soft)}
code{font-family:var(--mono);font-size:.86em}
a{color:var(--accent);text-decoration:none}
.layout{display:flex;min-height:100vh}
.sidebar{position:sticky;top:0;height:100vh;width:232px;flex:none;display:flex;flex-direction:column;background:var(--panel);border-right:1px solid var(--border-soft)}
.brand{display:flex;align-items:center;gap:11px;padding:20px 20px 18px;border-bottom:1px solid var(--border-soft)}
.brand-mark{display:grid;place-items:center;width:32px;height:32px;border-radius:9px;background:linear-gradient(135deg,#5b8cff,#7c5cff);box-shadow:0 4px 14px #5b8cff40}
.brand-mark svg{width:17px;height:17px;stroke:#fff}
.brand-name{font-weight:700;font-size:14.5px;letter-spacing:-.01em}
.brand-sub{font-size:11px;color:var(--text-3);letter-spacing:.02em}
.nav{flex:1;overflow-y:auto;padding:14px 12px}
.nav-label{margin:16px 10px 6px;font-size:10.5px;font-weight:700;letter-spacing:.12em;text-transform:uppercase;color:var(--text-3)}
.nav-label:first-child{margin-top:0}
.nav-item{display:flex;align-items:center;gap:10px;width:100%;padding:8px 10px;border-radius:8px;color:var(--text-2);font-weight:500;text-align:left;transition:background .12s,color .12s}
.nav-item svg{width:16px;height:16px;stroke:currentColor;flex:none}
.nav-item:hover{background:var(--panel-2);color:var(--text)}
.nav-item.active{background:var(--accent-soft);color:var(--accent);font-weight:600}
.nav-item .count{margin-left:auto;font-size:11px;font-weight:600;color:var(--text-3);background:var(--panel-2);border-radius:99px;padding:1px 7px}
.nav-item.active .count{color:var(--accent);background:#5b8cff26}
.side-foot{padding:14px;border-top:1px solid var(--border-soft)}
.user-chip{display:flex;align-items:center;gap:10px;padding:9px 10px;border-radius:9px;background:var(--panel-2)}
.avatar{display:grid;place-items:center;width:28px;height:28px;border-radius:8px;background:linear-gradient(135deg,#3fce8a,#2ea56c);color:#08130d;font-weight:800;font-size:12px;flex:none}
.user-chip b{display:block;font-size:12.5px;line-height:1.25}
.user-chip span{display:block;font-size:11px;color:var(--text-3)}
.back-link{display:block;margin-top:9px;padding:7px 10px;color:var(--text-3);font-size:12.5px;text-decoration:none;border-radius:8px}
.back-link:hover{color:var(--text);background:var(--panel-2)}
.main{flex:1;min-width:0;display:flex;flex-direction:column}
.topbar{position:sticky;top:0;z-index:5;display:flex;align-items:center;justify-content:space-between;gap:16px;padding:16px 32px;background:#0b0e13e6;backdrop-filter:blur(10px);border-bottom:1px solid var(--border-soft)}
.topbar h1{font-size:17px;font-weight:700;letter-spacing:-.015em}
.topbar p{font-size:12.5px;color:var(--text-3);margin-top:1px}
.env-badge{display:flex;align-items:center;gap:8px;font-size:12px;color:var(--text-2);border:1px solid var(--border);border-radius:99px;padding:5px 12px;background:var(--panel)}
.env-badge i{width:7px;height:7px;border-radius:50%;background:var(--green);box-shadow:0 0 8px var(--green)}
.content{flex:1;padding:26px 32px 60px;max-width:1120px;width:100%;margin:0 auto}
.page{display:none}.page.active{display:block;animation:fade .18s ease}
@keyframes fade{from{opacity:0;transform:translateY(4px)}to{opacity:1}}
.card{background:var(--panel);border:1px solid var(--border-soft);border-radius:13px}
.card-pad{padding:20px}
.card-head{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:15px 20px;border-bottom:1px solid var(--border-soft)}
.card-head h3{font-size:13.5px;font-weight:650}
.card-head span{font-size:12px;color:var(--text-3)}
.grid-kpi{display:grid;grid-template-columns:repeat(4,1fr);gap:14px;margin-bottom:14px}
.kpi{background:var(--panel);border:1px solid var(--border-soft);border-radius:13px;padding:16px 18px}
.kpi .k-label{display:flex;align-items:center;gap:7px;font-size:11.5px;font-weight:600;letter-spacing:.06em;text-transform:uppercase;color:var(--text-3)}
.kpi .k-value{margin-top:8px;font-size:26px;font-weight:700;letter-spacing:-.03em;line-height:1}
.kpi .k-sub{margin-top:6px;font-size:12px;color:var(--text-2)}
.pill{display:inline-flex;align-items:center;gap:6px;border-radius:99px;padding:3px 10px;font-size:11.5px;font-weight:650;white-space:nowrap}
.pill i{width:6px;height:6px;border-radius:50%;background:currentColor}
.pill-success{background:var(--green-soft);color:var(--green)}
.pill-active{background:var(--accent-soft);color:var(--accent)}
.pill-waiting{background:#ffffff12;color:var(--text-2)}
.pill-danger{background:var(--red-soft);color:var(--red)}
.pill-amber{background:var(--amber-soft);color:var(--amber)}
.btn{display:inline-flex;align-items:center;gap:7px;border-radius:8px;padding:8px 14px;font-weight:600;font-size:13px;border:1px solid transparent;transition:filter .12s,background .12s}
.btn svg{width:14px;height:14px;stroke:currentColor}
.btn-primary{background:var(--accent);color:#0a1222}
.btn-ghost{background:var(--panel-2);border-color:var(--border);color:var(--text-2)}
.btn-ghost:hover{color:var(--text);border-color:#2a3446}
.btn-danger{background:var(--red-soft);color:var(--red)}
.btn-sm{padding:5px 10px;font-size:12px;border-radius:7px}
table{width:100%;border-collapse:collapse}
th{padding:10px 20px;text-align:left;font-size:11px;font-weight:650;letter-spacing:.07em;text-transform:uppercase;color:var(--text-3);border-bottom:1px solid var(--border-soft)}
td{padding:12px 20px;border-bottom:1px solid var(--border-soft);vertical-align:middle}
tr:last-child td{border-bottom:0}
tbody tr{transition:background .1s}
tbody tr:hover{background:var(--panel-2)}
.t-main{font-weight:600;font-size:13.5px}
.t-sub{font-size:11.5px;color:var(--text-3);font-family:var(--mono)}
.bar{height:5px;width:130px;border-radius:99px;background:#ffffff10;overflow:hidden}
.bar span{display:block;height:100%;border-radius:inherit;background:linear-gradient(90deg,#5b8cff,#3fce8a)}
.bar-label{font-size:11.5px;color:var(--text-2);font-family:var(--mono);margin-top:5px}
.tabs{display:flex;gap:4px;padding:4px;background:var(--panel);border:1px solid var(--border-soft);border-radius:10px;width:max-content;margin-bottom:14px;flex-wrap:wrap}
.tab{padding:6px 13px;border-radius:7px;font-size:12.5px;font-weight:600;color:var(--text-2)}
.tab:hover{color:var(--text)}
.tab.active{background:var(--panel-2);color:var(--text);box-shadow:inset 0 0 0 1px var(--border)}
.tab .n{color:var(--text-3);font-weight:500;margin-left:5px}
.tab.active .n{color:var(--accent)}
.section-title{margin:26px 0 12px;font-size:13px;font-weight:700;letter-spacing:-.01em;color:var(--text)}
.section-title:first-child{margin-top:0}
.section-note{font-size:12px;color:var(--text-3);margin:-8px 0 12px}
.kv{display:grid;grid-template-columns:210px 1fr;row-gap:0}
.kv dt{padding:11px 20px;font-size:12.5px;color:var(--text-3);border-bottom:1px solid var(--border-soft)}
.kv dd{padding:11px 20px;font-size:13px;border-bottom:1px solid var(--border-soft)}
.kv dt:last-of-type,.kv dd:last-of-type{border-bottom:0}
.stack{display:grid;gap:14px}
.split{display:grid;grid-template-columns:1fr 1fr;gap:14px}
.storage-bar{display:flex;height:10px;border-radius:99px;overflow:hidden;margin:14px 20px 6px}
.storage-bar div{height:100%}
.legend{display:flex;gap:20px;padding:10px 20px 18px;flex-wrap:wrap}
.legend span{display:flex;align-items:center;gap:7px;font-size:12px;color:var(--text-2)}
.legend i{width:9px;height:9px;border-radius:3px}
.footer-row{display:flex;align-items:center;justify-content:space-between;padding:11px 20px;font-size:12px;color:var(--text-3)}
.pager{display:flex;gap:4px}
.pager button{width:26px;height:26px;border-radius:7px;font-size:12px;color:var(--text-2)}
.pager button.cur{background:var(--accent-soft);color:var(--accent);font-weight:700}
.pager button:disabled{opacity:.35;cursor:default}
.chart{width:100%;height:auto;display:block}
.chart-bar{fill:#2c3a52;rx:4}
.chart-bar.hot{fill:var(--accent)}
.chart-grid{stroke:#ffffff08}
.chart-label{font:10px var(--mono);fill:var(--text-3)}
.placeholder{border:1px dashed #2c3a52;border-radius:13px;padding:34px 24px;text-align:center;color:var(--text-2)}
.placeholder b{display:block;color:var(--text);margin-bottom:6px}
.empty{padding:26px;text-align:center;color:var(--text-3);font-size:13px}
.sec-label{font-size:11px;font-weight:650;letter-spacing:.06em;text-transform:uppercase;color:var(--text-3);margin:14px 20px 6px}
.sec-label:first-child{margin-top:18px}
.field-row{display:flex;gap:10px;align-items:center;margin-bottom:10px}
.field-row label{font-size:12px;color:var(--text-2);width:200px;flex:none}
@media(max-width:960px){.sidebar{display:none}.grid-kpi,.split{grid-template-columns:1fr 1fr}}
</style>
</head>
<body>
<main class="page">
<header class="top">
<div><p class="eyebrow">Admin panel</p><h1>User &amp; run control</h1></div>
<div style="display:flex;gap:10px;align-items:center"><a href="/ui">← Dashboard</a><a href="/ui/profile">Profile</a><form method="post" action="/ui/logout" style="margin:0"><button class="btn btn-muted" type="submit">Log out</button></form></div>
</header>
<p class="lead">Signed in as <strong>{{.Role}}</strong>. Promote or verify a user by their id, and control every job from the dashboard.</p>
<div class="layout">
{{if .Msg}}<div class="notice ok">{{.Msg}}</div>{{end}}
{{if .Error}}<div class="notice err">{{.Error}}</div>{{end}}
<aside class="sidebar">
<div class="brand">
<div class="brand-mark"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round"><circle cx="6" cy="6" r="2.4"/><circle cx="18" cy="7" r="2.4"/><circle cx="12" cy="17" r="2.4"/><path d="M8 7.5l7.6-1M7 8.3l3.7 6.5M16.6 9.2l-3.2 5.6"/></svg></div>
<div><div class="brand-name">SciMesh</div><div class="brand-sub">Coordinator Admin</div></div>
</div>
<nav class="nav" id="nav">
<div class="nav-label">Operate</div>
<button class="nav-item active" data-page="system"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><rect x="3" y="3" width="8" height="8" rx="2"/><rect x="13" y="3" width="8" height="5" rx="2"/><rect x="13" y="10" width="8" height="11" rx="2"/><rect x="3" y="13" width="8" height="8" rx="2"/></svg>System</button>
<button class="nav-item" data-page="jobs"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><path d="M4 6h16M4 12h16M4 18h10"/></svg>Jobs</button>
<button class="nav-item" data-page="workers"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8M12 16v4"/></svg>Workers</button>
<div class="nav-label">Access</div>
<button class="nav-item" data-page="users"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><circle cx="9" cy="8" r="3.2"/><path d="M3.5 19c.7-3 2.9-4.5 5.5-4.5s4.8 1.5 5.5 4.5"/><circle cx="17" cy="9" r="2.4"/><path d="M15.5 14.6c2.6.2 4.3 1.7 5 4.4"/></svg>Users &amp; keys</button>
<div class="nav-label">Platform</div>
<button class="nav-item" data-page="workloads"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><path d="M12 3l8 4.5v9L12 21l-8-4.5v-9L12 3z"/><path d="M12 12l8-4.5M12 12v9M12 12L4 7.5"/></svg>Workloads</button>
<button class="nav-item" data-page="metrics"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><path d="M4 19V5M4 19h16"/><path d="M8 15v-4M12 15V7M16 15v-6M20 15V9"/></svg>Metrics</button>
<button class="nav-item" data-page="settings"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><circle cx="12" cy="12" r="3"/><path d="M19 12a7 7 0 0 0-.1-1.2l2-1.6-2-3.4-2.4 1a7 7 0 0 0-2-1.2L14 3h-4l-.5 2.6a7 7 0 0 0-2 1.2l-2.4-1-2 3.4 2 1.6A7 7 0 0 0 5 12c0 .4 0 .8.1 1.2l-2 1.6 2 3.4 2.4-1a7 7 0 0 0 2 1.2L10 21h4l.5-2.6a7 7 0 0 0 2-1.2l2.4 1 2-3.4-2-1.6c.1-.4.1-.8.1-1.2z"/></svg>Settings</button>
</nav>
<div class="side-foot">
<div class="user-chip"><div class="avatar">{{.Role}}</div><div><b>Signed in as {{.Role}}</b><span>cluster administrator</span></div></div>
<a class="back-link" href="/ui/jobs/new"> New computation</a>
</div>
</aside>
<section class="card">
<h2>Manage a user</h2>
<p>Paste the user id (the JWT <code>sub</code> / the value shown at registration). Actions are applied immediately.</p>
<form method="post" action="/ui/admin/user-action">
<label for="user_id">User id</label>
<input id="user_id" name="user_id" placeholder="00000000-0000-0000-0000-000000000000" autocomplete="off" required>
<p class="hint">Promote makes them an admin; Verify marks them a trusted contributor (their workers skip quorum).</p>
<div class="actions">
<button class="btn btn-primary" name="action" value="promote" type="submit">Make admin</button>
<button class="btn btn-muted" name="action" value="demote" type="submit">Remove admin</button>
<button class="btn btn-primary" name="action" value="verify" type="submit">Verify</button>
<button class="btn btn-muted" name="action" value="unverify" type="submit">Unverify</button>
<div class="main">
<header class="topbar">
<div><h1 id="page-title">System</h1><p id="page-sub">Cluster state and node information</p></div>
<div style="display:flex;gap:10px;align-items:center"><a class="btn btn-primary" href="/ui/jobs/new"> New computation</a><div class="env-badge"><i></i><span id="env-label">admin console</span></div></div>
</header>
<div class="content">
<!-- ═══ SYSTEM ═══ -->
<section class="page active" id="page-system">
<div class="grid-kpi">
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M12 3l8 4.5v9L12 21l-8-4.5v-9L12 3z"/></svg>Version</div><div class="k-value" id="k-version"></div><div class="k-sub" id="k-version-sub">loading…</div></div>
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="12" cy="12" r="9"/><path d="M12 7v5l3 3"/></svg>Uptime</div><div class="k-value" id="k-uptime"></div><div class="k-sub" id="k-started">loading…</div></div>
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M4 6h16M4 12h16M4 18h10"/></svg>Active jobs</div><div class="k-value" id="k-active"></div><div class="k-sub" id="k-active-sub">loading…</div></div>
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8"/></svg>Workers online</div><div class="k-value" id="k-workers"></div><div class="k-sub" id="k-workers-sub">loading…</div></div>
</div>
<div class="split">
<div class="card">
<div class="card-head"><h3>Storage usage</h3><span id="storage-total"></span></div>
<div class="storage-bar" id="storage-bar"><div style="width:0;background:#5b8cff"></div><div style="width:0;background:#7c5cff"></div><div style="width:0;background:#3fce8a"></div></div>
<div class="legend">
<span><i style="background:#5b8cff"></i>Datasets · <b id="storage-datasets"></b></span>
<span><i style="background:#7c5cff"></i>Artifacts · <b id="storage-artifacts"></b></span>
<span><i style="background:#3fce8a"></i>Database · <b id="storage-db"></b></span>
</div>
</form>
</section>
</div>
<div class="card">
<div class="card-head"><h3>Health</h3><span id="health-state"></span></div>
<dl class="kv">
<dt>Database</dt><dd id="h-db"></dd>
<dt>Userservice</dt><dd id="h-users"></dd>
<dt>Reducer</dt><dd id="h-reducer"></dd>
</dl>
</div>
</div>
<div class="section-title">Node information</div>
<div class="card">
<dl class="kv">
<dt>Binary</dt><dd><code id="n-binary"></code></dd>
<dt>Listen address</dt><dd><code id="n-addr"></code></dd>
<dt>Data directory</dt><dd><code id="n-datadir"></code></dd>
<dt>Database engine</dt><dd id="n-engine"></dd>
<dt>Public URL</dt><dd><code id="n-public"></code></dd>
</dl>
</div>
</section>
<section class="card">
<h2>Jobs &amp; tasks</h2>
<p>As an admin you already see <strong>every user's jobs</strong> on the dashboard, with per-task status and job cancellation. A regular user sees only their own.</p>
<div class="actions"><a class="btn btn-muted" href="/ui" style="text-decoration:none">Open the dashboard →</a></div>
</section>
</main>
<!-- ═══ JOBS ═══ -->
<section class="page" id="page-jobs">
<div class="tabs" id="job-tabs"></div>
<div class="card">
<table>
<thead><tr><th>Job</th><th>Workload</th><th>Owner</th><th>Status</th><th>Progress</th><th>Submitted</th><th></th></tr></thead>
<tbody id="job-rows"></tbody>
</table>
<div class="footer-row"><span id="job-range"></span><div class="pager"><button id="pg-prev" aria-label="previous"></button><button id="pg-next" aria-label="next"></button></div></div>
</div>
</section>
<!-- ═══ WORKERS ═══ -->
<section class="page" id="page-workers">
<div class="section-note">Workers register themselves. Trust decides whether a machine's results are accepted directly or need quorum.</div>
<div class="card">
<table>
<thead><tr><th>Worker</th><th>Status</th><th>Capabilities</th><th>Trust</th><th>Owner</th><th>Last signal</th><th></th></tr></thead>
<tbody id="worker-rows"></tbody>
</table>
</div>
</section>
<!-- ═══ USERS & KEYS ═══ -->
<section class="page" id="page-users">
<div class="section-title">Users</div>
<div class="card">
<table>
<thead><tr><th>Email</th><th>Role</th><th>Verified</th><th>Created</th></tr></thead>
<tbody id="user-rows"></tbody>
</table>
<div class="footer-row"><span id="user-count"></span></div>
</div>
<div class="section-title">Worker keys</div>
<div class="section-note">Keys let lab machines register as workers under a user account. Served instances can also use the cluster token.</div>
<div class="card">
<table>
<thead><tr><th>Label</th><th>Prefix</th><th>Owner</th><th>Created</th><th>Last used</th><th></th></tr></thead>
<tbody id="key-rows"></tbody>
</table>
</div>
<div class="section-title">Quick user action</div>
<div class="card">
<form method="post" action="/ui/admin/user-action" style="padding:16px 20px">
<div class="field-row"><label for="user_id">User id</label><input id="user_id" name="user_id" placeholder="00000000-0000-0000-0000-000000000000" autocomplete="off" required style="flex:1"></div>
<div style="display:flex;gap:8px;margin-left:210px;flex-wrap:wrap">
<button class="btn btn-primary" name="action" value="promote" type="submit">Make admin</button>
<button class="btn btn-ghost" name="action" value="demote" type="submit">Remove admin</button>
<button class="btn btn-primary" name="action" value="verify" type="submit">Verify</button>
<button class="btn btn-ghost" name="action" value="unverify" type="submit">Unverify</button>
</div>
{{if .Msg}}<div style="margin-left:210px;margin-top:10px;color:var(--green);font-size:13px">✓ {{.Msg}}</div>{{end}}
{{if .Error}}<div style="margin-left:210px;margin-top:10px;color:var(--red);font-size:13px">✗ {{.Error}}</div>{{end}}
</form>
</div>
</section>
<!-- ═══ WORKLOADS ═══ -->
<section class="page" id="page-workloads">
<div class="section-note">Disabled workloads are rejected at submit time and hidden from the job form. Settings persist in the database.</div>
<div class="card">
<table>
<thead><tr><th>Workload</th><th>Reduction</th><th>Parameters</th><th>Dataset upload</th><th>Enabled</th></tr></thead>
<tbody id="workload-rows"></tbody>
</table>
</div>
</section>
<!-- ═══ METRICS ═══ -->
<section class="page" id="page-metrics">
<div class="grid-kpi">
<div class="kpi"><div class="k-label">Jobs · 7 days</div><div class="k-value" id="m-jobs7"></div><div class="k-sub">created in the last week</div></div>
<div class="kpi"><div class="k-label">Shards completed</div><div class="k-value" id="m-shards"></div><div class="k-sub" id="m-shards-sub">across all workers</div></div>
<div class="kpi"><div class="k-label">Avg shard time</div><div class="k-value" id="m-avg"></div><div class="k-sub">completed shards only</div></div>
<div class="kpi"><div class="k-label">Failure rate</div><div class="k-value" id="m-failrate"></div><div class="k-sub" id="m-failrate-sub"></div></div>
</div>
<div class="split">
<div class="card">
<div class="card-head"><h3>Jobs per day</h3><span>last 7 days</span></div>
<div style="padding:16px 20px 10px" id="chart-days"></div>
</div>
<div class="card">
<div class="card-head"><h3>Jobs by workload</h3><span>all time</span></div>
<dl class="kv" id="chart-workloads"></dl>
</div>
</div>
</section>
<!-- ═══ SETTINGS ═══ -->
<section class="page" id="page-settings">
<div class="warn-strip" style="display:flex;gap:10px;align-items:flex-start;background:var(--amber-soft);border:1px solid #e5b64f33;border-radius:10px;padding:12px 14px;font-size:12.5px;color:#eecf8d"><span>The cluster token below authenticates <b>any</b> worker. Reveal it only on a trusted machine.</span></div>
<div class="section-title" style="color:var(--red)">Danger zone</div>
<div class="card" style="border-color:#f2647c33">
<dl class="kv">
<dt>Prune artifacts</dt><dd style="display:flex;justify-content:space-between;align-items:center;gap:14px"><span style="color:var(--text-2);font-size:12.5px">Delete finished jobs older than a cutoff and all their artifacts (blob files included).</span><span style="display:flex;gap:8px;align-items:center"><input id="prune-days" type="number" min="1" max="3650" value="30" style="width:80px"><button class="btn btn-danger btn-sm" id="prune">Prune…</button></span></dd>
</dl>
</div>
<div class="section-title">Cluster</div>
<div class="card">
<dl class="kv">
<dt>Worker token</dt><dd><div class="secret"><code id="tok">••••••••••••••••••••••••</code><button class="btn btn-ghost btn-sm" id="reveal">Reveal</button></div></dd>
<dt>Public URL</dt><dd><code id="s-public"></code></dd>
<dt>Listen address</dt><dd><code id="s-addr"></code></dd>
<dt>Data directory</dt><dd><code id="s-datadir"></code></dd>
<dt>Database engine</dt><dd id="s-engine"></dd>
<dt>Binary</dt><dd><code id="s-binary"></code></dd>
</dl>
</div>
</section>
</div>
</div>
</div>
<script>
const titles={system:['System','Cluster state and node information'],jobs:['Jobs','Every computation, filterable and paginated'],workers:['Workers','Fleet overview and trust management'],users:['Users & keys','Accounts, roles and worker keys'],workloads:['Workloads','Catalog entries and availability'],metrics:['Metrics','Throughput and reliability, last 7 days'],settings:['Settings','Cluster, storage and security']};
const statusLabel={pending:'Waiting',leased:'Assigned',running:'Running',reducing:'Merging',completed:'Completed',failed:'Failed',cancelled:'Cancelled'};
const statusClass={pending:'pill-waiting',leased:'pill-active',running:'pill-active',reducing:'pill-active',completed:'pill-success',failed:'pill-danger',cancelled:'pill-waiting'};
const fmtBytes=b=>{if(b==null||b<0)return '';if(b<1024)return b+' B';if(b<1048576)return (b/1024).toFixed(1)+' KB';if(b<1073741824)return (b/1048576).toFixed(1)+' MB';return (b/1073741824).toFixed(2)+' GB'};
const fmtTime=t=>t?new Date(t).toLocaleString():'—';
const esc=s=>String(s).replace(/[&<>"]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
let timer=null,current={page:'system',jobsStatus:'',jobsPage:1};
const setPage=page=>{current.page=page;document.querySelectorAll('.nav-item').forEach(b=>b.classList.toggle('active',b.dataset.page===page));document.querySelectorAll('.page').forEach(p=>p.classList.toggle('active',p.id==='page-'+page));const [t,s]=titles[page];document.getElementById('page-title').textContent=t;document.getElementById('page-sub').textContent=s;if(timer){clearInterval(timer);timer=null}refresh();timer=setInterval(refresh,5000)};
document.querySelectorAll('.nav-item').forEach(b=>b.addEventListener('click',()=>setPage(b.dataset.page)));
const refresh=()=>{if(document.hidden)return;const p=current.page;if(p==='system')loadSystem();else if(p==='jobs')loadJobs();else if(p==='metrics')loadMetrics();else if(p==='workers')loadWorkers();else if(p==='users')loadUsers();else if(p==='workloads')loadWorkloads();else if(p==='settings')loadSettings()};
document.addEventListener('visibilitychange',()=>{if(!document.hidden)refresh()});
async function loadSystem(){
const r=await fetch('/ui/admin/api/system',{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
document.getElementById('k-version').textContent=v.version;
document.getElementById('k-version-sub').textContent=v.node.db_engine+' · '+navigator.platform;
const secs=v.uptime_seconds;
const uptime=secs<3600?(secs/60).toFixed(0)+' min':secs<86400?(secs/3600).toFixed(1)+' h':(secs/86400).toFixed(1)+' d';
document.getElementById('k-uptime').textContent=uptime;
document.getElementById('k-started').textContent='since '+fmtTime(v.started_at);
document.getElementById('k-active').textContent=v.active_jobs;
document.getElementById('k-active-sub').textContent=v.running_jobs+' running · '+v.waiting_jobs+' waiting';
document.getElementById('k-workers').textContent=v.workers_online;
document.getElementById('k-workers-sub').textContent=(v.workers_total-v.workers_online)+' offline · '+v.workers_busy+' busy';
const total=v.storage.datasets_bytes+v.storage.artifacts_bytes+v.storage.database_bytes;
document.getElementById('storage-total').textContent=fmtBytes(total);
const pct=b=>total?Math.round(b*100/total)+'%':'0%';
document.getElementById('storage-bar').children[0].style.width=pct(v.storage.datasets_bytes);
document.getElementById('storage-bar').children[1].style.width=pct(v.storage.artifacts_bytes);
document.getElementById('storage-bar').children[2].style.width=pct(v.storage.database_bytes);
document.getElementById('storage-datasets').textContent=fmtBytes(v.storage.datasets_bytes);
document.getElementById('storage-artifacts').textContent=fmtBytes(v.storage.artifacts_bytes);
document.getElementById('storage-db').textContent=fmtBytes(v.storage.database_bytes);
document.getElementById('h-db').innerHTML=pill(v.health.database==='connected'?'Connected':'Error','pill-success',v.health.database==='connected'?'pill-danger':null);
document.getElementById('h-users').innerHTML=pill(cap(v.health.userservice),'pill-success',null);
document.getElementById('h-reducer').innerHTML=pill(cap(v.health.reducer),'pill-waiting',null);
document.getElementById('health-state').textContent=v.health.database==='connected'?'all checks pass':'database unreachable';
document.getElementById('n-binary').textContent=v.node.binary||'—';
document.getElementById('n-addr').textContent=v.node.addr;
document.getElementById('n-datadir').textContent=v.node.data_dir||'—';
document.getElementById('n-engine').textContent=v.node.db_engine;
document.getElementById('n-public').textContent=v.node.public_url||'—';
document.getElementById('env-label').textContent=(v.node.public_url||'').replace(/^https?:\/\//,'')+' · '+v.node.db_engine+' · '+v.node.addr;
}
const cap=s=>s?s.charAt(0).toUpperCase()+s.slice(1):'—';
const pill=(text,cls,fail)=>{const c=fail||cls;return '<span class="pill '+c+'"><i></i>'+esc(text)+'</span>'};
let jobFilter={};
async function loadJobs(){
const page=current.jobsPage,status=current.jobsStatus;
const r=await fetch('/ui/admin/api/jobs?page='+page+'&per_page=10'+(status?'&status='+status:''),{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
const tabs=document.getElementById('job-tabs');
const defs=[['', 'All'],['running','Running'],['pending','Waiting'],['completed','Completed'],['failed','Failed'],['cancelled','Cancelled']];
tabs.replaceChildren();
for(const [key,label] of defs){
const n=v.counts[key]||0;
const b=document.createElement('button');
b.className='tab'+(key===status?' active':'');
b.innerHTML=label+'<span class="n">'+n+'</span>';
b.addEventListener('click',()=>{current.jobsStatus=key;current.jobsPage=1;loadJobs()});
tabs.append(b);
}
const rows=document.getElementById('job-rows');
rows.replaceChildren();
if(!v.jobs.length){const tr=document.createElement('tr');tr.innerHTML='<td colspan="6"><div class="empty">No jobs'+(status?' with this status':'')+'.</div></td>';rows.append(tr)}
for(const j of v.jobs){
const tr=document.createElement('tr');
const pct=j.total?Math.min(100,Math.round((j.completed+j.failed)*100/j.total)):0;
tr.innerHTML='<td><div class="t-main">'+esc(j.workload)+'</div><div class="t-sub">'+esc(j.id.slice(0,8))+'…</div></td>'+
'<td><code style="color:var(--text-2)">'+esc(j.workload)+'</code></td>'+
'<td style="color:var(--text-2)">'+esc(j.owner)+'</td>'+
'<td>'+pill(statusLabel[j.status]||j.status,statusClass[j.status]||'pill-waiting',null)+'</td>'+
'<td><div class="bar"><span style="width:'+pct+'%"></span></div><div class="bar-label">'+j.completed+' / '+j.total+' shards'+(j.failed?' · '+j.failed+' failed':'')+'</div></td>'+
'<td style="color:var(--text-2);font-size:12.5px">'+fmtTime(j.created_at)+'</td>'+
'<td><a class="btn btn-ghost btn-sm" href="/ui/jobs/'+encodeURIComponent(j.id)+'">Open</a></td>';
rows.append(tr);
}
const from=(v.page-1)*v.per_page+1,to=Math.min(v.page*v.per_page,v.total);
document.getElementById('job-range').textContent=v.total?(from+''+to+' of '+v.total+' jobs'):'no jobs';
const prev=document.getElementById('pg-prev'),next=document.getElementById('pg-next');
prev.disabled=v.page<=1;next.disabled=v.page*v.per_page>=v.total;
prev.onclick=()=>{current.jobsPage--;loadJobs()};
next.onclick=()=>{current.jobsPage++;loadJobs()};
}
function dayChart(days){
const max=Math.max(1,...days.map(d=>d.count));
const w=460,h=150,bw=44,gap=18,base=118;
let bars='';
days.forEach((d,i)=>{const x=12+i*(bw+gap),bh=Math.round(d.count*base/max);bars+='<rect class="chart-bar'+(d.count===max&&d.count>0?' hot':'')+'" x="'+x+'" y="'+(base-bh+6)+'" width="'+bw+'" height="'+bh+'"/>';bars+='<text class="chart-label" x="'+x+'" y="146">'+d.day.slice(5)+'</text>'});
return '<svg class="chart" viewBox="0 0 '+w+' '+h+'"><line class="chart-grid" x1="0" y1="30" x2="'+w+'" y2="30"/><line class="chart-grid" x1="0" y1="60" x2="'+w+'" y2="60"/><line class="chart-grid" x1="0" y1="90" x2="'+w+'" y2="90"/>'+bars+'</svg>';
}
async function loadMetrics(){
const r=await fetch('/ui/admin/api/metrics',{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
document.getElementById('m-jobs7').textContent=v.jobs_last_7_days;
document.getElementById('m-shards').textContent=v.shards_completed.toLocaleString();
document.getElementById('m-shards-sub').textContent=(v.shards_completed+v.shards_failed)+' shards · '+v.shards_failed+' failed';
document.getElementById('m-avg').textContent=v.avg_shard_seconds?v.avg_shard_seconds.toFixed(1)+'s':'—';
document.getElementById('m-failrate').textContent=(v.failure_rate*100).toFixed(1)+'%';
document.getElementById('m-failrate-sub').textContent=v.shards_failed+' of '+(v.shards_completed+v.shards_failed)+' shards failed';
document.getElementById('chart-days').innerHTML=dayChart(v.jobs_by_day);
const wl=document.getElementById('chart-workloads');
wl.replaceChildren();
if(!v.jobs_by_workload.length){const p=document.createElement('p');p.className='empty';p.textContent='No workloads used yet.';wl.append(p)}
const max=Math.max(1,...v.jobs_by_workload.map(w=>w.count));
for(const w of v.jobs_by_workload){
const dt=document.createElement('dt');dt.textContent=w.workload;
const dd=document.createElement('dd');
dd.innerHTML='<div class="bar" style="width:100%"><span style="width:'+Math.round(w.count*100/max)+'%"></span></div>';
wl.append(dt,dd);
}
}
setPage('system');
const workerStatusPill=s=>({online:['Online','pill-success'],busy:['Busy','pill-active'],offline:['Offline','pill-waiting']}[s]||[s,'pill-waiting']);
async function loadWorkers(){
const r=await fetch('/ui/admin/api/workers',{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
const rows=document.getElementById('worker-rows');
rows.replaceChildren();
if(!v.workers.length){const tr=document.createElement('tr');tr.innerHTML='<td colspan="6"><div class="empty">No worker is registered yet.</div></td>';rows.append(tr);return}
for(const w of v.workers){
const tr=document.createElement('tr');
const [label,cls]=workerStatusPill(w.status);
const trustSel='<select data-id="'+w.id+'" class="trust-sel" '+(w.status==='offline'?'disabled':'')+'><option value="trusted" '+(w.trust==='trusted'?'selected':'')+'>Trusted</option><option value="untrusted" '+(w.trust==='untrusted'?'selected':'')+'>Untrusted</option></select>';
tr.innerHTML='<td><div class="t-main">'+esc(w.name)+'</div><div class="t-sub">'+esc(w.id.slice(0,8))+'…</div></td>'+
'<td>'+pill(label,cls,null)+'</td>'+
'<td>'+(w.capabilities||[]).map(c=>'<span class="cap">'+esc(c)+'</span>').join('')+'</td>'+
'<td>'+trustSel+'</td>'+
'<td style="color:var(--text-2)">'+esc(w.owner)+'</td>'+
'<td style="color:var(--text-2);font-size:12.5px">'+fmtTime(w.last_heartbeat_at)+'</td>'+
'<td>'+(w.status==='offline'?'<button class="btn btn-danger btn-sm worker-remove" data-id="'+w.id+'" data-name="'+esc(w.name)+'">Remove</button>':'')+'</td>';
rows.append(tr);
}
document.querySelectorAll('.worker-remove').forEach(btn=>btn.addEventListener('click',async()=>{
if(!confirm('Remove the offline worker "'+btn.dataset.name+'" from the registry? This cannot be undone.'))return;
await fetch('/ui/admin/api/workers/'+btn.dataset.id+'/remove',{method:'POST'});
loadWorkers();
}));
document.querySelectorAll('.trust-sel').forEach(sel=>sel.addEventListener('change',async()=>{
await fetch('/ui/admin/api/workers/'+sel.dataset.id+'/trust',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({trusted:sel.value==='trusted'})});
loadWorkers();
}));
}
async function loadUsers(){
const r=await fetch('/ui/admin/api/users',{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
const rows=document.getElementById('user-rows');
rows.replaceChildren();
for(const u of v.users||[]){
const tr=document.createElement('tr');
const roleSel='<select class="role-sel" data-id="'+u.id+'"><option value="user" '+(u.role==='user'?'selected':'')+'>user</option><option value="admin" '+(u.role==='admin'?'selected':'')+'>admin</option></select>';
tr.innerHTML='<td><div class="t-main">'+esc(u.email)+'</div></td>'+
'<td>'+roleSel+'</td>'+
'<td>'+pill(u.verified?'Verified':'—',u.verified?'pill-success':'pill-waiting',null)+'</td>'+
'<td style="color:var(--text-2);font-size:12.5px">'+fmtTime(u.created_at)+'</td>';
rows.append(tr);
}
document.getElementById('user-count').textContent=(v.users||[]).length+' users';
document.querySelectorAll('.role-sel').forEach(sel=>sel.addEventListener('change',async()=>{
await fetch('/ui/admin/api/users/'+sel.dataset.id+'/role',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({role:sel.value})});
loadUsers();
}));
const keys=await (await fetch('/ui/admin/api/worker-keys',{headers:{Accept:'application/json'}})).json();
const keyRows=document.getElementById('key-rows');
keyRows.replaceChildren();
const emailOf={};for(const u of v.users||[])emailOf[u.id]=u.email;
for(const k of keys.worker_keys||[]){
const tr=document.createElement('tr');
tr.innerHTML='<td class="t-main">'+esc(k.name)+'</td>'+
'<td><code style="color:var(--text-2)">'+esc(k.prefix)+'…</code></td>'+
'<td style="color:var(--text-2)">'+esc(emailOf[k.user_id]||k.user_id.slice(0,8)+'…')+'</td>'+
'<td style="color:var(--text-2);font-size:12.5px">'+fmtTime(k.created_at)+'</td>'+
'<td style="color:var(--text-2);font-size:12.5px">'+(k.last_used_at?fmtTime(k.last_used_at):'never')+'</td>'+
'<td>'+((k.revoked_at)?'<span class="pill pill-danger"><i></i>Revoked</span>':'<button class="btn btn-danger btn-sm key-revoke" data-id="'+k.id+'">Revoke</button>')+'</td>';
keyRows.append(tr);
}
document.querySelectorAll('.key-revoke').forEach(btn=>btn.addEventListener('click',async()=>{
if(!confirm('Revoke this worker key? The machine will be cut off on its next refresh.'))return;
await fetch('/ui/admin/api/worker-keys/'+btn.dataset.id+'/revoke',{method:'POST'});
loadUsers();
}));
}
async function loadWorkloads(){
const r=await fetch('/ui/admin/api/workloads',{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
const rows=document.getElementById('workload-rows');
rows.replaceChildren();
for(const w of v.workloads||[]){
const tr=document.createElement('tr');
tr.innerHTML='<td><div class="t-main">'+esc(w.name)+'</div><div class="t-sub" style="font-family:inherit">'+esc(w.description||'')+'</div></td>'+
'<td><span class="pill pill-active"><i></i>'+esc(w.reduction)+'</span></td>'+
'<td style="color:var(--text-2);font-size:12.5px">'+w.parameters+' declared</td>'+
'<td>'+pill(w.upload_ready?'ready':'—',w.upload_ready?'pill-success':'pill-waiting',null)+'</td>'+
'<td><button class="toggle wl-toggle '+(w.enabled?'on':'')+'" data-name="'+w.name+'" aria-label="enabled"></button></td>';
rows.append(tr);
}
document.querySelectorAll('.wl-toggle').forEach(t=>t.addEventListener('click',async()=>{
const enabled=!t.classList.contains('on');
await fetch('/ui/admin/api/workloads/'+encodeURIComponent(t.dataset.name)+'/enabled',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({enabled})});
t.classList.toggle('on',enabled);
}));
}
async function loadSettings(){
const r=await fetch('/ui/admin/api/settings',{headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
document.getElementById('s-public').textContent=v.public_url||'—';
document.getElementById('s-addr').textContent=v.addr;
document.getElementById('s-datadir').textContent=v.data_dir||'—';
document.getElementById('s-engine').textContent=v.db_engine;
document.getElementById('s-binary').textContent=v.binary||'—';
}
document.getElementById('reveal').addEventListener('click',async e=>{
const tok=document.getElementById('tok');
if(tok.textContent.startsWith('•')){
const r=await fetch('/ui/admin/api/token/reveal',{method:'POST',headers:{Accept:'application/json'}});
if(!r.ok)return;
const v=await r.json();
tok.textContent=v.token||'(none)';
e.target.textContent='Hide';
}else{tok.textContent='••••••••••••••••••••••••';e.target.textContent='Reveal'}
});
document.querySelectorAll('.toggle').forEach(t=>t.addEventListener('click',()=>t.classList.toggle('on')));
document.getElementById('prune').addEventListener('click',async()=>{
const days=parseInt(document.getElementById('prune-days').value||'30',10);
if(!confirm('Delete finished jobs older than '+days+' days with all their artifacts? This cannot be undone.'))return;
const btn=document.getElementById('prune');btn.disabled=true;btn.textContent='Pruning…';
const r=await fetch('/ui/admin/api/prune',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({older_than_days:days})});
btn.disabled=false;btn.textContent='Prune…';
if(!r.ok){alert('Prune failed.');return}
const v=await r.json();
alert('Removed '+v.jobs+' jobs and '+v.artifacts+' artifacts, freed '+fmtBytes(v.freed_bytes)+'.');
if(current.page==='settings')loadSettings();
});
</script>
</body>
</html>
{{end}}
@@ -1,42 +0,0 @@
{{define "dashboard.html"}}
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SciMesh control room</title>
<style>
:root{color:#dce8ff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 15% -10%,#163d77 0,transparent 32rem),radial-gradient(circle at 95% 5%,#123e39 0,transparent 29rem),#08111f}.page{max-width:1240px;margin:auto;padding:28px 22px 64px}.top{display:flex;align-items:flex-start;justify-content:space-between;gap:22px}.eyebrow{margin:0;color:#78a9ff;font-size:.77rem;font-weight:750;letter-spacing:.14em;text-transform:uppercase}.title{max-width:700px;margin:8px 0;font-size:clamp(2rem,5vw,3.6rem);line-height:1.04;letter-spacing:-.055em}.lead{max-width:690px;margin:0;color:#aabbd5;font-size:1.05rem}.button{display:inline-flex;align-items:center;gap:8px;border:0;border-radius:10px;padding:12px 16px;background:#4f8cff;color:#071224;font:inherit;font-weight:800;text-decoration:none;box-shadow:0 12px 30px #163d7766}.live{display:inline-flex;align-items:center;gap:7px;margin-top:18px;color:#8ba2c2;font-size:.87rem}.pulse{width:8px;height:8px;border-radius:50%;background:#5ee6a6;box-shadow:0 0 0 5px #5ee6a622}.summary{display:grid;grid-template-columns:1.4fr repeat(3,1fr);gap:13px;margin:32px 0}.panel,.metric,.flow-step,.job,.worker{border:1px solid #26415f;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #0000001f}.panel{padding:20px}.pipeline{display:grid;grid-template-columns:repeat(4,1fr);gap:10px;margin-top:16px}.flow-step{position:relative;padding:14px;min-height:116px}.flow-step:not(:last-child):after{content:"";position:absolute;top:35px;right:-10px;width:10px;height:2px;background:#3c5d83}.flow-step b{display:block;color:#f2f7ff}.flow-step small{display:block;margin-top:6px;color:#91a8c6}.flow-step .dot{display:inline-block;width:9px;height:9px;margin-right:7px;border-radius:50%;background:#5ee6a6}.metric{padding:16px}.metric b{display:block;margin-top:7px;color:#f5f8ff;font-size:2rem;line-height:1}.metric span{color:#9bb0cc;font-size:.84rem}.section-head{display:flex;align-items:baseline;justify-content:space-between;gap:12px;margin:36px 0 12px}.section-head h2{margin:0;color:#f3f7ff;font-size:1.18rem}.section-head p{margin:0;color:#8fa6c3;font-size:.9rem}.jobs{display:grid;gap:10px}.job{display:grid;grid-template-columns:minmax(210px,1.35fr) minmax(150px,.9fr) minmax(180px,1fr) auto;gap:18px;align-items:center;padding:17px 18px;text-decoration:none;color:inherit;transition:border-color .15s,transform .15s}.job:hover{border-color:#5d96ee;transform:translateY(-1px)}.job-name{color:#f3f7ff;font-weight:750}.job-id{margin-top:3px;color:#8196b3;font-family:ui-monospace,SFMono-Regular,monospace;font-size:.76rem}.badge{display:inline-flex;align-items:center;border-radius:999px;padding:4px 9px;font-size:.78rem;font-weight:800}.badge-waiting{background:#23344d;color:#b9cce9}.badge-active{background:#173d77;color:#9fc7ff}.badge-success{background:#123f34;color:#76efb5}.badge-danger{background:#552334;color:#ff9bad}.bar{height:7px;margin-top:8px;overflow:hidden;border-radius:999px;background:#20344e}.bar span{display:block;height:100%;border-radius:inherit;background:linear-gradient(90deg,#5a92ff,#61e1bd)}.numbers{color:#afc0d9;font-size:.88rem}.arrow{color:#79aaff;font-size:1.35rem}.workers{display:grid;grid-template-columns:repeat(3,1fr);gap:11px}.worker{padding:15px}.worker-head{display:flex;justify-content:space-between;gap:8px}.worker strong{color:#f3f7ff}.worker small,.worker p{color:#95a9c4}.worker p{margin:12px 0 0}.cap{display:inline-block;margin:4px 5px 0 0;border:1px solid #365576;border-radius:5px;padding:2px 6px;color:#a9c9f4;font:.75rem ui-monospace,SFMono-Regular,monospace}.empty{padding:30px;border:1px dashed #35516f;border-radius:14px;color:#9ab0cb;text-align:center}.offline{color:#faafbd}.sr{position:absolute;width:1px;height:1px;clip:rect(0,0,0,0);overflow:hidden;white-space:nowrap}@media(max-width:820px){.top,.section-head{display:block}.button{margin-top:18px}.summary{grid-template-columns:1fr 1fr}.panel{grid-column:span 2}.pipeline{grid-template-columns:1fr 1fr}.flow-step:not(:last-child):after{display:none}.job{grid-template-columns:1fr 1fr}.arrow{display:none}.workers{grid-template-columns:1fr 1fr}}@media(max-width:540px){.page{padding:22px 14px}.summary,.workers{grid-template-columns:1fr}.panel{grid-column:auto}.pipeline,.job{grid-template-columns:1fr}.title{font-size:2.35rem}}
</style>
</head>
<body>
<main class="page">
<header class="top">
<div><p class="eyebrow">Local scientific compute</p><h1 class="title">SciMesh control room</h1><p class="lead">Follow the real path from a molecular TSV to a globally reduced similarity result—without reading coordinator logs.</p><div class="live"><i class="pulse"></i><span id="refresh-state">Live overview · refreshes every 2 seconds</span></div></div>
<div style="display:flex;gap:10px;align-items:center;flex-wrap:wrap">{{if .Session}}<span class="live" style="margin-top:0">Signed in · {{.Session.Role}}</span>{{end}}{{if .Session}}<a class="button" href="/ui/profile" style="background:#23344d;color:#dce8ff;box-shadow:none">Profile</a>{{end}}{{if and .Session (eq .Session.Role "admin")}}<a class="button" href="/ui/admin" style="background:#23344d;color:#dce8ff;box-shadow:none">Admin</a>{{end}}{{if .Session}}<a class="button" href="/ui/workloads" style="background:#23344d;color:#dce8ff;box-shadow:none">Workloads</a>{{end}}{{if .Session}}<a class="button" href="/ui/docs/" style="background:#23344d;color:#dce8ff;box-shadow:none">Docs</a>{{end}}{{if .Session}}<a class="button" href="/ui/workers/new" style="background:#23344d;color:#dce8ff;box-shadow:none">🖥 Add your machine</a>{{end}}<a class="button" href="/ui/jobs/new"> New computation</a>{{if .Session}}<form method="post" action="/ui/logout" style="margin:0"><button class="button" type="submit" style="background:#23344d;color:#dce8ff;box-shadow:none">Log out</button></form>{{end}}</div>
</header>
<section class="summary" aria-label="Pipeline summary">
<div class="panel"><strong>How a search becomes a result</strong><div class="pipeline"><div class="flow-step"><span><i class="dot"></i>01</span><b>Upload TSV</b><small>The coordinator validates and slices the dataset.</small></div><div class="flow-step"><span><i class="dot"></i>02</span><b>Run shards</b><small>Workers fingerprint molecules and return shard top-k CSVs.</small></div><div class="flow-step"><span><i class="dot"></i>03</span><b>Merge exactly</b><small>The coordinator ranks retained candidates deterministically.</small></div><div class="flow-step"><span><i class="dot"></i>04</span><b>Download CSV</b><small>A checksum-protected global result is ready.</small></div></div></div>
<div class="metric"><span>Active runs</span><b id="active-jobs">{{.ActiveJobs}}</b><small>waiting, running, or merging</small></div>
<div class="metric"><span>Available workers</span><b id="online-workers">{{.OnlineWorkers}}</b><small>recently registered</small></div>
<div class="metric"><span>Finished runs</span><b id="finished-jobs">{{.FinishedJobs}}</b><small>in the latest 20</small></div>
</section>
<section><div class="section-head"><h2>Recent computations</h2><p id="job-count">{{len .Jobs}} shown · newest first</p></div><div id="jobs" class="jobs">{{range .Jobs}}<a class="job" href="/ui/jobs/{{.ID}}"><div><div class="job-name">{{workloadLabel .Workload}}</div><div class="job-id">{{.ID}}</div></div><div><span class="badge badge-{{statusClass .Status}}">{{statusLabel .Status}}</span><div class="job-id">{{statusHint .Status}}</div></div><div><div class="numbers"><b>{{.Completed}}</b> / {{.Total}} shards complete{{if gt .Failed 0}} · <span class="offline">{{.Failed}} failed</span>{{end}}</div><div class="bar"><span style="width:{{progressPercent .Completed .Failed .Cancelled .Total}}%"></span></div></div><span class="arrow" aria-hidden="true"></span></a>{{else}}<div class="empty"><strong>No computations yet.</strong><br>Start a computation, then keep one or more workers running to watch this dashboard come alive.</div>{{end}}</div></section>
{{if and .Session (ne .Session.Role "admin")}}<section><div class="section-head"><h2>My machines</h2><p>Workers you registered. <a href="/ui/workers/new" style="color:#79aaff">Add your machine →</a></p></div><div id="my-workers" class="workers">{{range .MyWorkers}}<article class="worker"><div class="worker-head"><strong>{{.Name}}</strong><span class="badge badge-{{workerStatusClass .Status}}">{{workerStatusLabel .Status}}</span></div><small>{{.ID}}</small><p>{{range .Capabilities}}<span class="cap">{{.}}</span>{{end}}</p><p>Last signal · {{time .LastHeartbeatAt}}</p></article>{{else}}<div class="empty"><strong>No machine of yours is connected.</strong><br><a href="/ui/workers/new" style="color:#79aaff">Turn this computer into a worker →</a></div>{{end}}</div></section>{{end}}
<section><div class="section-head"><h2>Worker fleet</h2><p>Workers register themselves; this page never controls their processes.</p></div><div id="workers" class="workers">{{range .Workers}}<article class="worker"><div class="worker-head"><strong>{{.Name}}</strong><span class="badge badge-{{workerStatusClass .Status}}">{{workerStatusLabel .Status}}</span></div><small>{{.ID}}</small><p>{{range .Capabilities}}<span class="cap">{{.}}</span>{{end}}</p><p>Last signal · {{time .LastHeartbeatAt}}</p></article>{{else}}<div class="empty"><strong>No worker is registered.</strong><br>Start <code>scimesh-worker</code> in another terminal, then return here.</div>{{end}}</div></section>
</main>
<script>
const statusInfo={pending:['Waiting for a worker','waiting'],leased:['Assigned to a worker','active'],running:['Running','active'],reducing:['Merging results','active'],completed:['Completed','success'],failed:['Needs attention','danger'],cancelled:['Stopped','waiting']};
const pct=j=>j.total?Math.min(100,Math.floor((j.completed+j.failed+j.cancelled)*100/j.total)):0;
const node=(tag,text,cls)=>{const n=document.createElement(tag);if(text!==undefined)n.textContent=text;if(cls)n.className=cls;return n};
const renderJobs=jobs=>{const box=document.querySelector('#jobs');box.replaceChildren();if(!jobs.length){const empty=node('div',undefined,'empty');empty.append(node('strong','No computations yet.'),document.createElement('br'),document.createTextNode('Start a computation, then keep one or more workers running to watch this dashboard come alive.'));box.append(empty);return}for(const job of jobs){const info=statusInfo[job.status]||[job.status,'waiting'],link=node('a',undefined,'job');link.href='/ui/jobs/'+encodeURIComponent(job.id);const intro=node('div');intro.append(node('div',job.workload==='similarity-search'?'Molecule similarity search':job.workload,'job-name'),node('div',job.id,'job-id'));const state=node('div');state.append(node('span',info[0],'badge badge-'+info[1]),node('div',job.status==='reducing'?'Every shard is complete; coordinator is ranking the global top-k.':'Live coordinator state','job-id'));const progress=node('div'),numbers=node('div',undefined,'numbers');numbers.append(node('b',String(job.completed)),document.createTextNode(' / '+job.total+' shards complete'));if(job.failed){numbers.append(document.createTextNode(' · '),node('span',job.failed+' failed','offline'))}const bar=node('div',undefined,'bar'),fill=node('span');fill.style.width=pct(job)+'%';bar.append(fill);progress.append(numbers,bar);link.append(intro,state,progress,node('span','→','arrow'));box.append(link)}};
const workerCard=worker=>{const card=node('article',undefined,'worker'),head=node('div',undefined,'worker-head'),left=node('div'),workerInfo=worker.status==='online'?['Available','success']:worker.status==='busy'?['Busy','active']:['Offline','waiting'];left.append(node('strong',worker.name),node('small',worker.id));head.append(left,node('span',workerInfo[0],'badge badge-'+workerInfo[1]));const caps=node('p');for(const capability of worker.capabilities||[])caps.append(node('span',capability,'cap'));card.append(head,caps,node('p','Last signal · '+new Date(worker.last_heartbeat_at).toLocaleString()));return card};
const renderWorkers=workers=>{const box=document.querySelector('#workers');box.replaceChildren();if(!workers.length){const empty=node('div',undefined,'empty');empty.append(node('strong','No worker is registered.'),document.createElement('br'),document.createTextNode('Start scimesh-worker in another terminal, then return here.'));box.append(empty);return}for(const worker of workers)box.append(workerCard(worker))};
const renderMyWorkers=workers=>{const box=document.querySelector('#my-workers');if(!box)return;box.replaceChildren();if(!workers.length){const empty=node('div',undefined,'empty'),link=node('a','Turn this computer into a worker →');link.href='/ui/workers/new';link.style.color='#79aaff';empty.append(node('strong','No machine of yours is connected.'),document.createElement('br'),link);box.append(empty);return}for(const worker of workers)box.append(workerCard(worker))};
let timer;const refresh=async()=>{try{const response=await fetch('/ui/api/overview',{headers:{Accept:'application/json'}});if(!response.ok)throw Error();const view=await response.json();document.querySelector('#active-jobs').textContent=view.active_jobs;document.querySelector('#online-workers').textContent=view.online_workers;document.querySelector('#finished-jobs').textContent=view.finished_jobs;document.querySelector('#job-count').textContent=view.jobs.length+' shown · newest first';renderJobs(view.jobs);renderWorkers(view.workers);renderMyWorkers(view.my_workers||[]);document.querySelector('#refresh-state').textContent='Live overview · updated just now'}catch(_){document.querySelector('#refresh-state').textContent='Connection interrupted · trying again automatically'}};
const start=()=>{if(!timer&&!document.hidden)timer=setInterval(refresh,2000)};document.addEventListener('visibilitychange',()=>{if(document.hidden&&timer){clearInterval(timer);timer=undefined}else start()});start();
</script>
</body>
</html>
{{end}}
@@ -16,7 +16,7 @@
<div class="card">
<p>The documentation site has not been built or the coordinator has not been pointed at it. From the repository root, run:</p>
<p><code>make docs</code> &nbsp;then restart the coordinator with <code>SCIMESH_DOCS_DIR</code> set to the generated <code>site/</code> directory (the <code>make demo-ui</code> demo does this automatically).</p>
<a class="back" href="/ui">← Back to the control room</a>
<a class="back" href="/ui/admin">← Back to the control room</a>
</div>
</main>
</body>
File diff suppressed because one or more lines are too long
@@ -13,13 +13,17 @@
<p class="eyebrow">SciMesh</p>
<h1>Sign in</h1>
<form method="post" action="/ui/login">
{{if .Next}}<input type="hidden" name="next" value="{{.Next}}">{{end}}
<label for="email">Email</label>
<input id="email" name="email" type="email" autocomplete="username" required autofocus>
<label for="password">Password</label>
<input id="password" name="password" type="password" autocomplete="current-password" required>
<button class="button" type="submit">Sign in</button>
</form>
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
{{if eq .Error "admin role required"}}
<p class="error">The admin console is reserved for the cluster administrator.</p>
<p class="alt">You are signed in as a non-admin. <a href="/ui/logout-form">Log out</a>, then sign in with the admin account — its login is printed by <code>coordinator serve</code> on first start and stored in <code>~/.scimesh/admin.password</code>.</p>
{{else if .Error}}<p class="error">{{.Error}}</p>{{end}}
<p class="alt">No account? <a href="/ui/register">Register</a></p>
</main>
</body>
@@ -0,0 +1,24 @@
{{define "logout-form.html"}}
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Sign out · SciMesh</title>
<style>
:root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;min-height:100vh;display:grid;place-items:center;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}a{color:#94bdff}.card{width:min(92vw,380px);border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021;padding:28px}.eyebrow{margin:0 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0 0 12px;color:#f4f8ff;font-size:1.5rem;letter-spacing:-.03em}p{margin:0 0 18px;color:#9fb3cf;font-size:.92rem}code{color:#cfe0ff}.button{display:block;width:100%;margin-top:4px;border:0;border-radius:10px;padding:12px 16px;background:#ff7d92;color:#230810;font:inherit;font-weight:850;cursor:pointer}.alt{margin:18px 0 0;color:#9fb3cf;font-size:.92rem}
</style>
</head>
<body>
<main class="card">
<p class="eyebrow">SciMesh</p>
<h1>Sign out</h1>
<p>End the current session so you can sign in with a different account (for example the cluster administrator).</p>
<form method="post" action="/ui/logout">
<button class="button" type="submit">Log out</button>
</form>
<p class="alt">Changed your mind? <a href="/ui/login">Back to sign in</a></p>
</main>
</body>
</html>
{{end}}
@@ -5,13 +5,11 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>New computation · SciMesh</title>
<style>
:root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:980px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.back{text-decoration:none}.eyebrow{margin:28px 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3.25rem);letter-spacing:-.055em}.lead{max-width:720px;margin:10px 0 0;color:#aabed9;font-size:1.06rem}.layout{display:grid;grid-template-columns:1.45fr .8fr;gap:15px;margin-top:28px}.card,.aside,.notice{border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021}.card{padding:22px}.aside,.notice{padding:18px}.aside h2,.notice h2{margin:0;color:#f1f6ff;font-size:1rem}.aside p,.notice p{color:#9fb3cf}.aside ol{margin:13px 0 0;padding-left:20px;color:#aebfda}.aside li{margin:10px 0}label{display:block;margin:18px 0 5px;color:#eaf2ff;font-weight:750}input,select{width:100%;border:1px solid #42617f;border-radius:9px;padding:10px 11px;background:#0a1626;color:#e5efff;font:inherit}input:focus,select:focus{outline:2px solid #5d97f5;outline-offset:1px}input[type=file]{padding:8px}.checkbox-row{display:flex;align-items:flex-start;gap:10px;margin-top:16px}.checkbox-row input[type=checkbox]{width:18px;height:18px;margin-top:4px;accent-color:#67e3b8}.checkbox-row label{margin:0}.checkbox-row .hint{margin:0}.hint{margin:5px 0;color:#92a9c6;font-size:.87rem}.split{display:grid;grid-template-columns:1fr 1fr;gap:12px}.run-preview{margin-top:18px;border:1px solid #2c8c70;border-radius:11px;padding:13px;background:#0c2b2a;color:#a8f1d0}.run-preview strong{color:#e6fff4}.button{display:inline-flex;margin-top:22px;border:0;border-radius:10px;padding:12px 16px;background:#67e3b8;color:#062018;font:inherit;font-weight:850;cursor:pointer}.button:disabled{opacity:.6;cursor:wait}.working{margin:14px 0 0;color:#9fc5ff}.error{margin:12px 0 0;color:#ffacba}.hidden{display:none}.cap{display:inline-block;margin:4px 5px 0 0;border:1px solid #375978;border-radius:5px;padding:2px 6px;color:#bcd2f0;font-size:.75rem;font-weight:700}.req{color:#ffb4c0}.workload-meta{margin:6px 0 0;color:#8fa7c8;font-size:.9rem}
</style>
{{template "ui-styles"}}
</head>
<body>
<main class="page">
<a class="back" href="/ui">← Back to control room</a><p class="eyebrow">New computation</p><h1>Any workload, end to end</h1><p class="lead">Choose a workload from the installed library. Upload its dataset; workers compute shards; the coordinator reduces the partials into one final artifact.</p>
<a class="back" href="/ui/admin">← Back to control room</a><p class="eyebrow">New computation</p><h1>Any workload, end to end</h1><p class="lead">Choose a workload from the installed library. Upload its dataset; workers compute shards; the coordinator reduces the partials into one final artifact.</p>
<div class="layout"><form id="run" class="card" novalidate><label for="workload">Workload</label><select id="workload" name="workload"></select><p id="workload-meta" class="workload-meta hidden"></p><div id="params"></div><div class="split"><div><label for="chunk-rows">Rows per shard</label><input id="chunk-rows" name="chunk_rows" type="number" min="1" max="100000" value="1000" required><p class="hint">Smaller shards make more visible tasks.</p></div><div><label for="max-rows">Maximum dataset rows <small>(optional)</small></label><input id="max-rows" name="max_rows" type="number" min="1" max="10000000" placeholder="For example: 500"><p class="hint">Only the first N data rows become shards; the upload stays stored.</p></div></div><label for="file">Dataset file</label><input id="file" type="file" name="file" required accept=".tsv,.txt,.csv,text/tab-separated-values,text/csv"><p class="hint">A delimited table with a header row. The workload defines the required columns.</p><div id="preview" class="run-preview"><strong>Ready to plan a run.</strong><br>Select a dataset to see the file that will be sent to the coordinator.</div><button class="button" id="submit" type="submit">Create pipeline run →</button><p id="working" class="working hidden" aria-live="polite">Uploading dataset and creating coordinator-owned shards…</p><p id="error" class="error" role="alert"></p></form><aside class="aside"><h2>What you will observe</h2><ol><li><strong>Input accepted</strong><br>Dataset is validated and split into durable shard artifacts.</li><li><strong>Workers claim tasks</strong><br>Each worker downloads one shard, computes, and uploads a partial result.</li><li><strong>Global reduction</strong><br>The coordinator reduces all partials into one final artifact.</li><li><strong>Final download</strong><br>The result page exposes a checksum-protected result file.</li></ol><h2 style="margin-top:26px">Before you submit</h2><p>Keep at least one worker running in another terminal. The browser cannot start or control worker processes.</p><p>The form controls come from the workload's own declarations in the SDK library.</p></aside></div>
</main>
<script>
@@ -5,14 +5,13 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Profile · SciMesh</title>
<style>
:root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:720px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.top{display:flex;justify-content:space-between;align-items:center;gap:12px}.eyebrow{margin:0;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:6px 0 0;color:#f4f8ff;font-size:clamp(1.8rem,4vw,2.6rem);letter-spacing:-.04em}.btn{border:0;border-radius:9px;padding:10px 14px;font:inherit;font-weight:800;cursor:pointer;text-decoration:none}.btn-muted{background:#23344d;color:#dce8ff}.card{margin-top:24px;border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021;padding:6px 22px}.err{margin-top:24px;border-radius:10px;padding:12px 14px;background:#552334;color:#ff9bad;font-weight:700}.row{display:flex;justify-content:space-between;gap:16px;padding:15px 0;border-bottom:1px solid #1d3350}.row:last-child{border-bottom:0}.k{color:#9fb3cf}.v{color:#f2f7ff;font-weight:700;text-align:right;word-break:break-all}.mono{font-family:ui-monospace,SFMono-Regular,monospace;font-size:.9rem}.pill{display:inline-block;border-radius:999px;padding:3px 10px;font-size:.82rem;font-weight:800}.pill-yes{background:#123f34;color:#76efb5}.pill-no{background:#23344d;color:#b9cce9}.hint{margin-top:14px;color:#8ba2c2;font-size:.86rem}</style>
{{template "ui-styles"}}
</head>
<body>
<main class="page">
<header class="top">
<div><p class="eyebrow">Account</p><h1>Your profile</h1></div>
<div style="display:flex;gap:10px;align-items:center"><a href="/ui">← Dashboard</a><form method="post" action="/ui/logout" style="margin:0"><button class="btn btn-muted" type="submit">Log out</button></form></div>
<div style="display:flex;gap:10px;align-items:center"><a href="/ui/admin">← Dashboard</a><form method="post" action="/ui/logout" style="margin:0"><button class="btn btn-muted" type="submit">Log out</button></form></div>
</header>
{{if .Error}}<div class="err">{{.Error}}</div>{{end}}
@@ -0,0 +1,141 @@
{{define "ui-styles"}}<style>
:root{--bg:#0b0e13;--panel:#11151d;--panel-2:#161c27;--border:#1f2634;--border-soft:#181f2b;--text:#e9ecf3;--text-2:#98a2b5;--text-3:#5d6879;--accent:#5b8cff;--accent-soft:#5b8cff1f;--accent-strong:#3f6fe0;--green:#3fce8a;--green-soft:#3fce8a1a;--amber:#e5b64f;--amber-soft:#e5b64f1a;--red:#f2647c;--red-soft:#f2647c1a;--mono:ui-monospace,"SF Mono",SFMono-Regular,Menlo,monospace;color-scheme:dark}
*{box-sizing:border-box;margin:0;padding:0}
body{background:var(--bg);color:var(--text);font:14px/1.55 Inter,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
a{color:var(--accent);text-decoration:none}
a:hover{text-decoration:underline}
code{font-family:var(--mono);font-size:.86em}
.page{max-width:1100px;margin:auto;padding:30px 26px 70px}
.back{display:inline-flex;align-items:center;gap:6px;color:var(--text-2);font-size:13px;text-decoration:none}
.back:hover{color:var(--text)}
.eyebrow{margin:30px 0 4px;color:var(--accent);font-size:.72rem;font-weight:750;letter-spacing:.14em;text-transform:uppercase}
h1{margin:0;color:var(--text);font-size:clamp(1.7rem,4vw,2.6rem);letter-spacing:-.04em;line-height:1.1}
.lead{max-width:760px;margin:10px 0 0;color:var(--text-2);font-size:1rem}
.top{display:flex;justify-content:space-between;gap:24px;align-items:flex-start}
.title{margin:0;font-size:clamp(1.7rem,4vw,2.6rem);letter-spacing:-.04em}
.subtitle{margin:9px 0 0;color:var(--text-2)}
.live{color:var(--text-3);font-size:.85rem}
.hidden{display:none!important}
.card,.panel,.aside,.run-note,.notice,.artifact,.workload{background:var(--panel);border:1px solid var(--border-soft);border-radius:13px}
.card{padding:22px}
.panel{padding:20px}
.aside,.notice{padding:18px}
.aside h2,.notice h2{margin:0;color:var(--text);font-size:.95rem;font-weight:650}
.aside p,.notice p{color:var(--text-2)}
.aside ol{margin:13px 0 0;padding-left:20px;color:var(--text-2)}
.aside li{margin:10px 0}
.layout{display:grid;grid-template-columns:1.45fr .8fr;gap:15px;margin-top:28px}
label{display:block;margin:18px 0 5px;color:var(--text);font-weight:650}
input,select,textarea{width:100%;border:1px solid var(--border);border-radius:8px;padding:9px 11px;background:var(--panel-2);color:var(--text);font:inherit;outline:none}
input:focus,select:focus,textarea:focus{border-color:var(--accent);box-shadow:0 0 0 3px var(--accent-soft)}
input[type=file]{padding:8px}
.checkbox-row{display:flex;align-items:flex-start;gap:10px;margin-top:16px}
.checkbox-row input[type=checkbox]{width:17px;height:17px;margin-top:3px;accent-color:var(--accent)}
.checkbox-row label{margin:0}
.checkbox-row .hint{margin:0}
.hint{margin:5px 0;color:var(--text-3);font-size:.86rem}
.split{display:grid;grid-template-columns:1fr 1fr;gap:12px}
.run-preview{margin-top:18px;border:1px solid var(--green-soft);border-radius:10px;padding:13px 14px;background:var(--panel-2);color:var(--green)}
.run-preview strong{color:var(--text)}
.button,.btn{display:inline-flex;align-items:center;gap:7px;border:0;border-radius:8px;padding:10px 16px;background:var(--accent);color:#0a1222;font:inherit;font-weight:700;cursor:pointer;text-decoration:none}
.button:hover,.btn:hover{background:var(--accent-strong);color:#fff;text-decoration:none}
.button:disabled{opacity:.55;cursor:default}
.btn-muted{background:var(--panel-2);color:var(--text-2);border:1px solid var(--border)}
.btn-muted:hover{color:var(--text);border-color:#2a3446;background:var(--panel-2)}
.working{margin-top:14px;color:var(--text-2)}
.error{margin-top:12px;color:var(--red)}
.workload-meta{margin-top:8px;color:var(--text-2);font-size:.9rem}
.badge,.pill{display:inline-flex;align-items:center;gap:6px;border-radius:999px;padding:3px 10px;font-size:.78rem;font-weight:700;white-space:nowrap}
.badge i,.pill i{width:6px;height:6px;border-radius:50%;background:currentColor}
.badge-waiting,.pill-no{background:#ffffff12;color:var(--text-2)}
.badge-active,.pill-yes{background:var(--accent-soft);color:var(--accent)}
.badge-success{background:var(--green-soft);color:var(--green)}
.badge-danger{background:var(--red-soft);color:var(--red)}
.section-head{display:flex;align-items:baseline;justify-content:space-between;gap:12px;margin:34px 0 12px}
.section-head h2{margin:0;color:var(--text);font-size:1.05rem;font-weight:700}
.section-head p{margin:0;color:var(--text-3);font-size:.86rem}
.summary{margin-top:26px}
.summary-top{display:flex;justify-content:space-between;gap:20px;align-items:flex-start}
.bar{height:7px;margin:22px 0 9px;overflow:hidden;border-radius:999px;background:#ffffff10}
.bar span{display:block;height:100%;border-radius:inherit;background:linear-gradient(90deg,var(--accent),var(--green));transition:width .25s}
.progress-line{margin:0;color:var(--text-2)}
.metrics{display:grid;grid-template-columns:repeat(6,1fr);gap:10px;margin-top:17px}
.metric{padding:13px 15px}
.metric b{display:block;margin-top:5px;color:var(--text);font-size:1.35rem;line-height:1}
.metric small{color:var(--text-3)}
.stop{border:1px solid var(--red);border-radius:8px;padding:8px 12px;background:var(--red-soft);color:var(--red);font:inherit;font-weight:700;cursor:pointer}
.stop:disabled{opacity:.55}
.pipeline{display:grid;grid-template-columns:repeat(5,1fr);gap:10px}
.stage{position:relative;padding:13px 14px;min-height:104px;background:var(--panel);border:1px solid var(--border-soft);border-radius:11px}
.stage:not(:last-child):after{content:"";position:absolute;top:32px;right:-10px;width:10px;height:2px;background:#2a3446}
.stage .index{color:var(--text-3);font-size:.74rem;font-weight:700}
.stage b{display:block;margin-top:6px;color:var(--text);font-size:.92rem}
.stage p{margin-top:6px;color:var(--text-3);font-size:.82rem}
.stage-done{border-color:#3fce8a33;background:var(--green-soft)}
.stage-done .index{color:var(--green)}
.stage-active{border-color:var(--accent)}
.stage-active .index{color:var(--accent)}
.stage-waiting{opacity:.75}
.two-col{display:grid;grid-template-columns:1fr 1fr;gap:14px}
.parameter-list{display:grid;gap:8px;margin-top:12px}
.parameter{display:flex;justify-content:space-between;gap:14px;padding:8px 0;border-bottom:1px solid var(--border-soft)}
.parameter:last-child{border-bottom:0}
.parameter span{color:var(--text-2)}
.parameter code{color:var(--text)}
.result{border-color:var(--green)}
.result h3{color:var(--green)}
.alert{border-color:var(--red)}
.alert h3{color:var(--red)}
.download{display:inline-block;margin:10px 12px 0 0;color:var(--accent);font-weight:650}
.table-wrap{overflow-x:auto;background:var(--panel);border:1px solid var(--border-soft);border-radius:13px}
table{width:100%;border-collapse:collapse}
th{padding:10px 18px;text-align:left;font-size:11px;font-weight:650;letter-spacing:.07em;text-transform:uppercase;color:var(--text-3);border-bottom:1px solid var(--border-soft)}
td{padding:12px 18px;border-bottom:1px solid var(--border-soft);vertical-align:middle}
tr:last-child td{border-bottom:0}
tbody tr:hover{background:var(--panel-2)}
.muted{color:var(--text-3)}
.empty{padding:26px;text-align:center;color:var(--text-3)}
.artifact-grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(240px,1fr));gap:12px}
.artifact{padding:16px 18px}
.artifact strong{display:block;margin:7px 0 3px;color:var(--text)}
.artifact code{display:block;color:var(--text-3);font-size:.76rem;word-break:break-all}
.artifact a{margin-right:12px;font-weight:650}
.artifact-type{color:var(--text-3);font-size:.74rem;font-weight:700;letter-spacing:.06em;text-transform:uppercase}
.artifact-final{border-color:var(--green)}
.artifact-final .artifact-type{color:var(--green)}
.technical{margin-top:26px;color:var(--text-3);font-size:.86rem}
.technical summary{cursor:pointer}
.copy{border:1px solid var(--border);border-radius:6px;padding:2px 8px;margin-left:8px;background:var(--panel-2);color:var(--text-2);font:inherit;font-size:.8rem;cursor:pointer}
.speed-card{padding:18px}
.speed-stats{display:flex;gap:28px;flex-wrap:wrap}
.speed-stat b{display:block;color:var(--green);font-size:1.3rem}
.speed-stat small{color:var(--text-3)}
.speed-chart{display:block;width:100%;height:auto;margin-top:16px;border:1px solid var(--border);border-radius:10px;background:var(--panel-2)}
.speed-grid{stroke:#ffffff12;stroke-width:1}
.speed-line{fill:none;stroke:var(--green);stroke-width:2.5;stroke-linecap:round;stroke-linejoin:round}
.speed-point{fill:var(--green);stroke:var(--panel-2);stroke-width:2}
.speed-label{fill:var(--text-3);font-size:10px}
.library{display:grid;gap:14px;margin-top:30px}
.workload{padding:18px 22px}
.workload-head{display:flex;align-items:baseline;justify-content:space-between;gap:14px;flex-wrap:wrap}
.workload-head h2{margin:0;color:var(--text);font-size:1.05rem;letter-spacing:-.01em}
.version{margin:0;color:var(--text-3);font:.8rem var(--mono)}
.description{margin:8px 0 0;color:var(--text-2);max-width:860px}
.cap{display:inline-block;margin:12px 5px 0 0;border:1px solid var(--border);border-radius:6px;padding:2px 8px;color:var(--text-2);font:.75rem var(--mono)}
.schema-grid{display:grid;gap:10px;margin-top:14px}
.schema{margin:0;color:var(--text-2);font-size:.86rem}
.schema code{color:var(--accent)}
.schema-grid pre{background:var(--panel-2);border:1px solid var(--border);border-radius:8px;padding:12px;overflow-x:auto;color:var(--text-2);font:.8rem var(--mono)}
.params{margin-top:14px}
.keys{display:grid;gap:10px;margin-top:12px}
.keys .key{background:var(--panel-2);border:1px solid var(--border);border-radius:8px;padding:11px 13px;color:var(--text-2);font:.82rem var(--mono);word-break:break-all}
.command{background:var(--panel-2);border:1px solid var(--border);border-radius:8px;padding:12px 14px;color:var(--text-2);font:.82rem var(--mono);overflow-x:auto;white-space:pre-wrap}
.mono{font-family:var(--mono)}
.row{display:flex;justify-content:space-between;gap:16px;padding:14px 0;border-bottom:1px solid var(--border-soft)}
.row:last-child{border-bottom:0}
.k{color:var(--text-2)}
.v{color:var(--text);font-weight:650;text-align:right;word-break:break-all}
.err{margin-top:24px;border-radius:10px;padding:12px 14px;background:var(--red-soft);color:var(--red);font-weight:650}
.meta{color:var(--text-3)}
@media(max-width:900px){.layout,.two-col,.split{grid-template-columns:1fr}.pipeline{grid-template-columns:1fr 1fr}.metrics{grid-template-columns:repeat(3,1fr)}}
</style>{{end}}
@@ -5,9 +5,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Workload library · SciMesh</title>
<style>
:root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 12% -8%,#183f77 0,transparent 32rem),radial-gradient(circle at 95% 5%,#123e39 0,transparent 29rem),#08111f}.page{max-width:1240px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.back{text-decoration:none}.eyebrow{margin:28px 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3.25rem);letter-spacing:-.055em}.lead{max-width:760px;margin:10px 0 0;color:#aabed9;font-size:1.06rem}.library{display:grid;gap:14px;margin-top:30px}.workload{border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021;padding:20px 22px}.workload-head{display:flex;align-items:baseline;justify-content:space-between;gap:14px;flex-wrap:wrap}.workload-head h2{margin:0;color:#f2f7ff;font-size:1.22rem;letter-spacing:-.02em}.version{margin:0;color:#7d93b2;font:0.82rem ui-monospace,SFMono-Regular,monospace}.description{margin:8px 0 0;color:#b9c9e2;max-width:860px}.cap{display:inline-block;margin:12px 5px 0 0;border:1px solid #375978;border-radius:5px;padding:2px 7px;color:#b5d3f5;font:.76rem ui-monospace,SFMono-Regular,monospace}.badge{display:inline-flex;align-items:center;border-radius:999px;padding:4px 10px;font-size:.78rem;font-weight:800}.badge-success{background:#123f34;color:#76efb5}.badge-waiting{background:#23344d;color:#b9cce9}.meta{display:flex;gap:9px;flex-wrap:wrap;margin-top:14px}.meta span{border:1px solid #2b4a6b;border-radius:7px;padding:3px 8px;color:#a9c3e2;font-size:.8rem}.meta b{color:#dbe9fb;font-weight:750}.schema-grid{display:grid;grid-template-columns:1fr 1fr;gap:13px;margin-top:16px}.schema{border:1px solid #233e5c;border-radius:11px;background:#091627;padding:13px}.schema h3{margin:0 0 8px;color:#cfe1f7;font-size:.86rem;letter-spacing:.04em;text-transform:uppercase}.schema pre{margin:0;overflow:auto;max-height:300px;color:#9fc1e8;font:.76rem ui-monospace,SFMono-Regular,monospace;white-space:pre-wrap;word-break:break-word}.params{padding:13px}.params h3{margin:0 0 8px;color:#cfe1f7;font-size:.86rem;letter-spacing:.04em;text-transform:uppercase}.param{margin:0;padding:6px 0;border-bottom:1px dashed #223a56;color:#b9c9e2;font-size:.9rem}.param:last-child{border-bottom:0}.param b{color:#e8f2ff}.param small{display:block;margin-top:2px;color:#7f96b5}.empty{padding:30px;border:1px dashed #35516f;border-radius:14px;color:#9ab0cb;text-align:center}@media(max-width:820px){.schema-grid{grid-template-columns:1fr}}
</style>
{{template "ui-styles"}}
</head>
<body>
<main class="page">
@@ -48,7 +46,7 @@
<div class="empty"><strong>No workloads are installed.</strong><br>Install an SDK workload package and run <code>scimesh workload export</code> to republish this catalog.</div>
{{end}}
</div>
<p class="lead" style="margin-top:26px"><a class="back" href="/ui">← Back to the control room</a></p>
<p class="lead" style="margin-top:26px"><a class="back" href="/ui/admin">← Back to the control room</a></p>
</main>
</body>
</html>
+9 -21
View File
@@ -217,29 +217,17 @@ func (s *Server) renderUI(w http.ResponseWriter, name string, data any) {
}
}
// handleUIHome lands the operator on the real UI. In session mode that is the
// admin console; under basic auth (no userservice, no roles) it is the job
// form, since the console does not exist there. The old demo control room is
// gone; /ui is a plain redirect so stale bookmarks still arrive somewhere
// useful.
func (s *Server) handleUIHome(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
view, err := s.uc.Dashboard.Overview(ctx, 20)
if err != nil {
s.writeError(w, r, err)
return
target := "/ui/jobs/new"
if s.uiSessionMode() {
target = "/ui/admin"
}
s.renderUI(w, "dashboard.html", view)
}
// handleUIOverviewJSON is the bounded polling projection used by the operator
// dashboard. It intentionally returns only the safe UI read model, never
// worker tokens, storage keys, or database entities.
func (s *Server) handleUIOverviewJSON(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
view, err := s.uc.Dashboard.Overview(ctx, 20)
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, view)
http.Redirect(w, r, target, http.StatusSeeOther)
}
func (s *Server) handleUINewJob(w http.ResponseWriter, r *http.Request) {
+322 -3
View File
@@ -2,14 +2,18 @@ package http
import (
"context"
"encoding/json"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
)
// adminUserActions are the userservice endpoints the admin panel may invoke, by
@@ -23,12 +27,19 @@ var adminUserActions = map[string]bool{
}
// requireAdmin gates a route on the session caller being an admin. It runs
// inside withUISession, which has already stamped the requester. A non-admin is
// sent back to the dashboard rather than shown the panel.
// inside withUISession, which has already stamped the requester. A signed-in
// non-admin is told why (and bounced to the login with the message); an
// unauthenticated caller never gets here — the gate has already sent them to
// the login page with the intended destination.
func requireAdmin(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if req, ok := authctx.From(r.Context()); !ok || !req.IsAdmin() {
http.Redirect(w, r, "/ui", http.StatusSeeOther)
target := "/ui/login?error=admin+role+required"
// Keep the destination so a successful login lands straight back.
if strings.HasPrefix(r.URL.Path, "/ui/") {
target += "&next=" + url.QueryEscape(r.URL.Path)
}
http.Redirect(w, r, target, http.StatusSeeOther)
return
}
next.ServeHTTP(w, r)
@@ -112,3 +123,311 @@ func (s *Server) callUserserviceAuthed(ctx context.Context, method, path, bearer
}
return resp.StatusCode, body, nil
}
// handleUIAdminSystemJSON serves the admin "System" page: process info,
// storage figures and health. Admin-only via the route chain.
func (s *Server) handleUIAdminSystemJSON(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
view, err := s.uc.Admin.System(ctx)
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, view)
}
// handleUIAdminJobsJSON serves one page of the admin jobs table. The owner
// emails are resolved from the userservice when it is reachable; the resolver
// failing is not fatal (cards fall back to short ids).
func (s *Server) handleUIAdminJobsJSON(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
page, _ := strconv.Atoi(r.URL.Query().Get("page"))
perPage, _ := strconv.Atoi(r.URL.Query().Get("per_page"))
status := strings.TrimSpace(r.URL.Query().Get("status"))
view, err := s.uc.Admin.Jobs(ctx, status, page, perPage, s.adminOwnerEmails(r))
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, view)
}
// handleUIAdminMetricsJSON serves the admin "Metrics" page.
func (s *Server) handleUIAdminMetricsJSON(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
view, err := s.uc.Admin.Metrics(ctx)
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, view)
}
// handleUIAdminWorkersJSON serves the admin "Workers" page.
func (s *Server) handleUIAdminWorkersJSON(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
view, err := s.uc.Admin.Workers(ctx, s.adminOwnerEmails(r))
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, view)
}
// handleUIAdminSetTrustJSON flips one worker's trust level.
func (s *Server) handleUIAdminSetTrustJSON(w http.ResponseWriter, r *http.Request) {
id, err := uuid.Parse(r.PathValue("id"))
if err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
var body struct {
Trusted bool `json:"trusted"`
}
if err := decodeJSON(r, &body); err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
ctx, cancel := s.reqCtx(r)
defer cancel()
if err := s.uc.Admin.SetTrust(ctx, id, body.Trusted); err != nil {
s.writeError(w, r, err)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleUIAdminWorkloadsJSON serves the catalog with persisted enable flags.
func (s *Server) handleUIAdminWorkloadsJSON(w http.ResponseWriter, r *http.Request) {
ctx, cancel := s.reqCtx(r)
defer cancel()
view, err := s.uc.Admin.Workloads(ctx)
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, view)
}
// handleUIAdminSetWorkloadEnabledJSON flips a workload's enable flag.
func (s *Server) handleUIAdminSetWorkloadEnabledJSON(w http.ResponseWriter, r *http.Request) {
name := r.PathValue("name")
var body struct {
Enabled bool `json:"enabled"`
}
if err := decodeJSON(r, &body); err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
ctx, cancel := s.reqCtx(r)
defer cancel()
if err := s.uc.Admin.SetWorkloadEnabled(ctx, name, body.Enabled); err != nil {
s.writeError(w, r, err)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleUIAdminSettingsJSON serves the read-only cluster settings.
func (s *Server) handleUIAdminSettingsJSON(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, s.uc.Admin.Settings())
}
// handleUIAdminRevealTokenJSON reveals the shared worker token, auditing the
// reveal. Admin-only via the route chain.
func (s *Server) handleUIAdminRevealTokenJSON(w http.ResponseWriter, r *http.Request) {
actor := "admin"
if req, ok := authctx.From(r.Context()); ok {
actor = req.Role + ":" + req.UserID.String()
}
ctx, cancel := s.reqCtx(r)
defer cancel()
writeJSON(w, http.StatusOK, map[string]string{"token": s.uc.Admin.RevealWorkerToken(ctx, actor)})
}
// handleUIAdminUsersJSON serves the account table, proxied from the
// userservice. The userservice projects away password hashes; a failure here
// is a 502 rather than a silent empty table.
func (s *Server) handleUIAdminUsersJSON(w http.ResponseWriter, r *http.Request) {
c, err := r.Cookie(sessionCookie)
if err != nil {
redirectToLogin(w, r)
return
}
status, body, err := s.callUserserviceAuthed(r.Context(), http.MethodGet, "/users", c.Value)
if err != nil {
s.writeError(w, r, err)
return
}
if status != http.StatusOK {
writeJSON(w, status, map[string]string{"error": "userservice: unexpected response"})
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(body)
}
// handleUIAdminSetUserRoleJSON changes a user's role through the userservice
// promote/demote actions.
func (s *Server) handleUIAdminSetUserRoleJSON(w http.ResponseWriter, r *http.Request) {
id, err := uuid.Parse(r.PathValue("id"))
if err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
var body struct {
Role string `json:"role"`
}
if err := decodeJSON(r, &body); err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
action := ""
switch body.Role {
case "admin":
action = "promote"
case "user":
action = "demote"
default:
s.writeError(w, r, domain.ErrInvalidInput)
return
}
c, err := r.Cookie(sessionCookie)
if err != nil {
redirectToLogin(w, r)
return
}
status, _, err := s.callUserserviceAuthed(r.Context(), http.MethodPost, "/users/"+id.String()+"/"+action, c.Value)
if err != nil {
s.writeError(w, r, err)
return
}
if status != http.StatusNoContent {
writeJSON(w, status, map[string]string{"error": "userservice: unexpected response"})
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleUIAdminWorkerKeysJSON serves every worker key with its owning user,
// proxied from the userservice.
func (s *Server) handleUIAdminWorkerKeysJSON(w http.ResponseWriter, r *http.Request) {
c, err := r.Cookie(sessionCookie)
if err != nil {
redirectToLogin(w, r)
return
}
status, body, err := s.callUserserviceAuthed(r.Context(), http.MethodGet, "/worker-keys/all", c.Value)
if err != nil {
s.writeError(w, r, err)
return
}
if status != http.StatusOK {
writeJSON(w, status, map[string]string{"error": "userservice: unexpected response"})
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(body)
}
// handleUIAdminRevokeKeyJSON revokes any worker key through the userservice
// (whose DELETE endpoint already lets an admin revoke keys of any owner).
func (s *Server) handleUIAdminRevokeKeyJSON(w http.ResponseWriter, r *http.Request) {
id, err := uuid.Parse(r.PathValue("id"))
if err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
c, err := r.Cookie(sessionCookie)
if err != nil {
redirectToLogin(w, r)
return
}
status, _, err := s.callUserserviceAuthed(r.Context(), http.MethodDelete, "/worker-keys/"+id.String(), c.Value)
if err != nil {
s.writeError(w, r, err)
return
}
if status != http.StatusNoContent {
writeJSON(w, status, map[string]string{"error": "userservice: unexpected response"})
return
}
w.WriteHeader(http.StatusNoContent)
}
// adminOwnerEmails resolves job owner ids to emails through the userservice,
// which is the only place email addresses live. It never blocks the page on
// failure: an empty map leaves the admin jobs table on short ids.
func (s *Server) adminOwnerEmails(r *http.Request) map[uuid.UUID]string {
if s.userserviceURL == "" {
return nil
}
c, err := r.Cookie(sessionCookie)
if err != nil {
return nil
}
status, body, err := s.callUserserviceAuthed(r.Context(), http.MethodGet, "/users", c.Value)
if err != nil || status != http.StatusOK {
return nil
}
var users []struct {
ID string `json:"id"`
Email string `json:"email"`
}
if err := json.Unmarshal(body, &users); err != nil {
return nil
}
out := make(map[uuid.UUID]string, len(users))
for _, user := range users {
id, err := uuid.Parse(user.ID)
if err != nil {
continue
}
out[id] = user.Email
}
return out
}
// handleUIAdminPruneJSON deletes finished jobs older than the requested
// number of days (with all their artifacts) and reports what was freed.
func (s *Server) handleUIAdminPruneJSON(w http.ResponseWriter, r *http.Request) {
var body struct {
OlderThanDays int `json:"older_than_days"`
}
if err := decodeJSON(r, &body); err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
if body.OlderThanDays < 1 || body.OlderThanDays > 3650 {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
ctx, cancel := s.reqCtx(r)
defer cancel()
result, err := s.uc.PruneArtifacts.Execute(ctx, time.Duration(body.OlderThanDays)*24*time.Hour)
if err != nil {
s.writeError(w, r, err)
return
}
writeJSON(w, http.StatusOK, result)
}
// handleUIAdminRemoveWorkerJSON deletes an offline worker.
func (s *Server) handleUIAdminRemoveWorkerJSON(w http.ResponseWriter, r *http.Request) {
id, err := uuid.Parse(r.PathValue("id"))
if err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
ctx, cancel := s.reqCtx(r)
defer cancel()
if err := s.uc.Admin.RemoveWorker(ctx, id); err != nil {
s.writeError(w, r, err)
return
}
w.WriteHeader(http.StatusNoContent)
}
@@ -11,7 +11,6 @@ import (
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
func adminReq(t *testing.T, role string) *http.Request {
@@ -30,15 +29,15 @@ func TestRequireAdminAllowsAdminOnly(t *testing.T) {
t.Error("admin must reach the handler")
}
// Plain user is redirected to the dashboard.
// Plain user is redirected to the login with the reason and the destination.
reached = false
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminReq(t, "user"))
if reached {
t.Error("non-admin must not reach the handler")
}
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/ui" {
t.Errorf("non-admin got %d -> %q, want 303 -> /ui", rec.Code, rec.Header().Get("Location"))
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/ui/login?error=admin+role+required&next=%2Fui%2Fadmin" {
t.Errorf("non-admin got %d -> %q, want 303 -> login with the admin-required error and next", rec.Code, rec.Header().Get("Location"))
}
}
@@ -101,13 +100,24 @@ func TestAdminUserActionRejectsBadID(t *testing.T) {
}
}
func TestDashboardAdminLinkOnlyForAdmin(t *testing.T) {
admin := render(t, "dashboard.html", usecase.DashboardView{Session: &usecase.SessionView{Role: "admin"}})
if !strings.Contains(admin, "/ui/admin") {
t.Error("admin must see the Admin link")
func TestLoginPageExplainsAdminRequiredError(t *testing.T) {
html := render(t, "login.html", map[string]any{"Error": "admin role required"})
if !strings.Contains(html, "/ui/logout-form") {
t.Error("the admin-required error must offer a logout path to switch accounts")
}
user := render(t, "dashboard.html", usecase.DashboardView{Session: &usecase.SessionView{Role: "user"}})
if strings.Contains(user, "/ui/admin") {
t.Error("a plain user must not see the Admin link")
if !strings.Contains(html, "cluster administrator") {
t.Error("the admin-required error must name the admin account")
}
// Other errors keep the plain message, no logout teaser.
plain := render(t, "login.html", map[string]any{"Error": "invalid email or password"})
if strings.Contains(plain, "/ui/logout-form") {
t.Error("plain login errors must not advertise logout")
}
}
func TestLogoutFormRendersPostButton(t *testing.T) {
html := render(t, "logout-form.html", map[string]any{})
if !strings.Contains(html, `action="/ui/logout"`) || !strings.Contains(html, "Log out") {
t.Error("logout form must POST /ui/logout")
}
}
+29 -3
View File
@@ -6,6 +6,8 @@ import (
"encoding/json"
"io"
"net/http"
"net/url"
"strings"
"time"
"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
@@ -53,7 +55,14 @@ type tokenVerifier interface {
}
func (s *Server) handleUILoginForm(w http.ResponseWriter, r *http.Request) {
s.renderUI(w, "login.html", map[string]any{"Error": r.URL.Query().Get("error")})
s.renderUI(w, "login.html", map[string]any{"Error": r.URL.Query().Get("error"), "Next": r.URL.Query().Get("next")})
}
// handleUILogoutForm renders a small confirm page for ending the current
// session. The actual logout stays a POST (/ui/logout); this page exists so a
// signed-in non-admin who hit an admin-only page can switch accounts.
func (s *Server) handleUILogoutForm(w http.ResponseWriter, r *http.Request) {
s.renderUI(w, "logout-form.html", map[string]any{})
}
func (s *Server) handleUIRegisterForm(w http.ResponseWriter, r *http.Request) {
@@ -85,7 +94,14 @@ func (s *Server) handleUILogin(w http.ResponseWriter, r *http.Request) {
return
}
setSessionCookie(w, r, resp.Token)
http.Redirect(w, r, "/ui", http.StatusSeeOther)
// Land back where the user was headed (e.g. /ui/admin); never follow a
// value that escapes the UI prefix — that would be an open redirect.
next := strings.TrimSpace(r.FormValue("next"))
if next == "" || !strings.HasPrefix(next, "/ui/") {
next = "/ui/admin"
}
//nolint:gosec // G710: next is validated to start with /ui/ just above
http.Redirect(w, r, next, http.StatusSeeOther)
}
// handleUIRegister creates an account through the userservice, then sends the
@@ -167,5 +183,15 @@ func clearSessionCookie(w http.ResponseWriter, r *http.Request) {
}
func redirectToLogin(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ui/login", http.StatusSeeOther)
// Remember where the user was headed so a successful login lands back
// there (e.g. /ui/admin) instead of the control room.
next := r.URL.Path
if !strings.HasPrefix(next, "/ui/") {
next = ""
}
target := "/ui/login"
if next != "" {
target += "?next=" + url.QueryEscape(next)
}
http.Redirect(w, r, target, http.StatusSeeOther)
}
@@ -115,8 +115,8 @@ func TestHandleUILoginSetsCookieOnSuccess(t *testing.T) {
rec := httptest.NewRecorder()
s.handleUILogin(rec, postForm("/ui/login", url.Values{"email": {"a@b.com"}, "password": {"password123"}}))
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/ui" {
t.Fatalf("got %d -> %q, want 303 -> /ui", rec.Code, rec.Header().Get("Location"))
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/ui/admin" {
t.Fatalf("got %d -> %q, want 303 -> /ui/admin", rec.Code, rec.Header().Get("Location"))
}
cookies := rec.Result().Cookies()
if len(cookies) == 0 || cookies[0].Name != sessionCookie || cookies[0].Value != "issued.jwt.here" {
@@ -173,3 +173,56 @@ func TestHandleUILogoutClearsCookie(t *testing.T) {
t.Error("logout must clear the session cookie")
}
}
func TestHandleUILoginRedirectsToNext(t *testing.T) {
stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"token":"t"}`))
}))
defer stub.Close()
s := newLoginServer(stub)
// A UI-scoped next is honoured: the admin lands back on the console.
rec := httptest.NewRecorder()
s.handleUILogin(rec, postForm("/ui/login", url.Values{"email": {"a@b.com"}, "password": {"p"}, "next": {"/ui/admin"}}))
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/ui/admin" {
t.Errorf("got %d -> %q, want 303 -> /ui/admin", rec.Code, rec.Header().Get("Location"))
}
// Anything outside the UI prefix must not become a redirect target.
for _, next := range []string{"https://evil.example", "/", "//evil.example", "/api/jobs"} {
rec = httptest.NewRecorder()
s.handleUILogin(rec, postForm("/ui/login", url.Values{"email": {"a@b.com"}, "password": {"p"}, "next": {next}}))
if loc := rec.Header().Get("Location"); loc != "/ui/admin" {
t.Errorf("next=%q landed on %q, want /ui/admin (no open redirect)", next, loc)
}
}
}
func TestRedirectToLoginCarriesNext(t *testing.T) {
rec := httptest.NewRecorder()
req := newReq(http.MethodGet, "/ui/admin", nil)
redirectToLogin(rec, req)
if loc := rec.Header().Get("Location"); loc != "/ui/login?next=%2Fui%2Fadmin" {
t.Errorf("location = %q, want /ui/login?next=%%2Fui%%2Fadmin", loc)
}
// Paths outside the UI stay on the plain login.
rec = httptest.NewRecorder()
req = newReq(http.MethodGet, "/health", nil)
redirectToLogin(rec, req)
if loc := rec.Header().Get("Location"); loc != "/ui/login" {
t.Errorf("location = %q, want /ui/login", loc)
}
}
func TestLoginFormRendersNext(t *testing.T) {
html := render(t, "login.html", map[string]any{"Next": "/ui/admin"})
if !strings.Contains(html, `name="next" value="/ui/admin"`) {
t.Error("login form must carry the next field")
}
html = render(t, "login.html", map[string]any{})
if strings.Contains(html, `name="next"`) {
t.Error("login form must not render next when absent")
}
}
@@ -17,15 +17,15 @@ func render(t *testing.T, name string, data any) string {
return buf.String()
}
func TestDashboardLogoutOnlyInSession(t *testing.T) {
withSession := render(t, "dashboard.html", usecase.DashboardView{Session: &usecase.SessionView{Role: "admin"}})
func TestJobPageLogoutOnlyInSession(t *testing.T) {
withSession := render(t, "job.html", usecase.JobDetailView{Session: &usecase.SessionView{Role: "admin"}})
if !strings.Contains(withSession, "/ui/logout") || !strings.Contains(withSession, "Log out") {
t.Error("dashboard must show a logout control in session mode")
t.Error("job page must show a logout control in session mode")
}
noSession := render(t, "dashboard.html", usecase.DashboardView{})
noSession := render(t, "job.html", usecase.JobDetailView{})
if strings.Contains(noSession, "/ui/logout") {
t.Error("dashboard must not show logout under basic auth (no session)")
t.Error("job page must not show logout under basic auth (no session)")
}
}
@@ -9,6 +9,7 @@ import (
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
@@ -116,3 +117,38 @@ func (s *Server) callUserserviceAuthedBody(ctx context.Context, method, path, be
}
return resp.StatusCode, respBody, nil
}
// handleWorkerTokenExchangeProxy forwards a worker-key exchange to the
// userservice. The key itself is the credential, so this route is public —
// exactly like the userservice's own endpoint. In `serve` mode the embedded
// userservice binds loopback only, so workers need the coordinator to front
// the exchange for them.
func (s *Server) handleWorkerTokenExchangeProxy(w http.ResponseWriter, r *http.Request) {
if s.userserviceURL == "" {
writeJSON(w, http.StatusNotFound, map[string]string{"error": "not found"})
return
}
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err != nil {
s.writeError(w, r, domain.ErrInvalidInput)
return
}
req, err := http.NewRequestWithContext(r.Context(), http.MethodPost, s.userserviceURL+"/worker-tokens/exchange", bytes.NewReader(body)) //nolint:gosec // G704: path is fixed, host is config
if err != nil {
s.writeError(w, r, err)
return
}
req.Header.Set("Content-Type", "application/json")
resp, err := s.httpClient.Do(req) //nolint:gosec // G704: see above
if err != nil {
s.writeError(w, r, err)
return
}
defer func() { _ = resp.Body.Close() }()
respBody, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if err != nil {
s.writeError(w, r, err)
return
}
proxyJSON(w, resp.StatusCode, respBody)
}
+540
View File
@@ -0,0 +1,540 @@
package usecase
import (
"context"
"log/slog"
"sort"
"strings"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
)
// AdminReadRepository is the bounded read projection behind the coordinator
// admin console. Like UIReadRepository it exposes no storage paths or
// credentials; unlike it, every method is admin-scoped (no owner filter).
type AdminReadRepository interface {
// ListJobsPaginated returns one page of jobs filtered by stored status;
// an empty status returns all. total counts the filtered set (for the
// pager).
ListJobsPaginated(ctx context.Context, status string, limit, offset int) (jobs []domain.Job, total int, err error)
// CountJobsByStatus powers the status tabs: every stored status, all jobs.
CountJobsByStatus(ctx context.Context) (map[string]int, error)
// TaskCountsByJobs aggregates task statuses per job for progress bars.
TaskCountsByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID]map[string]int, error)
// JobCountsByDay buckets jobs created since `since` by UTC day
// ("2006-01-02").
JobCountsByDay(ctx context.Context, since time.Time) (map[string]int, error)
// JobCountsByWorkload counts all jobs per workload name.
JobCountsByWorkload(ctx context.Context) (map[string]int, error)
// TaskStats totals shard execution: completed/failed counts and the mean
// run duration of completed shards (seconds; 0 when nothing completed).
TaskStats(ctx context.Context) (completed, failed int64, avgSeconds float64, err error)
// ArtifactSizeByKind sums stored bytes per artifact kind.
ArtifactSizeByKind(ctx context.Context) (map[string]int64, error)
// DatabaseSizeBytes reports the engine's own size figure (sqlite pages,
// pg_database_size); 0 when the engine cannot say.
DatabaseSizeBytes(ctx context.Context) (int64, error)
}
// AdminNodeInfo describes the running coordinator process to the admin
// console. It is static for the process lifetime and assembled at startup.
type AdminNodeInfo struct {
Version string
StartedAt time.Time
Binary string
Addr string
DataDir string
DBEngine string
PublicURL string
Userservice string // base URL; empty when the UI runs without user auth
// WorkerToken reads the shared worker token for the Settings page. It is a
// func so serve mode can read the token file lazily after provisioning.
WorkerToken func() string
}
type AdminStorageView struct {
DatasetsBytes int64 `json:"datasets_bytes"`
ArtifactsBytes int64 `json:"artifacts_bytes"`
DatabaseBytes int64 `json:"database_bytes"`
}
type AdminHealthView struct {
Database string `json:"database"` // connected | error
Reducer string `json:"reducer"` // idle | active
Userservice string `json:"userservice"` // embedded | external | disabled
}
type AdminNodeView struct {
Binary string `json:"binary"`
Addr string `json:"addr"`
DataDir string `json:"data_dir"`
DBEngine string `json:"db_engine"`
PublicURL string `json:"public_url"`
}
type AdminSystemView struct {
Version string `json:"version"`
StartedAt time.Time `json:"started_at"`
UptimeSeconds int64 `json:"uptime_seconds"`
ActiveJobs int `json:"active_jobs"`
RunningJobs int `json:"running_jobs"`
WaitingJobs int `json:"waiting_jobs"`
WorkersOnline int `json:"workers_online"`
WorkersBusy int `json:"workers_busy"`
WorkersTotal int `json:"workers_total"`
Storage AdminStorageView `json:"storage"`
Health AdminHealthView `json:"health"`
Node AdminNodeView `json:"node"`
}
// AdminJobCard is one row of the admin jobs table. Owner is a display string
// resolved by the caller (email when the userservice is reachable, a short id
// or "cluster token" otherwise).
type AdminJobCard struct {
ID string `json:"id"`
Workload string `json:"workload"`
Status string `json:"status"`
OwnerID string `json:"owner_id,omitempty"`
Owner string `json:"owner"`
Total int `json:"total"`
Completed int `json:"completed"`
Failed int `json:"failed"`
CreatedAt time.Time `json:"created_at"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
}
type AdminJobsView struct {
Jobs []AdminJobCard `json:"jobs"`
Total int `json:"total"`
Page int `json:"page"`
PerPage int `json:"per_page"`
// Counts holds every stored status for the filter tabs (all jobs, not
// just the current filter).
Counts map[string]int `json:"counts"`
}
type AdminDayCount struct {
Day string `json:"day"`
Count int `json:"count"`
}
type AdminWorkloadCount struct {
Workload string `json:"workload"`
Count int `json:"count"`
}
type AdminMetricsView struct {
JobsLast7Days int `json:"jobs_last_7_days"`
JobsByDay []AdminDayCount `json:"jobs_by_day"`
JobsByWorkload []AdminWorkloadCount `json:"jobs_by_workload"`
ShardsCompleted int64 `json:"shards_completed"`
ShardsFailed int64 `json:"shards_failed"`
AvgShardSeconds float64 `json:"avg_shard_seconds"`
FailureRate float64 `json:"failure_rate"`
}
// Admin answers the coordinator admin console from the bounded read model
// plus process info supplied at startup.
type Admin struct {
read AdminReadRepository
uiRead UIReadRepository
workers WorkerRepository
settings WorkloadSettingsRepository
catalog *workloads.Catalog
node AdminNodeInfo
ready func(context.Context) error
now func() time.Time
log *slog.Logger
audit func(ctx context.Context, action, detail string)
}
func NewAdmin(read AdminReadRepository, uiRead UIReadRepository, workers WorkerRepository,
settings WorkloadSettingsRepository, catalog *workloads.Catalog, node AdminNodeInfo,
ready func(context.Context) error, now func() time.Time) *Admin {
if now == nil {
now = time.Now
}
return &Admin{read: read, uiRead: uiRead, workers: workers, settings: settings, catalog: catalog, node: node, ready: ready, now: now}
}
// WithAuditLog attaches an audit sink for sensitive actions (token reveal).
// Without it the admin usecase stays silent about them.
func (a *Admin) WithAuditLog(log *slog.Logger, audit func(ctx context.Context, action, detail string)) *Admin {
a.log = log
a.audit = audit
return a
}
func (a *Admin) revealToken(ctx context.Context, actor string) string {
if a.node.WorkerToken != nil {
return a.node.WorkerToken()
}
return ""
}
func (a *Admin) System(ctx context.Context) (AdminSystemView, error) {
counts, err := a.read.CountJobsByStatus(ctx)
if err != nil {
return AdminSystemView{}, err
}
workers, err := a.uiRead.ListWorkers(ctx, 100)
if err != nil {
return AdminSystemView{}, err
}
sizes, err := a.read.ArtifactSizeByKind(ctx)
if err != nil {
return AdminSystemView{}, err
}
dbSize, err := a.read.DatabaseSizeBytes(ctx)
if err != nil {
return AdminSystemView{}, err
}
out := AdminSystemView{
Version: a.node.Version,
StartedAt: a.node.StartedAt,
WaitingJobs: counts[string(domain.JobPending)],
RunningJobs: counts[string(domain.JobRunning)] + counts[string(domain.JobReducing)],
}
out.ActiveJobs = out.WaitingJobs + out.RunningJobs
out.UptimeSeconds = int64(a.now().Sub(a.node.StartedAt).Seconds())
if out.UptimeSeconds < 0 {
out.UptimeSeconds = 0
}
for _, w := range workers {
out.WorkersTotal++
switch w.Status {
case domain.WorkerOnline:
out.WorkersOnline++
case domain.WorkerBusy:
out.WorkersOnline++
out.WorkersBusy++
}
}
for kind, size := range sizes {
if kind == string(domain.ArtifactInput) {
out.Storage.DatasetsBytes += size
} else {
out.Storage.ArtifactsBytes += size
}
}
out.Storage.DatabaseBytes = dbSize
out.Health.Database = "connected"
if a.ready != nil {
if err := a.ready(ctx); err != nil {
out.Health.Database = "error"
}
}
out.Health.Reducer = "idle"
if counts[string(domain.JobReducing)] > 0 {
out.Health.Reducer = "active"
}
out.Health.Userservice = "disabled"
if a.node.Userservice != "" {
out.Health.Userservice = "external"
// The embedded userservice always binds the loopback interface.
if strings.Contains(a.node.Userservice, "127.0.0.1") || strings.Contains(a.node.Userservice, "localhost") {
out.Health.Userservice = "embedded"
}
}
out.Node = AdminNodeView{
Binary: a.node.Binary,
Addr: a.node.Addr,
DataDir: a.node.DataDir,
DBEngine: a.node.DBEngine,
PublicURL: a.node.PublicURL,
}
return out, nil
}
// Jobs returns one page of the admin jobs table. The owner emails map may be
// nil; cards then fall back to a short id or "cluster token".
func (a *Admin) Jobs(ctx context.Context, status string, page, perPage int, ownerEmails map[uuid.UUID]string) (AdminJobsView, error) {
if page < 1 {
page = 1
}
if perPage < 1 || perPage > 100 {
perPage = 20
}
jobs, total, err := a.read.ListJobsPaginated(ctx, status, perPage, (page-1)*perPage)
if err != nil {
return AdminJobsView{}, err
}
counts, err := a.read.CountJobsByStatus(ctx)
if err != nil {
return AdminJobsView{}, err
}
jobIDs := make([]uuid.UUID, 0, len(jobs))
for _, job := range jobs {
jobIDs = append(jobIDs, job.ID)
}
taskCounts, err := a.read.TaskCountsByJobs(ctx, jobIDs)
if err != nil {
return AdminJobsView{}, err
}
out := AdminJobsView{
Jobs: make([]AdminJobCard, 0, len(jobs)),
Total: total,
Page: page,
PerPage: perPage,
Counts: counts,
}
for _, job := range jobs {
tc := taskCounts[job.ID]
card := AdminJobCard{
ID: job.ID.String(),
Workload: job.Workload,
CreatedAt: job.CreatedAt,
CompletedAt: job.CompletedAt,
Owner: "cluster token",
}
var pending, leased, cancelled int
for status, n := range tc {
card.Total += n
switch domain.TaskStatus(status) {
case domain.TaskCompleted:
card.Completed = n
case domain.TaskFailed:
card.Failed = n
case domain.TaskPending:
pending = n
case domain.TaskLeased, domain.TaskRunning:
leased += n
case domain.TaskCancelled:
cancelled = n
}
}
// Derive the status exactly like the operator dashboard does, so the
// two views never disagree about the same job.
progress := domain.JobProgress{Job: job, Total: card.Total, Pending: pending, Leased: leased, Done: card.Completed, Failed: card.Failed, Cancelled: cancelled}
card.Status = string(progress.DeriveStatus())
if job.OwnerID != nil {
card.OwnerID = job.OwnerID.String()
card.Owner = "user " + shortID(job.OwnerID.String())
if email, ok := ownerEmails[*job.OwnerID]; ok && email != "" {
card.Owner = email
}
}
out.Jobs = append(out.Jobs, card)
}
return out, nil
}
func (a *Admin) Metrics(ctx context.Context) (AdminMetricsView, error) {
since := a.now().Add(-6 * 24 * time.Hour).Truncate(24 * time.Hour)
byDay, err := a.read.JobCountsByDay(ctx, since)
if err != nil {
return AdminMetricsView{}, err
}
byWorkload, err := a.read.JobCountsByWorkload(ctx)
if err != nil {
return AdminMetricsView{}, err
}
completed, failed, avg, err := a.read.TaskStats(ctx)
if err != nil {
return AdminMetricsView{}, err
}
out := AdminMetricsView{
JobsByDay: make([]AdminDayCount, 0, 7),
JobsByWorkload: make([]AdminWorkloadCount, 0, len(byWorkload)),
ShardsCompleted: completed,
ShardsFailed: failed,
AvgShardSeconds: avg,
}
if completed+failed > 0 {
out.FailureRate = float64(failed) / float64(completed+failed)
}
for i := 0; i < 7; i++ {
day := since.Add(time.Duration(i) * 24 * time.Hour).UTC().Format("2006-01-02")
count := byDay[day]
out.JobsByDay = append(out.JobsByDay, AdminDayCount{Day: day, Count: count})
out.JobsLast7Days += count
}
for workload, count := range byWorkload {
out.JobsByWorkload = append(out.JobsByWorkload, AdminWorkloadCount{Workload: workload, Count: count})
}
sort.Slice(out.JobsByWorkload, func(i, j int) bool {
if out.JobsByWorkload[i].Count != out.JobsByWorkload[j].Count {
return out.JobsByWorkload[i].Count > out.JobsByWorkload[j].Count
}
return out.JobsByWorkload[i].Workload < out.JobsByWorkload[j].Workload
})
return out, nil
}
// AdminWorkerCard is one row of the admin workers table.
type AdminWorkerCard struct {
ID string `json:"id"`
Name string `json:"name"`
Status string `json:"status"`
Capabilities []string `json:"capabilities"`
Trust string `json:"trust"`
OwnerID string `json:"owner_id,omitempty"`
Owner string `json:"owner"`
Completed int `json:"completed"`
LastHeartbeatAt time.Time `json:"last_heartbeat_at"`
}
type AdminWorkersView struct {
Workers []AdminWorkerCard `json:"workers"`
}
// Workers lists the whole fleet for the admin console. Owner emails are
// resolved through the same map as the jobs table (userservice-backed).
func (a *Admin) Workers(ctx context.Context, ownerEmails map[uuid.UUID]string) (AdminWorkersView, error) {
workers, err := a.uiRead.ListWorkers(ctx, 100)
if err != nil {
return AdminWorkersView{}, err
}
out := AdminWorkersView{Workers: make([]AdminWorkerCard, 0, len(workers))}
for _, w := range workers {
card := AdminWorkerCard{
ID: w.ID.String(),
Name: w.Name,
Status: string(w.Status),
Capabilities: w.Capabilities,
Trust: string(w.TrustLevel),
LastHeartbeatAt: w.LastHeartbeatAt,
Owner: "cluster token",
}
if w.OwnerID != nil {
card.OwnerID = w.OwnerID.String()
card.Owner = "user " + shortID(w.OwnerID.String())
if email, ok := ownerEmails[*w.OwnerID]; ok && email != "" {
card.Owner = email
}
}
out.Workers = append(out.Workers, card)
}
return out, nil
}
// SetTrust reclassifies one worker (trusted/untrusted).
func (a *Admin) SetTrust(ctx context.Context, id uuid.UUID, trusted bool) error {
trust := domain.WorkerUntrusted
if trusted {
trust = domain.WorkerTrusted
}
return a.workers.SetTrust(ctx, id, trust)
}
// AdminWorkloadView is the catalog plus the persisted enable flag.
type AdminWorkloadView struct {
Name string `json:"name"`
Description string `json:"description"`
Reduction string `json:"reduction"`
Parameters int `json:"parameters"`
UploadReady bool `json:"upload_ready"`
Enabled bool `json:"enabled"`
DefaultOn bool `json:"default_on"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
type AdminWorkloadsView struct {
Workloads []AdminWorkloadView `json:"workloads"`
}
// Workloads lists the catalog with persisted enable/disable overrides.
func (a *Admin) Workloads(ctx context.Context) (AdminWorkloadsView, error) {
if a.catalog == nil {
return AdminWorkloadsView{}, domain.ErrInvalidInput
}
items := a.catalog.Items()
overrides, err := a.settings.List(ctx)
if err != nil {
return AdminWorkloadsView{}, err
}
enabled := make(map[string]WorkloadSetting, len(overrides))
for _, s := range overrides {
enabled[s.Workload] = s
}
out := AdminWorkloadsView{Workloads: make([]AdminWorkloadView, 0, len(items))}
for _, item := range items {
params := 0
if properties, ok := item.Parameters["properties"].(map[string]any); ok {
params = len(properties)
}
view := AdminWorkloadView{
Name: item.Name,
Description: item.Description,
Reduction: item.Reduction,
Parameters: params,
UploadReady: item.UploadReady,
Enabled: true,
DefaultOn: true,
}
if s, ok := enabled[item.Name]; ok {
view.Enabled = s.Enabled
view.DefaultOn = false
view.UpdatedAt = &s.UpdatedAt
}
out.Workloads = append(out.Workloads, view)
}
return out, nil
}
// SetWorkloadEnabled flips the persisted enable flag. An unknown workload is
// rejected: the admin console must not invent catalog entries.
func (a *Admin) SetWorkloadEnabled(ctx context.Context, name string, enabled bool) error {
if a.catalog == nil || a.catalog.ByName(name) == nil {
return domain.ErrInvalidInput
}
return a.settings.SetEnabled(ctx, name, enabled, a.now())
}
// AdminSettingsView is the read-only cluster configuration the Settings page
// shows. The token is never included; it is revealed only through
// RevealWorkerToken, which audits.
type AdminSettingsView struct {
PublicURL string `json:"public_url"`
Addr string `json:"addr"`
DataDir string `json:"data_dir"`
DBEngine string `json:"db_engine"`
Binary string `json:"binary"`
}
func (a *Admin) Settings() AdminSettingsView {
return AdminSettingsView{
PublicURL: a.node.PublicURL,
Addr: a.node.Addr,
DataDir: a.node.DataDir,
DBEngine: a.node.DBEngine,
Binary: a.node.Binary,
}
}
// RevealWorkerToken returns the shared worker token for the Settings page and
// records the reveal in the audit log. It must only be called for an admin
// session.
func (a *Admin) RevealWorkerToken(ctx context.Context, actor string) string {
token := a.revealToken(ctx, actor)
if a.audit != nil {
a.audit(ctx, "worker token revealed", "by "+actor)
}
if a.log != nil {
a.log.Warn("admin console revealed the worker token", "actor", actor)
}
return token
}
// RemoveWorker deletes an offline worker from the registry. Online or busy
// workers are refused: an admin console must never yank a live machine out
// from under a running task.
func (a *Admin) RemoveWorker(ctx context.Context, id uuid.UUID) error {
if a.workers == nil {
return domain.ErrWorkerNotFound
}
worker, err := a.workers.Get(ctx, id)
if err != nil {
return err
}
if worker.Status != domain.WorkerOffline {
return domain.ErrInvalidInput
}
return a.workers.Delete(ctx, id)
}
+365
View File
@@ -0,0 +1,365 @@
package usecase
import (
"context"
"errors"
"testing"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
)
type fakeAdminRead struct {
jobs []domain.Job
taskCounts map[uuid.UUID]map[string]int
sizes map[string]int64
byDay map[string]int
byWorkload map[string]int
completed int64
failed int64
avg float64
dbSize int64
}
func (f *fakeAdminRead) ListJobsPaginated(ctx context.Context, status string, limit, offset int) ([]domain.Job, int, error) {
var out []domain.Job
for _, j := range f.jobs {
if status == "" || string(j.Status) == status {
out = append(out, j)
}
}
total := len(out)
if offset >= len(out) {
return nil, total, nil
}
if offset+limit < len(out) {
out = out[offset : offset+limit]
} else {
out = out[offset:]
}
return out, total, nil
}
func (f *fakeAdminRead) CountJobsByStatus(ctx context.Context) (map[string]int, error) {
out := map[string]int{}
for _, j := range f.jobs {
out[string(j.Status)]++
}
return out, nil
}
func (f *fakeAdminRead) TaskCountsByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID]map[string]int, error) {
return f.taskCounts, nil
}
func (f *fakeAdminRead) JobCountsByDay(ctx context.Context, since time.Time) (map[string]int, error) {
return f.byDay, nil
}
func (f *fakeAdminRead) JobCountsByWorkload(ctx context.Context) (map[string]int, error) {
return f.byWorkload, nil
}
func (f *fakeAdminRead) TaskStats(ctx context.Context) (int64, int64, float64, error) {
return f.completed, f.failed, f.avg, nil
}
func (f *fakeAdminRead) ArtifactSizeByKind(ctx context.Context) (map[string]int64, error) {
return f.sizes, nil
}
func (f *fakeAdminRead) DatabaseSizeBytes(ctx context.Context) (int64, error) { return f.dbSize, nil }
type fakeUIRead struct {
UIReadRepository // embedded: only ListWorkers is exercised
workers []domain.Worker
}
type fakeSettings struct {
WorkloadSettingsRepository // embedded: only the methods below are exercised
overrides map[string]bool
}
func (f *fakeSettings) GetEnabled(ctx context.Context, workload string) (bool, error) {
if enabled, ok := f.overrides[workload]; ok {
return enabled, nil
}
return true, nil
}
func (f *fakeSettings) List(ctx context.Context) ([]WorkloadSetting, error) {
out := make([]WorkloadSetting, 0, len(f.overrides))
for name, enabled := range f.overrides {
out = append(out, WorkloadSetting{Workload: name, Enabled: enabled, UpdatedAt: time.Now()})
}
return out, nil
}
func (f *fakeSettings) SetEnabled(ctx context.Context, workload string, enabled bool, now time.Time) error {
if f.overrides == nil {
f.overrides = map[string]bool{}
}
f.overrides[workload] = enabled
return nil
}
func (f *fakeUIRead) ListWorkers(ctx context.Context, limit int) ([]domain.Worker, error) {
return f.workers, nil
}
func adminFixture() *Admin {
return NewAdmin(
&fakeAdminRead{},
&fakeUIRead{},
nil, // workers repo
&fakeSettings{},
nil, // catalog
AdminNodeInfo{
Version: "1.1.0-alpha.1", StartedAt: time.Unix(1_000_000, 0).UTC(),
Binary: "/usr/local/bin/coordinator", Addr: ":8080", DataDir: "/var/lib/scimesh",
DBEngine: "sqlite", PublicURL: "http://192.168.1.10:8080", Userservice: "http://127.0.0.1:41273",
},
func(context.Context) error { return nil },
func() time.Time { return time.Unix(1_000_000+3600*3, 0).UTC() },
)
}
func TestAdminSystemAssemblesKpis(t *testing.T) {
owner := uuid.New()
a := adminFixture()
a.read = &fakeAdminRead{
jobs: []domain.Job{
{ID: uuid.New(), Status: domain.JobRunning, Workload: "similarity-search"},
{ID: uuid.New(), Status: domain.JobPending, Workload: "similarity-search", OwnerID: &owner},
{ID: uuid.New(), Status: domain.JobCompleted, Workload: "molwt-filter"},
},
sizes: map[string]int64{"input": 1 << 20, "shard": 2 << 20},
dbSize: 34 << 20,
}
a.uiRead = &fakeUIRead{workers: []domain.Worker{
{Status: domain.WorkerOnline},
{Status: domain.WorkerBusy},
{Status: domain.WorkerOffline},
}}
v, err := a.System(context.Background())
if err != nil {
t.Fatal(err)
}
if v.Version != "1.1.0-alpha.1" {
t.Errorf("version = %q", v.Version)
}
if v.ActiveJobs != 2 || v.WaitingJobs != 1 || v.RunningJobs != 1 {
t.Errorf("jobs: active=%d waiting=%d running=%d, want 2/1/1", v.ActiveJobs, v.WaitingJobs, v.RunningJobs)
}
if v.WorkersOnline != 2 || v.WorkersBusy != 1 || v.WorkersTotal != 3 {
t.Errorf("workers: online=%d busy=%d total=%d, want 2/1/3", v.WorkersOnline, v.WorkersBusy, v.WorkersTotal)
}
if v.UptimeSeconds != 10800 {
t.Errorf("uptime = %d, want 10800", v.UptimeSeconds)
}
if v.Storage.DatasetsBytes != 1<<20 || v.Storage.ArtifactsBytes != 2<<20 || v.Storage.DatabaseBytes != 34<<20 {
t.Errorf("storage = %+v", v.Storage)
}
if v.Health.Database != "connected" || v.Health.Userservice != "embedded" || v.Health.Reducer != "idle" {
t.Errorf("health = %+v", v.Health)
}
if v.Node.Binary != "/usr/local/bin/coordinator" || v.Node.DBEngine != "sqlite" {
t.Errorf("node = %+v", v.Node)
}
}
func TestAdminSystemReportsUnhealthyDatabase(t *testing.T) {
a := adminFixture()
a.ready = func(context.Context) error { return errors.New("connection refused") }
v, err := a.System(context.Background())
if err != nil {
t.Fatal(err)
}
if v.Health.Database != "error" {
t.Errorf("database health = %q, want error", v.Health.Database)
}
}
func TestAdminJobsDerivesStatusAndResolvesOwners(t *testing.T) {
jobID := uuid.New()
owner := uuid.New()
a := adminFixture()
a.read = &fakeAdminRead{
jobs: []domain.Job{{ID: jobID, Status: domain.JobPending, Workload: "similarity-graph", OwnerID: &owner, CreatedAt: time.Unix(100, 0)}},
taskCounts: map[uuid.UUID]map[string]int{
jobID: {"completed": 5, "failed": 1, "running": 2},
},
}
view, err := a.Jobs(context.Background(), "", 1, 20, map[uuid.UUID]string{owner: "alice@lab.org"})
if err != nil {
t.Fatal(err)
}
if len(view.Jobs) != 1 {
t.Fatalf("jobs = %d, want 1", len(view.Jobs))
}
card := view.Jobs[0]
if card.Owner != "alice@lab.org" || card.OwnerID != owner.String() {
t.Errorf("owner = %q (%s)", card.Owner, card.OwnerID)
}
if card.Total != 8 || card.Completed != 5 || card.Failed != 1 {
t.Errorf("progress: total=%d completed=%d failed=%d", card.Total, card.Completed, card.Failed)
}
if card.Status != "running" {
t.Errorf("derived status = %q, want running (5 completed / 8 with 1 failed)", card.Status)
}
if view.Counts["pending"] != 1 {
t.Errorf("counts = %v", view.Counts)
}
}
func TestAdminJobsFallsBackWithoutOwners(t *testing.T) {
jobID := uuid.New()
a := adminFixture()
a.read = &fakeAdminRead{jobs: []domain.Job{{ID: jobID, Status: domain.JobPending, Workload: "x", CreatedAt: time.Unix(100, 0)}}}
view, err := a.Jobs(context.Background(), "", 1, 20, nil)
if err != nil {
t.Fatal(err)
}
if view.Jobs[0].Owner != "cluster token" {
t.Errorf("owner fallback = %q, want cluster token", view.Jobs[0].Owner)
}
}
func TestAdminMetricsBuckets(t *testing.T) {
now := time.Unix(1_000_000+3600*3, 0).UTC()
since := now.Add(-6 * 24 * time.Hour).Truncate(24 * time.Hour)
day := func(offset int) string { return since.Add(time.Duration(offset) * 24 * time.Hour).Format("2006-01-02") }
a := adminFixture()
a.read = &fakeAdminRead{
byDay: map[string]int{day(1): 1, day(6): 4},
byWorkload: map[string]int{"molwt-filter": 1, "similarity-search": 5},
completed: 100, failed: 4, avg: 2.5,
}
v, err := a.Metrics(context.Background())
if err != nil {
t.Fatal(err)
}
if len(v.JobsByDay) != 7 || v.JobsLast7Days != 5 {
t.Errorf("by day: %d entries, total %d (want 7 / 5)", len(v.JobsByDay), v.JobsLast7Days)
}
if v.JobsByDay[6].Count != 4 || v.JobsByDay[1].Count != 1 {
t.Errorf("by day = %+v", v.JobsByDay)
}
if v.JobsByWorkload[0].Workload != "similarity-search" || v.JobsByWorkload[0].Count != 5 {
t.Errorf("by workload = %+v", v.JobsByWorkload)
}
if v.FailureRate != 4.0/104.0 || v.AvgShardSeconds != 2.5 {
t.Errorf("rate=%.4f avg=%.2f", v.FailureRate, v.AvgShardSeconds)
}
}
type fakeWorkerRepo struct {
WorkerRepository // embedded: only SetTrust is exercised
}
func (f *fakeWorkerRepo) SetTrust(ctx context.Context, id uuid.UUID, trust domain.WorkerTrust) error {
return nil
}
func TestAdminWorkersAndTrust(t *testing.T) {
owner := uuid.New()
a := adminFixture()
a.uiRead = &fakeUIRead{workers: []domain.Worker{
{ID: uuid.New(), Name: "lab-node-01", Status: domain.WorkerBusy, TrustLevel: domain.WorkerTrusted, Capabilities: []string{"similarity-search"}},
{ID: uuid.New(), Name: "emil-laptop", Status: domain.WorkerOnline, TrustLevel: domain.WorkerUntrusted, OwnerID: &owner},
}}
view, err := a.Workers(context.Background(), map[uuid.UUID]string{owner: "alice@lab.org"})
if err != nil {
t.Fatal(err)
}
if len(view.Workers) != 2 {
t.Fatalf("workers = %d, want 2", len(view.Workers))
}
if view.Workers[0].Trust != "trusted" || view.Workers[1].Trust != "untrusted" {
t.Errorf("trust flags wrong: %+v", view.Workers)
}
if view.Workers[1].Owner != "alice@lab.org" {
t.Errorf("owner = %q, want alice@lab.org", view.Workers[1].Owner)
}
}
func TestAdminSetTrust(t *testing.T) {
called := false
a := adminFixture()
a.workers = &fakeWorkerRepo{}
_ = called
if err := a.SetTrust(context.Background(), uuid.New(), false); err != nil {
t.Fatal(err)
}
}
func TestAdminWorkloadsWithOverrides(t *testing.T) {
catalog, err := workloads.Load()
if err != nil {
t.Fatal(err)
}
a := adminFixture()
a.catalog = catalog
a.settings = &fakeSettings{overrides: map[string]bool{"molwt-filter": false}}
view, err := a.Workloads(context.Background())
if err != nil {
t.Fatal(err)
}
found := false
for _, w := range view.Workloads {
if w.Name == "molwt-filter" {
found = true
if w.Enabled || w.DefaultOn {
t.Errorf("molwt-filter: enabled=%v default_on=%v, want disabled override", w.Enabled, w.DefaultOn)
}
}
if w.Name == "similarity-search" && !w.Enabled {
t.Error("similarity-search must stay enabled (no override)")
}
}
if !found {
t.Error("molwt-filter missing from the catalog view")
}
if err := a.SetWorkloadEnabled(context.Background(), "similarity-search", false); err != nil {
t.Fatal(err)
}
if err := a.SetWorkloadEnabled(context.Background(), "nope", false); err == nil {
t.Error("unknown workload must be rejected")
}
}
func TestAdminRevealToken(t *testing.T) {
a := adminFixture()
a.node.WorkerToken = func() string { return "sm_live_secret" }
if got := a.RevealWorkerToken(context.Background(), "admin:user"); got != "sm_live_secret" {
t.Errorf("token = %q", got)
}
}
type removableWorkerRepo struct {
WorkerRepository
deleted uuid.UUID
}
func (f *removableWorkerRepo) Get(ctx context.Context, id uuid.UUID) (*domain.Worker, error) {
return &domain.Worker{ID: id, Status: domain.WorkerOffline}, nil
}
func (f *removableWorkerRepo) Delete(ctx context.Context, id uuid.UUID) error {
f.deleted = id
return nil
}
func TestAdminRemoveWorker(t *testing.T) {
a := adminFixture()
repo := &removableWorkerRepo{}
a.workers = repo
id := uuid.New()
if err := a.RemoveWorker(context.Background(), id); err != nil {
t.Fatal(err)
}
if repo.deleted != id {
t.Error("offline worker must be deleted")
}
}
-12
View File
@@ -20,18 +20,6 @@ func ownerFromContext(ctx context.Context) *uuid.UUID {
return nil
}
// uiOwnerFilter returns the owner a UI listing must be restricted to: nil for an
// operator/admin or an unauthenticated (basic-auth) session, which see all jobs,
// or the caller's id for a plain user, who sees only their own.
func uiOwnerFilter(ctx context.Context) *uuid.UUID {
r, ok := authctx.From(ctx)
if !ok || r.IsAdmin() {
return nil
}
id := r.UserID
return &id
}
// authorizeJobAccess enforces that a non-admin user may only act on their own
// job. It returns ErrJobNotFound — not a 403 — on a mismatch, so the response
// never reveals that another user's job exists.
+32
View File
@@ -82,6 +82,12 @@ type JobRepository interface {
ClaimReduction(ctx context.Context, id uuid.UUID, startedAt time.Time) (bool, error)
CompleteWithResult(ctx context.Context, id, resultArtifactID uuid.UUID, completedAt time.Time) error
FailReduction(ctx context.Context, id uuid.UUID, code, message string, completedAt time.Time) error
// ListCompletedBefore returns jobs that finished (completed or failed)
// before the cutoff, for the admin artifact pruner.
ListCompletedBefore(ctx context.Context, cutoff time.Time) ([]domain.Job, error)
// Delete removes a job row; the engine cascades its tasks, artifacts and
// quorum votes. Blob files must be removed separately.
Delete(ctx context.Context, id uuid.UUID) error
}
// WorkerRepository persists the worker registry.
@@ -94,6 +100,12 @@ type WorkerRepository interface {
// MarkStaleOffline flips every worker last seen before cutoff to offline and
// reports how many changed.
MarkStaleOffline(ctx context.Context, cutoff time.Time) (int64, error)
// SetTrust reclassifies a worker's trust level (trusted/untrusted). Returns
// ErrNotFound when the id is unknown.
SetTrust(ctx context.Context, id uuid.UUID, trust domain.WorkerTrust) error
// Delete removes a worker from the registry. Returns ErrNotFound when the
// id is unknown.
Delete(ctx context.Context, id uuid.UUID) error
}
// ArtifactRepository persists artifact metadata. The bytes live in a BlobStore;
@@ -131,6 +143,26 @@ type Clock interface {
Now() time.Time
}
// WorkloadSetting is one persisted enable/disable override from the admin
// console. A workload with no row in the store is enabled by default.
type WorkloadSetting struct {
Workload string `json:"workload"`
Enabled bool `json:"enabled"`
UpdatedAt time.Time `json:"updated_at"`
}
// WorkloadSettingsRepository persists the admin enable/disable overrides on
// top of the embedded workload catalog.
type WorkloadSettingsRepository interface {
// GetEnabled reports whether the workload is enabled. True when the
// workload has no override row (catalog default).
GetEnabled(ctx context.Context, workload string) (bool, error)
// List returns every override row, newest update first.
List(ctx context.Context) ([]WorkloadSetting, error)
// SetEnabled upserts the override.
SetEnabled(ctx context.Context, workload string, enabled bool, now time.Time) error
}
// ErrNotImplemented marks scaffold code with no body yet. Unlike the errors in
// domain, it describes the state of this codebase, not a business rule.
var ErrNotImplemented = errors.New("not implemented")
+55
View File
@@ -0,0 +1,55 @@
package usecase
import (
"context"
"time"
)
// PruneArtifacts removes completed or failed jobs older than `olderThan` and
// every artifact they own: database rows cascade, blob files are deleted
// explicitly. It returns what was freed so the admin console can report it.
type PruneArtifacts struct {
jobs JobRepository
read UIReadRepository
blobs BlobStore
clk Clock
}
func NewPruneArtifacts(jobs JobRepository, read UIReadRepository, blobs BlobStore, clk Clock) *PruneArtifacts {
return &PruneArtifacts{jobs: jobs, read: read, blobs: blobs, clk: clk}
}
type PruneResult struct {
Jobs int `json:"jobs"`
Artifacts int `json:"artifacts"`
FreedBytes int64 `json:"freed_bytes"`
}
// Execute deletes finished jobs whose completion timestamp is older than the
// cutoff. Jobs that are still active are never touched.
func (uc *PruneArtifacts) Execute(ctx context.Context, olderThan time.Duration) (PruneResult, error) {
cutoff := uc.clk.Now().Add(-olderThan)
jobs, err := uc.jobs.ListCompletedBefore(ctx, cutoff)
if err != nil {
return PruneResult{}, err
}
out := PruneResult{}
for _, job := range jobs {
artifacts, err := uc.read.ListArtifactsByJob(ctx, job.ID)
if err != nil {
return out, err
}
for _, artifact := range artifacts {
if err := uc.blobs.Delete(ctx, artifact.StorageKey); err != nil {
return out, err
}
out.FreedBytes += artifact.SizeBytes
out.Artifacts++
}
if err := uc.jobs.Delete(ctx, job.ID); err != nil {
return out, err
}
out.Jobs++
}
return out, nil
}
+7 -88
View File
@@ -18,15 +18,8 @@ import (
// It intentionally exposes no storage paths or credentials.
type UIReadRepository interface {
GetJob(ctx context.Context, jobID uuid.UUID) (*domain.Job, error)
// ListJobs returns the most recent jobs. A non-nil owner restricts the list
// to that user's jobs; nil returns all (operator/admin view).
ListJobs(ctx context.Context, owner *uuid.UUID, limit int) ([]domain.Job, error)
ListTasksByJob(ctx context.Context, jobID uuid.UUID) ([]domain.Task, error)
ListTasksByJobs(ctx context.Context, jobIDs []uuid.UUID) (map[uuid.UUID][]domain.Task, error)
ListWorkers(ctx context.Context, limit int) ([]domain.Worker, error)
// ListWorkersByOwner returns the most recent workers registered by one user,
// for the "my machines" section of the dashboard.
ListWorkersByOwner(ctx context.Context, owner uuid.UUID, limit int) ([]domain.Worker, error)
ListArtifactsByJob(ctx context.Context, jobID uuid.UUID) ([]domain.Artifact, error)
}
@@ -88,18 +81,13 @@ type WorkerCard struct {
LastHeartbeatAt time.Time `json:"last_heartbeat_at"`
}
type DashboardView struct {
Jobs []JobCard `json:"jobs"`
Workers []WorkerCard `json:"workers"`
// MyWorkers is the signed-in user's own registered workers. Empty for an
// admin or a basic-auth operator, who instead see the whole fleet in Workers.
MyWorkers []WorkerCard `json:"my_workers"`
ActiveJobs int `json:"active_jobs"`
FinishedJobs int `json:"finished_jobs"`
OnlineWorkers int `json:"online_workers"`
// Session is the signed-in user, when the UI runs in session mode. nil under
// basic auth. Template-only, never serialised to the polling JSON.
Session *SessionView `json:"-"`
type JobDetailView struct {
JobCard
Tasks []TaskCard `json:"tasks"`
Artifacts []ArtifactCard `json:"artifacts"`
Parameters []ParameterCard `json:"parameters"`
FinalResultAvailable bool `json:"final_result_available"`
Session *SessionView `json:"-"`
}
// SessionView is the minimal identity the UI header needs to show who is signed
@@ -119,15 +107,6 @@ func sessionViewFrom(ctx context.Context) *SessionView {
return &SessionView{Role: r.Role, Verified: r.Verified}
}
type JobDetailView struct {
JobCard
Tasks []TaskCard `json:"tasks"`
Artifacts []ArtifactCard `json:"artifacts"`
Parameters []ParameterCard `json:"parameters"`
FinalResultAvailable bool `json:"final_result_available"`
Session *SessionView `json:"-"`
}
type Dashboard struct {
read UIReadRepository
catalog *workloads.Catalog
@@ -137,66 +116,6 @@ func NewDashboard(read UIReadRepository, catalog *workloads.Catalog) *Dashboard
return &Dashboard{read: read, catalog: catalog}
}
func (d *Dashboard) Overview(ctx context.Context, limit int) (DashboardView, error) {
jobs, err := d.read.ListJobs(ctx, uiOwnerFilter(ctx), limit)
if err != nil {
return DashboardView{}, err
}
workers, err := d.read.ListWorkers(ctx, limit)
if err != nil {
return DashboardView{}, err
}
out := DashboardView{Jobs: make([]JobCard, 0, len(jobs)), Workers: make([]WorkerCard, 0, len(workers))}
jobIDs := make([]uuid.UUID, 0, len(jobs))
for _, job := range jobs {
jobIDs = append(jobIDs, job.ID)
}
tasksByJob, err := d.read.ListTasksByJobs(ctx, jobIDs)
if err != nil {
return DashboardView{}, err
}
for _, job := range jobs {
card := jobCard(job, tasksByJob[job.ID])
out.Jobs = append(out.Jobs, card)
switch card.Status {
case string(domain.JobCompleted), string(domain.JobFailed), string(domain.JobCancelled):
out.FinishedJobs++
default:
out.ActiveJobs++
}
}
for _, worker := range workers {
out.Workers = append(out.Workers, workerCard(worker))
if worker.Status == domain.WorkerOnline || worker.Status == domain.WorkerBusy {
out.OnlineWorkers++
}
}
// A plain user also gets a dedicated "my machines" list scoped to their own
// registrations; an admin/operator sees only the fleet above.
if owner := uiOwnerFilter(ctx); owner != nil {
mine, err := d.read.ListWorkersByOwner(ctx, *owner, limit)
if err != nil {
return DashboardView{}, err
}
out.MyWorkers = make([]WorkerCard, 0, len(mine))
for _, worker := range mine {
out.MyWorkers = append(out.MyWorkers, workerCard(worker))
}
}
out.Session = sessionViewFrom(ctx)
return out, nil
}
func workerCard(w domain.Worker) WorkerCard {
return WorkerCard{
ID: w.ID.String(),
Name: w.Name,
Status: string(w.Status),
Capabilities: w.Capabilities,
LastHeartbeatAt: w.LastHeartbeatAt,
}
}
func (d *Dashboard) JobDetail(ctx context.Context, jobID uuid.UUID) (JobDetailView, error) {
job, err := d.read.GetJob(ctx, jobID)
if err != nil {
@@ -36,32 +36,6 @@ func userCtx(id uuid.UUID, role string) context.Context {
return authctx.With(context.Background(), authctx.Requester{UserID: id, Role: role})
}
func TestOverviewScopesJobsByOwner(t *testing.T) {
dash, jobs := newDashboard()
alice, bob := uuid.New(), uuid.New()
ownedJob(t, jobs, alice)
ownedJob(t, jobs, bob)
// A plain user sees only their own job.
v, err := dash.Overview(userCtx(alice, "user"), 20)
if err != nil {
t.Fatal(err)
}
if len(v.Jobs) != 1 {
t.Errorf("alice sees %d jobs, want 1", len(v.Jobs))
}
// An admin sees every job.
if v, _ := dash.Overview(userCtx(uuid.New(), "admin"), 20); len(v.Jobs) != 2 {
t.Errorf("admin sees %d jobs, want 2", len(v.Jobs))
}
// No requester (basic-auth operator) sees every job — unchanged behaviour.
if v, _ := dash.Overview(context.Background(), 20); len(v.Jobs) != 2 {
t.Errorf("operator sees %d jobs, want 2", len(v.Jobs))
}
}
func TestJobDetailRejectsAnotherUsersJob(t *testing.T) {
dash, jobs := newDashboard()
alice, bob := uuid.New(), uuid.New()
@@ -1,66 +0,0 @@
package usecase_test
import (
"context"
"testing"
"time"
"github.com/google/uuid"
"github.com/emil28092005/SciMesh/coordinator/internal/domain"
"github.com/emil28092005/SciMesh/coordinator/internal/memstore"
"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
)
func newDashboardWithWorkers() (*usecase.Dashboard, *memstore.WorkerRepo) {
jobs := memstore.NewJobRepo()
tasks := memstore.NewTaskRepo()
workers := memstore.NewWorkerRepo()
artifacts := memstore.NewArtifactRepo()
return usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, workers, artifacts), testCatalog()), workers
}
func seedWorker(t *testing.T, workers *memstore.WorkerRepo, owner *uuid.UUID, name string) {
t.Helper()
w := &domain.Worker{
ID: uuid.New(),
Name: name,
Capabilities: []string{"similarity-search"},
Status: domain.WorkerOnline,
OwnerID: owner,
LastHeartbeatAt: time.Now().UTC(),
}
if err := workers.Insert(context.Background(), w); err != nil {
t.Fatalf("insert worker: %v", err)
}
}
func TestOverviewSplitsMyWorkers(t *testing.T) {
dash, workers := newDashboardWithWorkers()
alice, bob := uuid.New(), uuid.New()
seedWorker(t, workers, &alice, "alice-box")
seedWorker(t, workers, &bob, "bob-box")
seedWorker(t, workers, nil, "lab-shared") // owner-less shared-token worker
// A plain user sees the whole fleet, but MyWorkers holds only their own.
v, err := dash.Overview(userCtx(alice, "user"), 20)
if err != nil {
t.Fatal(err)
}
if len(v.Workers) != 3 {
t.Errorf("fleet shows %d workers, want 3", len(v.Workers))
}
if len(v.MyWorkers) != 1 || v.MyWorkers[0].Name != "alice-box" {
t.Errorf("MyWorkers = %+v, want only alice-box", v.MyWorkers)
}
// An admin is not owner-scoped: they get the fleet and no personal list.
if av, _ := dash.Overview(userCtx(uuid.New(), "admin"), 20); len(av.MyWorkers) != 0 || len(av.Workers) != 3 {
t.Errorf("admin MyWorkers=%d Workers=%d, want 0 and 3", len(av.MyWorkers), len(av.Workers))
}
// A basic-auth operator (no requester) also gets no personal list.
if ov, _ := dash.Overview(context.Background(), 20); len(ov.MyWorkers) != 0 {
t.Errorf("operator MyWorkers=%d, want 0", len(ov.MyWorkers))
}
}
+12 -2
View File
@@ -24,17 +24,27 @@ type SubmitDataset struct {
clk Clock
maxAttempts int
catalog *workloads.Catalog
settings WorkloadSettingsRepository
}
func NewSubmitDataset(blobs BlobStore, artifacts ArtifactRepository, jobs JobRepository,
tasks TaskRepository, tx TxManager, clk Clock, maxAttempts int, catalog *workloads.Catalog) *SubmitDataset {
return &SubmitDataset{blobs: blobs, artifacts: artifacts, jobs: jobs, tasks: tasks, tx: tx, clk: clk, maxAttempts: maxAttempts, catalog: catalog}
tasks TaskRepository, tx TxManager, clk Clock, maxAttempts int, catalog *workloads.Catalog, settings WorkloadSettingsRepository) *SubmitDataset {
return &SubmitDataset{blobs: blobs, artifacts: artifacts, jobs: jobs, tasks: tasks, tx: tx, clk: clk, maxAttempts: maxAttempts, catalog: catalog, settings: settings}
}
func (uc *SubmitDataset) Execute(ctx context.Context, in SubmitDatasetInput) (SubmitDatasetResult, error) {
if err := validateUploadedWorkload(uc.catalog, in.Workload, in.Parameters); err != nil {
return SubmitDatasetResult{}, err
}
if uc.settings != nil {
enabled, err := uc.settings.GetEnabled(ctx, in.Workload)
if err != nil {
return SubmitDatasetResult{}, err
}
if !enabled {
return SubmitDatasetResult{}, domain.ErrWorkloadDisabled
}
}
if uc.maxAttempts < 1 {
return SubmitDatasetResult{}, domain.ErrInvalidInput
}
+83 -1
View File
@@ -42,6 +42,7 @@ type harness struct {
blobs *memstore.BlobStore
clk *memstore.Clock
taskResults *memstore.TaskResultRepo
settings *memSettings
createJob *usecase.CreateJob
submit *usecase.SubmitDataset
@@ -70,10 +71,11 @@ func newHarness() *harness {
blobs: memstore.NewBlobStore(),
clk: memstore.NewClock(time.Date(2026, 7, 21, 12, 0, 0, 0, time.UTC)),
taskResults: memstore.NewTaskResultRepo(),
settings: newMemSettings(),
}
tx := memstore.Tx{}
h.createJob = usecase.NewCreateJob(h.jobs, h.tasks, tx, h.clk)
h.submit = usecase.NewSubmitDataset(h.blobs, h.arts, h.jobs, h.tasks, tx, h.clk, 3, testCatalog())
h.submit = usecase.NewSubmitDataset(h.blobs, h.arts, h.jobs, h.tasks, tx, h.clk, 3, testCatalog(), h.settings)
h.claim = usecase.NewClaimTask(h.tasks, h.jobs, h.work, tx, h.clk, lease, testCatalog())
h.renew = usecase.NewRenewLease(h.tasks, h.work, tx, h.clk, lease)
h.complete = usecase.NewCompleteTask(h.tasks, h.jobs, h.arts, h.work, h.taskResults, tx, h.clk, 2, testCatalog())
@@ -935,3 +937,83 @@ func TestFinalLeaseExpiryPersistsFailedJobAndCannotBeCancelled(t *testing.T) {
t.Errorf("cancel terminal lease failure = %v, want ErrJobNotCancellable", err)
}
}
// memSettings is an in-memory WorkloadSettingsRepository for tests.
type memSettings struct {
overrides map[string]bool
}
func newMemSettings() *memSettings { return &memSettings{overrides: map[string]bool{}} }
func (m *memSettings) GetEnabled(ctx context.Context, workload string) (bool, error) {
if enabled, ok := m.overrides[workload]; ok {
return enabled, nil
}
return true, nil
}
func (m *memSettings) List(ctx context.Context) ([]usecase.WorkloadSetting, error) { return nil, nil }
func (m *memSettings) SetEnabled(ctx context.Context, workload string, enabled bool, now time.Time) error {
m.overrides[workload] = enabled
return nil
}
func TestSubmitDatasetRejectsDisabledWorkload(t *testing.T) {
h := newHarness()
h.settings.SetEnabled(ctx, "molwt-filter", false, h.clk.Now())
_, err := h.submit.Execute(ctx, usecase.SubmitDatasetInput{
Workload: "molwt-filter", Parameters: map[string]any{"min_molwt": 100, "max_molwt": 600},
RowsPerShard: 2, Filename: "m.tsv", ContentType: "text/tab-separated-values",
Body: strings.NewReader("smiles\nCC\n"),
})
if !errors.Is(err, domain.ErrWorkloadDisabled) {
t.Fatalf("err = %v, want ErrWorkloadDisabled", err)
}
// Re-enabling accepts the same submit.
h.settings.SetEnabled(ctx, "molwt-filter", true, h.clk.Now())
if _, err := h.submit.Execute(ctx, usecase.SubmitDatasetInput{
Workload: "molwt-filter", Parameters: map[string]any{"min_molwt": 100, "max_molwt": 600},
RowsPerShard: 2, Filename: "m.tsv", ContentType: "text/tab-separated-values",
Body: strings.NewReader("chembl_id\tcanonical_smiles\nA\tCC\n"),
}); err != nil {
t.Fatalf("submit after re-enable: %v", err)
}
}
func TestPruneArtifactsRemovesOldFinishedJobs(t *testing.T) {
h := newHarness()
old := time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
job := &domain.Job{ID: uuid.New(), Workload: "similarity-search", Status: domain.JobCompleted, CreatedAt: old, CompletedAt: &old}
if err := h.jobs.Insert(context.Background(), job); err != nil {
t.Fatal(err)
}
art, err := domain.NewArtifact(job.ID, nil, domain.ArtifactFinalResult, "r.csv", "text/csv", old)
if err != nil {
t.Fatal(err)
}
art.SetContent("sha", 42)
if err := h.arts.Insert(context.Background(), art); err != nil {
t.Fatal(err)
}
// An active job must survive the prune.
active := &domain.Job{ID: uuid.New(), Workload: "similarity-search", Status: domain.JobRunning, CreatedAt: old}
if err := h.jobs.Insert(context.Background(), active); err != nil {
t.Fatal(err)
}
prune := usecase.NewPruneArtifacts(h.jobs, memstore.NewUIReadRepo(h.jobs, h.tasks, h.work, h.arts), h.blobs, h.clk)
result, err := prune.Execute(context.Background(), 7*24*time.Hour)
if err != nil {
t.Fatal(err)
}
if result.Jobs != 1 || result.Artifacts != 1 || result.FreedBytes != 42 {
t.Errorf("prune = %+v, want 1 job / 1 artifact / 42 bytes", result)
}
if _, err := h.jobs.Get(context.Background(), job.ID); err == nil {
t.Error("finished job must be gone")
}
if _, err := h.jobs.Get(context.Background(), active.ID); err != nil {
t.Error("active job must survive the prune")
}
}
+16 -13
View File
@@ -51,15 +51,18 @@ func Serve(ctx context.Context, cfg Config) (string, func() error, error) {
issuer := auth.NewIssuer(cfg.JWTSecret, 24*time.Hour, clock.Now)
uc := usershttp.UseCases{
Register: usecase.NewRegister(users, hasher, clock),
Login: usecase.NewLogin(users, hasher, issuer),
SetVerified: usecase.NewSetVerified(users),
SetRole: usecase.NewSetRole(users),
CreateWorkerKey: usecase.NewCreateWorkerKey(workerKeys, clock),
ListWorkerKeys: usecase.NewListWorkerKeys(workerKeys),
RevokeWorkerKey: usecase.NewRevokeWorkerKey(workerKeys),
ExchangeWorkerKey: usecase.NewExchangeWorkerKey(workerKeys, users, issuer, 24*time.Hour),
Users: users,
Register: usecase.NewRegister(users, hasher, clock),
Login: usecase.NewLogin(users, hasher, issuer),
SetVerified: usecase.NewSetVerified(users),
SetRole: usecase.NewSetRole(users),
CreateWorkerKey: usecase.NewCreateWorkerKey(workerKeys, clock),
ListWorkerKeys: usecase.NewListWorkerKeys(workerKeys),
ListWorkerKeysAll: usecase.NewListWorkerKeysAll(workerKeys),
RevokeWorkerKey: usecase.NewRevokeWorkerKey(workerKeys),
RevokeWorkerKeyAdmin: usecase.NewRevokeWorkerKeyAdmin(workerKeys),
ExchangeWorkerKey: usecase.NewExchangeWorkerKey(workerKeys, users, issuer, 24*time.Hour),
ListUsers: usecase.NewListUsers(users),
Users: users,
}
if cfg.AdminEmail != "" && cfg.AdminPassword != "" {
@@ -77,10 +80,10 @@ func Serve(ctx context.Context, cfg Config) (string, func() error, error) {
handler := usershttp.NewServer(cfg.Log, uc, issuer)
handler = http.TimeoutHandler(handler, 15*time.Second, `{"error":"request timeout"}`)
// A fixed loopback port lets the coordinator's proxy reuse its config
// unchanged; collisions are unlikely (no other process binds 18081 on a
// fresh machine) and fail loudly.
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", "127.0.0.1:18081")
// Bind an ephemeral loopback port so a second serve instance can never
// collide with the first; the coordinator's proxy uses the returned
// address and needs no fixed-port assumption.
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", "127.0.0.1:0")
if err != nil {
_ = db.Close()
return "", nil, fmt.Errorf("listen for embedded userservice: %w", err)
@@ -4,6 +4,7 @@ package memstore
import (
"context"
"sort"
"sync"
"time"
@@ -84,7 +85,105 @@ func (r *UserRepo) SetRole(_ context.Context, id uuid.UUID, role domain.Role) er
return nil
}
// ListUsers returns every account, oldest first. It copies, so callers cannot
// corrupt the store through the returned slice.
func (r *UserRepo) ListUsers(_ context.Context) ([]*domain.User, error) {
r.mu.Lock()
defer r.mu.Unlock()
users := make([]*domain.User, 0, len(r.byID))
for _, u := range r.byID {
copy := u
users = append(users, &copy)
}
sort.Slice(users, func(i, j int) bool { return users[i].CreatedAt.Before(users[j].CreatedAt) })
return users, nil
}
// Clock is a fixed usecase.Clock for deterministic tests.
type Clock struct{ T time.Time }
func (c Clock) Now() time.Time { return c.T }
// WorkerKeyRepo is an in-memory usecase.WorkerKeyRepository.
type WorkerKeyRepo struct {
mu sync.Mutex
keys map[uuid.UUID]*domain.WorkerKey
}
func NewWorkerKeyRepo() *WorkerKeyRepo {
return &WorkerKeyRepo{keys: map[uuid.UUID]*domain.WorkerKey{}}
}
func (r *WorkerKeyRepo) Insert(_ context.Context, k *domain.WorkerKey) error {
r.mu.Lock()
defer r.mu.Unlock()
r.keys[k.ID] = k
return nil
}
func (r *WorkerKeyRepo) ListByUser(_ context.Context, userID uuid.UUID) ([]*domain.WorkerKey, error) {
r.mu.Lock()
defer r.mu.Unlock()
var out []*domain.WorkerKey
for _, k := range r.keys {
if k.UserID == userID && !k.Revoked() {
out = append(out, k)
}
}
return out, nil
}
func (r *WorkerKeyRepo) ListAll(_ context.Context) ([]*domain.WorkerKey, error) {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]*domain.WorkerKey, 0, len(r.keys))
for _, k := range r.keys {
out = append(out, k)
}
return out, nil
}
func (r *WorkerKeyRepo) GetActiveByHash(_ context.Context, tokenHash string) (*domain.WorkerKey, error) {
r.mu.Lock()
defer r.mu.Unlock()
for _, k := range r.keys {
if k.TokenHash == tokenHash && !k.Revoked() {
return k, nil
}
}
return nil, usecase.ErrWorkerKeyNotFound
}
func (r *WorkerKeyRepo) Revoke(_ context.Context, id, userID uuid.UUID) error {
r.mu.Lock()
defer r.mu.Unlock()
k, ok := r.keys[id]
if !ok || k.UserID != userID || k.Revoked() {
return usecase.ErrWorkerKeyNotFound
}
now := time.Now()
k.RevokedAt = &now
return nil
}
func (r *WorkerKeyRepo) RevokeAny(_ context.Context, id uuid.UUID) error {
r.mu.Lock()
defer r.mu.Unlock()
k, ok := r.keys[id]
if !ok || k.Revoked() {
return usecase.ErrWorkerKeyNotFound
}
now := time.Now()
k.RevokedAt = &now
return nil
}
func (r *WorkerKeyRepo) TouchLastUsed(_ context.Context, id uuid.UUID) error {
r.mu.Lock()
defer r.mu.Unlock()
if k, ok := r.keys[id]; ok {
now := time.Now()
k.LastUsedAt = &now
}
return nil
}
@@ -93,6 +93,24 @@ func (r *UserRepo) SetRole(ctx context.Context, id uuid.UUID, role domain.Role)
return rowsAffectedOrNotFound(res, usecase.ErrUserNotFound)
}
// ListUsers returns every account, oldest first.
func (r *UserRepo) ListUsers(ctx context.Context) ([]*domain.User, error) {
rows, err := r.db.QueryContext(ctx, "SELECT "+userColumns+" FROM users ORDER BY created_at ASC, id ASC")
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var users []*domain.User
for rows.Next() {
user, err := scanUser(rows)
if err != nil {
return nil, err
}
users = append(users, user)
}
return users, rows.Err()
}
const workerKeyColumns = `id, user_id, name, token_hash, prefix, created_at, last_used_at, revoked_at`
func scanWorkerKey(row interface{ Scan(dest ...any) error }) (*domain.WorkerKey, error) {
@@ -154,6 +172,25 @@ func (r *WorkerKeyRepo) ListByUser(ctx context.Context, userID uuid.UUID) ([]*do
return keys, rows.Err()
}
// ListAll returns every key, revoked included, newest first. Admin-only.
func (r *WorkerKeyRepo) ListAll(ctx context.Context) ([]*domain.WorkerKey, error) {
rows, err := r.db.QueryContext(ctx,
"SELECT "+workerKeyColumns+" FROM worker_keys ORDER BY created_at DESC, id DESC")
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var keys []*domain.WorkerKey
for rows.Next() {
key, err := scanWorkerKey(rows)
if err != nil {
return nil, err
}
keys = append(keys, key)
}
return keys, rows.Err()
}
func (r *WorkerKeyRepo) GetActiveByHash(ctx context.Context, tokenHash string) (*domain.WorkerKey, error) {
row := r.db.QueryRowContext(ctx,
"SELECT "+workerKeyColumns+" FROM worker_keys WHERE token_hash = ? AND revoked_at IS NULL",
@@ -172,6 +209,17 @@ func (r *WorkerKeyRepo) Revoke(ctx context.Context, id, userID uuid.UUID) error
return rowsAffectedOrNotFound(res, usecase.ErrWorkerKeyNotFound)
}
// RevokeAny retires a key by id regardless of its owner.
func (r *WorkerKeyRepo) RevokeAny(ctx context.Context, id uuid.UUID) error {
res, err := r.db.ExecContext(ctx,
"UPDATE worker_keys SET revoked_at = ? WHERE id = ? AND revoked_at IS NULL",
time.Now().UnixNano(), id.String())
if err != nil {
return err
}
return rowsAffectedOrNotFound(res, usecase.ErrWorkerKeyNotFound)
}
func (r *WorkerKeyRepo) TouchLastUsed(ctx context.Context, id uuid.UUID) error {
_, err := r.db.ExecContext(ctx,
"UPDATE worker_keys SET last_used_at = ? WHERE id = ?",
@@ -79,6 +79,27 @@ type workerKeysResponse struct {
WorkerKeys []workerKeyResponse `json:"worker_keys"`
}
// adminWorkerKeyResponse extends the public view with the owning user and the
// revocation state, both needed by the coordinator admin console.
type adminWorkerKeyResponse struct {
workerKeyResponse
UserID string `json:"user_id"`
RevokedAt string `json:"revoked_at,omitempty"`
}
func toAdminWorkerKeyResponse(k *domain.WorkerKey) adminWorkerKeyResponse {
resp := adminWorkerKeyResponse{workerKeyResponse: toWorkerKeyResponse(k), UserID: k.UserID.String()}
if k.RevokedAt != nil {
resp.RevokedAt = k.RevokedAt.UTC().Format(time.RFC3339)
}
return resp
}
// usersResponse is the admin list of accounts, password hashes excluded.
type usersResponse struct {
Users []userResponse `json:"users"`
}
func toWorkerKeyResponse(k *domain.WorkerKey) workerKeyResponse {
resp := workerKeyResponse{
ID: k.ID.String(),
@@ -14,16 +14,19 @@ import (
// Handlers holds the use cases each endpoint drives.
type Handlers struct {
register *usecase.Register
login *usecase.Login
setVerified *usecase.SetVerified
setRole *usecase.SetRole
createWorkerKey *usecase.CreateWorkerKey
listWorkerKeys *usecase.ListWorkerKeys
revokeWorkerKey *usecase.RevokeWorkerKey
exchangeWorkerKey *usecase.ExchangeWorkerKey
users usecase.UserRepository
log *slog.Logger
register *usecase.Register
login *usecase.Login
setVerified *usecase.SetVerified
setRole *usecase.SetRole
createWorkerKey *usecase.CreateWorkerKey
listWorkerKeys *usecase.ListWorkerKeys
listWorkerKeysAll *usecase.ListWorkerKeysAll
revokeWorkerKey *usecase.RevokeWorkerKey
revokeWorkerKeyAdmin *usecase.RevokeWorkerKeyAdmin
exchangeWorkerKey *usecase.ExchangeWorkerKey
listUsers *usecase.ListUsers
users usecase.UserRepository
log *slog.Logger
}
// handleHealth is an unauthenticated liveness probe for the container and load
@@ -173,15 +176,42 @@ func (h *Handlers) handleListWorkerKeys(w http.ResponseWriter, r *http.Request)
writeJSON(w, http.StatusOK, workerKeysResponse{WorkerKeys: out})
}
// handleListWorkerKeysAll returns every key in the service — revoked included,
// with the owning user id — for the coordinator admin console. Admin-only.
func (h *Handlers) handleListWorkerKeysAll(w http.ResponseWriter, r *http.Request) {
keys, err := h.listWorkerKeysAll.Execute(r.Context())
if err != nil {
writeError(w, r, h.log, err)
return
}
out := make([]adminWorkerKeyResponse, 0, len(keys))
for _, k := range keys {
out = append(out, toAdminWorkerKeyResponse(k))
}
writeJSON(w, http.StatusOK, struct {
WorkerKeys []adminWorkerKeyResponse `json:"worker_keys"`
}{WorkerKeys: out})
}
// handleListUsers returns every account for the coordinator admin console.
// Password hashes never leave the service: only the public projection is sent.
func (h *Handlers) handleListUsers(w http.ResponseWriter, r *http.Request) {
users, err := h.listUsers.Execute(r.Context())
if err != nil {
writeError(w, r, h.log, err)
return
}
out := make([]userResponse, 0, len(users))
for _, u := range users {
out = append(out, toUserResponse(u))
}
writeJSON(w, http.StatusOK, usersResponse{Users: out})
}
// handleRevokeWorkerKey retires one of the caller's keys. The repository scopes
// the delete to the owner, so a mismatched id is a clean 404, not another user's
// key.
func (h *Handlers) handleRevokeWorkerKey(w http.ResponseWriter, r *http.Request) {
userID, ok := userIDFrom(r.Context())
if !ok {
unauthorized(w, r)
return
}
keyID, err := uuid.Parse(r.PathValue("id"))
if err != nil {
writeJSON(w, http.StatusBadRequest, errorResponse{
@@ -190,6 +220,20 @@ func (h *Handlers) handleRevokeWorkerKey(w http.ResponseWriter, r *http.Request)
})
return
}
// An admin may revoke any key; a plain user only their own.
if role, ok := r.Context().Value(roleKey).(domain.Role); ok && role == domain.RoleAdmin {
if err := h.revokeWorkerKeyAdmin.Execute(r.Context(), keyID); err != nil {
writeError(w, r, h.log, err)
return
}
w.WriteHeader(http.StatusNoContent)
return
}
userID, ok := userIDFrom(r.Context())
if !ok {
unauthorized(w, r)
return
}
if err := h.revokeWorkerKey.Execute(r.Context(), userID, keyID); err != nil {
writeError(w, r, h.log, err)
return
@@ -14,31 +14,37 @@ import (
// UseCases bundles the application services the handlers drive.
type UseCases struct {
Register *usecase.Register
Login *usecase.Login
SetVerified *usecase.SetVerified
SetRole *usecase.SetRole
CreateWorkerKey *usecase.CreateWorkerKey
ListWorkerKeys *usecase.ListWorkerKeys
RevokeWorkerKey *usecase.RevokeWorkerKey
ExchangeWorkerKey *usecase.ExchangeWorkerKey
Users usecase.UserRepository
Register *usecase.Register
Login *usecase.Login
SetVerified *usecase.SetVerified
SetRole *usecase.SetRole
CreateWorkerKey *usecase.CreateWorkerKey
ListWorkerKeys *usecase.ListWorkerKeys
ListWorkerKeysAll *usecase.ListWorkerKeysAll
RevokeWorkerKey *usecase.RevokeWorkerKey
RevokeWorkerKeyAdmin *usecase.RevokeWorkerKeyAdmin
ExchangeWorkerKey *usecase.ExchangeWorkerKey
ListUsers *usecase.ListUsers
Users usecase.UserRepository
}
// NewServer wires the routes and the middleware stack and returns the handler.
// The issuer verifies tokens for the JWT-protected routes.
func NewServer(log *slog.Logger, uc UseCases, issuer auth.Issuer) http.Handler {
h := &Handlers{
register: uc.Register,
login: uc.Login,
setVerified: uc.SetVerified,
setRole: uc.SetRole,
createWorkerKey: uc.CreateWorkerKey,
listWorkerKeys: uc.ListWorkerKeys,
revokeWorkerKey: uc.RevokeWorkerKey,
exchangeWorkerKey: uc.ExchangeWorkerKey,
users: uc.Users,
log: log,
register: uc.Register,
login: uc.Login,
setVerified: uc.SetVerified,
setRole: uc.SetRole,
createWorkerKey: uc.CreateWorkerKey,
listWorkerKeys: uc.ListWorkerKeys,
listWorkerKeysAll: uc.ListWorkerKeysAll,
revokeWorkerKey: uc.RevokeWorkerKey,
revokeWorkerKeyAdmin: uc.RevokeWorkerKeyAdmin,
exchangeWorkerKey: uc.ExchangeWorkerKey,
listUsers: uc.ListUsers,
users: uc.Users,
log: log,
}
mux := http.NewServeMux()
@@ -68,6 +74,12 @@ func NewServer(log *slog.Logger, uc UseCases, issuer auth.Issuer) http.Handler {
mux.Handle("POST /users/{id}/demote",
chain(h.handleSetRole(domain.RoleUser), withJWT(issuer), withAdmin))
// Admin console: lists of every account and every worker key, and the key
// revoke path the admin console calls (the same DELETE endpoint already
// lets an admin revoke any key).
mux.Handle("GET /users", chain(http.HandlerFunc(h.handleListUsers), withJWT(issuer), withAdmin))
mux.Handle("GET /worker-keys/all", chain(http.HandlerFunc(h.handleListWorkerKeysAll), withJWT(issuer), withAdmin))
// Outermost first: every request gets an ID and an access-log line.
return chain(mux, withRequestID, withAccessLog(log))
}
@@ -25,17 +25,25 @@ const secret = "server-test-secret-32-bytes-long!!!!"
func newTestServer() http.Handler {
users := memstore.NewUserRepo()
keys := memstore.NewWorkerKeyRepo()
hasher := auth.NewHasher(4)
clk := memstore.Clock{T: time.Date(2026, 7, 26, 0, 0, 0, 0, time.UTC)}
// Real clock for the issuer so tokens are valid at verification time.
issuer := auth.NewIssuer(secret, time.Hour, nil)
uc := apihttp.UseCases{
Register: usecase.NewRegister(users, hasher, clk),
Login: usecase.NewLogin(users, hasher, issuer),
SetVerified: usecase.NewSetVerified(users),
SetRole: usecase.NewSetRole(users),
Users: users,
Register: usecase.NewRegister(users, hasher, clk),
Login: usecase.NewLogin(users, hasher, issuer),
SetVerified: usecase.NewSetVerified(users),
SetRole: usecase.NewSetRole(users),
CreateWorkerKey: usecase.NewCreateWorkerKey(keys, clk),
ListWorkerKeys: usecase.NewListWorkerKeys(keys),
ListWorkerKeysAll: usecase.NewListWorkerKeysAll(keys),
RevokeWorkerKey: usecase.NewRevokeWorkerKey(keys),
RevokeWorkerKeyAdmin: usecase.NewRevokeWorkerKeyAdmin(keys),
ExchangeWorkerKey: usecase.NewExchangeWorkerKey(keys, users, issuer, time.Hour),
ListUsers: usecase.NewListUsers(users),
Users: users,
}
log := slog.New(slog.NewTextHandler(io.Discard, nil))
return apihttp.NewServer(log, uc, issuer)
@@ -216,10 +224,16 @@ func TestMeInternalError(t *testing.T) {
}
// mintToken issues a token with the package secret for a synthetic caller of the
// given role — enough to drive the admin-gated endpoints.
// given role — enough to drive the admin-gated endpoints. userID defaults to a
// fresh random id; pass one to act as an existing account.
func mintToken(t *testing.T, role domain.Role) string {
t.Helper()
token, err := auth.NewIssuer(secret, time.Hour, nil).Issue(&domain.User{ID: uuid.New(), Role: role})
return mintTokenFor(t, role, uuid.New())
}
func mintTokenFor(t *testing.T, role domain.Role, userID uuid.UUID) string {
t.Helper()
token, err := auth.NewIssuer(secret, time.Hour, nil).Issue(&domain.User{ID: userID, Role: role})
if err != nil {
t.Fatal(err)
}
@@ -370,3 +384,98 @@ func TestUnverifyRevokes(t *testing.T) {
t.Error("verified should be false after unverify")
}
}
func TestAdminListsUsersAndKeys(t *testing.T) {
h := newTestServer()
userID, _ := uuid.Parse(registerUser(t, h, "listed@example.com"))
userToken := mintTokenFor(t, domain.RoleUser, userID)
// Mint a worker key as the plain user.
keyRec := do(t, h, http.MethodPost, "/worker-keys", userToken, map[string]string{"name": "lab-node"})
if keyRec.Code != http.StatusCreated {
t.Fatalf("create key: got %d, body %s", keyRec.Code, keyRec.Body)
}
var created struct {
ID string `json:"id"`
}
_ = json.Unmarshal(keyRec.Body.Bytes(), &created)
// Admin lists users: emails present, password hashes absent.
rec := do(t, h, http.MethodGet, "/users", mintToken(t, domain.RoleAdmin), nil)
if rec.Code != http.StatusOK {
t.Fatalf("list users: got %d, body %s", rec.Code, rec.Body)
}
var users struct {
Users []struct {
Email string `json:"email"`
Role string `json:"role"`
PasswordHash string `json:"password_hash"`
} `json:"users"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &users); err != nil {
t.Fatal(err)
}
found := false
for _, u := range users.Users {
if u.PasswordHash != "" {
t.Error("password hash leaked through the admin users list")
}
if u.Email == "listed@example.com" {
found = true
}
}
if !found {
t.Error("listed user missing from the admin list")
}
// Admin lists all keys: the owner is attached, no secret.
rec = do(t, h, http.MethodGet, "/worker-keys/all", mintToken(t, domain.RoleAdmin), nil)
if rec.Code != http.StatusOK {
t.Fatalf("list all keys: got %d", rec.Code)
}
var keys struct {
WorkerKeys []struct {
ID string `json:"id"`
UserID string `json:"user_id"`
Name string `json:"name"`
} `json:"worker_keys"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &keys); err != nil {
t.Fatal(err)
}
if len(keys.WorkerKeys) != 1 || keys.WorkerKeys[0].UserID != userID.String() {
t.Errorf("all keys = %+v, want the one key owned by %s", keys.WorkerKeys, userID)
}
// Plain users cannot see either list.
for _, path := range []string{"/users", "/worker-keys/all"} {
if rec := do(t, h, http.MethodGet, path, mintToken(t, domain.RoleUser), nil); rec.Code != http.StatusForbidden {
t.Errorf("%s as user: got %d, want 403", path, rec.Code)
}
}
// An admin revokes a key that belongs to another user; the plain owner of
// that key could not (it would be a 404, scoped to their own keys).
// The owner of the key revokes it themselves: 204.
if rec := do(t, h, http.MethodDelete, "/worker-keys/"+created.ID, userToken, nil); rec.Code != http.StatusNoContent {
t.Errorf("user revoke own key: got %d, want 204", rec.Code)
}
// Another plain user cannot revoke it: scoped to their own keys, so a
// mismatch reads as 404.
otherID, _ := uuid.Parse(registerUser(t, h, "other@example.com"))
if rec := do(t, h, http.MethodDelete, "/worker-keys/"+created.ID, mintTokenFor(t, domain.RoleUser, otherID), nil); rec.Code != http.StatusNotFound {
t.Errorf("other user revoke: got %d, want 404", rec.Code)
}
// An admin revokes a key that belongs to someone else: 204.
keyRec = do(t, h, http.MethodPost, "/worker-keys", userToken, map[string]string{"name": "lab-node-2"})
var second struct {
ID string `json:"id"`
}
_ = json.Unmarshal(keyRec.Body.Bytes(), &second)
if rec := do(t, h, http.MethodDelete, "/worker-keys/"+second.ID, mintToken(t, domain.RoleAdmin), nil); rec.Code != http.StatusNoContent {
t.Errorf("admin revoke other's key: got %d, want 204", rec.Code)
}
// Admin cannot revoke an unknown key.
if rec := do(t, h, http.MethodDelete, "/worker-keys/"+uuid.NewString(), mintToken(t, domain.RoleAdmin), nil); rec.Code != http.StatusNotFound {
t.Errorf("admin revoke unknown key: got %d, want 404", rec.Code)
}
}
@@ -35,6 +35,7 @@ func (s stubRepo) SetVerified(context.Context, uuid.UUID, bool) error {
func (s stubRepo) SetRole(context.Context, uuid.UUID, domain.Role) error {
return usecase.ErrUserNotFound
}
func (s stubRepo) ListUsers(context.Context) ([]*domain.User, error) { return nil, nil }
type stubHasher struct {
hashErr error
@@ -28,6 +28,10 @@ type UserRepository interface {
// SetRole changes a user's role, returning ErrUserNotFound if no such user
// exists.
SetRole(ctx context.Context, id uuid.UUID, role domain.Role) error
// ListUsers returns every account, oldest first. Admin-only: used by the
// coordinator admin console; the response must never carry password hashes
// (the caller projects the entity).
ListUsers(ctx context.Context) ([]*domain.User, error)
}
// WorkerKeyRepository persists and looks up the long-lived worker keys a user
@@ -38,12 +42,18 @@ type WorkerKeyRepository interface {
Insert(ctx context.Context, k *domain.WorkerKey) error
// ListByUser returns a user's live (non-revoked) keys, newest first.
ListByUser(ctx context.Context, userID uuid.UUID) ([]*domain.WorkerKey, error)
// ListAll returns every key (revoked included), newest first. Admin-only:
// backs the coordinator admin console's key table.
ListAll(ctx context.Context) ([]*domain.WorkerKey, error)
// GetActiveByHash returns the non-revoked key with the given hash, or
// ErrWorkerKeyNotFound.
GetActiveByHash(ctx context.Context, tokenHash string) (*domain.WorkerKey, error)
// Revoke retires a key the user owns, returning ErrWorkerKeyNotFound when no
// live key with that id belongs to the user.
Revoke(ctx context.Context, id, userID uuid.UUID) error
// RevokeAny retires a key by id regardless of its owner. Admin-only; the
// coordinator admin console uses it to cut a key immediately.
RevokeAny(ctx context.Context, id uuid.UUID) error
// TouchLastUsed records a successful exchange. Best-effort: a failure here
// must not fail the exchange itself.
TouchLastUsed(ctx context.Context, id uuid.UUID) error
@@ -0,0 +1,21 @@
package usecase
import (
"context"
"github.com/emil28092005/SciMesh/coordinator/internal/userservice/domain"
)
// ListUsers returns every account for the coordinator admin console. The
// handler must project the entities so password hashes never leave the service.
type ListUsers struct {
users UserRepository
}
func NewListUsers(users UserRepository) *ListUsers {
return &ListUsers{users: users}
}
func (uc *ListUsers) Execute(ctx context.Context) ([]*domain.User, error) {
return uc.users.ListUsers(ctx)
}
@@ -47,6 +47,34 @@ func (uc *ListWorkerKeys) Execute(ctx context.Context, userID uuid.UUID) ([]*dom
return uc.keys.ListByUser(ctx, userID)
}
// ListWorkerKeysAll returns every key in the service, revoked included, for
// the coordinator admin console. Admin-only.
type ListWorkerKeysAll struct {
keys WorkerKeyRepository
}
func NewListWorkerKeysAll(keys WorkerKeyRepository) *ListWorkerKeysAll {
return &ListWorkerKeysAll{keys: keys}
}
func (uc *ListWorkerKeysAll) Execute(ctx context.Context) ([]*domain.WorkerKey, error) {
return uc.keys.ListAll(ctx)
}
// RevokeWorkerKeyAdmin retires any key, regardless of owner. Admin-only; used
// by the coordinator admin console when a key must be cut immediately.
type RevokeWorkerKeyAdmin struct {
keys WorkerKeyRepository
}
func NewRevokeWorkerKeyAdmin(keys WorkerKeyRepository) *RevokeWorkerKeyAdmin {
return &RevokeWorkerKeyAdmin{keys: keys}
}
func (uc *RevokeWorkerKeyAdmin) Execute(ctx context.Context, id uuid.UUID) error {
return uc.keys.RevokeAny(ctx, id)
}
// RevokeWorkerKey retires one of the caller's keys.
type RevokeWorkerKey struct {
keys WorkerKeyRepository
@@ -64,6 +64,24 @@ func (r *fakeKeyRepo) TouchLastUsed(_ context.Context, id uuid.UUID) error {
return nil
}
func (r *fakeKeyRepo) ListAll(_ context.Context) ([]*domain.WorkerKey, error) {
out := make([]*domain.WorkerKey, 0, len(r.byID))
for _, k := range r.byID {
out = append(out, k)
}
return out, nil
}
func (r *fakeKeyRepo) RevokeAny(_ context.Context, id uuid.UUID) error {
k, ok := r.byID[id]
if !ok || k.Revoked() {
return usecase.ErrWorkerKeyNotFound
}
now := time.Now()
k.RevokedAt = &now
return nil
}
func newKeyFixtures(t *testing.T) (*usecase.CreateWorkerKey, *usecase.ExchangeWorkerKey, *usecase.RevokeWorkerKey, *usecase.ListWorkerKeys, *fakeKeyRepo, *domain.User) {
t.Helper()
users := memstore.NewUserRepo()
@@ -176,6 +176,12 @@ func (c *Catalog) Enabled() []*Workload {
return result
}
// Items returns every workload in the catalog, sorted by name (the same
// ordering as Enabled). The caller must not mutate the entries.
func (c *Catalog) Items() []*Workload {
return c.workloads
}
// ByName returns the workload with the given name, or nil.
func (c *Catalog) ByName(name string) *Workload {
for _, workload := range c.workloads {
@@ -597,6 +597,206 @@
"upload_ready": true,
"verifier": "exact-artifact@1",
"version": "1.0.0"
},
{
"capabilities": [
"similarity-search-parallel"
],
"description": "Exact top-k Tanimoto molecular similarity search over deterministic TSV shards with a bounded merge; each shard is fingerprinted and scored across a thread pool. Output is byte-identical to similarity-search.",
"determinism": "byte_exact",
"enabled": true,
"inputs": {
"input": {
"allow_nested_collections": false,
"canonicalizer": "scimesh-tsv-v1",
"encoding": "utf-8",
"max_bytes": 10737418240,
"max_dimensions": [],
"max_records": 100000000,
"media_type": "text/tab-separated-values",
"privacy_class": "project",
"ref": "molecule-table@1",
"retention_class": "durable",
"streaming": false,
"validator": "delimited-table@1",
"validator_configuration": {
"required_columns": [
"canonical_smiles",
"chembl_id"
]
}
}
},
"name": "similarity-search-parallel",
"outputs": {
"result": {
"allow_nested_collections": false,
"canonicalizer": "scimesh-search-result-v1",
"encoding": "utf-8",
"max_bytes": 1073741824,
"max_dimensions": [],
"max_records": 100000,
"media_type": "text/csv",
"privacy_class": "project",
"ref": "similarity-search-result@1",
"retention_class": "durable",
"streaming": false,
"validator": "delimited-table@1",
"validator_configuration": {
"columns": [
"rank",
"chembl_id",
"canonical_smiles",
"similarity"
]
}
}
},
"parameters_schema": {
"additionalProperties": false,
"oneOf": [
{
"not": {
"required": [
"query_smiles"
]
},
"required": [
"query_id"
]
},
{
"not": {
"required": [
"query_id"
]
},
"required": [
"query_smiles"
]
}
],
"properties": {
"max_rows": {
"minimum": 1,
"type": "integer"
},
"progress_every": {
"minimum": 0,
"type": "integer"
},
"query_id": {
"maxLength": 200,
"minLength": 1,
"type": "string"
},
"query_smiles": {
"maxLength": 200,
"minLength": 1,
"type": "string"
},
"threads": {
"description": "Threads used to fingerprint and score one shard (default: CPU count).",
"minimum": 1,
"type": "integer"
},
"threshold": {
"maximum": 1,
"minimum": 0,
"type": "number"
},
"threshold_direction": {
"enum": [
"greater",
"less"
]
},
"top_k": {
"minimum": 1,
"type": "integer"
}
},
"type": "object"
},
"reduction": "top-k",
"trust_modes": [
"trusted",
"untrusted_quorum"
],
"ui_elements": [
{
"default": null,
"field": "query_id",
"group": "",
"help": "ChEMBL id of the query molecule. Provide exactly one of id or SMILES.",
"label": "Query molecule id",
"options": [],
"order": 1,
"placeholder": "",
"widget": "text"
},
{
"default": null,
"field": "query_smiles",
"group": "",
"help": "SMILES of the query molecule. Provide exactly one of id or SMILES.",
"label": "Query molecule SMILES",
"options": [],
"order": 2,
"placeholder": "",
"widget": "text"
},
{
"default": 20,
"field": "top_k",
"group": "",
"help": "Number of most similar molecules to keep per shard (global merge keeps the best of these).",
"label": "Top k",
"options": [],
"order": 3,
"placeholder": "",
"widget": "number"
},
{
"default": "greater",
"field": "threshold_direction",
"group": "",
"help": "Keep molecules with similarity greater or less than the threshold.",
"label": "Direction",
"options": [
"greater",
"less"
],
"order": 4,
"placeholder": "",
"widget": "select"
},
{
"default": null,
"field": "threshold",
"group": "",
"help": "Optional similarity bound: results are filtered to this direction.",
"label": "Similarity threshold",
"options": [],
"order": 5,
"placeholder": "e.g. 0.8",
"widget": "number"
},
{
"default": null,
"field": "threads",
"group": "",
"help": "Threads used to fingerprint and score one shard (default: CPU count).",
"label": "Threads per shard",
"options": [],
"order": 6,
"placeholder": "auto",
"widget": "number"
}
],
"upload_ready": true,
"verifier": "exact-artifact@1",
"version": "1.0.0"
}
]
}
+181
View File
@@ -0,0 +1,181 @@
# План реализации: Coordinator Admin UI и Worker Setup UI
> Статус: **реализовано (M1–M3), E2E проверено**. Визуал утверждён — мокапы:
> [`ui-mockups/coordinator-admin.html`](ui-mockups/coordinator-admin.html),
> [`ui-mockups/worker-setup.html`](ui-mockups/worker-setup.html).
> Два UI живут в **разных бинарниках**: админка кластера — в `coordinator`
> (`/ui/admin`), визард установки воркера — в `worker-agent`
> (`worker-agent setup`, локально на 127.0.0.1).
## 1. Цель
1. **Coordinator Admin UI** — консоль админа распределённого кластера:
System, Jobs, Workers, Users & keys, Workloads, Metrics, Settings.
Реальные данные, никакой симуляции; только роль `admin`.
2. **Worker Setup UI** — локальный визард в бинарнике `worker-agent` для
машины, на которой стоит только воркер: подключение (URL + токен/ключ),
параметры машины, preflight-проверка, запуск/остановка, статус и лог.
## 2. Что уже есть в коде (фундамент)
- `requireAdmin` middleware и `/ui/admin` (`internal/transport/http/ui_admin.go`)
— сейчас минимальная панель: прокси `promote/demote/verify/unverify` в
userservice.
- Userservice (`internal/userservice/`): `POST /users/{id}/promote|demote|
verify|unverify` (admin), `GET/POST/DELETE /worker-keys` (per-user),
`POST /worker-tokens/exchange`. **Нет**: list users, list all keys.
- `domain.Worker.TrustLevel` (`trusted`/`untrusted`) + quorum для untrusted —
колонка в БД есть, нужен только метод смены и отображение.
- `domain.Job.OwnerID *uuid.UUID` — владелец джобы из JWT `sub` (может быть nil).
- Каталог ворклоадов `internal/workloads` (embedded `workloads.json`) —
enable/disable кладём поверх через таблицу настроек.
- UIReadRepo (`ui_read_repo.go` в обоих движках) — bounded read model для UI.
- Агент целиком конфигурируется env (таблица в `mkdocs/sdk/worker-integration.md`);
`--config` маппится на ту же поверхность `agent.Config`.
## 3. Coordinator Admin UI
### 3.1 Страницы ↔ API ↔ данные
Все API — JSON, под `withUISession` + `requireAdmin`, префикс
`/ui/admin/api/`. Страница `admin.html` — оболочка из мокапа (сайдбар,
7 разделов), данные подтягивает JS через fetch (паттерн dashboard).
| Раздел | API | Данные |
| --- | --- | --- |
| System | `GET /system` | version, uptime, storage stats, health (db/userservice/reducer), node info |
| Jobs | `GET /jobs?status=&page=` | пагинированный список + счётчики по статусам; owner email резолвится через userservice |
| Workers | `GET /workers`, `POST /workers/{id}/trust` | список + trust, смена trust (trusted/untrusted) |
| Users & keys | `GET /users`, `POST /users/{id}/role`, `GET /worker-keys`, `POST /worker-keys/{id}/revoke` | прокси/агрегация userservice |
| Workloads | `GET /workloads`, `POST /workloads/{name}/enabled` | каталог + persisted enabled-флаг |
| Metrics | `GET /metrics` | jobs/day (7d), jobs by workload, shards, failure rate, avg shard time |
| Settings | `GET /settings`, `POST /token/reveal` | read-only конфиг + reveal токена (audit-лог) |
### 3.2 Userservice — новые эндпоинты
- `GET /users` (admin) — `id, email, role, verified, created_at`.
- `GET /worker-keys/all` (admin) — все ключи + owner email.
- Репозитории: `ListUsers(ctx)`, `ListWorkerKeysAll(ctx)` в `memstore` и
`storage/sqlite` (+ тесты). Координатор вызывает их через
`callUserserviceAuthed` и отдаёт в свой bounded API — браузер юзерсервис
не касается.
### 3.3 Storage координатора — новые методы
- `WorkerRepository.SetTrust(ctx, id, trust)` — оба движка + тесты.
- `UIReadRepo.ListJobsPaginated(ctx, status string, limit, offset int)`
`([]UIJob, total int, counts map[string]int)` — фильтр по статусу,
счётчики для табов.
- `UIReadRepo.JobMetrics(ctx, since time.Time)` → jobs/day, by workload,
shards completed/failed, avg shard duration (из `tasks`).
- `UIReadRepo.StorageStats(ctx)` → суммы байт по kind артефактов
(datasets/artifacts) + размер файла БД (sqlite: `page_count*page_size`;
postgres: `pg_database_size(current_database())`).
- Миграция `0002_workload_settings` (оба движка):
`workload_settings(workload TEXT PRIMARY KEY, enabled BOOL NOT NULL,
updated_at TIMESTAMP NOT NULL)`; отсутствие строки = enabled (default).
Repo: `WorkloadSettingsRepo{ List, Set }`.
- **Enforcement**: `SubmitDataset` отклоняет disabled-ворклоад;
`/ui/api/workloads` и форма new-job помечают disabled (скрываем из выбора).
### 3.4 Usecase — `internal/usecase/admin.go`
`AdminSystem`, `ListJobsAdmin(status,page)` (+owner emails), `ListWorkersAdmin`,
`SetWorkerTrust`, `ListUsersAdmin`, `SetUserRole` (через userservice promote/demote),
`ListWorkerKeysAdmin`, `RevokeWorkerKeyAdmin`, `ListWorkloadsAdmin`,
`SetWorkloadEnabled`, `AdminMetrics`, `RevealWorkerToken` (читает
`worker.token`/env, пишет audit-лог).
### 3.5 Шаблон
`templates/admin.html` заменяется на дизайн мокапа: сайдбар (Operate /
Access / Platform), topbar с env-бейджем (serve/postgres, addr), 7 разделов,
рендер через JS. Имя ворклоада — реальное `descriptor-batch`.
## 4. Worker Setup UI (бинарник `worker-agent`)
### 4.1 Новые файлы
```
coordinator/internal/agent/
configfile.go # ConfigFile (json), Load/Save (0600), путь по умолчанию
check.go # CheckCoordinator(url, auth) — health + версии + python/scimesh
setupui/
server.go # локальный HTTP 127.0.0.1:12700, API + запуск/остановка
template.html # визард + статус (по мокапу), go:embed
coordinator/cmd/worker-agent/main.go # + setup / --config / --check
```
### 4.2 CLI
- `worker-agent setup [--port 12700] [--no-open]` — визард; печатает URL,
открывает браузер.
- `worker-agent --config <path>` — демон из JSON-конфига; env имеет приоритет.
- `worker-agent --check [--coordinator-url URL]` — пинг `/health`, auth
(token → claim-endpoint 401/200 probe или exchange для ключа), python3 +
`import scimesh`; exit 0/1. Используется визардом на шаге 3.
### 4.3 config.json
`~/.scimesh-worker/config.json` (переопределяется `WORKER_CONFIG`), права 0600:
```json
{
"coordinator_url": "http://192.168.1.10:8080",
"token": "…", // или
"worker_key": "…", "userservice_url": "http://…:8081",
"work_dir": "…", "worker_name": "emil-laptop",
"cpu_count": 8, "memory_mb": 16384,
"task_runner": ["python", "-m", "scimesh.worker.task"]
}
```
### 4.4 API визарда (только 127.0.0.1, без auth)
| Метод | Путь | Назначение |
| --- | --- | --- |
| GET | `/api/status` | конфиг (секрет маскирован), running (pid alive), статистика из лога, runtime |
| POST | `/api/config` | валидация + сохранение `config.json` (0600) |
| POST | `/api/test` | preflight: coordinator reachable, auth ok, python, scimesh |
| POST | `/api/start` | spawn `worker-agent --config <path>` (лог → `worker.log`, pid-файл) |
| POST | `/api/stop` | SIGTERM по pid-файлу |
| GET | `/api/logs?tail=` | хвост `worker.log` |
Статистика — парсинг лога (registered/claimed/completed/failed/heartbeat).
Spawner — интерфейс, в тестах подменяется.
## 5. Милестоуны
- **M1. Admin-фундамент** — ✅ done: оболочка `admin.html` по мокапу +
`GET /system` (storage stats, health, node) + `GET /jobs`
(фильтр+пагинация+счётчики) + `GET /metrics`; тесты на sqlite, 403 для
не-админа.
- **M2. Access & Platform** — ✅ done: userservice `ListUsers`/
`ListWorkerKeysAll` + admin revoke (mem+sqlite+http), `SetTrust` (оба
движка), users/keys/trust разделы, `workload_settings` (миграции 0002/
0014) + enforcement в `SubmitDataset` и форме, settings + audited token
reveal. Все секции админки на живых данных.
- **M3. Worker Setup** — ✅ done: `configfile.go` (0600), `--config` (env
wins), `--check`, `setupui` (сервер на 127.0.0.1, 6 API, шаблон по мокапу),
тесты (roundtrip, права, check, API с fake supervisor).
- **M4. E2E + docs** — ✅ done: браузерный E2E (admin видит реальные
данные; визард запустил реального воркера → `wizard-machine online` в
админке), обновлены `mkdocs/index.md` и `mkdocs/standalone.md`
(`worker-agent setup`), README, статусы CTX-19/CTX-20 в `PLAN.md`.
## 6. Тестирование
- **Go unit**: usecase admin (fakes + sqlite), repos обоих движков
(SetTrust, pagination, metrics, settings, storage stats), userservice
List/ListAll (mem+sqlite), HTTP permission (user → 403 на все `/ui/admin/api/*`),
enforcement disabled-ворклоада, configfile/check/setupui агента.
- **Go integration (postgres)**: миграция 0002 и parity новых методов —
через существующий docker-хелпер, skip без docker.
- **E2E браузер**: M4.
## 7. Открытые решения (зафиксировано)
- Локальный визард без аутентификации — слушает только 127.0.0.1.
- Settings-раздел v1 — read-only + reveal токена; редактирование конфигурации
и prune/reset (danger zone) — v2, в UI помечены как таковые.
- `owner` джобы: email из userservice по `OwnerID`; при nil — «cluster token».
- Статистика воркера в визарде — из лога, без новых эндпоинтов координатора.
+388
View File
@@ -0,0 +1,388 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SciMesh · Coordinator Admin</title>
<style>
:root{
--bg:#0b0e13;--panel:#11151d;--panel-2:#161c27;--border:#1f2634;--border-soft:#181f2b;
--text:#e9ecf3;--text-2:#98a2b5;--text-3:#5d6879;
--accent:#5b8cff;--accent-soft:#5b8cff1f;--accent-strong:#3f6fe0;
--green:#3fce8a;--green-soft:#3fce8a1a;--amber:#e5b64f;--amber-soft:#e5b64f1a;--red:#f2647c;--red-soft:#f2647c1a;
--mono:ui-monospace,"SF Mono",SFMono-Regular,Menlo,monospace;
color-scheme:dark;
}
*{box-sizing:border-box;margin:0;padding:0}
body{background:var(--bg);color:var(--text);font:14px/1.55 Inter,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
button{font:inherit;color:inherit;background:none;border:0;cursor:pointer}
input,select{font:inherit;color:var(--text);background:var(--panel-2);border:1px solid var(--border);border-radius:8px;padding:8px 11px;outline:none}
input:focus,select:focus{border-color:var(--accent);box-shadow:0 0 0 3px var(--accent-soft)}
code{font-family:var(--mono);font-size:.86em}
.layout{display:flex;min-height:100vh}
/* ── sidebar ─────────────────────────────── */
.sidebar{position:sticky;top:0;height:100vh;width:232px;flex:none;display:flex;flex-direction:column;background:var(--panel);border-right:1px solid var(--border-soft)}
.brand{display:flex;align-items:center;gap:11px;padding:20px 20px 18px;border-bottom:1px solid var(--border-soft)}
.brand-mark{display:grid;place-items:center;width:32px;height:32px;border-radius:9px;background:linear-gradient(135deg,#5b8cff,#7c5cff);box-shadow:0 4px 14px #5b8cff40}
.brand-mark svg{width:17px;height:17px;stroke:#fff}
.brand-name{font-weight:700;font-size:14.5px;letter-spacing:-.01em}
.brand-sub{font-size:11px;color:var(--text-3);letter-spacing:.02em}
.nav{flex:1;overflow-y:auto;padding:14px 12px}
.nav-label{margin:16px 10px 6px;font-size:10.5px;font-weight:700;letter-spacing:.12em;text-transform:uppercase;color:var(--text-3)}
.nav-label:first-child{margin-top:0}
.nav-item{display:flex;align-items:center;gap:10px;width:100%;padding:8px 10px;border-radius:8px;color:var(--text-2);font-weight:500;text-align:left;transition:background .12s,color .12s}
.nav-item svg{width:16px;height:16px;stroke:currentColor;flex:none}
.nav-item:hover{background:var(--panel-2);color:var(--text)}
.nav-item.active{background:var(--accent-soft);color:var(--accent);font-weight:600}
.nav-item .count{margin-left:auto;font-size:11px;font-weight:600;color:var(--text-3);background:var(--panel-2);border-radius:99px;padding:1px 7px}
.nav-item.active .count{color:var(--accent);background:#5b8cff26}
.side-foot{padding:14px;border-top:1px solid var(--border-soft)}
.user-chip{display:flex;align-items:center;gap:10px;padding:9px 10px;border-radius:9px;background:var(--panel-2)}
.avatar{display:grid;place-items:center;width:28px;height:28px;border-radius:8px;background:linear-gradient(135deg,#3fce8a,#2ea56c);color:#08130d;font-weight:800;font-size:12px;flex:none}
.user-chip b{display:block;font-size:12.5px;line-height:1.25}
.user-chip span{display:block;font-size:11px;color:var(--text-3)}
.back-link{display:block;margin-top:9px;padding:7px 10px;color:var(--text-3);font-size:12.5px;text-decoration:none;border-radius:8px}
.back-link:hover{color:var(--text);background:var(--panel-2)}
/* ── main ────────────────────────────────── */
.main{flex:1;min-width:0;display:flex;flex-direction:column}
.topbar{position:sticky;top:0;z-index:5;display:flex;align-items:center;justify-content:space-between;gap:16px;padding:16px 32px;background:#0b0e13e6;backdrop-filter:blur(10px);border-bottom:1px solid var(--border-soft)}
.topbar h1{font-size:17px;font-weight:700;letter-spacing:-.015em}
.topbar p{font-size:12.5px;color:var(--text-3);margin-top:1px}
.env-badge{display:flex;align-items:center;gap:8px;font-size:12px;color:var(--text-2);border:1px solid var(--border);border-radius:99px;padding:5px 12px;background:var(--panel)}
.env-badge i{width:7px;height:7px;border-radius:50%;background:var(--green);box-shadow:0 0 8px var(--green)}
.content{flex:1;padding:26px 32px 60px;max-width:1120px;width:100%;margin:0 auto}
.page{display:none}.page.active{display:block;animation:fade .18s ease}
@keyframes fade{from{opacity:0;transform:translateY(4px)}to{opacity:1}}
/* ── shared components ───────────────────── */
.card{background:var(--panel);border:1px solid var(--border-soft);border-radius:13px}
.card-pad{padding:20px}
.card-head{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:15px 20px;border-bottom:1px solid var(--border-soft)}
.card-head h3{font-size:13.5px;font-weight:650}
.card-head span{font-size:12px;color:var(--text-3)}
.grid-kpi{display:grid;grid-template-columns:repeat(4,1fr);gap:14px;margin-bottom:14px}
.kpi{background:var(--panel);border:1px solid var(--border-soft);border-radius:13px;padding:16px 18px}
.kpi .k-label{display:flex;align-items:center;gap:7px;font-size:11.5px;font-weight:600;letter-spacing:.06em;text-transform:uppercase;color:var(--text-3)}
.kpi .k-label svg{width:14px;height:14px;stroke:currentColor}
.kpi .k-value{margin-top:8px;font-size:26px;font-weight:700;letter-spacing:-.03em;line-height:1}
.kpi .k-sub{margin-top:6px;font-size:12px;color:var(--text-2)}
.kpi .k-sub.up{color:var(--green)}
.pill{display:inline-flex;align-items:center;gap:6px;border-radius:99px;padding:3px 10px;font-size:11.5px;font-weight:650;white-space:nowrap}
.pill i{width:6px;height:6px;border-radius:50%;background:currentColor}
.pill-success{background:var(--green-soft);color:var(--green)}
.pill-active{background:var(--accent-soft);color:var(--accent)}
.pill-waiting{background:#ffffff12;color:var(--text-2)}
.pill-danger{background:var(--red-soft);color:var(--red)}
.pill-amber{background:var(--amber-soft);color:var(--amber)}
.btn{display:inline-flex;align-items:center;gap:7px;border-radius:8px;padding:8px 14px;font-weight:600;font-size:13px;border:1px solid transparent;transition:filter .12s,background .12s}
.btn svg{width:14px;height:14px;stroke:currentColor}
.btn-primary{background:var(--accent);color:#0a1222}
.btn-primary:hover{background:var(--accent-strong);color:#fff}
.btn-ghost{background:var(--panel-2);border-color:var(--border);color:var(--text-2)}
.btn-ghost:hover{color:var(--text);border-color:#2a3446}
.btn-danger{background:var(--red-soft);color:var(--red)}
.btn-danger:hover{background:#f2647c2e}
.btn-sm{padding:5px 10px;font-size:12px;border-radius:7px}
table{width:100%;border-collapse:collapse}
th{padding:10px 20px;text-align:left;font-size:11px;font-weight:650;letter-spacing:.07em;text-transform:uppercase;color:var(--text-3);border-bottom:1px solid var(--border-soft)}
td{padding:12px 20px;border-bottom:1px solid var(--border-soft);vertical-align:middle}
tr:last-child td{border-bottom:0}
tbody tr{transition:background .1s}
tbody tr:hover{background:var(--panel-2)}
.t-main{font-weight:600;font-size:13.5px}
.t-sub{font-size:11.5px;color:var(--text-3);font-family:var(--mono)}
.cap{display:inline-block;margin:2px 4px 2px 0;padding:2px 7px;border:1px solid var(--border);border-radius:6px;font:11px var(--mono);color:var(--text-2)}
.bar{height:5px;width:130px;border-radius:99px;background:#ffffff10;overflow:hidden}
.bar span{display:block;height:100%;border-radius:inherit;background:linear-gradient(90deg,#5b8cff,#3fce8a)}
.bar-label{font-size:11.5px;color:var(--text-2);font-family:var(--mono);margin-top:5px}
.tabs{display:flex;gap:4px;padding:4px;background:var(--panel);border:1px solid var(--border-soft);border-radius:10px;width:max-content;margin-bottom:14px}
.tab{padding:6px 13px;border-radius:7px;font-size:12.5px;font-weight:600;color:var(--text-2)}
.tab:hover{color:var(--text)}
.tab.active{background:var(--panel-2);color:var(--text);box-shadow:inset 0 0 0 1px var(--border)}
.tab .n{color:var(--text-3);font-weight:500;margin-left:5px}
.tab.active .n{color:var(--accent)}
.section-title{margin:26px 0 12px;font-size:13px;font-weight:700;letter-spacing:-.01em;color:var(--text)}
.section-title:first-child{margin-top:0}
.section-note{font-size:12px;color:var(--text-3);margin:-8px 0 12px}
.kv{display:grid;grid-template-columns:210px 1fr;row-gap:0}
.kv dt{padding:11px 20px;font-size:12.5px;color:var(--text-3);border-bottom:1px solid var(--border-soft)}
.kv dd{padding:11px 20px;font-size:13px;border-bottom:1px solid var(--border-soft)}
.kv dt:last-of-type,.kv dd:last-of-type{border-bottom:0}
.stack{display:grid;gap:14px}
.split{display:grid;grid-template-columns:1fr 1fr;gap:14px}
.toggle{position:relative;width:36px;height:20px;border-radius:99px;background:#ffffff17;transition:background .15s;flex:none}
.toggle:after{content:"";position:absolute;top:2px;left:2px;width:16px;height:16px;border-radius:50%;background:#8b95a5;transition:transform .15s,background .15s}
.toggle.on{background:var(--accent)}
.toggle.on:after{transform:translateX(16px);background:#fff}
.storage-bar{display:flex;height:10px;border-radius:99px;overflow:hidden;margin:14px 20px 6px}
.storage-bar div{height:100%}
.legend{display:flex;gap:20px;padding:10px 20px 18px;flex-wrap:wrap}
.legend span{display:flex;align-items:center;gap:7px;font-size:12px;color:var(--text-2)}
.legend i{width:9px;height:9px;border-radius:3px}
.footer-row{display:flex;align-items:center;justify-content:space-between;padding:11px 20px;font-size:12px;color:var(--text-3)}
.pager{display:flex;gap:4px}
.pager button{width:26px;height:26px;border-radius:7px;font-size:12px;color:var(--text-2)}
.pager button.cur{background:var(--accent-soft);color:var(--accent);font-weight:700}
.secret{display:flex;align-items:center;gap:10px}
.secret code{background:var(--panel-2);border:1px solid var(--border);border-radius:7px;padding:6px 10px;letter-spacing:.08em}
.chart{width:100%;height:auto;display:block}
.chart-bar{fill:#2c3a52;rx:4}
.chart-bar.hot{fill:var(--accent)}
.chart-grid{stroke:#ffffff08}
.chart-label{font:10px var(--mono);fill:var(--text-3)}
.warn-strip{display:flex;gap:10px;align-items:flex-start;background:var(--amber-soft);border:1px solid #e5b64f33;border-radius:10px;padding:12px 14px;font-size:12.5px;color:#eecf8d}
.warn-strip svg{width:15px;height:15px;stroke:var(--amber);flex:none;margin-top:1px}
@media(max-width:960px){.sidebar{display:none}.grid-kpi,.split{grid-template-columns:1fr 1fr}}
</style>
</head>
<body>
<div class="layout">
<aside class="sidebar">
<div class="brand">
<div class="brand-mark"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round"><circle cx="6" cy="6" r="2.4"/><circle cx="18" cy="7" r="2.4"/><circle cx="12" cy="17" r="2.4"/><path d="M8 7.5l7.6-1M7 8.3l3.7 6.5M16.6 9.2l-3.2 5.6"/></svg></div>
<div><div class="brand-name">SciMesh</div><div class="brand-sub">Coordinator Admin</div></div>
</div>
<nav class="nav" id="nav">
<div class="nav-label">Operate</div>
<button class="nav-item active" data-page="system"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><rect x="3" y="3" width="8" height="8" rx="2"/><rect x="13" y="3" width="8" height="5" rx="2"/><rect x="13" y="10" width="8" height="11" rx="2"/><rect x="3" y="13" width="8" height="8" rx="2"/></svg>System</button>
<button class="nav-item" data-page="jobs"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><path d="M4 6h16M4 12h16M4 18h10"/></svg>Jobs<span class="count">26</span></button>
<button class="nav-item" data-page="workers"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8M12 16v4"/></svg>Workers<span class="count">3</span></button>
<div class="nav-label">Access</div>
<button class="nav-item" data-page="users"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><circle cx="9" cy="8" r="3.2"/><path d="M3.5 19c.7-3 2.9-4.5 5.5-4.5s4.8 1.5 5.5 4.5"/><circle cx="17" cy="9" r="2.4"/><path d="M15.5 14.6c2.6.2 4.3 1.7 5 4.4"/></svg>Users &amp; keys</button>
<div class="nav-label">Platform</div>
<button class="nav-item" data-page="workloads"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><path d="M12 3l8 4.5v9L12 21l-8-4.5v-9L12 3z"/><path d="M12 12l8-4.5M12 12v9M12 12L4 7.5"/></svg>Workloads</button>
<button class="nav-item" data-page="metrics"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><path d="M4 19V5M4 19h16"/><path d="M8 15v-4M12 15V7M16 15v-6M20 15V9"/></svg>Metrics</button>
<button class="nav-item" data-page="settings"><svg viewBox="0 0 24 24" fill="none" stroke-width="1.8" stroke-linecap="round"><circle cx="12" cy="12" r="3"/><path d="M19 12a7 7 0 0 0-.1-1.2l2-1.6-2-3.4-2.4 1a7 7 0 0 0-2-1.2L14 3h-4l-.5 2.6a7 7 0 0 0-2 1.2l-2.4-1-2 3.4 2 1.6A7 7 0 0 0 5 12c0 .4 0 .8.1 1.2l-2 1.6 2 3.4 2.4-1a7 7 0 0 0 2 1.2L10 21h4l.5-2.6a7 7 0 0 0 2-1.2l2.4 1 2-3.4-2-1.6c.1-.4.1-.8.1-1.2z"/></svg>Settings</button>
</nav>
<div class="side-foot">
<div class="user-chip"><div class="avatar">A</div><div><b>admin@scimesh.local</b><span>admin · serve bootstrap</span></div></div>
<a class="back-link" href="#">← Back to control room</a>
</div>
</aside>
<div class="main">
<header class="topbar">
<div><h1 id="page-title">System</h1><p id="page-sub">Cluster state and node information</p></div>
<div class="env-badge"><i></i>serve mode · sqlite · 127.0.0.1:8080</div>
</header>
<div class="content">
<!-- ═══ SYSTEM ═══ -->
<section class="page active" id="page-system">
<div class="grid-kpi">
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M12 3l8 4.5v9L12 21l-8-4.5v-9L12 3z"/></svg>Version</div><div class="k-value">1.1.0</div><div class="k-sub">alpha.1 · linux/amd64</div></div>
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="12" cy="12" r="9"/><path d="M12 7v5l3 3"/></svg>Uptime</div><div class="k-value">3d 14h</div><div class="k-sub">since Jul 30, 09:12</div></div>
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M4 6h16M4 12h16M4 18h10"/></svg>Active jobs</div><div class="k-value">2</div><div class="k-sub up">1 running · 1 waiting</div></div>
<div class="kpi"><div class="k-label"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8"/></svg>Workers online</div><div class="k-value">3<span style="font-size:15px;color:var(--text-3);font-weight:500"> / 5</span></div><div class="k-sub">2 busy · 1 idle</div></div>
</div>
<div class="split">
<div class="card">
<div class="card-head"><h3>Storage usage</h3><span>2.1 GB total</span></div>
<div class="storage-bar"><div style="width:38%;background:#5b8cff"></div><div style="width:57%;background:#7c5cff"></div><div style="width:5%;background:#3fce8a"></div></div>
<div class="legend">
<span><i style="background:#5b8cff"></i>Datasets · 812 MB</span>
<span><i style="background:#7c5cff"></i>Artifacts · 1.2 GB</span>
<span><i style="background:#3fce8a"></i>Database · 34 MB</span>
</div>
</div>
<div class="card">
<div class="card-head"><h3>Health</h3><span>all checks pass</span></div>
<dl class="kv">
<dt>Database</dt><dd><span class="pill pill-success"><i></i>Connected</span> <span style="color:var(--text-3);font-size:12px">sqlite · wal</span></dd>
<dt>Userservice</dt><dd><span class="pill pill-success"><i></i>Embedded</span> <span style="color:var(--text-3);font-size:12px">127.0.0.1:41273</span></dd>
<dt>Reducer</dt><dd><span class="pill pill-waiting"><i></i>Idle</span> <span style="color:var(--text-3);font-size:12px">last merge 12 min ago</span></dd>
</dl>
</div>
</div>
<div class="section-title">Node information</div>
<div class="card">
<dl class="kv">
<dt>Binary</dt><dd><code>/usr/local/bin/coordinator</code></dd>
<dt>Listen address</dt><dd><code>127.0.0.1:8080</code></dd>
<dt>Data directory</dt><dd><code>~/.scimesh</code> <span style="color:var(--text-3)">(jwt.secret · worker.token · venv)</span></dd>
<dt>Database engine</dt><dd>sqlite · <code>~/.scimesh/scimesh.db</code></dd>
<dt>Public URL</dt><dd><code>http://192.168.1.10:8080</code></dd>
</dl>
</div>
</section>
<!-- ═══ JOBS ═══ -->
<section class="page" id="page-jobs">
<div class="tabs">
<button class="tab active">All<span class="n">26</span></button>
<button class="tab">Running<span class="n">1</span></button>
<button class="tab">Waiting<span class="n">1</span></button>
<button class="tab">Completed<span class="n">21</span></button>
<button class="tab">Failed<span class="n">2</span></button>
<button class="tab">Cancelled<span class="n">1</span></button>
</div>
<div class="card">
<table>
<thead><tr><th>Job</th><th>Workload</th><th>Owner</th><th>Status</th><th>Progress</th><th>Submitted</th><th></th></tr></thead>
<tbody>
<tr><td><div class="t-main">kinase-inhibitors-topk</div><div class="t-sub">7f3a…c91e</div></td><td><code style="color:var(--text-2)">similarity-search</code></td><td style="color:var(--text-2)">alice@lab.org</td><td><span class="pill pill-active"><i></i>Running</span></td><td><div class="bar"><span style="width:62%"></span></div><div class="bar-label">124 / 200 shards</div></td><td style="color:var(--text-2);font-size:12.5px">12 min ago</td><td><button class="btn btn-ghost btn-sm">Open</button></td></tr>
<tr><td><div class="t-main">chembl-2d-descriptors</div><div class="t-sub">a102…77bd</div></td><td><code style="color:var(--text-2)">descriptors</code></td><td style="color:var(--text-2)">bob@lab.org</td><td><span class="pill pill-waiting"><i></i>Waiting</span></td><td><div class="bar"><span style="width:0%"></span></div><div class="bar-label">0 / 48 shards</div></td><td style="color:var(--text-2);font-size:12.5px">4 min ago</td><td><button class="btn btn-ghost btn-sm">Open</button></td></tr>
<tr><td><div class="t-main">fragment-library-graph</div><div class="t-sub">55e0…0a3f</div></td><td><code style="color:var(--text-2)">similarity-graph</code></td><td style="color:var(--text-2)">alice@lab.org</td><td><span class="pill pill-success"><i></i>Completed</span></td><td><div class="bar"><span style="width:100%"></span></div><div class="bar-label">96 / 96 shards · 3m 41s</div></td><td style="color:var(--text-2);font-size:12.5px">1 h ago</td><td><button class="btn btn-ghost btn-sm">Open</button></td></tr>
<tr><td><div class="t-main">leadopt-molwt-450</div><div class="t-sub">9b41…e2c8</div></td><td><code style="color:var(--text-2)">molwt-filter</code></td><td style="color:var(--text-2)">admin@scimesh.local</td><td><span class="pill pill-success"><i></i>Completed</span></td><td><div class="bar"><span style="width:100%"></span></div><div class="bar-label">12 / 12 shards · 24s</div></td><td style="color:var(--text-2);font-size:12.5px">3 h ago</td><td><button class="btn btn-ghost btn-sm">Open</button></td></tr>
<tr><td><div class="t-main">hts-plate-14-search</div><div class="t-sub">c7d9…4b60</div></td><td><code style="color:var(--text-2)">similarity-search</code></td><td style="color:var(--text-2)">bob@lab.org</td><td><span class="pill pill-danger"><i></i>Failed</span></td><td><div class="bar"><span style="width:34%;background:linear-gradient(90deg,#5b8cff,#f2647c)"></span></div><div class="bar-label">18 / 53 shards · 2 failed</div></td><td style="color:var(--text-2);font-size:12.5px">yesterday</td><td><button class="btn btn-ghost btn-sm">Open</button></td></tr>
</tbody>
</table>
<div class="footer-row"><span>15 of 26 jobs</span><div class="pager"><button></button><button class="cur">1</button><button>2</button><button>3</button><button></button></div></div>
</div>
</section>
<!-- ═══ WORKERS ═══ -->
<section class="page" id="page-workers">
<div class="section-note">Workers register themselves. Trust controls whether a machine may claim tasks from this cluster.</div>
<div class="card">
<table>
<thead><tr><th>Worker</th><th>Status</th><th>Capabilities</th><th>Trust</th><th>Completed</th><th>Last signal</th></tr></thead>
<tbody>
<tr><td><div class="t-main">lab-node-01</div><div class="t-sub">w-01HZZ…a4</div></td><td><span class="pill pill-active"><i></i>Busy</span></td><td><span class="cap">similarity-search</span><span class="cap">similarity-graph</span></td><td><select><option selected>Trusted</option><option>Untrusted</option></select></td><td style="font-family:var(--mono);font-size:12.5px">341</td><td style="color:var(--text-2);font-size:12.5px">4 s ago</td></tr>
<tr><td><div class="t-main">emil-laptop</div><div class="t-sub">w-01HZZ…9c</div></td><td><span class="pill pill-success"><i></i>Idle</span></td><td><span class="cap">similarity-search</span><span class="cap">descriptors</span><span class="cap">molwt-filter</span></td><td><select><option selected>Trusted</option><option>Untrusted</option></select></td><td style="font-family:var(--mono);font-size:12.5px">87</td><td style="color:var(--text-2);font-size:12.5px">2 s ago</td></tr>
<tr><td><div class="t-main">lab-node-02</div><div class="t-sub">w-01HZZ…d1</div></td><td><span class="pill pill-active"><i></i>Busy</span></td><td><span class="cap">similarity-search</span></td><td><select><option>Trusted</option><option selected>Untrusted</option></select></td><td style="font-family:var(--mono);font-size:12.5px">12</td><td style="color:var(--text-2);font-size:12.5px">6 s ago</td></tr>
<tr><td><div class="t-main">old-workstation</div><div class="t-sub">w-01HZX…7f</div></td><td><span class="pill pill-waiting"><i></i>Offline</span></td><td><span class="cap">similarity-search</span></td><td><select><option selected>Trusted</option><option>Untrusted</option></select></td><td style="font-family:var(--mono);font-size:12.5px">1 204</td><td style="color:var(--text-2);font-size:12.5px">2 days ago</td></tr>
</tbody>
</table>
</div>
</section>
<!-- ═══ USERS & KEYS ═══ -->
<section class="page" id="page-users">
<div class="section-title">Users</div>
<div class="card">
<table>
<thead><tr><th>Email</th><th>Role</th><th>Jobs</th><th>Created</th><th>Last active</th><th></th></tr></thead>
<tbody>
<tr><td><div class="t-main">admin@scimesh.local</div></td><td><span class="pill pill-amber"><i></i>admin</span></td><td style="font-family:var(--mono);font-size:12.5px">9</td><td style="color:var(--text-2);font-size:12.5px">Jul 30</td><td style="color:var(--text-2);font-size:12.5px">now</td><td><button class="btn btn-ghost btn-sm">Edit</button></td></tr>
<tr><td><div class="t-main">alice@lab.org</div></td><td><select><option selected>user</option><option>admin</option></select></td><td style="font-family:var(--mono);font-size:12.5px">11</td><td style="color:var(--text-2);font-size:12.5px">Jul 30</td><td style="color:var(--text-2);font-size:12.5px">12 min ago</td><td><button class="btn btn-ghost btn-sm">Edit</button></td></tr>
<tr><td><div class="t-main">bob@lab.org</div></td><td><select><option selected>user</option><option>admin</option></select></td><td style="font-family:var(--mono);font-size:12.5px">6</td><td style="color:var(--text-2);font-size:12.5px">Jul 31</td><td style="color:var(--text-2);font-size:12.5px">1 h ago</td><td><button class="btn btn-ghost btn-sm">Edit</button></td></tr>
</tbody>
</table>
<div class="footer-row"><span>3 users · bootstrap admin cannot be demoted</span><button class="btn btn-ghost btn-sm"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M12 5v14M5 12h14"/></svg>Invite user</button></div>
</div>
<div class="section-title">Worker keys</div>
<div class="section-note">Keys let lab machines register as workers under a user account. Served instances can also use the cluster token.</div>
<div class="card">
<table>
<thead><tr><th>Label</th><th>Key prefix</th><th>Owner</th><th>Created</th><th>Last used</th><th></th></tr></thead>
<tbody>
<tr><td class="t-main">lab-node-01</td><td><code style="color:var(--text-2)">smk_7f3a…</code></td><td style="color:var(--text-2)">alice@lab.org</td><td style="color:var(--text-2);font-size:12.5px">Jul 30</td><td style="color:var(--text-2);font-size:12.5px">4 s ago</td><td><button class="btn btn-danger btn-sm">Revoke</button></td></tr>
<tr><td class="t-main">emil-laptop</td><td><code style="color:var(--text-2)">smk_b291…</code></td><td style="color:var(--text-2)">admin@scimesh.local</td><td style="color:var(--text-2);font-size:12.5px">Jul 31</td><td style="color:var(--text-2);font-size:12.5px">2 s ago</td><td><button class="btn btn-danger btn-sm">Revoke</button></td></tr>
<tr><td class="t-main" style="color:var(--text-3)">old-workstation</td><td><code style="color:var(--text-3)">smk_44cd…</code></td><td style="color:var(--text-3)">alice@lab.org</td><td style="color:var(--text-3);font-size:12.5px">Jul 30</td><td style="color:var(--text-3);font-size:12.5px">2 days ago</td><td><span class="pill pill-danger"><i></i>Revoked</span></td></tr>
</tbody>
</table>
<div class="footer-row"><span>2 active keys</span><button class="btn btn-primary btn-sm"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M12 5v14M5 12h14"/></svg>Issue key</button></div>
</div>
</section>
<!-- ═══ WORKLOADS ═══ -->
<section class="page" id="page-workloads">
<div class="section-note">Disabled workloads are rejected at submit time and hidden from the job form. The catalog is compiled into the binary; settings persist in the database.</div>
<div class="card">
<table>
<thead><tr><th>Workload</th><th>Reduction</th><th>Parameters</th><th>Dataset upload</th><th>Enabled</th></tr></thead>
<tbody>
<tr><td><div class="t-main">similarity-search</div><div class="t-sub">top-k Tanimoto search over a TSV of SMILES</div></td><td><span class="pill pill-active"><i></i>top-k</span></td><td style="color:var(--text-2);font-size:12.5px">4 declared</td><td><span class="pill pill-success"><i></i>ready</span></td><td><button class="toggle on" aria-label="enabled"></button></td></tr>
<tr><td><div class="t-main">similarity-graph</div><div class="t-sub">threshold graph edges between all pairs</div></td><td><span class="pill pill-active"><i></i>ordered-concat</span></td><td style="color:var(--text-2);font-size:12.5px">2 declared</td><td><span class="pill pill-success"><i></i>ready</span></td><td><button class="toggle on" aria-label="enabled"></button></td></tr>
<tr><td><div class="t-main">descriptors</div><div class="t-sub">2D RDKit descriptors per molecule</div></td><td><span class="pill pill-active"><i></i>ordered-concat</span></td><td style="color:var(--text-2);font-size:12.5px">1 declared</td><td><span class="pill pill-success"><i></i>ready</span></td><td><button class="toggle" aria-label="disabled"></button></td></tr>
<tr><td><div class="t-main">molwt-filter</div><div class="t-sub">keep molecules inside a weight window</div></td><td><span class="pill pill-active"><i></i>ordered-concat</span></td><td style="color:var(--text-2);font-size:12.5px">2 declared</td><td><span class="pill pill-success"><i></i>ready</span></td><td><button class="toggle on" aria-label="enabled"></button></td></tr>
</tbody>
</table>
</div>
</section>
<!-- ═══ METRICS ═══ -->
<section class="page" id="page-metrics">
<div class="grid-kpi">
<div class="kpi"><div class="k-label">Jobs · 7 days</div><div class="k-value">26</div><div class="k-sub up">+8 vs previous week</div></div>
<div class="kpi"><div class="k-label">Shards completed</div><div class="k-value">1 644</div><div class="k-sub">across 5 workers</div></div>
<div class="kpi"><div class="k-label">Avg shard time</div><div class="k-value">2.3s</div><div class="k-sub up">0.4s this week</div></div>
<div class="kpi"><div class="k-label">Failure rate</div><div class="k-value" style="color:var(--amber)">1.2%</div><div class="k-sub">20 failed shards</div></div>
</div>
<div class="split">
<div class="card">
<div class="card-head"><h3>Jobs per day</h3><span>last 7 days</span></div>
<div style="padding:16px 20px 10px">
<svg class="chart" viewBox="0 0 460 150">
<line class="chart-grid" x1="0" y1="30" x2="460" y2="30"/><line class="chart-grid" x1="0" y1="70" x2="460" y2="70"/><line class="chart-grid" x1="0" y1="110" x2="460" y2="110"/>
<rect class="chart-bar" x="18" y="70" width="44" height="60"/><rect class="chart-bar" x="82" y="50" width="44" height="80"/>
<rect class="chart-bar" x="146" y="90" width="44" height="40"/><rect class="chart-bar" x="210" y="40" width="44" height="90"/>
<rect class="chart-bar hot" x="274" y="20" width="44" height="110"/><rect class="chart-bar" x="338" y="60" width="44" height="70"/>
<rect class="chart-bar" x="402" y="80" width="44" height="50"/>
<text class="chart-label" x="18" y="145">Mon</text><text class="chart-label" x="82" y="145">Tue</text><text class="chart-label" x="146" y="145">Wed</text>
<text class="chart-label" x="210" y="145">Thu</text><text class="chart-label" x="274" y="145">Fri</text><text class="chart-label" x="338" y="145">Sat</text>
<text class="chart-label" x="402" y="145">Sun</text>
</svg>
</div>
</div>
<div class="card">
<div class="card-head"><h3>Jobs by workload</h3><span>all time</span></div>
<dl class="kv">
<dt>similarity-search</dt><dd><div class="bar" style="width:100%"><span style="width:78%"></span></div></dd>
<dt>similarity-graph</dt><dd><div class="bar" style="width:100%"><span style="width:41%"></span></div></dd>
<dt>molwt-filter</dt><dd><div class="bar" style="width:100%"><span style="width:22%"></span></div></dd>
<dt>descriptors</dt><dd><div class="bar" style="width:100%"><span style="width:9%"></span></div></dd>
</dl>
</div>
</div>
</section>
<!-- ═══ SETTINGS ═══ -->
<section class="page" id="page-settings">
<div class="warn-strip"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round"><path d="M12 9v4M12 17h.01M10.3 3.9L2.6 17a2 2 0 0 0 1.7 3h15.4a2 2 0 0 0 1.7-3L13.7 3.9a2 2 0 0 0-3.4 0z"/></svg><div>The cluster token below authenticates <b>any</b> worker. Reveal it only on a trusted machine, and rotate it if it may have leaked.</div></div>
<div class="section-title">Cluster</div>
<div class="card">
<dl class="kv">
<dt>Worker token</dt><dd><div class="secret"><code id="tok">••••••••••••••••••••••••</code><button class="btn btn-ghost btn-sm" id="reveal">Reveal</button><button class="btn btn-ghost btn-sm">Rotate…</button></div></dd>
<dt>Public URL</dt><dd><input value="http://192.168.1.10:8080" style="width:320px"></dd>
<dt>Listen address</dt><dd><input value="127.0.0.1:8080" style="width:200px"> <span style="color:var(--text-3);font-size:12px">takes effect after restart</span></dd>
</dl>
</div>
<div class="section-title">Storage</div>
<div class="card">
<dl class="kv">
<dt>Data directory</dt><dd><code>~/.scimesh</code></dd>
<dt>Database engine</dt><dd><select><option selected>sqlite (embedded)</option><option>postgres (external)</option></select></dd>
<dt>Auto-migrate on start</dt><dd><button class="toggle on"></button></dd>
</dl>
<div class="footer-row"><span></span><button class="btn btn-primary btn-sm">Save changes</button></div>
</div>
<div class="section-title" style="color:var(--red)">Danger zone</div>
<div class="card" style="border-color:#f2647c33">
<dl class="kv">
<dt>Prune artifacts</dt><dd style="display:flex;justify-content:space-between;align-items:center"><span style="color:var(--text-2);font-size:12.5px">Delete results of completed jobs older than 30 days · frees ~400 MB</span><button class="btn btn-ghost btn-sm">Prune…</button></dd>
<dt>Reset cluster</dt><dd style="display:flex;justify-content:space-between;align-items:center"><span style="color:var(--text-2);font-size:12.5px">Drop all jobs, workers and keys. Users are kept.</span><button class="btn btn-danger btn-sm">Reset…</button></dd>
</dl>
</div>
</section>
</div>
</div>
</div>
<script>
const titles={system:['System','Cluster state and node information'],jobs:['Jobs','Every computation, filterable and paginated'],workers:['Workers','Fleet overview and trust management'],users:['Users & keys','Accounts, roles and worker keys'],workloads:['Workloads','Catalog entries and availability'],metrics:['Metrics','Throughput and reliability, last 7 days'],settings:['Settings','Cluster, storage and security']};
document.querySelectorAll('.nav-item').forEach(btn=>btn.addEventListener('click',()=>{
document.querySelectorAll('.nav-item').forEach(b=>b.classList.remove('active'));
document.querySelectorAll('.page').forEach(p=>p.classList.remove('active'));
btn.classList.add('active');
document.getElementById('page-'+btn.dataset.page).classList.add('active');
const [t,s]=titles[btn.dataset.page];
document.getElementById('page-title').textContent=t;
document.getElementById('page-sub').textContent=s;
}));
document.querySelectorAll('.toggle').forEach(t=>t.addEventListener('click',()=>t.classList.toggle('on')));
document.getElementById('reveal').addEventListener('click',e=>{
const tok=document.getElementById('tok');
const shown=tok.textContent.startsWith('•');
tok.textContent=shown?'sm_live_9f2c7ab41d06e8f3c5a7':'••••••••••••••••••••••••';
e.target.textContent=shown?'Hide':'Reveal';
});
</script>
</body>
</html>
+244
View File
@@ -0,0 +1,244 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SciMesh Worker · Setup</title>
<style>
:root{
--bg:#0b0e13;--panel:#11151d;--panel-2:#161c27;--border:#1f2634;--border-soft:#181f2b;
--text:#e9ecf3;--text-2:#98a2b5;--text-3:#5d6879;
--accent:#5b8cff;--accent-soft:#5b8cff1f;--accent-strong:#3f6fe0;
--green:#3fce8a;--green-soft:#3fce8a1a;--amber:#e5b64f;--amber-soft:#e5b64f1a;--red:#f2647c;--red-soft:#f2647c1a;
--mono:ui-monospace,"SF Mono",SFMono-Regular,Menlo,monospace;
color-scheme:dark;
}
*{box-sizing:border-box;margin:0;padding:0}
body{background:radial-gradient(900px 500px at 50% -180px,#16233d66,transparent),var(--bg);color:var(--text);font:14px/1.55 Inter,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased;min-height:100vh}
button{font:inherit;color:inherit;background:none;border:0;cursor:pointer}
input{font:inherit;color:var(--text);background:var(--panel-2);border:1px solid var(--border);border-radius:9px;padding:10px 13px;width:100%;outline:none;transition:border-color .12s,box-shadow .12s}
input:focus{border-color:var(--accent);box-shadow:0 0 0 3px var(--accent-soft)}
input::placeholder{color:var(--text-3)}
code{font-family:var(--mono);font-size:.86em}
.shell{max-width:660px;margin:0 auto;padding:44px 22px 70px}
.brand{display:flex;align-items:center;justify-content:center;gap:11px;margin-bottom:8px}
.brand-mark{display:grid;place-items:center;width:34px;height:34px;border-radius:10px;background:linear-gradient(135deg,#5b8cff,#7c5cff);box-shadow:0 4px 16px #5b8cff40}
.brand-mark svg{width:18px;height:18px;stroke:#fff}
.brand-name{font-weight:700;font-size:16px;letter-spacing:-.01em}
.brand-name span{color:var(--text-3);font-weight:500}
.tagline{text-align:center;color:var(--text-3);font-size:12.5px;margin-bottom:34px}
.tagline code{color:var(--text-2)}
/* step indicator */
.steps{display:flex;align-items:center;justify-content:center;gap:0;margin-bottom:30px}
.step{display:flex;flex-direction:column;align-items:center;gap:7px;width:96px}
.step-dot{display:grid;place-items:center;width:30px;height:30px;border-radius:50%;border:1.5px solid var(--border);background:var(--panel);color:var(--text-3);font-size:12.5px;font-weight:700;transition:all .2s}
.step-label{font-size:11px;font-weight:600;color:var(--text-3);letter-spacing:.02em}
.step.active .step-dot{border-color:var(--accent);background:var(--accent-soft);color:var(--accent);box-shadow:0 0 0 4px #5b8cff14}
.step.active .step-label{color:var(--text)}
.step.done .step-dot{border-color:var(--green);background:var(--green-soft);color:var(--green)}
.step.done .step-label{color:var(--text-2)}
.step-line{flex:1;max-width:44px;height:1.5px;background:var(--border);margin:0 6px 22px;position:relative;overflow:hidden}
.step-line.done:after{content:"";position:absolute;inset:0;background:var(--green)}
.card{background:var(--panel);border:1px solid var(--border-soft);border-radius:15px;padding:26px 28px;box-shadow:0 24px 60px #0000004d}
.card h1{font-size:18px;font-weight:700;letter-spacing:-.02em;margin-bottom:4px}
.card .sub{color:var(--text-2);font-size:13px;margin-bottom:22px}
.field{margin-bottom:16px}
.field label{display:block;font-size:12px;font-weight:650;letter-spacing:.04em;text-transform:uppercase;color:var(--text-3);margin-bottom:7px}
.field .hint{margin-top:6px;font-size:12px;color:var(--text-3)}
.field .hint code{color:var(--text-2)}
.radio-grid{display:grid;grid-template-columns:1fr 1fr;gap:10px}
.radio-card{border:1px solid var(--border);border-radius:11px;padding:13px 14px;cursor:pointer;transition:all .13s;background:var(--panel-2)}
.radio-card:hover{border-color:#2a3446}
.radio-card.sel{border-color:var(--accent);background:var(--accent-soft);box-shadow:0 0 0 3px #5b8cff14}
.radio-card b{display:flex;align-items:center;gap:8px;font-size:13.5px}
.radio-card b svg{width:15px;height:15px;stroke:var(--accent)}
.radio-card p{margin-top:4px;font-size:12px;color:var(--text-2)}
.check-row{display:flex;align-items:center;gap:12px;padding:11px 14px;border:1px solid var(--border-soft);border-radius:10px;margin-bottom:9px;background:var(--panel-2)}
.check-ic{display:grid;place-items:center;width:24px;height:24px;border-radius:50%;flex:none}
.check-ic svg{width:13px;height:13px;stroke-width:2.6}
.check-ok{background:var(--green-soft)}.check-ok svg{stroke:var(--green)}
.check-bad{background:var(--red-soft)}.check-bad svg{stroke:var(--red)}
.check-wait{background:#ffffff10}.check-wait svg{stroke:var(--text-3)}
.check-row b{font-size:13.5px;font-weight:600}
.check-row span{display:block;font-size:12px;color:var(--text-3)}
.check-row .ms{margin-left:auto;font:11.5px var(--mono);color:var(--text-3)}
.actions{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-top:24px}
.btn{display:inline-flex;align-items:center;gap:8px;border-radius:9px;padding:10px 18px;font-weight:650;font-size:13.5px;border:1px solid transparent;transition:all .13s}
.btn svg{width:15px;height:15px;stroke:currentColor}
.btn-primary{background:var(--accent);color:#0a1222}
.btn-primary:hover{background:var(--accent-strong);color:#fff}
.btn-ghost{border-color:var(--border);color:var(--text-2);background:var(--panel-2)}
.btn-ghost:hover{color:var(--text);border-color:#2a3446}
.btn-danger{background:var(--red-soft);color:var(--red)}
.btn-lg{padding:12px 24px;font-size:14.5px;border-radius:10px}
.link{color:var(--text-3);font-size:13px}
.link:hover{color:var(--text)}
/* status view */
.status-head{display:flex;align-items:center;gap:14px;margin-bottom:22px}
.pulse{position:relative;width:12px;height:12px;border-radius:50%;background:var(--green);flex:none}
.pulse:after{content:"";position:absolute;inset:-5px;border-radius:50%;border:2px solid var(--green);opacity:.5;animation:ping 1.6s ease-out infinite}
@keyframes ping{from{transform:scale(.6);opacity:.7}to{transform:scale(1.4);opacity:0}}
.status-head h1{font-size:19px}
.status-head .sub{font-size:12.5px;color:var(--text-3)}
.stat-row{display:grid;grid-template-columns:repeat(4,1fr);gap:10px;margin-bottom:18px}
.stat{background:var(--panel-2);border:1px solid var(--border-soft);border-radius:11px;padding:12px 14px}
.stat b{display:block;font-size:20px;font-weight:700;letter-spacing:-.02em}
.stat span{font-size:11px;font-weight:600;letter-spacing:.05em;text-transform:uppercase;color:var(--text-3)}
.stat.bad b{color:var(--red)}
.logbox{background:#0a0d12;border:1px solid var(--border-soft);border-radius:11px;padding:14px 16px;font:12px/1.7 var(--mono);color:#8fa3bf;max-height:190px;overflow-y:auto}
.logbox .t{color:var(--text-3)}
.logbox .ok{color:var(--green)}
.logbox .hl{color:#c9d6ea}
.meta-line{display:flex;gap:8px;flex-wrap:wrap;margin-bottom:16px}
.chip{display:inline-flex;align-items:center;gap:6px;border:1px solid var(--border);border-radius:99px;padding:4px 11px;font-size:12px;color:var(--text-2);background:var(--panel-2)}
.chip svg{width:12px;height:12px;stroke:var(--text-3)}
.wizard-page{display:none}.wizard-page.active{display:block;animation:fade .18s ease}
@keyframes fade{from{opacity:0;transform:translateY(5px)}to{opacity:1}}
.demo-toggle{position:fixed;bottom:14px;right:16px;font-size:11.5px;color:var(--text-3);background:var(--panel);border:1px solid var(--border);border-radius:99px;padding:6px 13px;opacity:.8}
.demo-toggle:hover{color:var(--text)}
</style>
</head>
<body>
<div class="shell">
<div class="brand">
<div class="brand-mark"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round"><circle cx="6" cy="6" r="2.4"/><circle cx="18" cy="7" r="2.4"/><circle cx="12" cy="17" r="2.4"/><path d="M8 7.5l7.6-1M7 8.3l3.7 6.5M16.6 9.2l-3.2 5.6"/></svg></div>
<div class="brand-name">SciMesh <span>· Worker setup</span></div>
</div>
<p class="tagline">Local wizard served by <code>worker-agent setup</code> · <code>127.0.0.1:12700</code></p>
<!-- ═══ WIZARD VIEW ═══ -->
<div id="view-wizard">
<div class="steps">
<div class="step done" id="st1"><div class="step-dot"></div><div class="step-label">Connect</div></div>
<div class="step-line done" id="sl1"></div>
<div class="step active" id="st2"><div class="step-dot">2</div><div class="step-label">Machine</div></div>
<div class="step-line" id="sl2"></div>
<div class="step" id="st3"><div class="step-dot">3</div><div class="step-label">Check</div></div>
<div class="step-line" id="sl3"></div>
<div class="step" id="st4"><div class="step-dot">4</div><div class="step-label">Run</div></div>
</div>
<!-- step 1 (kept in DOM for the mockup; step 2 shown) -->
<div class="wizard-page" id="wp1">
<div class="card">
<h1>Connect to a coordinator</h1>
<p class="sub">The coordinator hands out work and collects results. Ask your cluster admin for its address.</p>
<div class="field"><label>Coordinator URL</label><input value="http://192.168.1.10:8080" placeholder="http://192.168.1.10:8080"><p class="hint">For a served instance this is the address printed by <code>coordinator serve</code>.</p></div>
<div class="field"><label>Authentication</label>
<div class="radio-grid">
<div class="radio-card sel"><b><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><rect x="4" y="10" width="16" height="10" rx="2"/><path d="M8 10V7a4 4 0 0 1 8 0v3"/></svg>Cluster token</b><p>Serve instances: one token for every worker.</p></div>
<div class="radio-card"><b><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="8" cy="14" r="4"/><path d="M10.8 11.2L20 2M15 4l3 3"/></svg>Worker key</b><p>Shared clusters: a key tied to your account.</p></div>
</div>
</div>
<div class="field"><label>Token</label><input type="password" value="sm_live_9f2c7ab41d06e8f3c5a7"><p class="hint">The admin can copy it from <code>coordinator token</code> on the server.</p></div>
<div class="actions"><span class="link">Step 1 of 4</span><button class="btn btn-primary" onclick="goto(2)">Continue →</button></div>
</div>
</div>
<!-- step 2 -->
<div class="wizard-page active" id="wp2">
<div class="card">
<h1>This machine</h1>
<p class="sub">Where tasks run and how the machine appears in the cluster.</p>
<div class="field"><label>Worker name</label><input value="emil-laptop"><p class="hint">Shown in the coordinators worker list.</p></div>
<div class="field"><label>Work directory</label><input value="/home/emil/scimesh-worker"><p class="hint">Datasets and shard results live here. ~1 GB free space recommended.</p></div>
<div class="field"><label>Compute resources advertised</label>
<div class="radio-grid">
<div class="radio-card sel"><b>8 CPUs · 16 GB</b><p>Detected automatically.</p></div>
<div class="radio-card"><b>Custom…</b><p>Limit what this machine advertises.</p></div>
</div>
</div>
<div class="actions"><button class="btn btn-ghost" onclick="goto(1)">← Back</button><button class="btn btn-primary" onclick="goto(3)">Continue →</button></div>
</div>
</div>
<!-- step 3 -->
<div class="wizard-page" id="wp3">
<div class="card">
<h1>Preflight check</h1>
<p class="sub">Making sure this machine can reach the coordinator and run SciMesh workloads.</p>
<div class="check-row"><div class="check-ic check-ok"><svg viewBox="0 0 24 24" fill="none" stroke-linecap="round"><path d="M4 12l5 5L20 6"/></svg></div><div><b>Coordinator reachable</b><span>http://192.168.1.10:8080 · coordinator 1.1.0-alpha.1</span></div><span class="ms">34 ms</span></div>
<div class="check-row"><div class="check-ic check-ok"><svg viewBox="0 0 24 24" fill="none" stroke-linecap="round"><path d="M4 12l5 5L20 6"/></svg></div><div><b>Token accepted</b><span>authenticated as cluster worker</span></div><span class="ms">51 ms</span></div>
<div class="check-row"><div class="check-ic check-ok"><svg viewBox="0 0 24 24" fill="none" stroke-linecap="round"><path d="M4 12l5 5L20 6"/></svg></div><div><b>Python 3.12 found</b><span>/usr/bin/python3</span></div><span class="ms">8 ms</span></div>
<div class="check-row"><div class="check-ic check-bad"><svg viewBox="0 0 24 24" fill="none" stroke-linecap="round"><path d="M6 6l12 12M18 6L6 18"/></svg></div><div><b>scimesh package missing</b><span>run: pip install scimesh — or let the wizard install it</span></div><button class="btn btn-ghost" style="margin-left:auto;padding:6px 12px;font-size:12px">Install</button></div>
<div class="actions"><button class="btn btn-ghost" onclick="goto(2)">← Back</button><button class="btn btn-primary" onclick="goto(4)">Continue anyway →</button></div>
</div>
</div>
<!-- step 4 -->
<div class="wizard-page" id="wp4">
<div class="card" style="text-align:center;padding:40px 28px">
<div class="brand-mark" style="margin:0 auto 18px;width:46px;height:46px;border-radius:13px"><svg viewBox="0 0 24 24" fill="none" stroke-width="2" stroke-linecap="round" style="width:22px;height:22px"><path d="M6 4l14 8-14 8V4z"/></svg></div>
<h1 style="font-size:20px">Ready to join the cluster</h1>
<p class="sub" style="max-width:380px;margin:8px auto 26px">The wizard will save your configuration to <code>~/.scimesh-worker/config.json</code> and start the worker as a background process.</p>
<div class="meta-line" style="justify-content:center">
<span class="chip"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="12" cy="12" r="9"/></svg>emil-laptop</span>
<span class="chip"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M4 6h16M4 12h16M4 18h16"/></svg>8 CPUs · 16 GB</span>
<span class="chip"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><path d="M12 3l8 4.5v9L12 21l-8-4.5v-9L12 3z"/></svg>4 workloads</span>
</div>
<div class="actions" style="justify-content:space-between;margin-top:30px"><button class="btn btn-ghost" onclick="goto(3)">← Back</button><button class="btn btn-primary btn-lg" onclick="showStatus()">Start worker</button></div>
</div>
</div>
</div>
<!-- ═══ STATUS VIEW ═══ -->
<div id="view-status" style="display:none">
<div class="card">
<div class="status-head">
<div class="pulse"></div>
<div><h1>emil-laptop is working</h1><div class="sub">worker id <code>w-01HZZ4T9C6</code> · registered with 192.168.1.10:8080 · up 42 min</div></div>
<button class="btn btn-danger" style="margin-left:auto"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><rect x="6" y="6" width="12" height="12" rx="2"/></svg>Stop</button>
</div>
<div class="stat-row">
<div class="stat"><b>14</b><span>claimed</span></div>
<div class="stat"><b>12</b><span>completed</span></div>
<div class="stat bad"><b>0</b><span>failed</span></div>
<div class="stat"><b>2.1s</b><span>avg shard</span></div>
</div>
<div class="meta-line">
<span class="chip">similarity-search</span><span class="chip">similarity-graph</span><span class="chip">descriptors</span><span class="chip">molwt-filter</span>
<span class="chip" style="margin-left:auto"><svg viewBox="0 0 24 24" fill="none" stroke-width="2"><circle cx="12" cy="12" r="9"/><path d="M12 7v5l3 3"/></svg>heartbeat 2 s ago</span>
</div>
<div class="logbox">
<div><span class="t">14:02:11</span> agent 1.1.0-alpha.1 · config ~/.scimesh-worker/config.json</div>
<div><span class="t">14:02:11</span> coordinator http://192.168.1.10:8080 · token auth</div>
<div><span class="t">14:02:12</span> <span class="ok">registered</span> as <span class="hl">emil-laptop</span> (w-01HZZ4T9C6)</div>
<div><span class="t">14:02:12</span> capabilities: similarity-search, similarity-graph, descriptors, molwt-filter</div>
<div><span class="t">14:03:40</span> claimed task <span class="hl">7f3a…c91e/007</span> · similarity-search</div>
<div><span class="t">14:03:42</span> <span class="ok">completed</span> 7f3a…c91e/007 · uploaded shard (2.1s)</div>
<div><span class="t">14:05:03</span> claimed task <span class="hl">7f3a…c91e/031</span> · similarity-search</div>
<div><span class="t">14:05:05</span> <span class="ok">completed</span> 7f3a…c91e/031 · uploaded shard (1.9s)</div>
</div>
<div class="actions"><span class="link">Configuration: <code>~/.scimesh-worker/config.json</code></span><button class="btn btn-ghost" onclick="showWizard()">Reconfigure…</button></div>
</div>
</div>
</div>
<button class="demo-toggle" onclick="toggleView()">mockup · toggle wizard / status</button>
<script>
const pages=['wp1','wp2','wp3','wp4'];
function goto(n){
pages.forEach((id,i)=>document.getElementById(id).classList.toggle('active',i===n-1));
for(let i=1;i<=4;i++){
const st=document.getElementById('st'+i);
st.classList.toggle('done',i<n);st.classList.toggle('active',i===n);
st.querySelector('.step-dot').textContent=i<n?'':i;
if(i<4)document.getElementById('sl'+i).classList.toggle('done',i<n);
}
}
function showStatus(){document.getElementById('view-wizard').style.display='none';document.getElementById('view-status').style.display='block'}
function showWizard(){document.getElementById('view-wizard').style.display='block';document.getElementById('view-status').style.display='none'}
function toggleView(){document.getElementById('view-status').style.display==='none'?showStatus():showWizard()}
document.querySelectorAll('.radio-grid').forEach(g=>g.querySelectorAll('.radio-card').forEach(c=>c.addEventListener('click',()=>{g.querySelectorAll('.radio-card').forEach(x=>x.classList.remove('sel'));c.classList.add('sel')})));
</script>
</body>
</html>
+67 -10
View File
@@ -9,12 +9,23 @@
$ErrorActionPreference = "Stop"
$Repo = "emil28092005/SciMesh"
$Component = if ($env:SCIMESH_COMPONENT) { $env:SCIMESH_COMPONENT } else { "coordinator" }
$Version = if ($env:SCIMESH_VERSION) { $env:SCIMESH_VERSION } else { "latest" }
$InstallDir = if ($env:SCIMESH_INSTALL_DIR) {
$env:SCIMESH_INSTALL_DIR
} else {
Join-Path $env:LOCALAPPDATA "SciMesh"
}
# Auto-start the component right after install and open its UI (the control
# room for the coordinator, the local setup wizard for the worker). Set
# SCIMESH_AUTO_START=0 to install only.
$AutoStart = if ($env:SCIMESH_AUTO_START) { $env:SCIMESH_AUTO_START } else { "1" }
switch ($Component) {
"coordinator" { $Binary = "coordinator" }
"worker" { $Binary = "worker-agent" }
default { throw "unknown component: $Component (use 'coordinator' or 'worker')" }
}
$Arch = switch ($env:PROCESSOR_ARCHITECTURE) {
"AMD64" { "amd64" }
@@ -23,24 +34,70 @@ $Arch = switch ($env:PROCESSOR_ARCHITECTURE) {
}
if ($Version -eq "latest") {
Write-Host "Resolving the latest SciMesh release..."
$Url = "https://github.com/$Repo/releases/latest/download/coordinator-windows-$Arch.exe"
} else {
$Url = "https://github.com/$Repo/releases/download/$Version/coordinator-windows-$Arch.exe"
Write-Host "Resolving the newest SciMesh release (including pre-releases)..."
# /releases/latest only sees stable releases; the API list is newest-first
# across all channels.
try {
$Releases = Invoke-RestMethod -Uri "https://api.github.com/repos/$Repo/releases?per_page=1"
if ($Releases -and $Releases[0].tag_name) {
$Version = $Releases[0].tag_name
Write-Host " -> $Version"
} else {
Write-Host " -> falling back to the stable latest release"
}
} catch {
Write-Host " -> falling back to the stable latest release"
}
}
$Url = "https://github.com/$Repo/releases/download/$Version/$Binary-windows-$Arch.exe"
New-Item -ItemType Directory -Force -Path $InstallDir | Out-Null
$Target = Join-Path $InstallDir "coordinator.exe"
$Target = Join-Path $InstallDir "$Binary.exe"
Write-Host "Downloading $Url"
Invoke-WebRequest -Uri $Url -OutFile "$Target.tmp"
Move-Item -Force "$Target.tmp" $Target
Write-Host ""
Write-Host "SciMesh installed: $Target"
Write-Host "SciMesh $Component installed: $Target"
& $Target --version
Write-Host ""
Write-Host "Start the platform (one command, everything embedded):"
Write-Host " $Target serve --open"
Write-Host ""
Write-Host "Your data lives in ~\.scimesh. The admin login is printed on first start."
if ($Component -eq "coordinator") {
if ($AutoStart -eq "1") {
Write-Host "Starting the platform and opening the admin console in your browser..."
Write-Host "(stop it with Ctrl-C; it keeps your data in ~\.scimesh)"
Write-Host ""
& $Target serve --open
} else {
Write-Host "Start the platform (one command, everything embedded):"
Write-Host " $Target serve --open"
Write-Host ""
Write-Host "Your data lives in ~\.scimesh. The admin login is printed on first start."
}
} else {
if ($AutoStart -eq "1") {
Write-Host "Starting the local setup wizard in your browser..."
Write-Host "(stop it with Ctrl-C; it keeps the configuration in ~\.scimesh-worker)"
Write-Host ""
& $Target setup
} else {
Write-Host "The worker needs Python 3 with the scimesh package, then a coordinator"
Write-Host "to connect to. Point the local wizard at it:"
Write-Host ""
Write-Host " $Target setup"
Write-Host ""
Write-Host "Or run it with environment variables:"
Write-Host ""
Write-Host " set COORDINATOR_URL=http://COORDINATOR_HOST:8080"
Write-Host " set WORKER_AUTH_TOKEN=<worker token from the coordinator>"
Write-Host " set WORK_DIR=%USERPROFILE%\scimesh-worker"
Write-Host " $Target"
Write-Host ""
Write-Host "For a coordinator started with 'coordinator serve', the worker token is"
Write-Host "in ~\.scimesh\worker.token on that machine. Set SCIMESH_PIP_PACKAGE to"
Write-Host "install scimesh into a managed venv, or install it yourself:"
Write-Host " set SCIMESH_PIP_PACKAGE=<your wheel or index>"
Write-Host " $Target setup"
}
}
+82 -17
View File
@@ -1,18 +1,32 @@
#!/usr/bin/env bash
# SciMesh installer: downloads the coordinator binary for this OS/architecture
# from the latest GitHub release and installs it locally. One command, no
# picking from a list of files:
# SciMesh installer: downloads a binary for this OS/architecture from the
# newest GitHub release, installs it locally, then starts it and opens its UI
# in the browser (SCIMESH_AUTO_START=0 installs only). One command:
#
# curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash
# curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash -s worker
#
# Installed to ~/.local/bin/coordinator (Linux/macOS). Then run:
#
# coordinator serve --open
# The first form installs the coordinator (the whole platform in one binary:
# databases, userservice, local workers) and opens the admin console. The
# second installs a standalone worker agent that joins an existing
# coordinator, and opens its local setup wizard. Installed to ~/.local/bin
# (Linux/macOS) or %LOCALAPPDATA%\SciMesh (Windows).
set -eu
REPO="emil28092005/SciMesh"
COMPONENT="${1:-coordinator}"
VERSION="${SCIMESH_VERSION:-latest}"
INSTALL_DIR="${SCIMESH_INSTALL_DIR:-$HOME/.local/bin}"
# Auto-start the component right after install and open its UI (the control
# room for the coordinator, the local setup wizard for the worker). Set
# SCIMESH_AUTO_START=0 to install only.
AUTO_START="${SCIMESH_AUTO_START:-1}"
case "$COMPONENT" in
coordinator) BINARY="coordinator" ;;
worker) BINARY="worker-agent" ;;
*) echo "unknown component: $COMPONENT (use 'coordinator' or 'worker')" >&2; exit 1 ;;
esac
case "$(uname -s)" in
Linux) OS="linux" ;;
@@ -27,25 +41,76 @@ case "$(uname -m)" in
esac
if [ "$VERSION" = "latest" ]; then
echo "Resolving the latest SciMesh release..."
URL="https://github.com/${REPO}/releases/latest/download/coordinator-${OS}-${ARCH}"
else
URL="https://github.com/${REPO}/releases/download/${VERSION}/coordinator-${OS}-${ARCH}"
echo "Resolving the newest SciMesh release (including pre-releases)..."
# GitHub's /releases/latest only sees stable releases; the API list is
# newest-first across all channels. Without jq, pull the first tag_name.
RESOLVED=$(curl -fsSL "https://api.github.com/repos/${REPO}/releases?per_page=1" \
| sed -n 's/.*"tag_name"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' | head -1)
if [ -n "$RESOLVED" ]; then
VERSION="$RESOLVED"
echo " -> $VERSION"
else
echo " -> falling back to the stable latest release"
fi
fi
mkdir -p "$INSTALL_DIR"
TARGET="$INSTALL_DIR/coordinator"
TARGET="$INSTALL_DIR/$BINARY"
URL="https://github.com/${REPO}/releases/download/${VERSION}/${BINARY}-${OS}-${ARCH}"
echo "Downloading $URL"
curl -fsSL -o "$TARGET.tmp" "$URL"
chmod +x "$TARGET.tmp"
mv "$TARGET.tmp" "$TARGET"
echo
echo "SciMesh installed: $TARGET"
echo "SciMesh $COMPONENT installed: $TARGET"
INSTALLED_VERSION=$("$TARGET" --version 2>/dev/null | awk '{print $2}')
"$TARGET" --version
echo
echo "Start the platform (one command, everything embedded):"
echo " $TARGET serve --open"
echo
echo "Your data lives in ~/.scimesh. The admin login is printed on first start."
if [ -n "$INSTALLED_VERSION" ] && [ "$INSTALLED_VERSION" != "${VERSION#v}" ]; then
echo
echo "WARNING: expected version $VERSION but got $INSTALLED_VERSION."
echo "This is usually a stale download cache. Re-run the installer in a few"
echo "minutes, or pin the version explicitly:"
echo " SCIMESH_VERSION=${VERSION} bash <(curl -fsSL https://raw.githubusercontent.com/${REPO}/main/install.sh)"
fi
if [ "$COMPONENT" = "coordinator" ]; then
if [ "$AUTO_START" = "1" ]; then
echo
echo "Starting the platform and opening the admin console in your browser..."
echo "(stop it with Ctrl-C; it keeps your data in ~/.scimesh)"
echo
exec "$TARGET" serve --open
fi
echo
echo "Start the platform (one command, everything embedded):"
echo " $TARGET serve --open"
echo
echo "Your data lives in ~/.scimesh. The admin login is printed on first start."
else
if [ "$AUTO_START" = "1" ]; then
echo
echo "Starting the local setup wizard in your browser..."
echo "(stop it with Ctrl-C; it keeps the configuration in ~/.scimesh-worker)"
echo
exec "$TARGET" setup
fi
echo
echo "The worker needs Python 3 with the scimesh package, then a coordinator"
echo "to connect to. Point the local wizard at it:"
echo
echo " $TARGET setup"
echo
echo "Or run it with environment variables:"
echo
echo " export COORDINATOR_URL=http://COORDINATOR_HOST:8080"
echo " export WORKER_AUTH_TOKEN=<worker token from the coordinator>"
echo " export WORK_DIR=~/scimesh-worker"
echo " $TARGET"
echo
echo "For a coordinator started with 'coordinator serve', the worker token is"
echo "in ~/.scimesh/worker.token on that machine. Set SCIMESH_PIP_PACKAGE to"
echo "install scimesh into a managed venv, or install it yourself:"
echo " SCIMESH_PIP_PACKAGE=<your wheel or index> worker-agent setup"
fi
+1
View File
@@ -47,6 +47,7 @@ markdown_extensions:
nav:
- Home: index.md
- Standalone setup: standalone.md
- SDK:
- Overview: sdk/overview.md
- Authoring workloads: sdk/authoring-workloads.md
+38 -14
View File
@@ -31,10 +31,12 @@ The two halves of the project:
- **A distributed worker** that executes the same SDK workload handlers on
tasks claimed from the coordinator, with digest-pinned `TaskSpec`s,
resource reservation, and allowlist-driven workload discovery.
- **An operator UI** served by the coordinator: the control room, a workload
library page, a workload-agnostic "new computation" form whose controls come
from each workload's own `UIElement` declarations, and this documentation
site at `/ui/docs/`.
- **An operator UI** served by the coordinator: the **admin console**
(`/ui/admin`) for cluster operators — system/storage/health, jobs,
worker trust, users and worker keys, workload enable/disable, metrics and
the worker token — plus a workload-agnostic "new computation" form whose
controls come from each workload's own `UIElement` declarations, job detail
pages, a workload library page, and this documentation site at `/ui/docs/`.
## Quick start
@@ -44,17 +46,16 @@ local workers — is embedded in a single binary; no PostgreSQL, no Docker, no
Python setup.
```bash
# Linux / macOS
# Linux / macOS — installs and opens the admin console automatically
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash
coordinator serve --open
# Windows (PowerShell)
powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex"
coordinator serve --open
```
The first start prints the admin login (also stored under `~/.scimesh`), and
`--open` opens the UI in the browser. `coordinator serve --workers 2`
The installer starts the platform and opens the admin console in your browser
(set `SCIMESH_AUTO_START=0` to install only). The first start prints the admin
login (also stored under `~/.scimesh`). `coordinator serve --workers 2`
spawns two local workers; `SCIMESH_PIP_PACKAGE` points the managed venv at
your scimesh wheel so scientific workloads can run.
@@ -90,11 +91,34 @@ curl -L -o coordinator https://github.com/emil28092005/SciMesh/releases/latest/d
chmod +x coordinator
```
- **worker-agent** runs anywhere with Python: it spawns
`python -m scimesh.worker.task`, so the machine needs the `scimesh`
package in a venv (`pip install scimesh`) and the usual environment:
`COORDINATOR_URL`, `WORKER_AUTH_TOKEN`, `WORK_DIR`. The same binary can run
it via `coordinator agent`.
- **worker-agent** is installed separately and joins an existing coordinator.
Point its **local setup wizard** at the cluster — no need to have the
coordinator on this machine:
```bash
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash -s worker
# the installer opens the local wizard at http://127.0.0.1:12700 automatically
```
The wizard collects the coordinator URL and token (or worker key), runs a
preflight check, saves the configuration under `~/.scimesh-worker/` and
starts the worker as a background process — with a live status page and log.
Everything can also be done by hand:
```bash
export COORDINATOR_URL=http://COORDINATOR_HOST:8080
export WORKER_AUTH_TOKEN=<worker token from the coordinator>
export WORK_DIR=~/scimesh-worker
worker-agent
```
It spawns `python -m scimesh.worker.task`, so the machine needs Python 3
with the `scimesh` package. The wizard installs it into its own venv; the
package must come from your wheel, checkout or index — point
`SCIMESH_PIP_PACKAGE` at it (the PyPI name `scimesh` belongs to an unrelated
project). For a `coordinator serve` instance, the worker token is in
`~/.scimesh/worker.token`. On Windows set `SCIMESH_COMPONENT=worker` for
`install.ps1`.
- **coordinator** needs no external services at all in its default mode:
`coordinator serve` embeds SQLite (both databases), the userservice, and
local workers. The `SCIMESH_DB=postgres` engine remains for cluster
+143
View File
@@ -0,0 +1,143 @@
# Standalone (split) setup
The single-binary `coordinator serve` mode is the default for a scientist on
one machine. The **standalone setup** splits the platform into separate
processes — the coordinator, the userservice, and any number of workers —
typically on different machines, backed by PostgreSQL. This is the cluster
deployment.
```text
Browser (operator)
┌──────────────────────────────┐ ┌───────────────────────────┐
│ coordinator (port 8080) │ │ userservice (port 8081) │
│ jobs · tasks · artifacts │◄─────►│ users · roles · keys │
│ PostgreSQL DB "scimesh" │ JWT │ PostgreSQL DB │
└──────────────┬───────────────┘ secret │ "scimesh_users" │
│ └───────────────────────────┘
│ HTTP (workers connect out)
worker-agent × N (COORDINATOR_URL, WORKER_AUTH_TOKEN)
└─ python -m scimesh.worker.task (needs Python + scimesh)
```
Each component is a separate process; workers never touch a database — they
only talk to the coordinator over HTTP.
## What each component needs
| Component | Binary | Configuration |
| --- | --- | --- |
| Coordinator | `coordinator` (`SCIMESH_DB=postgres`) | `DATABASE_URL`, `COORDINATOR_ADDR`, `COORDINATOR_TOKEN`, `COORDINATOR_STORAGE_DIR`, `JWT_SECRET`, `USERSERVICE_URL`, `PUBLIC_COORDINATOR_URL` |
| Userservice | the `users/` service | `USERSERVICE_ADDR`, `DATABASE_URL` (its own DB), `JWT_SECRET` (must match the coordinator), `BOOTSTRAP_ADMIN_EMAIL` / `BOOTSTRAP_ADMIN_PASSWORD` |
| Worker | `worker-agent` | `COORDINATOR_URL`, `WORKER_AUTH_TOKEN`, `WORK_DIR`, `TASK_RUNNER` (JSON array), `CPU_COUNT`, `MEMORY_MB` |
`JWT_SECRET` is the one secret shared between the coordinator and the
userservice: the userservice signs tokens with it, the coordinator verifies
them. It must be at least 32 bytes and identical on both.
## Option A — Docker Compose (fastest)
The repository ships compose files for the whole split stack: PostgreSQL for
both services, migrations, the coordinator, and the userservice:
```bash
cd coordinator
JWT_SECRET='change-me-32-bytes-minimum' \
BOOTSTRAP_ADMIN_EMAIL='root@scimesh.local' \
BOOTSTRAP_ADMIN_PASSWORD='choose-a-strong-password' \
docker compose -f docker-compose.yml -f docker-compose.users.yml up -d --build
```
Then install workers on any machines with Python. The worker's own setup
wizard walks through the rest — URL, token or worker key, work directory —
and starts the worker for you:
```bash
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash -s worker
# the installer opens the local wizard at http://127.0.0.1:12700 automatically
```
On a headless server (no desktop environment), RDKit needs a few X11
libraries that desktops already ship — install them once with apt:
```bash
sudo apt-get install -y libxrender1 libxext6 libxcursor1 libxfixes3 libxi6 libxrandr2
```
Or configure by hand:
```bash
export COORDINATOR_URL=http://COORDINATOR_HOST:8080
export WORKER_AUTH_TOKEN="$COORDINATOR_TOKEN" # the coordinator's shared token
export WORK_DIR=~/scimesh-worker
worker-agent
```
## Option B — Manual binaries
1. **Provision PostgreSQL** (two databases, or one server and `CREATE
DATABASE`):
```bash
./coordinator setup --yes \
--db 'postgres://scimesh:scimesh@db-host:5432/scimesh?sslmode=disable' \
--env-file /etc/scimesh/coordinator.env
```
The wizard creates the database when missing and writes the `.env` with a
generated `JWT_SECRET`. The userservice needs its own database — create it
and apply `users/migrations` (e.g. with the migrate CLI):
```bash
createdb scimesh_users
migrate -path users/migrations \
-database 'postgres://scimesh:scimesh@db-host:5432/scimesh_users?sslmode=disable' up
```
2. **Run the userservice** with the *same* `JWT_SECRET`:
```bash
cd users && make build # builds the binary into users/bin/
USERSERVICE_ADDR=':8081' \
DATABASE_URL='postgres://scimesh:scimesh@db-host:5432/scimesh_users?sslmode=disable' \
JWT_SECRET='<same secret>' \
BOOTSTRAP_ADMIN_EMAIL='root@scimesh.local' \
BOOTSTRAP_ADMIN_PASSWORD='choose-a-strong-password' \
./bin/userservice
```
3. **Run the coordinator**:
```bash
ENV_FILE=/etc/scimesh/coordinator.env ./coordinator
# or, without the .env:
SCIMESH_DB=postgres \
DATABASE_URL='postgres://scimesh:scimesh@db-host:5432/scimesh?sslmode=disable' \
COORDINATOR_ADDR=':8080' \
COORDINATOR_TOKEN='a-worker-token' \
COORDINATOR_STORAGE_DIR='/var/lib/scimesh/artifacts' \
JWT_SECRET='<same secret>' \
USERSERVICE_URL='http://127.0.0.1:8081' \
PUBLIC_COORDINATOR_URL='http://coordinator.example:8080' \
./coordinator
```
The binary applies its embedded schema migrations on startup
(`AUTO_MIGRATE=false` to disable when you manage them out of band).
4. **Attach workers** as in Option A (or via `worker-agent setup`). The UI
login uses the userservice session; the workers use `COORDINATOR_TOKEN` or
a worker key. The coordinator's admin console (`/ui/admin`) shows the
whole cluster: jobs, worker fleet with trust controls, accounts and keys,
workload switches and metrics.
## Notes
- The coordinator and the userservice each keep their own PostgreSQL database
— different bounded contexts, deliberately not shared.
- A worker can join by hostname or IP; only outbound HTTP from the worker to
the coordinator is required (no inbound firewall rules on workers).
- For a quick all-in-one alternative, `coordinator serve` embeds all of this
on one machine — see the [home page](index.md).
+5 -2
View File
@@ -1,10 +1,10 @@
[build-system]
requires = ["setuptools>=68"]
requires = ["setuptools>=68", "setuptools-scm>=8"]
build-backend = "setuptools.build_meta"
[project]
name = "scimesh"
version = "0.1.0"
dynamic = ["version"]
description = "Local scientific workloads for molecular similarity analysis"
readme = "README.md"
requires-python = ">=3.10"
@@ -23,6 +23,7 @@ scimesh = "scimesh.cli:main"
[project.entry-points."scimesh.workloads"]
"similarity-search@1.0.0" = "scimesh.workloads.search:workload_definition"
"similarity-search-parallel@1.0.0" = "scimesh.workloads.search_parallel:workload_definition"
"similarity-graph@1.0.0" = "scimesh.workloads.graph:workload_definition"
"descriptor-batch@1.0.0" = "scimesh.workloads.descriptors:workload_definition"
"molwt-filter@1.0.0" = "scimesh.workloads.molwt_filter:workload_definition"
@@ -40,3 +41,5 @@ venv = ".venv"
pythonVersion = "3.10"
typeCheckingMode = "basic"
exclude = ["SciMesh", "site", "coordinator"]
[tool.setuptools_scm]
+6
View File
@@ -21,6 +21,7 @@ from .environment import current_environment_digest
from .graph import similarity_graph_sdk_definition
from .molwt_filter import molwt_filter_sdk_definition
from .search import similarity_search_sdk_definition
from .search_parallel import similarity_search_parallel_sdk_definition
__all__ = [
"default_sdk_registry",
@@ -46,6 +47,10 @@ def default_sdk_registry(
similarity_search_sdk_definition(shard_rows=shard_rows).definition(),
enabled=True,
)
registry.register(
similarity_search_parallel_sdk_definition(shard_rows=shard_rows).definition(),
enabled=True,
)
registry.register(
similarity_graph_sdk_definition().definition(),
enabled=True,
@@ -82,6 +87,7 @@ def default_sdk_runtime(
workload_capabilities
or (
"similarity-search",
"similarity-search-parallel",
"similarity-graph",
"descriptor-batch",
"molwt-filter",
@@ -0,0 +1,28 @@
"""SDK-built ``similarity-search-parallel`` workload.
Same contract as ``similarity-search`` with a per-shard thread pool. See
``core.py`` for the parallel scoring core and ``definition.py`` for the
manifest-backed handlers.
"""
from .core import (
run_search_shard_parallel,
search_similar_parallel,
write_search_shards,
)
from .definition import (
MAP_ENTRY_POINT,
SimilaritySearchParallelSDKWorkload,
similarity_search_parallel_sdk_definition,
workload_definition,
)
__all__ = [
"MAP_ENTRY_POINT",
"SimilaritySearchParallelSDKWorkload",
"similarity_search_parallel_sdk_definition",
"workload_definition",
"run_search_shard_parallel",
"search_similar_parallel",
"write_search_shards",
]
+199
View File
@@ -0,0 +1,199 @@
"""Scientific core for the SDK-built ``similarity-search-parallel`` workload.
The exact same semantics as ``similarity-search`` identical partial format,
identical bounded merge, byte-identical output but the per-molecule
fingerprinting and Tanimoto scoring of one shard run across a thread pool
(``threads`` parameter, default = CPU count).
Parallelism is confined to the scoring phase: ``ThreadPoolExecutor.map`` keeps
the input row order, so the results are merged exactly like the sequential
reference (same ``_HeapEntry`` logic), which makes the output byte-identical
for every thread count by construction.
"""
from __future__ import annotations
import heapq
import os
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Mapping
from rdkit import Chem
from rdkit.Chem import DataStructs
from scimesh.chemistry.dataset import MoleculeRecord, parse_smiles
from scimesh.chemistry.fingerprints import fingerprint
from scimesh.workloads.search.core import (
run_search_shard,
write_search_partial,
write_search_shards,
)
from scimesh.workloads.similarity_search import (
DatasetStats,
SearchResult,
SimilarityMatch,
_HeapEntry,
iter_valid_molecules,
)
def search_similar_parallel(
tsv_path: Path,
query: MoleculeRecord,
top_k: int,
*,
threads: int = 0,
max_rows: int | None = None,
threshold: float | None = None,
threshold_direction: str = "greater",
) -> SearchResult:
"""Exact top-k matches with a bounded heap, scored by a thread pool.
Identical selection and ordering to ``search_similar`` for every thread
count: the merge runs in row order over the parallel-computed scores.
"""
if top_k < 1:
raise ValueError("--top-k must be a positive integer")
if threads < 0:
raise ValueError("threads must be a non-negative integer")
if threshold is not None and not 0.0 <= threshold <= 1.0:
raise ValueError("--threshold must be between 0 and 1")
if threshold_direction not in {"greater", "less"}:
raise ValueError("--threshold-direction must be 'greater' or 'less'")
workers = threads or (os.cpu_count() or 1)
query_fingerprint = fingerprint(query.molecule)
query_canonical_smiles = Chem.MolToSmiles(query.molecule, canonical=True)
stats = DatasetStats()
records = list(iter_valid_molecules(tsv_path, stats, max_rows=max_rows))
def score(record: MoleculeRecord):
candidate_smiles = Chem.MolToSmiles(record.molecule, canonical=True)
if (
record.molecule_id == query.molecule_id
or candidate_smiles == query_canonical_smiles
):
return None
similarity = DataStructs.TanimotoSimilarity(
query_fingerprint, fingerprint(record.molecule)
)
if threshold is not None and (
similarity < threshold
if threshold_direction == "greater"
else similarity > threshold
):
return None
return similarity
# map preserves the input order, so the merge below is exactly the
# sequential reference's merge, just over precomputed scores.
with ThreadPoolExecutor(max_workers=workers) as pool:
scored = pool.map(score, records)
heap: list[_HeapEntry] = []
for record, similarity in zip(records, scored):
if similarity is None:
continue
match = SimilarityMatch(similarity, record.molecule_id, record.smiles)
rank_key = match.sort_key(threshold_direction)
entry = _HeapEntry(match, rank_key)
if len(heap) < top_k:
heapq.heappush(heap, entry)
elif rank_key < heap[0].rank_key:
heapq.heapreplace(heap, entry)
matches = [entry.match for entry in sorted(heap, key=lambda e: e.rank_key)]
return SearchResult(matches=matches, stats=stats)
def run_search_shard_parallel(
input_path: Path,
parameters: Mapping[str, object],
output_path: Path,
) -> dict[str, int]:
"""Run one planned shard with the parallel scoring core.
Accepts the same parameters as ``similarity-search`` plus ``threads``.
"""
allowed = {
"query_id",
"query_smiles",
"top_k",
"threshold",
"threshold_direction",
"progress_every",
"threads",
}
unknown = set(parameters) - allowed
if unknown:
raise ValueError(
f"unsupported similarity-search-parallel parameters: {', '.join(sorted(unknown))}"
)
query_smiles = parameters.get("query_smiles")
query_id = parameters.get("query_id")
if isinstance(query_id, str) and not isinstance(query_smiles, str):
from rdkit import Chem
from scimesh.chemistry.dataset import find_molecule_by_id
record = find_molecule_by_id(input_path, query_id)
query_smiles = Chem.MolToSmiles(record.molecule, canonical=True)
if not isinstance(query_smiles, str) or not query_smiles.strip():
raise ValueError("query_smiles is required for a distributed shard")
molecule = parse_smiles(query_smiles)
if molecule is None:
raise ValueError("query_smiles is invalid")
top_k = _positive_int(parameters.get("top_k", 20), "top_k")
threads = _nonnegative_int(parameters.get("threads", 0), "threads")
threshold = None
if "threshold" in parameters:
threshold = _unit_interval(parameters["threshold"], "threshold")
direction = parameters.get("threshold_direction", "greater")
if direction not in {"greater", "less"}:
raise ValueError("threshold_direction must be 'greater' or 'less'")
assert isinstance(direction, str)
result = search_similar_parallel(
input_path,
MoleculeRecord("query", query_smiles, molecule),
top_k=top_k,
threads=threads,
threshold=threshold,
threshold_direction=direction,
)
write_search_partial(output_path, result.matches)
return {
"scanned_rows": result.stats.scanned,
"valid_molecules": result.stats.valid,
"invalid_smiles": result.stats.invalid,
"matches_emitted": len(result.matches),
}
def _positive_int(value: object, name: str) -> int:
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
raise ValueError(f"{name} must be a positive integer")
return value
def _nonnegative_int(value: object, name: str) -> int:
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
raise ValueError(f"{name} must be a non-negative integer")
return value
def _unit_interval(value: object, name: str) -> float:
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ValueError(f"{name} must be a number between 0 and 1")
return float(value)
# The shared shard writer is re-exported so the workload definition can reuse
# the deterministic partitioning without importing search internals.
__all__ = [
"search_similar_parallel",
"run_search_shard_parallel",
"write_search_shards",
"run_search_shard",
]

Some files were not shown because too many files have changed in this diff Show More