Commit Graph
140 Commits
Author SHA1 Message Date
Emil b4fe7aaf72 Publish pre-release tags as GitHub pre-releases 2026-08-02 21:01:32 +03:00
Emil e308085fe2 Mark the session goal as completed 2026-08-02 21:00:10 +03:00
Emil 988101afab Track session goal progress 2026-08-02 20:06:17 +03:00
Emil 215325a9e0 Document the single-binary platform and update the roadmap 2026-08-02 20:06:08 +03:00
Emil c06d8673ce Add one-command installers and serve/agent make targets 2026-08-02 20:06:08 +03:00
Emil 1473bbe2a8 Embed the userservice and add serve/agent subcommands for one-binary operation 2026-08-02 19:49:54 +03:00
Emil 9883def0c2 Add an embedded SQLite storage backend for single-binary deployments 2026-08-02 19:23:44 +03:00
Emil 079eca071e Update the README for releases, setup wizard, and the workload-agnostic UI 2026-08-02 18:57:25 +03:00
Emil 8c5350abdf Filter release artifacts to binaries in the download step
coordinator / test (push) Canceled after 0s
python / test (push) Canceled after 0s
release / binaries (amd64, darwin) (push) Canceled after 0s
release / binaries (amd64, linux) (push) Canceled after 0s
release / binaries (amd64, windows) (push) Canceled after 0s
release / binaries (arm64, darwin) (push) Canceled after 0s
release / binaries (arm64, linux) (push) Canceled after 0s
release / binaries (arm64, windows) (push) Canceled after 0s
release / image (push) Canceled after 0s
users / test (push) Canceled after 0s
release / release (push) Canceled after 0s
v1.0.0
2026-08-02 18:48:57 +03:00
Emil 9fe81bc531 Use golang-migrate-compatible watermark tracking in migrations 2026-08-02 18:37:15 +03:00
Emil b281d5811a Fix migration path in the coordinator CI workflow 2026-08-02 18:32:20 +03:00
Emil 771952e22e Fix golangci-lint findings across the coordinator and agent 2026-08-02 18:23:43 +03:00
Emil 565d4466e4 Run gofmt on the setup wizard and migration test 2026-08-02 18:16:44 +03:00
Emil 59e7fb0155 Add the coordinator setup wizard for self-provisioned databases 2026-08-02 18:14:36 +03:00
Emil 320f52615e Embed schema migrations and self-provision the schema on startup 2026-08-02 18:04:00 +03:00
Emil f2977a990e Add versioned release builds for coordinator and worker 2026-08-02 17:54:06 +03:00
Emil 5665e7df98 Document workload-declared UI and the upload contract 2026-08-02 17:47:52 +03:00
Emil 361eb2e344 Advertise all catalog workloads from the worker by default 2026-08-02 17:47:52 +03:00
Emil 749396da05 Drive coordinator upload and reduction from the workload catalog 2026-08-02 17:47:47 +03:00
Emil 700a96a259 Load every enabled built-in workload in the worker by default 2026-08-02 17:47:47 +03:00
Emil 5d738e0a14 Add SDK-declared workload UI elements and reduction metadata 2026-08-02 17:47:43 +03:00
Emil a18b8b8ae4 Build the coordinator server as a static binary 2026-08-02 16:56:06 +03:00
Emil 706bc85e17 Replace Python worker daemon with the Go worker agent 2026-08-02 16:53:04 +03:00
Emil 9a8221163a Remove dead code and wire the two-worker smoke script 2026-08-02 16:36:05 +03:00
Emil 644c287002 Add Go worker agent prototype 2026-08-02 16:27:58 +03:00
Emil f20cc7fe00 Serve documentation from the operator UI 2026-08-02 15:50:07 +03:00
Emil 284aef5d6f Set up MkDocs Material documentation site 2026-08-02 01:22:45 +03:00
Emil f059ac626c Add workload library page to the operator UI 2026-08-02 01:18:03 +03:00
Emil 5c5a2af0a1 Add molwt-filter workload with default scaffold hooks 2026-08-02 01:09:13 +03:00
Emil bc76f386e5 Add MapReduceWorkload scaffold and generic workload execution 2026-08-02 01:02:24 +03:00
Emil 19fbb8e926 Replace legacy distributed protocol with SDK-built workloads 2026-08-01 23:57:41 +03:00
Emil 96169086f0 Add descriptor-batch SDK reference workload 2026-08-01 23:27:53 +03:00
Emil c43af32495 Add workload SDK foundation 2026-08-01 23:22:20 +03:00
Emil 11e9333033 Define generalized workload SDK contract 2026-08-01 16:30:25 +03:00
Emil 0a759a3f01 Update workload SDK roadmap 2026-08-01 16:17:19 +03:00
Emil b9a975b0ea Merge self-service worker enrollment 2026-07-27 22:28:04 +03:00
Emil a201dd5ef9 Reject stale sessions before worker key creation
users / test (push) Canceled after 0s
2026-07-27 22:28:02 +03:00
Emil dc75411907 Record user service integration 2026-07-27 22:23:34 +03:00
Emil 6cdc115d60 Merge user service and coordinator integration 2026-07-27 22:23:21 +03:00
Emil fa76133efc Secure user worker operations
coordinator / test (push) Canceled after 0s
users / test (push) Canceled after 0s
2026-07-27 22:23:08 +03:00
Emil 7d8998408c Merge branch 'main' into feat/users 2026-07-27 22:19:43 +03:00
Efremenko Arhip e2a57175a0 fix(users): use errors.Is in worker key name test (errorlint) 2026-07-27 16:57:49 +03:00
Efremenko Arhip 2991ed202b feat(demo): add demo-reset to wipe the demo's data volumes
demo-down / stop intentionally keep the Postgres and artifact volumes, so a
fresh start still carried old workers/jobs/tasks. Add a reset action (compose
down -v) and a make demo-reset target for a truly pristine restart.
2026-07-27 16:48:06 +03:00
Efremenko Arhip 9b235282fc fix(coordinator/ui): use the entered machine name in the generated worker command
The Add-your-machine page sent the typed name only to the worker key, while
the copyable command hard-coded --worker-name my-machine, so every self-service
worker registered as my-machine regardless of what the user entered. Thread the
name into the command (shell-quoted; falls back to my-machine when blank).
2026-07-27 16:48:06 +03:00
Efremenko Arhip 3a1461315f feat: self-service worker enrollment bound to a user account
Let a signed-in user turn their own machine into a worker without the
shared token. The coordinator already binds a JWT-authenticated
registration to owner_id as untrusted; this adds the missing pieces.

userservice: long-lived worker keys (scimesh_wk_live_*, hash-at-rest)
with create/list/revoke and a public /worker-tokens/exchange that trades
a key for a short-lived JWT carrying the owner current role/verified.

python worker: SCIMESH_WORKER_KEY + SCIMESH_USERSERVICE_URL; a token
provider exchanges the key and refreshes the JWT proactively and on 401,
so a long-running worker survives token expiry. Static bearer token path
is unchanged.

coordinator UI: an "add your machine" page that mints a key and shows a
ready-to-run command, proxying key management to the userservice; the
dashboard gains an owner-scoped "my machines" section.

docs: how to run a worker from your account, plus the untrusted/quorum/
verified trust model.
2026-07-27 16:11:07 +03:00
Efremenko Arhip 172ff76fb8 fix(test): update NewCompleteTask call to new signature
Pass workers/results repos and quorum in the postgres integration
test, matching the constructor change that added quorum voting.
2026-07-27 11:29:52 +03:00
Efremenko Arhip 6f14eeb32e fix(coordinator): silence nilerr on unresolvable-worker trust fallback 2026-07-27 11:05:36 +03:00
Emil 87a483c2fb Plan user service integration 2026-07-27 01:39:26 +03:00
Efremenko Arhip 18d58cce84 feat(coordinator): quorum verification for untrusted (volunteer) results
Replaces the C1 quarantine with real verification. Trusted results (lab token,
verified, or admin worker) are accepted directly as before. An untrusted
worker's result is recorded as one vote per (task, owner) in a new task_results
table; the task only completes once QUORUM_SIZE distinct owners submit the same
result hash, otherwise it returns to the queue for another independent compute.

- migration 0013 task_results (one vote per owner, quorum by result_sha256)
- CompleteTask branches on worker trust; unknown worker defaults trusted (safe:
  completing needs the lease, whose owner is always a known registered worker)
- claim drops the quarantine and excludes chunks the owner already voted on
- domain Task.ReleaseAfterVote; QUORUM_SIZE config (default 2)
- unit tests: trusted direct-complete, untrusted needs-quorum, can-claim

Reducer and job done/total logic untouched — still one completed task per chunk.
2026-07-26 22:55:51 +03:00
Efremenko Arhip dcabfcd0c3 feat(coordinator): business metrics — jobs/tasks/workers by status
A scrape-time collector reports scimesh_tasks/jobs/workers gauges keyed by
status, sourced from cheap GROUP BY queries (StatsRepo), zero-filled across all
known statuses so the dashboard shows flat zeros instead of gaps. A failed
query yields no samples for that scrape rather than crashing it.

Metrics is now built in main so the DB-backed collector can be registered
(NewServer takes *metrics.Metrics; nil self-provisions for tests). Grafana
dashboard gains a Domain state row: tasks/jobs/workers by status and a queue-
depth stat.
2026-07-26 22:16:35 +03:00