demo-down / stop intentionally keep the Postgres and artifact volumes, so a
fresh start still carried old workers/jobs/tasks. Add a reset action (compose
down -v) and a make demo-reset target for a truly pristine restart.
The Add-your-machine page sent the typed name only to the worker key, while
the copyable command hard-coded --worker-name my-machine, so every self-service
worker registered as my-machine regardless of what the user entered. Thread the
name into the command (shell-quoted; falls back to my-machine when blank).
Let a signed-in user turn their own machine into a worker without the
shared token. The coordinator already binds a JWT-authenticated
registration to owner_id as untrusted; this adds the missing pieces.
userservice: long-lived worker keys (scimesh_wk_live_*, hash-at-rest)
with create/list/revoke and a public /worker-tokens/exchange that trades
a key for a short-lived JWT carrying the owner current role/verified.
python worker: SCIMESH_WORKER_KEY + SCIMESH_USERSERVICE_URL; a token
provider exchanges the key and refreshes the JWT proactively and on 401,
so a long-running worker survives token expiry. Static bearer token path
is unchanged.
coordinator UI: an "add your machine" page that mints a key and shows a
ready-to-run command, proxying key management to the userservice; the
dashboard gains an owner-scoped "my machines" section.
docs: how to run a worker from your account, plus the untrusted/quorum/
verified trust model.
Replaces the C1 quarantine with real verification. Trusted results (lab token,
verified, or admin worker) are accepted directly as before. An untrusted
worker's result is recorded as one vote per (task, owner) in a new task_results
table; the task only completes once QUORUM_SIZE distinct owners submit the same
result hash, otherwise it returns to the queue for another independent compute.
- migration 0013 task_results (one vote per owner, quorum by result_sha256)
- CompleteTask branches on worker trust; unknown worker defaults trusted (safe:
completing needs the lease, whose owner is always a known registered worker)
- claim drops the quarantine and excludes chunks the owner already voted on
- domain Task.ReleaseAfterVote; QUORUM_SIZE config (default 2)
- unit tests: trusted direct-complete, untrusted needs-quorum, can-claim
Reducer and job done/total logic untouched — still one completed task per chunk.
A scrape-time collector reports scimesh_tasks/jobs/workers gauges keyed by
status, sourced from cheap GROUP BY queries (StatsRepo), zero-filled across all
known statuses so the dashboard shows flat zeros instead of gaps. A failed
query yields no samples for that scrape rather than crashing it.
Metrics is now built in main so the DB-backed collector can be registered
(NewServer takes *metrics.Metrics; nil self-provisions for tests). Grafana
dashboard gains a Domain state row: tasks/jobs/workers by status and a queue-
depth stat.
The metrics deps pull in modules whose go directive is 1.25, so go mod tidy
raised the module to go 1.25.0. The build image is bumped golang:1.24 ->
golang:1.25-alpine to match (the image runs GOTOOLCHAIN=local and can't
auto-fetch a newer toolchain). CI reads go-version-file, so it follows along.
client_golang pinned to v1.19.1 (stable, same API).
make demo-ui now also starts Prometheus (scrapes coordinator:8080/metrics) and
Grafana with a provisioned datasource and a SciMesh Coordinator dashboard
(request rate & p95 by route, status mix, goroutines, RSS). Grafana allows
anonymous viewing so the dashboard opens without a login; admin/admin to edit.
- monitoring/prometheus.yml + grafana provisioning + dashboard JSON
- docker-compose.monitoring.yml overlay (third -f in demo-ui.sh)
- demo prints the Grafana and Prometheus URLs
Adds internal/metrics: a private registry with the Go runtime + process
collectors and HTTP RED instrumentation (scimesh_http_requests_total and
request_duration_seconds), labelled by method/status and a normalized route so
per-id paths collapse to {id} and never blow up label cardinality.
/metrics is unauthenticated (like /health) for a Prometheus scraper; the
middleware wraps the whole router so every request is measured once.
Shows the signed-in user's id, email, role, verified status, and created-at by
proxying the session token to the userservice /me endpoint (email/created_at
are not in the JWT). Profile link added to the dashboard, job, and admin headers.
Tests: /me proxy forwards the bearer and renders the account; redirect without a session.
Admin-only page to promote/demote/verify/unverify a user by id. Actions proxy
to the userservice forwarding the admin's session JWT, which the userservice
re-checks (defense in depth). Non-admins are redirected off the panel; the
Admin link shows only for admins. Admins already see all jobs on the dashboard.
Tests: requireAdmin gate, bearer forwarding, action/id validation, admin link.
The dashboard and job pages show a 'Log out' control (POST /ui/logout) and a
'Signed in · <role>' label when a userservice session is active. Under basic
auth (no session) neither appears, so the fallback UI is unchanged.
Threads a template-only Session view (json:"-") from authctx into the
dashboard and job views. Tests assert the control renders only in session mode.
make demo-ui now starts the userservice (own Postgres + migrations) beside the
coordinator on a shared JWT secret, and seeds a root admin. The coordinator runs
in UI session mode, so /ui opens a login page instead of a basic-auth prompt.
- docker-compose.users.yml overlay: userservice stack + coordinator JWT wiring
- demo-ui.sh: waits for the userservice, logs in as the seeded admin to poll the
now session-gated dashboard, and prints the admin credentials
- validated with docker compose config (6 services, merged env)
With a UI session, the dashboard and job pages are scoped to the caller:
- Overview lists only the user's own jobs (admin/basic-auth operator: all)
- JobDetail, artifact download, and preview 404 on another user's job
- scoping keys off authctx: no requester (basic auth) still sees everything,
so the fallback operator UI is unchanged
ListJobs gains an owner filter (SQL WHERE) so paging stays correct per user.
Tests cover Overview scoping and cross-user JobDetail rejection.
When JWT_SECRET + USERSERVICE_URL are set, the operator UI authenticates
through userservice login/registration instead of the static UI_AUTH_TOKEN:
- /ui/login, /ui/register, /ui/logout pages proxy to the userservice
- successful login stores the JWT in an httpOnly, /ui-scoped cookie
- withUISession verifies the cookie locally and stamps the requester
- unset -> falls back to basic auth, so the team's existing flow is unchanged
Tests cover the session gate, cookie set/clear, and the login/register proxy.
The trust tier was read off the caller-supplied worker_id, so a JWT user who
knew any trusted worker's id could claim as it — draining and poisoning the
trusted queue and bypassing the untrusted-worker quarantine entirely.
Claim now requires a JWT caller to own the worker it acts as; a shared-token
caller (lab operator) may still act as any worker. Claim is the sole grantor of
a lease, so this also protects the downstream heartbeat/result/failure paths.
Tests: reject claim as another user's worker; allow claim as own worker.
- migration 0012: workers.owner_id + trust_level (trusted/untrusted)
- verifier/authctx read the JWT verified claim; IsTrusted() = admin||verified
- /workers/register resolves trust from auth: service token or verified/admin
JWT -> trusted; plain user JWT -> untrusted, tagged with owner_id
- claim quarantines untrusted workers (no tasks) until quorum (C2) lands
- unit tests for trust resolution, quarantine, and the verified claim
Additive and backward compatible: shared-token workers stay trusted, so the
existing worker flow and team tests are unchanged. Quorum verification (C2)
is deferred.
jobColumns gained owner_id but ListJobs' Scan still read 12 targets,
so the query returned 13 columns and pgx failed at runtime. Only the
integration tests (real DB) caught it; memstore-backed unit tests did not.
Emil