Replaces the C1 quarantine with real verification. Trusted results (lab token,
verified, or admin worker) are accepted directly as before. An untrusted
worker's result is recorded as one vote per (task, owner) in a new task_results
table; the task only completes once QUORUM_SIZE distinct owners submit the same
result hash, otherwise it returns to the queue for another independent compute.
- migration 0013 task_results (one vote per owner, quorum by result_sha256)
- CompleteTask branches on worker trust; unknown worker defaults trusted (safe:
completing needs the lease, whose owner is always a known registered worker)
- claim drops the quarantine and excludes chunks the owner already voted on
- domain Task.ReleaseAfterVote; QUORUM_SIZE config (default 2)
- unit tests: trusted direct-complete, untrusted needs-quorum, can-claim
Reducer and job done/total logic untouched — still one completed task per chunk.
The trust tier was read off the caller-supplied worker_id, so a JWT user who
knew any trusted worker's id could claim as it — draining and poisoning the
trusted queue and bypassing the untrusted-worker quarantine entirely.
Claim now requires a JWT caller to own the worker it acts as; a shared-token
caller (lab operator) may still act as any worker. Claim is the sole grantor of
a lease, so this also protects the downstream heartbeat/result/failure paths.
Tests: reject claim as another user's worker; allow claim as own worker.
- migration 0012: workers.owner_id + trust_level (trusted/untrusted)
- verifier/authctx read the JWT verified claim; IsTrusted() = admin||verified
- /workers/register resolves trust from auth: service token or verified/admin
JWT -> trusted; plain user JWT -> untrusted, tagged with owner_id
- claim quarantines untrusted workers (no tasks) until quorum (C2) lands
- unit tests for trust resolution, quarantine, and the verified claim
Additive and backward compatible: shared-token workers stay trusted, so the
existing worker flow and team tests are unchanged. Quorum verification (C2)
is deferred.
Polish pass hardening the queue and closing plan gaps.
- Task state machine gains `running`: the first heartbeat moves a task from
leased to running (migrations 0006/0007 add the enum value and extend the
lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job
progress all treat leased and running alike.
- Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps
its last_heartbeat_at online; a second background reaper marks workers offline
after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic).
- Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads
capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader.
- Tests cover the running transition, liveness + offline reaper (unit over
memstore and integration over Postgres).
Task results are now coordinator-owned artifacts end to end.
- domain.Task carries ResultArtifactID instead of ResultURI/ResultSHA256;
CompleteWith and its idempotency key are keyed on the artifact id.
- CompleteTask verifies the referenced artifact was stored for this exact
task (rule 10): a worker cannot finish task B with task A's artifact, nor
name an id that isn't a partial_result. Mismatch → 409.
- POST /tasks/{id}/result takes {result:{artifact_id,...}}; ListResults and
ResultManifest follow.
- migration 0004 drops result_uri/result_sha256 and requires a completed task
to reference its result_artifact_id.
- smoke and requests.http exercise upload → complete-by-id → replay → conflict.
Replaces the repository stubs with real pgx queries, so the queue now works end
to end: a job is split into tasks, leased to workers one at a time, heartbeated,
completed, and reflected in job progress.
Task claiming is a single statement — SELECT ... FOR UPDATE SKIP LOCKED feeding
an UPDATE — so concurrent coordinators lease different rows instead of blocking
on the same one. Writes use optimistic concurrency: the entity increments its
version in memory, and the UPDATE guards on the previous value.
Retries moved to the transaction level. Once Postgres aborts a transaction with
a serialization failure, replaying one statement inside it cannot help; the unit
of retry is Begin -> fn -> Commit, which is safe because each attempt re-reads
its rows through GetForUpdate.
Adds integration tests behind the `integration` build tag, run against a real
PostgreSQL through TEST_DATABASE_URL: concurrent claiming hands each task to
exactly one worker, job creation rolls back whole, stale writes are refused,
completed results keep chunk order, and expired leases return to the queue.
Two bugs the tests caught:
- a nil parameters map reached a NOT NULL jsonb column as SQL NULL, since pgx
sends NULL rather than omitting the column and letting DEFAULT '{}' apply;
- replaying an already-recorded result returned 409. The idempotent path leaves
the entity untouched, so the version guard matched nothing and a successful
no-op looked like a conflict. CompleteTask now skips the write when the
entity did not change.
Adds the SciMesh coordinator: a durable task-queue server on PostgreSQL
that owns all database access, with workers reaching it over HTTP only.
Structured as a modular monolith following Clean Architecture:
domain entities and their invariants, no I/O
usecase business operations + repository/clock ports
transport HTTP handlers, DTOs, auth, error mapping
storage PostgreSQL repositories, transactions carried in context
infra config, pool, clock, server, lease reaper
Dependencies point strictly inward; domain imports nothing from the module.
Working: layer wiring, routing, shared-token auth, access logging, request
IDs, domain-error to status-code mapping, transactional boundaries,
graceful shutdown (HTTP drain -> reaper stop -> pool close), migrations,
and a Compose stack starting Postgres -> migrations -> coordinator.
The domain is complete and covered by unit tests that need no database:
lease ownership, stale attempts, idempotent result replay, retry budgets,
and lease expiry.
Repository methods are stubs returning ErrNotImplemented (HTTP 501). The
SQL for atomic claiming (FOR UPDATE SKIP LOCKED) and for lease expiry is
written and ready to wire up.
See coordinator/ARCHITECTURE.md for the layer map and a request traced
through every layer.
Emil