demo-down / stop intentionally keep the Postgres and artifact volumes, so a
fresh start still carried old workers/jobs/tasks. Add a reset action (compose
down -v) and a make demo-reset target for a truly pristine restart.
make demo-ui now starts the userservice (own Postgres + migrations) beside the
coordinator on a shared JWT secret, and seeds a root admin. The coordinator runs
in UI session mode, so /ui opens a login page instead of a basic-auth prompt.
- docker-compose.users.yml overlay: userservice stack + coordinator JWT wiring
- demo-ui.sh: waits for the userservice, logs in as the seeded admin to poll the
now session-gated dashboard, and prints the admin credentials
- validated with docker compose config (6 services, merged env)
vet + lint + race unit tests, then bring up the stack and run the integration
suite and the end-to-end smoke test. One command for a reviewer to verify
everything.
Two ways to exercise every endpoint, both living in the repo rather than in a
personal Postman workspace:
- scripts/smoke.sh walks the full lifecycle and asserts each status, exiting
non-zero on the first surprise, so it works in CI as well as by hand;
- api/requests.http drives the same calls from an editor's REST client, with
later requests reusing ids captured from earlier responses. It doubles as
API documentation for the worker author.
The script claims until it sees its own job's chunks instead of assuming an
empty queue: a shared development database usually holds pending tasks from
earlier runs, and it takes the attempt number from the claim response, since a
task requeued after an expired lease comes back with attempt 2 or 3.
Note for whoever extends the validation cases: Go matches JSON field names
case-insensitively, so "worker_ID" is accepted as "worker_id". Only a genuinely
unknown key trips DisallowUnknownFields.
Replaces the repository stubs with real pgx queries, so the queue now works end
to end: a job is split into tasks, leased to workers one at a time, heartbeated,
completed, and reflected in job progress.
Task claiming is a single statement — SELECT ... FOR UPDATE SKIP LOCKED feeding
an UPDATE — so concurrent coordinators lease different rows instead of blocking
on the same one. Writes use optimistic concurrency: the entity increments its
version in memory, and the UPDATE guards on the previous value.
Retries moved to the transaction level. Once Postgres aborts a transaction with
a serialization failure, replaying one statement inside it cannot help; the unit
of retry is Begin -> fn -> Commit, which is safe because each attempt re-reads
its rows through GetForUpdate.
Adds integration tests behind the `integration` build tag, run against a real
PostgreSQL through TEST_DATABASE_URL: concurrent claiming hands each task to
exactly one worker, job creation rolls back whole, stale writes are refused,
completed results keep chunk order, and expired leases return to the queue.
Two bugs the tests caught:
- a nil parameters map reached a NOT NULL jsonb column as SQL NULL, since pgx
sends NULL rather than omitting the column and letting DEFAULT '{}' apply;
- replaying an already-recorded result returned 409. The idempotent path leaves
the entity untouched, so the version guard matched nothing and a successful
no-op looked like a conflict. CompleteTask now skips the write when the
entity did not change.
Adds the SciMesh coordinator: a durable task-queue server on PostgreSQL
that owns all database access, with workers reaching it over HTTP only.
Structured as a modular monolith following Clean Architecture:
domain entities and their invariants, no I/O
usecase business operations + repository/clock ports
transport HTTP handlers, DTOs, auth, error mapping
storage PostgreSQL repositories, transactions carried in context
infra config, pool, clock, server, lease reaper
Dependencies point strictly inward; domain imports nothing from the module.
Working: layer wiring, routing, shared-token auth, access logging, request
IDs, domain-error to status-code mapping, transactional boundaries,
graceful shutdown (HTTP drain -> reaper stop -> pool close), migrations,
and a Compose stack starting Postgres -> migrations -> coordinator.
The domain is complete and covered by unit tests that need no database:
lease ownership, stale attempts, idempotent result replay, retry budgets,
and lease expiry.
Repository methods are stubs returning ErrNotImplemented (HTTP 501). The
SQL for atomic claiming (FOR UPDATE SKIP LOCKED) and for lease expiry is
written and ready to wire up.
See coordinator/ARCHITECTURE.md for the layer map and a request traced
through every layer.
Emil