Efremenko Arhip
f1c3163be4
Replaces the repository stubs with real pgx queries, so the queue now works end
to end: a job is split into tasks, leased to workers one at a time, heartbeated,
completed, and reflected in job progress.
Task claiming is a single statement — SELECT ... FOR UPDATE SKIP LOCKED feeding
an UPDATE — so concurrent coordinators lease different rows instead of blocking
on the same one. Writes use optimistic concurrency: the entity increments its
version in memory, and the UPDATE guards on the previous value.
Retries moved to the transaction level. Once Postgres aborts a transaction with
a serialization failure, replaying one statement inside it cannot help; the unit
of retry is Begin -> fn -> Commit, which is safe because each attempt re-reads
its rows through GetForUpdate.
Adds integration tests behind the `integration` build tag, run against a real
PostgreSQL through TEST_DATABASE_URL: concurrent claiming hands each task to
exactly one worker, job creation rolls back whole, stale writes are refused,
completed results keep chunk order, and expired leases return to the queue.
Two bugs the tests caught:
- a nil parameters map reached a NOT NULL jsonb column as SQL NULL, since pgx
sends NULL rather than omitting the column and letting DEFAULT '{}' apply;
- replaying an already-recorded result returned 409. The idempotent path leaves
the entity untouched, so the version guard matched nothing and a successful
no-op looked like a conflict. CompleteTask now skips the write when the
entity did not change.
145 lines
6.2 KiB
Markdown
145 lines
6.2 KiB
Markdown
# SciMesh Coordinator
|
|
|
|
Durable task-queue server for SciMesh, in Go on PostgreSQL. It owns all database
|
|
access; workers talk to it only over HTTP and never receive DB credentials.
|
|
|
|
Built as a **modular monolith following Clean Architecture** — one binary, four
|
|
layers, dependencies pointing strictly inward. See
|
|
`docs/database-integration-task.md` and `docs/worker-daemon-task.md` in the repo
|
|
root for the full contract.
|
|
|
|
## Layers
|
|
|
|
```
|
|
infra config, pgxpool, http.Server, clock ← frameworks & drivers
|
|
transport http handlers ← inbound: who calls us
|
|
storage sql repositories ← outbound: who we call
|
|
usecase business operations + PORTS ← application rules
|
|
domain Task, Job + their invariants ← enterprise rules
|
|
|
|
┌── transport ──┐
|
|
domain ◄── usecase ◄┤ ├◄── infra
|
|
└── storage ────┘
|
|
```
|
|
|
|
`transport` and `storage` are one layer — the "interface adapters" ring — split
|
|
by direction rather than by category, so a file's path tells you its role.
|
|
|
|
The rule that matters: **source dependencies point only inward**. `domain`
|
|
imports nothing from this module; `usecase` sees only `domain`; `transport` and
|
|
`storage` know nothing of each other. Verify it at any time with:
|
|
|
|
```sh
|
|
go list -f '{{range .Imports}}{{.}}{{"\n"}}{{end}}' ./internal/domain | grep internal # must be empty
|
|
```
|
|
|
|
## Layout
|
|
|
|
```
|
|
coordinator/
|
|
cmd/coordinator/main.go # composition root: the only place with concrete types
|
|
internal/
|
|
domain/ # entities + rules, no I/O
|
|
task.go Task, lease/complete/fail/expire transitions
|
|
job.go Job, chunk fan-out, status derivation
|
|
errors.go business-rule violations
|
|
usecase/ # one type per operation, dependencies injected
|
|
ports.go TaskRepository, JobRepository, TxManager, Clock
|
|
dto.go use-case boundary inputs
|
|
task.go claim, renew, complete, fail, expire
|
|
job.go create, status, results, stitch
|
|
transport/http/ # routing, DTOs, middleware, error mapping
|
|
storage/postgres/ # SQL behind the ports; TxManager via context
|
|
infra/ # config.go db.go clock.go server.go
|
|
migrations/ # golang-migrate SQL, run as an explicit command
|
|
```
|
|
|
|
A full map — file-by-file table, a request traced through every layer, and a
|
|
"where do I add X" guide — lives in [ARCHITECTURE.md](ARCHITECTURE.md).
|
|
|
|
## Quickstart
|
|
|
|
### With Docker (nothing to install but Docker)
|
|
|
|
```sh
|
|
make up # Postgres → migrations → coordinator
|
|
curl localhost:8080/health
|
|
make logs # follow the coordinator
|
|
make down # stop (add down-clean to drop the DB volume)
|
|
```
|
|
|
|
`up` starts three services in order: Postgres waits until `pg_isready` passes, a
|
|
one-shot `migrate` container applies the schema and exits, and only then does the
|
|
coordinator start — so it never queries a database that has no tables.
|
|
|
|
> **Needs BuildKit.** The Dockerfile uses `RUN --mount=type=cache` to reuse the
|
|
> Go module and compiler caches between builds. If the build fails with
|
|
> *"the --mount option requires BuildKit"*, install the buildx plugin —
|
|
> `pacman -S docker-buildx` on Arch, `apt install docker-buildx-plugin` on Debian.
|
|
|
|
### Locally, against your own Postgres
|
|
|
|
```sh
|
|
cp .env.example .env # then edit DATABASE_URL / WORKER_AUTH_TOKEN
|
|
# it is loaded automatically — no export needed
|
|
|
|
make tidy # fetch deps (needs network once)
|
|
make migrate-up # apply schema (needs the migrate CLI)
|
|
make run # start the server
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Settings come from the environment. A `.env` file is loaded at startup via
|
|
`godotenv` as a local-dev convenience (override its path with `ENV_FILE`):
|
|
|
|
- a missing `.env` is not an error — production injects real env vars;
|
|
- **real environment variables always win** over the file, so an orchestrator's
|
|
values are never shadowed by a stale `.env` baked into an image.
|
|
|
|
See `.env.example`; only `DATABASE_URL` is required.
|
|
|
|
## Endpoints
|
|
|
|
| Method | Path | Purpose |
|
|
| ------ | ----------------------------- | ------------------------------------------ |
|
|
| POST | `/jobs` | Create job + pending tasks transactionally |
|
|
| POST | `/tasks/claim` | Atomically lease one task (`204` if none) |
|
|
| POST | `/tasks/{task_id}/heartbeat` | Renew the caller's lease |
|
|
| POST | `/tasks/{task_id}/result` | Record a completed result (idempotent) |
|
|
| POST | `/tasks/{task_id}/failure` | Record failure / retryable state |
|
|
| GET | `/jobs/{job_id}` | Aggregate job progress |
|
|
| GET | `/health` | Liveness (unauthenticated) |
|
|
|
|
## Status
|
|
|
|
The queue works end to end: a job can be submitted, split into tasks, leased to
|
|
workers one at a time, heartbeated, completed, and reflected in job progress.
|
|
|
|
Roadmap:
|
|
|
|
1. schema + migrations ✅
|
|
2. `ClaimNext`, `InsertBatch` — atomic claim via `FOR UPDATE SKIP LOCKED` ✅
|
|
3. `GetForUpdate`, `Update`, `CountByStatus` — result/failure paths ✅
|
|
4. file upload / chunk download — **next**
|
|
5. `ExpireLeases` ✅ (reaper + a sweep before every claim)
|
|
6. stitcher: merge per-chunk top-k into the final CSV
|
|
7. more integration coverage as features land
|
|
|
|
Still stubbed: `StitchJob.Execute`, and there is no `POST /upload` or
|
|
`GET /download_chunk` yet — so chunk files must be referenced by URI for now.
|
|
|
|
## Tests
|
|
|
|
`internal/domain` is covered by unit tests that need **no database** — lease
|
|
ownership, stale attempts, idempotent replays, retry budgets, and expiry are all
|
|
pure functions of entity state:
|
|
|
|
```sh
|
|
go test ./...
|
|
go vet ./...
|
|
```
|
|
|
|
Integration tests (concurrent claiming, migrations) come in phase 7 and require
|
|
a real PostgreSQL instance supplied through `TEST_DATABASE_URL`.
|