vet + lint + race unit tests, then bring up the stack and run the integration
suite and the end-to-end smoke test. One command for a reviewer to verify
everything.
A self-contained handoff for whoever (human or agent) implements the Python
worker: what the coordinator already does, where the contract lives
(openapi.yaml, building-workers.md), how to run it locally, the deliverable
(CTX-06), acceptance criteria, and the do-not-break rules.
- .github/workflows/coordinator.yml: vet, gofmt, race unit tests, golangci-lint,
and the integration suite against a Postgres service — on every push/PR that
touches coordinator/.
- docs/building-workers.md: a from-scratch guide for implementing a worker (the
claim/heartbeat/upload/complete loop, auth, lease semantics, status codes,
and the do-not-break rules), pointing at openapi.yaml for client generation.
- README: refresh the endpoint table, status, and test sections to match reality.
Polish pass hardening the queue and closing plan gaps.
- Task state machine gains `running`: the first heartbeat moves a task from
leased to running (migrations 0006/0007 add the enum value and extend the
lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job
progress all treat leased and running alike.
- Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps
its last_heartbeat_at online; a second background reaper marks workers offline
after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic).
- Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads
capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader.
- Tests cover the running transition, liveness + offline reaper (unit over
memstore and integration over Postgres).
docs/openapi.yaml is a machine-readable mirror of the api-contract, covering
every coordinator endpoint with request/response schemas, auth, and error
codes. The Python author can generate a typed client or Pydantic models from it.
Its 12 paths match the router exactly; all $refs resolve.
- domain: NewJobWithTasks, DeriveStatus, NewUploadedJob, NewShardTask,
NewWorker, NewArtifact/SetContent (domain 47% -> 88%).
- internal/memstore: in-memory implementations of every usecase port, so
orchestration can be tested without Postgres or a filesystem.
- usecase: claim/renew/complete/fail/create/register/upload/submit-dataset
flows over memstore, including rule-10 cross-task rejection, idempotent
replay, lease sweep-on-claim, and dataset chunking (usecase 0% -> 73%).
- transport: httptest end-to-end over real use cases + memstore — auth,
readiness, full lifecycle, multipart upload + shard input, error mappings
(0% -> 70%).
- postgres integration: fix the tests broken by the artifact_id switch and add
worker-repo, artifact-repo, and shard-task (nullable input_uri) round-trips.
go test -race ./... is clean; golangci-lint (incl. integration tag) reports 0.
The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.
- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
splits it into shard artifacts, and creates one shard task per shard, all in
one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
The container runs as uid 10001, but the ./data and ./logs bind mounts were
root-owned, so blob storage failed with "mkdir .staging: permission denied"
and the coordinator crash-looped.
Pre-create the storage and log directories in the image owned by the
coordinator user, and switch the bind mounts to named volumes, which inherit
that ownership. The process can now write to them without running as root.
Task results are now coordinator-owned artifacts end to end.
- domain.Task carries ResultArtifactID instead of ResultURI/ResultSHA256;
CompleteWith and its idempotency key are keyed on the artifact id.
- CompleteTask verifies the referenced artifact was stored for this exact
task (rule 10): a worker cannot finish task B with task A's artifact, nor
name an id that isn't a partial_result. Mismatch → 409.
- POST /tasks/{id}/result takes {result:{artifact_id,...}}; ListResults and
ResultManifest follow.
- migration 0004 drops result_uri/result_sha256 and requires a completed task
to reference its result_artifact_id.
- smoke and requests.http exercise upload → complete-by-id → replay → conflict.
Wire the artifact storage foundation to HTTP.
- PUT /tasks/{id}/artifacts/{filename}: a worker streams a partial result;
the coordinator verifies lease ownership (foreign worker → 409), streams
the bytes to blob storage while hashing, and records the metadata. An
orphaned blob from a failed metadata insert is cleaned up.
- GET /artifacts/{id}/download: streams an artifact back with its content
type, length, and checksum.
- Ownership is read with a new non-locking TaskRepository.Get, so no row lock
is held across a long upload. Identity travels in X-Worker-ID / X-Task-Attempt
headers per the contract; upload/download bypass the short request timeout.
- docker-compose mounts ./data for durable artifact storage; smoke and
requests.http exercise an upload → foreign-409 → download round-trip.
Introduce durable, coordinator-owned artifacts — the model the master plan
requires instead of trusting worker-supplied result URIs.
- migration 0003: artifacts table + artifact_kind enum, plus nullable
input/result_artifact_id columns on jobs and tasks.
- domain.Artifact with NewArtifact/SetContent; storage keys derive from a
fresh UUID, never from a client filename (no path traversal).
- BlobStore and ArtifactRepository ports.
- blob.FSStore: filesystem blob storage that streams while hashing (SHA-256),
fsyncs, and atomically renames into place — a failed upload leaves neither a
committed artifact nor a staging file. Covered by unit tests.
- ArtifactRepo (squirrel) and COORDINATOR_STORAGE_DIR config.
HTTP upload/download handlers and the switch of result submission to
artifact_id come in the next parts.
Replace the positional-placeholder SQL in the repositories with the
Masterminds/squirrel builder, so column lists and $N numbering are no
longer maintained by hand. The optimistic-lock guard on task Update is now
a readable Where(id, version-1) instead of a $13-1 expression.
Two genuinely set-based statements stay as raw SQL on purpose — claimNext
(a FOR UPDATE SKIP LOCKED CTE) and expireLeases (CASE logic in the SET) —
because a builder cannot express them more clearly.
Align the coordinator with the master PLAN.md (CTX-00, CTX-04) and harden
process startup.
- CTX-00: freeze docs/api-contract.md as the v1 source of truth for the
Go coordinator and Python worker.
- CTX-04: worker registry — workers table (migration 0002), domain.Worker,
RegisterWorker use case, WorkerRepository, and POST /workers/register.
- Contract alignment: claim uses `capabilities` (was `workloads`),
COORDINATOR_TOKEN env (WORKER_AUTH_TOKEN kept as fallback), and
GET /health now reports database readiness (503 when the DB is down).
- Logging: logs are teed to stdout and an optional rotated file (LOG_FILE)
via lumberjack, so they survive a container rebuild.
- Startup resilience: the initial DB connection is retried with backoff,
so the coordinator waits for Postgres to boot instead of crash-looping.
Unimplemented endpoints answered {"error":"internal error"} because the
detail-hiding branch covered everything >= 500, and 501 falls in that range.
That is actively misleading: it sent one debugging session looking for a
database fault when the endpoint simply had no implementation yet.
501 now says "not implemented". It leaks nothing — the client already knows the
status code — while genuine 5xx responses keep hiding their details behind a
request id.
Two ways to exercise every endpoint, both living in the repo rather than in a
personal Postman workspace:
- scripts/smoke.sh walks the full lifecycle and asserts each status, exiting
non-zero on the first surprise, so it works in CI as well as by hand;
- api/requests.http drives the same calls from an editor's REST client, with
later requests reusing ids captured from earlier responses. It doubles as
API documentation for the worker author.
The script claims until it sees its own job's chunks instead of assuming an
empty queue: a shared development database usually holds pending tasks from
earlier runs, and it takes the attempt number from the claim response, since a
task requeued after an expired lease comes back with attempt 2 or 3.
Note for whoever extends the validation cases: Go matches JSON field names
case-insensitively, so "worker_ID" is accepted as "worker_id". Only a genuinely
unknown key trips DisallowUnknownFields.
Replaces the repository stubs with real pgx queries, so the queue now works end
to end: a job is split into tasks, leased to workers one at a time, heartbeated,
completed, and reflected in job progress.
Task claiming is a single statement — SELECT ... FOR UPDATE SKIP LOCKED feeding
an UPDATE — so concurrent coordinators lease different rows instead of blocking
on the same one. Writes use optimistic concurrency: the entity increments its
version in memory, and the UPDATE guards on the previous value.
Retries moved to the transaction level. Once Postgres aborts a transaction with
a serialization failure, replaying one statement inside it cannot help; the unit
of retry is Begin -> fn -> Commit, which is safe because each attempt re-reads
its rows through GetForUpdate.
Adds integration tests behind the `integration` build tag, run against a real
PostgreSQL through TEST_DATABASE_URL: concurrent claiming hands each task to
exactly one worker, job creation rolls back whole, stale writes are refused,
completed results keep chunk order, and expired leases return to the queue.
Two bugs the tests caught:
- a nil parameters map reached a NOT NULL jsonb column as SQL NULL, since pgx
sends NULL rather than omitting the column and letting DEFAULT '{}' apply;
- replaying an already-recorded result returned 409. The idempotent path leaves
the entity untouched, so the version guard matched nothing and a successful
no-op looked like a conflict. CompleteTask now skips the write when the
entity did not change.
Adds the SciMesh coordinator: a durable task-queue server on PostgreSQL
that owns all database access, with workers reaching it over HTTP only.
Structured as a modular monolith following Clean Architecture:
domain entities and their invariants, no I/O
usecase business operations + repository/clock ports
transport HTTP handlers, DTOs, auth, error mapping
storage PostgreSQL repositories, transactions carried in context
infra config, pool, clock, server, lease reaper
Dependencies point strictly inward; domain imports nothing from the module.
Working: layer wiring, routing, shared-token auth, access logging, request
IDs, domain-error to status-code mapping, transactional boundaries,
graceful shutdown (HTTP drain -> reaper stop -> pool close), migrations,
and a Compose stack starting Postgres -> migrations -> coordinator.
The domain is complete and covered by unit tests that need no database:
lease ownership, stale attempts, idempotent result replay, retry budgets,
and lease expiry.
Repository methods are stubs returning ErrNotImplemented (HTTP 501). The
SQL for atomic claiming (FOR UPDATE SKIP LOCKED) and for lease expiry is
written and ready to wire up.
See coordinator/ARCHITECTURE.md for the layer map and a request traced
through every layer.
Emil