Polish pass hardening the queue and closing plan gaps.
- Task state machine gains `running`: the first heartbeat moves a task from
leased to running (migrations 0006/0007 add the enum value and extend the
lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job
progress all treat leased and running alike.
- Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps
its last_heartbeat_at online; a second background reaper marks workers offline
after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic).
- Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads
capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader.
- Tests cover the running transition, liveness + offline reaper (unit over
memstore and integration over Postgres).
Wire the artifact storage foundation to HTTP.
- PUT /tasks/{id}/artifacts/{filename}: a worker streams a partial result;
the coordinator verifies lease ownership (foreign worker → 409), streams
the bytes to blob storage while hashing, and records the metadata. An
orphaned blob from a failed metadata insert is cleaned up.
- GET /artifacts/{id}/download: streams an artifact back with its content
type, length, and checksum.
- Ownership is read with a new non-locking TaskRepository.Get, so no row lock
is held across a long upload. Identity travels in X-Worker-ID / X-Task-Attempt
headers per the contract; upload/download bypass the short request timeout.
- docker-compose mounts ./data for durable artifact storage; smoke and
requests.http exercise an upload → foreign-409 → download round-trip.
Align the coordinator with the master PLAN.md (CTX-00, CTX-04) and harden
process startup.
- CTX-00: freeze docs/api-contract.md as the v1 source of truth for the
Go coordinator and Python worker.
- CTX-04: worker registry — workers table (migration 0002), domain.Worker,
RegisterWorker use case, WorkerRepository, and POST /workers/register.
- Contract alignment: claim uses `capabilities` (was `workloads`),
COORDINATOR_TOKEN env (WORKER_AUTH_TOKEN kept as fallback), and
GET /health now reports database readiness (503 when the DB is down).
- Logging: logs are teed to stdout and an optional rotated file (LOG_FILE)
via lumberjack, so they survive a container rebuild.
- Startup resilience: the initial DB connection is retried with backoff,
so the coordinator waits for Postgres to boot instead of crash-looping.
Adds the SciMesh coordinator: a durable task-queue server on PostgreSQL
that owns all database access, with workers reaching it over HTTP only.
Structured as a modular monolith following Clean Architecture:
domain entities and their invariants, no I/O
usecase business operations + repository/clock ports
transport HTTP handlers, DTOs, auth, error mapping
storage PostgreSQL repositories, transactions carried in context
infra config, pool, clock, server, lease reaper
Dependencies point strictly inward; domain imports nothing from the module.
Working: layer wiring, routing, shared-token auth, access logging, request
IDs, domain-error to status-code mapping, transactional boundaries,
graceful shutdown (HTTP drain -> reaper stop -> pool close), migrations,
and a Compose stack starting Postgres -> migrations -> coordinator.
The domain is complete and covered by unit tests that need no database:
lease ownership, stale attempts, idempotent result replay, retry budgets,
and lease expiry.
Repository methods are stubs returning ErrNotImplemented (HTTP 501). The
SQL for atomic claiming (FOR UPDATE SKIP LOCKED) and for lease expiry is
written and ready to wire up.
See coordinator/ARCHITECTURE.md for the layer map and a request traced
through every layer.
Emil