- delete api/requests.http and ARCHITECTURE.md (coordinator content)
- rewrite README.md and .env.example for the userservice
- drop dead RunPeriodic (reaper machinery userservice has no use for)
- fix .gitignore/.dockerignore/.golangci.yml module + artifact names
- degeneralize stale copied comments that said "coordinator"
Mirrors the coordinator workflow against a scimesh_users Postgres service.
Switches the test request helper to http.NewRequestWithContext so the noctx
linter passes on the go1.22 module.
jobColumns gained owner_id but ListJobs' Scan still read 12 targets,
so the query returned 13 columns and pgx failed at runtime. Only the
integration tests (real DB) caught it; memstore-backed unit tests did not.
vet + lint + race unit tests, then bring up the stack and run the integration
suite and the end-to-end smoke test. One command for a reviewer to verify
everything.
A self-contained handoff for whoever (human or agent) implements the Python
worker: what the coordinator already does, where the contract lives
(openapi.yaml, building-workers.md), how to run it locally, the deliverable
(CTX-06), acceptance criteria, and the do-not-break rules.
- .github/workflows/coordinator.yml: vet, gofmt, race unit tests, golangci-lint,
and the integration suite against a Postgres service — on every push/PR that
touches coordinator/.
- docs/building-workers.md: a from-scratch guide for implementing a worker (the
claim/heartbeat/upload/complete loop, auth, lease semantics, status codes,
and the do-not-break rules), pointing at openapi.yaml for client generation.
- README: refresh the endpoint table, status, and test sections to match reality.
Polish pass hardening the queue and closing plan gaps.
- Task state machine gains `running`: the first heartbeat moves a task from
leased to running (migrations 0006/0007 add the enum value and extend the
lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job
progress all treat leased and running alike.
- Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps
its last_heartbeat_at online; a second background reaper marks workers offline
after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic).
- Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads
capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader.
- Tests cover the running transition, liveness + offline reaper (unit over
memstore and integration over Postgres).
docs/openapi.yaml is a machine-readable mirror of the api-contract, covering
every coordinator endpoint with request/response schemas, auth, and error
codes. The Python author can generate a typed client or Pydantic models from it.
Its 12 paths match the router exactly; all $refs resolve.
- domain: NewJobWithTasks, DeriveStatus, NewUploadedJob, NewShardTask,
NewWorker, NewArtifact/SetContent (domain 47% -> 88%).
- internal/memstore: in-memory implementations of every usecase port, so
orchestration can be tested without Postgres or a filesystem.
- usecase: claim/renew/complete/fail/create/register/upload/submit-dataset
flows over memstore, including rule-10 cross-task rejection, idempotent
replay, lease sweep-on-claim, and dataset chunking (usecase 0% -> 73%).
- transport: httptest end-to-end over real use cases + memstore — auth,
readiness, full lifecycle, multipart upload + shard input, error mappings
(0% -> 70%).
- postgres integration: fix the tests broken by the artifact_id switch and add
worker-repo, artifact-repo, and shard-task (nullable input_uri) round-trips.
go test -race ./... is clean; golangci-lint (incl. integration tag) reports 0.
The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.
- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
splits it into shard artifacts, and creates one shard task per shard, all in
one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
The container runs as uid 10001, but the ./data and ./logs bind mounts were
root-owned, so blob storage failed with "mkdir .staging: permission denied"
and the coordinator crash-looped.
Pre-create the storage and log directories in the image owned by the
coordinator user, and switch the bind mounts to named volumes, which inherit
that ownership. The process can now write to them without running as root.
Efremenko Arhip