The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.
- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
splits it into shard artifacts, and creates one shard task per shard, all in
one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
Adds the SciMesh coordinator: a durable task-queue server on PostgreSQL
that owns all database access, with workers reaching it over HTTP only.
Structured as a modular monolith following Clean Architecture:
domain entities and their invariants, no I/O
usecase business operations + repository/clock ports
transport HTTP handlers, DTOs, auth, error mapping
storage PostgreSQL repositories, transactions carried in context
infra config, pool, clock, server, lease reaper
Dependencies point strictly inward; domain imports nothing from the module.
Working: layer wiring, routing, shared-token auth, access logging, request
IDs, domain-error to status-code mapping, transactional boundaries,
graceful shutdown (HTTP drain -> reaper stop -> pool close), migrations,
and a Compose stack starting Postgres -> migrations -> coordinator.
The domain is complete and covered by unit tests that need no database:
lease ownership, stale attempts, idempotent result replay, retry budgets,
and lease expiry.
Repository methods are stubs returning ErrNotImplemented (HTTP 501). The
SQL for atomic claiming (FOR UPDATE SKIP LOCKED) and for lease expiry is
written and ready to wire up.
See coordinator/ARCHITECTURE.md for the layer map and a request traced
through every layer.
Efremenko Arhip