make demo-ui now also starts Prometheus (scrapes coordinator:8080/metrics) and
Grafana with a provisioned datasource and a SciMesh Coordinator dashboard
(request rate & p95 by route, status mix, goroutines, RSS). Grafana allows
anonymous viewing so the dashboard opens without a login; admin/admin to edit.
- monitoring/prometheus.yml + grafana provisioning + dashboard JSON
- docker-compose.monitoring.yml overlay (third -f in demo-ui.sh)
- demo prints the Grafana and Prometheus URLs
make demo-ui now starts the userservice (own Postgres + migrations) beside the
coordinator on a shared JWT secret, and seeds a root admin. The coordinator runs
in UI session mode, so /ui opens a login page instead of a basic-auth prompt.
- docker-compose.users.yml overlay: userservice stack + coordinator JWT wiring
- demo-ui.sh: waits for the userservice, logs in as the seeded admin to poll the
now session-gated dashboard, and prints the admin credentials
- validated with docker compose config (6 services, merged env)
The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.
- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
splits it into shard artifacts, and creates one shard task per shard, all in
one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
Task results are now coordinator-owned artifacts end to end.
- domain.Task carries ResultArtifactID instead of ResultURI/ResultSHA256;
CompleteWith and its idempotency key are keyed on the artifact id.
- CompleteTask verifies the referenced artifact was stored for this exact
task (rule 10): a worker cannot finish task B with task A's artifact, nor
name an id that isn't a partial_result. Mismatch → 409.
- POST /tasks/{id}/result takes {result:{artifact_id,...}}; ListResults and
ResultManifest follow.
- migration 0004 drops result_uri/result_sha256 and requires a completed task
to reference its result_artifact_id.
- smoke and requests.http exercise upload → complete-by-id → replay → conflict.
Wire the artifact storage foundation to HTTP.
- PUT /tasks/{id}/artifacts/{filename}: a worker streams a partial result;
the coordinator verifies lease ownership (foreign worker → 409), streams
the bytes to blob storage while hashing, and records the metadata. An
orphaned blob from a failed metadata insert is cleaned up.
- GET /artifacts/{id}/download: streams an artifact back with its content
type, length, and checksum.
- Ownership is read with a new non-locking TaskRepository.Get, so no row lock
is held across a long upload. Identity travels in X-Worker-ID / X-Task-Attempt
headers per the contract; upload/download bypass the short request timeout.
- docker-compose mounts ./data for durable artifact storage; smoke and
requests.http exercise an upload → foreign-409 → download round-trip.
Align the coordinator with the master PLAN.md (CTX-00, CTX-04) and harden
process startup.
- CTX-00: freeze docs/api-contract.md as the v1 source of truth for the
Go coordinator and Python worker.
- CTX-04: worker registry — workers table (migration 0002), domain.Worker,
RegisterWorker use case, WorkerRepository, and POST /workers/register.
- Contract alignment: claim uses `capabilities` (was `workloads`),
COORDINATOR_TOKEN env (WORKER_AUTH_TOKEN kept as fallback), and
GET /health now reports database readiness (503 when the DB is down).
- Logging: logs are teed to stdout and an optional rotated file (LOG_FILE)
via lumberjack, so they survive a container rebuild.
- Startup resilience: the initial DB connection is retried with backoff,
so the coordinator waits for Postgres to boot instead of crash-looping.
Two ways to exercise every endpoint, both living in the repo rather than in a
personal Postman workspace:
- scripts/smoke.sh walks the full lifecycle and asserts each status, exiting
non-zero on the first surprise, so it works in CI as well as by hand;
- api/requests.http drives the same calls from an editor's REST client, with
later requests reusing ids captured from earlier responses. It doubles as
API documentation for the worker author.
The script claims until it sees its own job's chunks instead of assuming an
empty queue: a shared development database usually holds pending tasks from
earlier runs, and it takes the attempt number from the claim response, since a
task requeued after an expired lease comes back with attempt 2 or 3.
Note for whoever extends the validation cases: Go matches JSON field names
case-insensitively, so "worker_ID" is accepted as "worker_id". Only a genuinely
unknown key trips DisallowUnknownFields.
Efremenko Arhip