SciMesh/coordinator

SciMesh Coordinator

Durable task-queue server for SciMesh, in Go on PostgreSQL. It owns all database access; workers talk to it only over HTTP and never receive DB credentials.

Built as a modular monolith following Clean Architecture — one binary, four layers, dependencies pointing strictly inward. See docs/database-integration-task.md and docs/worker-daemon-task.md in the repo root for the full contract.

Layers

        infra       config, pgxpool, http.Server, clock    ← frameworks & drivers
        transport   http handlers      ← inbound: who calls us
        storage     sql repositories   ← outbound: who we call
        usecase     business operations + PORTS            ← application rules
        domain      Task, Job + their invariants           ← enterprise rules

                            ┌── transport ──┐
        domain ◄── usecase ◄┤               ├◄── infra
                            └── storage ────┘

transport and storage are one layer — the "interface adapters" ring — split by direction rather than by category, so a file's path tells you its role.

The rule that matters: source dependencies point only inward. domain imports nothing from this module; usecase sees only domain; transport and storage know nothing of each other. Verify it at any time with:

go list -f '{{range .Imports}}{{.}}{{"\n"}}{{end}}' ./internal/domain | grep internal   # must be empty

Layout

coordinator/
  cmd/coordinator/main.go      # composition root: the only place with concrete types
  internal/
    domain/                    # entities + rules, no I/O
      task.go                    Task, lease/complete/fail/expire transitions
      job.go                     Job, chunk fan-out, status derivation
      errors.go                  business-rule violations
    usecase/                   # one type per operation, dependencies injected
      ports.go                   TaskRepository, JobRepository, TxManager, Clock
      dto.go                     use-case boundary inputs
      task.go                    claim, renew, complete, fail, expire
      job.go                     create, status, results, stitch
    transport/http/            # routing, DTOs, middleware, error mapping
    storage/postgres/          # SQL behind the ports; TxManager via context
    infra/                     # config.go db.go clock.go server.go
  migrations/                  # golang-migrate SQL, run as an explicit command

A full map — file-by-file table, a request traced through every layer, and a "where do I add X" guide — lives in ARCHITECTURE.md.

Quickstart

With Docker (nothing to install but Docker)

make up                       # Postgres → migrations → coordinator
curl localhost:8080/health
make logs                     # follow the coordinator
make down                     # stop (add down-clean to drop the DB volume)

To enable the local operator UI, set a separate credential before starting:

UI_AUTH_TOKEN='local-ui-secret' make up
# Open http://localhost:8080/ui and use any username with this value as password.

The UI is disabled by default and never accepts the worker bearer token. The control room shows live workers, recent runs, shard state/attempts, safe failures, coordinator artifacts, and the final CSV for completed similarity-search jobs. The job page follows the real stages: TSV accepted → shards execute → workers return CSVs → reducing → final deterministic global top-k result. It polls only its own coordinator read-model and never controls or exposes worker processes.

For a hands-on run, open /ui, choose New similarity search, select a small ChEMBL-style TSV, then leave one or more scimesh-worker processes running in separate terminals. The detail page updates every two seconds and stops polling after a completed, failed, or cancelled job. Use Preview CSV to inspect a bounded first page of a partial or completed final result before downloading it. The UI never exposes source datasets or shard inputs; partial CSVs remain available only as diagnostics.

One-command manual demo

From the repository root, create the Python environment once, then start a self-contained UI demo with two local reference workers:

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
make demo-ui

This uses a separate Docker project and ports 18080 (coordinator) and 55432 (PostgreSQL), so it does not conflict with the normal stack. Open http://localhost:18080/ui, use username operator and password demo-ui-secret, upload a small ChEMBL TSV, and observe the workers process it. Change the worker count with make demo-ui WORKERS=3; stop all demo services and workers with make demo-down.

The job page shows a live Processing speed graph in completed shards per minute. It uses the coordinator snapshots observed by the open browser tab, so it is a transparent local-session measurement rather than a persisted metric. Use Preview CSV before downloading a partial diagnostic or completed final result. Run make help from either the repository root or this directory for the full list of demo commands.

up starts three services in order: Postgres waits until pg_isready passes, a one-shot migrate container applies the schema and exits, and only then does the coordinator start — so it never queries a database that has no tables.

Needs BuildKit. The Dockerfile uses RUN --mount=type=cache to reuse the Go module and compiler caches between builds. If the build fails with "the --mount option requires BuildKit", install the buildx plugin — pacman -S docker-buildx on Arch, apt install docker-buildx-plugin on Debian.

Locally, against your own Postgres

cp .env.example .env          # then edit DATABASE_URL / WORKER_AUTH_TOKEN
                              # it is loaded automatically — no export needed

make tidy                     # fetch deps (needs network once)
make migrate-up               # apply schema (needs the migrate CLI)
make run                      # start the server

Configuration

Settings come from the environment. A .env file is loaded at startup via godotenv as a local-dev convenience (override its path with ENV_FILE):

• a missing .env is not an error — production injects real env vars;
• real environment variables always win over the file, so an orchestrator's values are never shadowed by a stale .env baked into an image.

See .env.example; only DATABASE_URL is required.

Endpoints

Method	Path	Purpose
POST	/workers/register	Register a worker, get its id
POST	/jobs	Create job + tasks from chunk URIs
POST	/jobs/upload	Upload a dataset; coordinator chunks it
GET	/jobs/{job_id}	Aggregate job progress
POST	/tasks/claim	Atomically lease one task (204 if none)
GET	/tasks/{task_id}/input	Download the task's input shard
POST	/tasks/{task_id}/heartbeat	Renew the caller's lease (→ running)
PUT	/tasks/{task_id}/artifacts/{name}	Upload a partial-result artifact
POST	/tasks/{task_id}/result	Complete with an artifact id (idempotent)
POST	/tasks/{task_id}/failure	Record failure / retryable state
GET	/artifacts/{artifact_id}/download	Download an artifact by id
GET	/health	Readiness incl. database (unauthenticated)

The full contract is in docs/api-contract.md and docs/openapi.yaml; a worker-author guide is in docs/building-workers.md.

Poking the API

Two ways, both checked in:

make smoke                    # every endpoint, asserted; non-zero exit on failure

api/requests.http runs the same calls one at a time from an editor with a REST client (VSCodium/VS Code "REST Client", JetBrains HTTP Client). Later requests reuse ids captured from earlier responses, so it doubles as API documentation.

Status

Works end to end: a worker registers, a dataset is uploaded and chunked into shard tasks (or a job is created from chunk URIs), tasks are leased one at a time, downloaded, heartbeated (leased → running), completed via uploaded result artifacts, and reflected in job progress. A reaper reclaims expired leases and marks silent workers offline.

Done: schema + migrations, atomic claim (FOR UPDATE SKIP LOCKED), optimistic concurrency, result/failure paths, lease expiry, worker registry + liveness, artifact storage, dataset upload + chunking, request-size limits.

Still stubbed: StitchJob.Execute — merging per-chunk top-k into the final CSV is workload semantics that belongs to the Python side (reducer).

Tests

Unit tests need no database — domain rules, use-case orchestration (over in-memory internal/memstore), and HTTP handlers (via httptest):

make test           # go test ./...
make vet
make lint
go test -race ./...

Integration tests run against a real PostgreSQL (the spec forbids mocks here — they verify FOR UPDATE SKIP LOCKED, optimistic concurrency, rollback):

docker compose up -d
make test-integration TEST_DATABASE_URL='postgres://scimesh:scimesh@localhost:5432/scimesh?sslmode=disable'

CI (.github/workflows/coordinator.yml) runs vet, gofmt, race tests, lint, and the integration suite against a Postgres service on every push and PR.

For the complete local verification, including an isolated Docker PostgreSQL and the HTTP smoke flow, run:

make check

It uses Compose project scimesh-check and ports 55432/18080 by default, so it does not connect to a PostgreSQL already running on 5432. Override CHECK_POSTGRES_PORT, CHECK_COORDINATOR_PORT, or CHECK_PROJECT if needed.