add users logic
This commit is contained in:
Efremenko Arhip
+230
@@ -0,0 +1,230 @@
|
||||
# SciMesh Coordinator
|
||||
|
||||
Durable task-queue server for SciMesh, in Go on PostgreSQL. It owns all database
|
||||
access; workers talk to it only over HTTP and never receive DB credentials.
|
||||
|
||||
Built as a **modular monolith following Clean Architecture** — one binary, four
|
||||
layers, dependencies pointing strictly inward. See
|
||||
`docs/database-integration-task.md` and `docs/worker-daemon-task.md` in the repo
|
||||
root for the full contract.
|
||||
|
||||
## Layers
|
||||
|
||||
```
|
||||
infra config, pgxpool, http.Server, clock ← frameworks & drivers
|
||||
transport http handlers ← inbound: who calls us
|
||||
storage sql repositories ← outbound: who we call
|
||||
usecase business operations + PORTS ← application rules
|
||||
domain Task, Job + their invariants ← enterprise rules
|
||||
|
||||
┌── transport ──┐
|
||||
domain ◄── usecase ◄┤ ├◄── infra
|
||||
└── storage ────┘
|
||||
```
|
||||
|
||||
`transport` and `storage` are one layer — the "interface adapters" ring — split
|
||||
by direction rather than by category, so a file's path tells you its role.
|
||||
|
||||
The rule that matters: **source dependencies point only inward**. `domain`
|
||||
imports nothing from this module; `usecase` sees only `domain`; `transport` and
|
||||
`storage` know nothing of each other. Verify it at any time with:
|
||||
|
||||
```sh
|
||||
go list -f '{{range .Imports}}{{.}}{{"\n"}}{{end}}' ./internal/domain | grep internal # must be empty
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
coordinator/
|
||||
cmd/coordinator/main.go # composition root: the only place with concrete types
|
||||
internal/
|
||||
domain/ # entities + rules, no I/O
|
||||
task.go Task, lease/complete/fail/expire transitions
|
||||
job.go Job, chunk fan-out, status derivation
|
||||
errors.go business-rule violations
|
||||
usecase/ # one type per operation, dependencies injected
|
||||
ports.go TaskRepository, JobRepository, TxManager, Clock
|
||||
dto.go use-case boundary inputs
|
||||
task.go claim, renew, complete, fail, expire
|
||||
job.go create, status, results, stitch
|
||||
transport/http/ # routing, DTOs, middleware, error mapping
|
||||
storage/postgres/ # SQL behind the ports; TxManager via context
|
||||
infra/ # config.go db.go clock.go server.go
|
||||
migrations/ # golang-migrate SQL, run as an explicit command
|
||||
```
|
||||
|
||||
A full map — file-by-file table, a request traced through every layer, and a
|
||||
"where do I add X" guide — lives in [ARCHITECTURE.md](ARCHITECTURE.md).
|
||||
|
||||
## Quickstart
|
||||
|
||||
### With Docker (nothing to install but Docker)
|
||||
|
||||
```sh
|
||||
make up # Postgres → migrations → coordinator
|
||||
curl localhost:8080/health
|
||||
make logs # follow the coordinator
|
||||
make down # stop (add down-clean to drop the DB volume)
|
||||
```
|
||||
|
||||
To enable the local operator UI, set a separate credential before starting:
|
||||
|
||||
```sh
|
||||
UI_AUTH_TOKEN='local-ui-secret' make up
|
||||
# Open http://localhost:8080/ui and use any username with this value as password.
|
||||
```
|
||||
|
||||
The UI is disabled by default and never accepts the worker bearer token.
|
||||
The **control room** shows live workers, recent runs, shard state/attempts,
|
||||
safe failures, coordinator artifacts, and the final CSV for completed
|
||||
similarity-search jobs. The job page follows the real stages: TSV accepted →
|
||||
shards execute → workers return CSVs → `reducing` → final deterministic global
|
||||
top-k result. It polls only its own coordinator read-model and never controls
|
||||
or exposes worker processes.
|
||||
|
||||
For a hands-on run, open `/ui`, choose **New similarity search**, select a
|
||||
small ChEMBL-style TSV, then leave one or more `scimesh-worker` processes
|
||||
running in separate terminals. The detail page updates every two seconds and
|
||||
stops polling after a completed, failed, or cancelled job. Use **Preview CSV**
|
||||
to inspect a bounded first page of a partial or completed final result before
|
||||
downloading it. The UI never exposes source datasets or shard inputs; partial
|
||||
CSVs remain available only as diagnostics.
|
||||
|
||||
### One-command manual demo
|
||||
|
||||
From the repository root, create the Python environment once, then start a
|
||||
self-contained UI demo with two local reference workers:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -e '.[dev]'
|
||||
make demo-ui
|
||||
```
|
||||
|
||||
This uses a separate Docker project and ports `18080` (coordinator) and
|
||||
`55432` (PostgreSQL), so it does not conflict with the normal stack. Open
|
||||
`http://localhost:18080/ui`, use username `operator` and password
|
||||
`demo-ui-secret`, upload a small ChEMBL TSV, and observe the workers process
|
||||
it. Change the worker count with `make demo-ui WORKERS=3`; stop all demo
|
||||
services and workers with `make demo-down`.
|
||||
|
||||
The job page shows a live **Processing speed** graph in completed shards per
|
||||
minute. It uses the coordinator snapshots observed by the open browser tab, so
|
||||
it is a transparent local-session measurement rather than a persisted metric.
|
||||
Use **Preview CSV** before downloading a partial diagnostic or completed final
|
||||
result. Run `make help` from either the repository root or this directory for
|
||||
the full list of demo commands.
|
||||
|
||||
`up` starts three services in order: Postgres waits until `pg_isready` passes, a
|
||||
one-shot `migrate` container applies the schema and exits, and only then does the
|
||||
coordinator start — so it never queries a database that has no tables.
|
||||
|
||||
> **Needs BuildKit.** The Dockerfile uses `RUN --mount=type=cache` to reuse the
|
||||
> Go module and compiler caches between builds. If the build fails with
|
||||
> *"the --mount option requires BuildKit"*, install the buildx plugin —
|
||||
> `pacman -S docker-buildx` on Arch, `apt install docker-buildx-plugin` on Debian.
|
||||
|
||||
### Locally, against your own Postgres
|
||||
|
||||
```sh
|
||||
cp .env.example .env # then edit DATABASE_URL / WORKER_AUTH_TOKEN
|
||||
# it is loaded automatically — no export needed
|
||||
|
||||
make tidy # fetch deps (needs network once)
|
||||
make migrate-up # apply schema (needs the migrate CLI)
|
||||
make run # start the server
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Settings come from the environment. A `.env` file is loaded at startup via
|
||||
`godotenv` as a local-dev convenience (override its path with `ENV_FILE`):
|
||||
|
||||
- a missing `.env` is not an error — production injects real env vars;
|
||||
- **real environment variables always win** over the file, so an orchestrator's
|
||||
values are never shadowed by a stale `.env` baked into an image.
|
||||
|
||||
See `.env.example`; only `DATABASE_URL` is required.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------ | ---------------------------------- | --------------------------------------------- |
|
||||
| POST | `/workers/register` | Register a worker, get its id |
|
||||
| POST | `/jobs` | Create job + tasks from chunk URIs |
|
||||
| POST | `/jobs/upload` | Upload a dataset; coordinator chunks it |
|
||||
| GET | `/jobs/{job_id}` | Aggregate job progress |
|
||||
| POST | `/tasks/claim` | Atomically lease one task (`204` if none) |
|
||||
| GET | `/tasks/{task_id}/input` | Download the task's input shard |
|
||||
| POST | `/tasks/{task_id}/heartbeat` | Renew the caller's lease (→ `running`) |
|
||||
| PUT | `/tasks/{task_id}/artifacts/{name}`| Upload a partial-result artifact |
|
||||
| POST | `/tasks/{task_id}/result` | Complete with an artifact id (idempotent) |
|
||||
| POST | `/tasks/{task_id}/failure` | Record failure / retryable state |
|
||||
| GET | `/artifacts/{artifact_id}/download`| Download an artifact by id |
|
||||
| GET | `/health` | Readiness incl. database (unauthenticated) |
|
||||
|
||||
The full contract is in [`docs/api-contract.md`](../docs/api-contract.md) and
|
||||
[`docs/openapi.yaml`](../docs/openapi.yaml); a worker-author guide is in
|
||||
[`docs/building-workers.md`](../docs/building-workers.md).
|
||||
|
||||
## Poking the API
|
||||
|
||||
Two ways, both checked in:
|
||||
|
||||
```sh
|
||||
make smoke # every endpoint, asserted; non-zero exit on failure
|
||||
```
|
||||
|
||||
`api/requests.http` runs the same calls one at a time from an editor with a REST
|
||||
client (VSCodium/VS Code "REST Client", JetBrains HTTP Client). Later requests
|
||||
reuse ids captured from earlier responses, so it doubles as API documentation.
|
||||
|
||||
## Status
|
||||
|
||||
Works end to end: a worker registers, a dataset is uploaded and chunked into
|
||||
shard tasks (or a job is created from chunk URIs), tasks are leased one at a
|
||||
time, downloaded, heartbeated (`leased → running`), completed via uploaded
|
||||
result artifacts, and reflected in job progress. A reaper reclaims expired
|
||||
leases and marks silent workers offline.
|
||||
|
||||
Done: schema + migrations, atomic claim (`FOR UPDATE SKIP LOCKED`), optimistic
|
||||
concurrency, result/failure paths, lease expiry, worker registry + liveness,
|
||||
artifact storage, dataset upload + chunking, request-size limits.
|
||||
|
||||
Still stubbed: `StitchJob.Execute` — merging per-chunk top-k into the final CSV
|
||||
is workload semantics that belongs to the Python side (reducer).
|
||||
|
||||
## Tests
|
||||
|
||||
Unit tests need **no database** — domain rules, use-case orchestration (over
|
||||
in-memory `internal/memstore`), and HTTP handlers (via `httptest`):
|
||||
|
||||
```sh
|
||||
make test # go test ./...
|
||||
make vet
|
||||
make lint
|
||||
go test -race ./...
|
||||
```
|
||||
|
||||
Integration tests run against a **real PostgreSQL** (the spec forbids mocks
|
||||
here — they verify `FOR UPDATE SKIP LOCKED`, optimistic concurrency, rollback):
|
||||
|
||||
```sh
|
||||
docker compose up -d
|
||||
make test-integration TEST_DATABASE_URL='postgres://scimesh:scimesh@localhost:5432/scimesh?sslmode=disable'
|
||||
```
|
||||
|
||||
CI (`.github/workflows/coordinator.yml`) runs vet, gofmt, race tests, lint, and
|
||||
the integration suite against a Postgres service on every push and PR.
|
||||
|
||||
For the complete local verification, including an isolated Docker PostgreSQL
|
||||
and the HTTP smoke flow, run:
|
||||
|
||||
```sh
|
||||
make check
|
||||
```
|
||||
|
||||
It uses Compose project `scimesh-check` and ports `55432`/`18080` by default,
|
||||
so it does not connect to a PostgreSQL already running on `5432`. Override
|
||||
`CHECK_POSTGRES_PORT`, `CHECK_COORDINATOR_PORT`, or `CHECK_PROJECT` if needed.
|
||||
Reference in New Issue
Block a user