From 215325a9e02c045a9a561ca9405640ad823811be Mon Sep 17 00:00:00 2001 From: Emil Date: Sun, 2 Aug 2026 20:06:08 +0300 Subject: [PATCH] Document the single-binary platform and update the roadmap --- PLAN.md | 50 ++++++++++++++++++++++++++++++++---------- README.md | 46 +++++++++++++++++++++++---------------- STATUS.md | 58 +++++++++++++++++++++++++++++-------------------- mkdocs/index.md | 57 +++++++++++++++++++++++++++++------------------- 4 files changed, 136 insertions(+), 75 deletions(-) diff --git a/PLAN.md b/PLAN.md index d438255..e92a32a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1003,11 +1003,12 @@ with as little manual configuration as possible: it provisions its own schema, and a `setup` command walks the operator through the remaining environment (database creation, secrets, admin account). -**Depends on:** the Go coordinator and the release build pipeline. Step 1 -(embedded migrations, `AUTO_MIGRATE`) and step 2 (the `coordinator setup` -wizard: database reachability and creation, schema migration, `.env` with a -generated `JWT_SECRET`, readiness summary) are implemented; step 3 — a fully -embedded userservice — is the remaining work. +**Depends on:** the Go coordinator and the release build pipeline. All steps +are implemented: embedded migrations with `AUTO_MIGRATE`, the `setup` wizard, +the embedded SQLite storage backend (`SCIMESH_DB=sqlite`), the embedded +userservice, and the `coordinator serve` single-binary mode with local worker +agents. The standalone `users/` service and the PostgreSQL engine remain for +cluster deployments. **Acceptance criteria:** @@ -1027,6 +1028,34 @@ embedded userservice — is the remaining work. --- +### CTX-18 — Single-binary platform (`coordinator serve`) + +**Goal:** A scientist installs one file, runs one command, and gets the whole +platform: coordinator, both databases, the userservice, and local workers — +no PostgreSQL, no Docker, no Python setup. + +**Depends on:** CTX-17 (embedded migrations, SQLite, embedded userservice). + +**Acceptance criteria:** + +- `coordinator serve` provisions `~/.scimesh` (databases, secrets chmod 0600, + generated admin password printed once, artifacts dir) and serves the UI on + 127.0.0.1:8080 by default; `--open` opens the browser; +- `--workers N` spawns N `coordinator agent` subprocesses that claim and + execute tasks locally; agents are stopped on shutdown; +- the embedded userservice listens on the loopback interface and shares the + JWT secret with the coordinator, so UI login/registration work unchanged; +- a managed scientific runtime venv (`~/.scimesh/venv`) is bootstrapped on + first start; `SCIMESH_PIP_PACKAGE` controls what gets installed (the PyPI + name is not ours), and `TASK_RUNNER` points at the venv python; +- `install.sh` / `install.ps1` detect the platform, download the release + binary, and print the start command; both are release assets; +- the PostgreSQL engine and the standalone userservice stay fully supported; +- `coordinator serve` passes the full local E2E without any external service: + health, login, upload, claim, compute, reduction, byte-exact result. + +--- + ## 10. Suggested assignment bundles These bundles minimize overlap. Do not run tasks from the same bundle in @@ -1166,12 +1195,11 @@ Do not start these before CTX-12 is accepted. - Add shard caching and content-addressed input deduplication. - Add job priority and fair scheduling. - Add a CLI for submitting and monitoring remote jobs. -- Implement CTX-17 step 3: a fully embedded userservice - (`coordinator userservice` subcommand) so one binary can serve the whole - platform without containers. -- Replace PostgreSQL with an embedded SQLite backend for fully self-contained - single-binary deployments (large storage-layer change; postgres row locks, - transactions, and integration tests must be re-derived). +- Publish the scimesh Python package to PyPI so the managed venv bootstrap + (`SCIMESH_PIP_PACKAGE`) works out of the box on a scientist's machine. +- Bundle a Python runtime (python-build-standalone) into the release so local + workers need no system Python at all. +- Native installers (.msi/.dmg/.deb) built by the release workflow. --- diff --git a/README.md b/README.md index 66e6e0c..aafdf9a 100644 --- a/README.md +++ b/README.md @@ -43,13 +43,34 @@ pip install -e . Every `v*` tag pushes a GitHub Release with static binaries for `coordinator` and `worker-agent` on linux/darwin/windows × amd64/arm64 (plus SHA-256 -checksums) and the `coordinator` image on GHCR: +checksums), the installer scripts, and the `coordinator` image on GHCR: ```bash docker pull ghcr.io/emil28092005/SciMesh/coordinator:latest ``` -Download and run a release binary: +For scientists: one command downloads the right binary and prints the start +instructions: + +```bash +# Linux / macOS +curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash +coordinator serve --open + +# Windows (PowerShell) +powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex" +coordinator serve --open +``` + +`coordinator serve` is the single-binary mode: it embeds SQLite (coordinator + +userservice databases), the userservice itself, and local worker agents +(`--workers N`, default 1). On first start it generates secrets and the admin +password under `~/.scimesh`, prints the login, and opens the UI. No +PostgreSQL, no Docker, no environment variables. The scientific runtime is a +managed venv (`~/.scimesh/venv`); point `SCIMESH_PIP_PACKAGE` at your scimesh +wheel to install it automatically. + +Manual download and run of a release binary: ```bash curl -L -o coordinator https://github.com/emil28092005/SciMesh/releases/latest/download/coordinator-linux-amd64 @@ -57,23 +78,10 @@ chmod +x coordinator ./coordinator --version ``` -- **worker-agent** runs anywhere with Python: it spawns - `python -m scimesh.worker.task`, so the machine needs the `scimesh` package - in a venv (`pip install scimesh`) plus `COORDINATOR_URL`, - `WORKER_AUTH_TOKEN`, and `WORK_DIR`. -- **coordinator** needs PostgreSQL running. The binary applies its embedded - schema migrations itself on startup (`AUTO_MIGRATE=false` opts out), and the - interactive wizard provisions the rest — database creation when missing, a - generated `JWT_SECRET`, and a `.env` file: - - ```bash - ./coordinator setup --yes --db 'postgres://user:pass@localhost:5432/scimesh?sslmode=disable' - ENV_FILE=.env ./coordinator - ``` - - `coordinator setup --help` lists all options (`--admin-db`, `--env-file`, - `--force`, non-interactive `--yes`). The UI login additionally requires a - userservice (`USERSERVICE_URL`, see the `users/` service). +Cluster deployments keep the PostgreSQL engine (`SCIMESH_DB=postgres` with +`DATABASE_URL`, or `coordinator setup` to provision it) and the standalone +userservice (`users/`). `coordinator agent` runs a worker agent from the same +binary. ## Quick start diff --git a/STATUS.md b/STATUS.md index 133eb70..c988b35 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,7 +1,7 @@ # SciMesh Status -**Updated:** 2026-08-01 -**Branch baseline:** `main`; this revision adds the Workload SDK foundation. +**Updated:** 2026-08-02 +**Branch baseline:** `main`; this revision adds the single-binary platform. ## Current state @@ -17,11 +17,22 @@ the reference behaviour for future distributed execution: The Go coordinator and its PostgreSQL-backed task lifecycle are implemented: registration, atomic claiming, lease renewal, artifact storage, dataset -chunking, result/failure reporting, and job progress. The Python worker now -uses the live coordinator contract. Completed similarity-search shard results -are reduced once into a checksum-protected final CSV, which is downloadable -through the coordinator. The full Go checks (including a fresh migration and -real PostgreSQL smoke test) passed on 2026-07-24. +chunking, result/failure reporting, and job progress. The Go worker agent now +uses the live coordinator contract. Completed shard results are reduced once +into a checksum-protected final CSV, downloadable through the coordinator. + +**Single-binary platform (`coordinator serve`)**: the coordinator now ships +an embedded SQLite storage backend (`SCIMESH_DB=sqlite`), an embedded +userservice, and `serve`/`agent` subcommands, so one downloaded binary runs +the whole platform — coordinator, both databases, UI logins, and local +workers — with no PostgreSQL, no Docker, and no environment variables. The +first start provisions `~/.scimesh` (secrets, admin password printed once, +managed scientific-runtime venv) and opens the UI. `install.sh` / `install.ps1` +download the right release binary in one command and are release assets. The +PostgreSQL engine, the `setup` wizard, and the standalone `users/` service +remain fully supported for cluster deployments. The full E2E passes with zero +external services: health, UI login, job upload, local agent compute, +reduction, and a byte-exact final CSV. The User Service is merged into `main`. It owns user accounts, authentication, roles, and verified-contributor status; the coordinator scopes user jobs and @@ -37,24 +48,25 @@ the complete result-artifact SHA-256 before a task is accepted. | --- | --- | --- | | CTX-00 API and error contract | Implemented | Contract, OpenAPI, and request examples are in `docs/`. | | CTX-01 Go coordinator bootstrap | Implemented | Go service and Docker runtime in `coordinator/`. | -| CTX-02 PostgreSQL migrations | Implemented | Applied by the Compose migration service. | -| CTX-03 Transactional queue | Implemented | Real-PostgreSQL integration tests cover atomic claims and concurrency. | +| CTX-02 PostgreSQL migrations | Implemented | Embedded into the binary (`AUTO_MIGRATE`); the CLI path is still available for managed databases. | +| CTX-03 Transactional queue | Implemented | Real-PostgreSQL integration tests cover atomic claims and concurrency; the SQLite backend mirrors the semantics. | | CTX-04 Worker registry and HTTP API | Implemented | Registration, claim, heartbeat, result, failure, and status endpoints. | | CTX-05 Artifact storage | Implemented | Coordinator-owned inputs/results, checksum verification, and upload flow. | -| CTX-06 Python Worker live-contract alignment | Superseded | The Python worker daemon was removed; the Go worker agent (`coordinator/internal/agent/` + `cmd/worker-agent`) now implements the lifecycle (register/claim/heartbeat/download/upload/submit/fail, token refresh, cleanup) and executes SDK workloads via the Python task entry `scimesh/worker/task.py`. E2E: `make smoke-two-worker` passes 4/4 shards with two agents. | -| CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering are in `scimesh/distributed/`. | -| CTX-08 Distributed similarity-search | Implemented | Python planner resolves `query_id` once, creates deterministic shard plans, worker adapter emits exact partial top-k CSVs/metrics, and reducer matches the local reference. | -| CTX-09 Reducer and final-result API | Implemented | Atomic `reducing` claim, deterministic coordinator-side top-k reducer, sanitized reducer failure, final artifact persistence, `result_uri`, and final CSV download. | -| CTX-10 Distributed similarity-graph | Not started | Local reference exists; the SDK-built local graph workload already enforces the pair-coverage invariant. | -| CTX-11 Dashboard/operator view | Implemented | Protected live control room: MkDocs documentation served at `/ui/docs/` (SCIMESH_DOCS_DIR; the demo mounts `site/` automatically), recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, bounded polling, a Workload library page rendering the embedded catalog from `scimesh workload export` (`/ui/workloads`, regenerated via `make workloads-export`). | -| CTX-12 Reliability, security, CI | In progress | Unit, race, PostgreSQL integration, and smoke checks exist; CI hardening remains. | -| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, self-service enrollment, and quorum-backed untrusted workers are merged; local Go/Python and Docker/PostgreSQL checks passed. | -| MkDocs documentation site | Implemented | A standalone documentation site (`mkdocs/`, `docs_dir: mkdocs`) covering the complete Workload SDK: guides (overview, authoring workloads, CLI, worker integration), the full auto-generated API reference for all 15 `scimesh.sdk` modules (mkdocstrings), and the writing rules (`mkdocs/approach.md`). Built with `make docs`, served inside the UI at `/ui/docs/`; the project's internal `docs/` directory is not part of the site. | -| CTX-16 Workload SDK foundation | Implemented | `scimesh.sdk` provides strict immutable manifests/plans/artifacts, digest/trust-pinned tasks, typed DAGs, compatibility negotiation, verifier primitives with owner/binding-safe quorum inputs, resource eligibility/local allocation, measured package discovery, a trusted local core-batch conformance harness, and strict package discovery. Enforcing coordinator/Worker profiles remain fail-closed. | -| SDK roadmap step 3: `descriptor-batch` | Implemented | The first SDK-built reference workload (`scimesh/workloads/descriptors/`): pinned 81-name RDKit 2D descriptor set, canonical one-row-per-input CSV, deterministic row-bounded shards, shard-index concatenation with one header, byte-identical local/distributed output, and a two-worker `untrusted_quorum` verifier test. | -| SDK-built `similarity-search` and `similarity-graph` | Implemented | Both workloads are SDK-built packages (`scimesh/workloads/search/`, `scimesh/workloads/graph/`) built on the `MapReduceWorkload` authoring scaffold (`scimesh/sdk/batch.py`); they reuse the local scientific cores and are byte-identical to the single-process references (search; graph for both threshold directions and any block size). The graph reducer enforces the CTX-10 pair-coverage invariant. `scimesh/workloads/library.py` composes the built-in registry/runtime. | -| SDK-built `molwt-filter` | Implemented | The minimal authoring example (`scimesh/workloads/molwt_filter/`): filters molecules by exact RDKit molecular weight with only one scientific hook, using the scaffold's new default sharding and concatenation hooks. Registered in the built-in library and as a `scimesh.workloads` entry point. | -| SDK authoring scaffold | Implemented | `MapReduceWorkload` (exported from `scimesh.sdk`) assembles manifest, map/reduce stages, workflow, and digest-pinned handlers from three scientific hooks (partition/compute/merge), with overridable hooks for domain validation, plan-time resolution, custom task planning, and partial-key policy. The generic `scimesh workload list|run` CLI and the worker's allowlist-driven loading (`SCIMESH_WORKLOAD_ALLOWLIST`, `SCIMESH_CAPABILITIES`) let new workloads run without touching other code. | +| CTX-06 Python Worker live-contract alignment | Superseded | The Python worker daemon was removed; the Go worker agent (`coordinator/internal/agent/` + `cmd/worker-agent`, or `coordinator agent`) implements the lifecycle and executes SDK workloads via `scimesh/worker/task.py`. | +| CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering. | +| CTX-08 Distributed similarity-search | Implemented | Planner/worker/reducer match the local reference byte-exactly. | +| CTX-09 Reducer and final-result API | Implemented | Atomic `reducing` claim, deterministic coordinator-side reducers (`top-k` and `ordered-concat`), sanitized failure, final artifact, `result_uri`. | +| CTX-10 Distributed similarity-graph | Not started | Local reference exists; the SDK-built local graph workload enforces the pair-coverage invariant. | +| CTX-11 Dashboard/operator view | Implemented | Protected live control room, workload library, workload-agnostic "New computation" form (SDK-declared `UIElement`s), MkDocs at `/ui/docs/`, final-result download. | +| CTX-12 Reliability, security, CI | In progress | vet, gofmt, race tests, golangci-lint (0 issues), PostgreSQL integration, and smoke checks exist. | +| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, quorum; also embeddable (`coordinator serve`). | +| CTX-16 Workload SDK foundation | Implemented | Strict immutable manifests/plans/artifacts, digest/trust-pinned tasks, negotiation, verifier primitives, conformance harness. | +| CTX-17 Self-provisioning + setup wizard | Implemented | Embedded migrations, `coordinator setup`, SQLite backend, embedded userservice, `serve` mode. | +| CTX-18 Single-binary platform | Implemented | `coordinator serve` (data dir, secrets, admin bootstrap, local agents, managed venv) + `install.sh`/`install.ps1`; full no-external-service E2E green. | +| SDK roadmap step 3: `descriptor-batch` | Implemented | Byte-identical local/distributed output, quorum verifier test. | +| SDK-built `similarity-search` and `similarity-graph` | Implemented | SDK-built packages, byte-identical to single-process references. | +| SDK-built `molwt-filter` | Implemented | Minimal authoring example; also the single-binary E2E workload. | +| SDK authoring scaffold | Implemented | `MapReduceWorkload` with `UIElement` declarations, `reduction`, `upload_ready`; generic `scimesh workload list|run|export|allowlist` CLI. | ## Next recommended assignment diff --git a/mkdocs/index.md b/mkdocs/index.md index 8522faa..b56eebb 100644 --- a/mkdocs/index.md +++ b/mkdocs/index.md @@ -38,15 +38,32 @@ The two halves of the project: ## Quick start +The fastest path for a scientist: install the platform with one command and +start it. Everything — the coordinator, its databases, the userservice, and +local workers — is embedded in a single binary; no PostgreSQL, no Docker, no +Python setup. + +```bash +# Linux / macOS +curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash +coordinator serve --open + +# Windows (PowerShell) +powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex" +coordinator serve --open +``` + +The first start prints the admin login (also stored under `~/.scimesh`), and +`--open` opens the UI in the browser. `coordinator serve --workers 2` +spawns two local workers; `SCIMESH_PIP_PACKAGE` points the managed venv at +your scimesh wheel so scientific workloads can run. + +For development, install the Python SDK and run workloads locally: + ```bash python -m venv .venv source .venv/bin/activate pip install -e '.[dev]' -``` - -List the installed SDK workloads and run one locally: - -```bash scimesh workload list scimesh workload run molwt-filter \ --input molecules.tsv \ @@ -54,12 +71,6 @@ scimesh workload run molwt-filter \ -o filtered.csv ``` -Run the local scientific CLI workloads: - -```bash -scimesh help -``` - Start the full demo (PostgreSQL, coordinator, UI, two workers): ```bash @@ -71,8 +82,8 @@ make demo-ui Every `v*` tag pushes a GitHub Release with static binaries for `coordinator` and `worker-agent` on linux/darwin/windows × amd64/arm64 -(plus SHA-256 checksums) and the `coordinator` image on GHCR. Download and -run: +(plus SHA-256 checksums), the installer scripts above, and the `coordinator` +image on GHCR. Download and run: ```bash curl -L -o coordinator https://github.com/emil28092005/SciMesh/releases/latest/download/coordinator-linux-amd64 @@ -82,15 +93,17 @@ chmod +x coordinator - **worker-agent** runs anywhere with Python: it spawns `python -m scimesh.worker.task`, so the machine needs the `scimesh` package in a venv (`pip install scimesh`) and the usual environment: - `COORDINATOR_URL`, `WORKER_AUTH_TOKEN`, `WORK_DIR`. -- **coordinator** needs PostgreSQL running (`DATABASE_URL`, - `COORDINATOR_STORAGE_DIR`, `JWT_SECRET`); the binary applies its embedded - schema migrations itself on startup (`AUTO_MIGRATE=false` opts out), so no - separate migration step is needed. The interactive wizard provisions the - rest: it checks the database, creates it when missing, generates a - `JWT_SECRET`, and writes a `.env` file (`coordinator setup --help`, or - `make setup`; `--yes` for non-interactive runs). The UI login additionally - requires `USERSERVICE_URL`. + `COORDINATOR_URL`, `WORKER_AUTH_TOKEN`, `WORK_DIR`. The same binary can run + it via `coordinator agent`. +- **coordinator** needs no external services at all in its default mode: + `coordinator serve` embeds SQLite (both databases), the userservice, and + local workers. The `SCIMESH_DB=postgres` engine remains for cluster + deployments (`DATABASE_URL`, `COORDINATOR_STORAGE_DIR`, `JWT_SECRET`); the + binary applies its embedded schema migrations itself on startup + (`AUTO_MIGRATE=false` opts out), and `coordinator setup` provisions a + PostgreSQL deployment interactively. The UI login uses the userservice + (`USERSERVICE_URL`) — embedded by `serve`, or the standalone `users/` + service otherwise. `coordinator --version` / `worker-agent --version` print the build tag. Build and serve this documentation site: