Emil_Shanaty/SciMesh
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document the single-binary platform and update the roadmap

Browse Source
This commit is contained in:
Emil
2026-08-02 20:06:08 +03:00
parent c06d8673ce
commit 215325a9e0
4 changed files with 136 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PLAN.md
+39 -11
View File
@@ -1003,11 +1003,12 @@ with as little manual configuration as possible: it provisions its own
schema, and a `setup` command walks the operator through the remaining
environment (database creation, secrets, admin account).
**Depends on:** the Go coordinator and the release build pipeline. Step 1
(embedded migrations, `AUTO_MIGRATE`) and step 2 (the `coordinator setup`
wizard: database reachability and creation, schema migration, `.env` with a
generated `JWT_SECRET`, readiness summary) are implemented; step 3 — a fully
embedded userservice — is the remaining work.
**Depends on:** the Go coordinator and the release build pipeline. All steps
are implemented: embedded migrations with `AUTO_MIGRATE`, the `setup` wizard,
the embedded SQLite storage backend (`SCIMESH_DB=sqlite`), the embedded
userservice, and the `coordinator serve` single-binary mode with local worker
agents. The standalone `users/` service and the PostgreSQL engine remain for
cluster deployments.
**Acceptance criteria:**
@@ -1027,6 +1028,34 @@ embedded userservice — is the remaining work.
---
### CTX-18 — Single-binary platform (`coordinator serve`)
**Goal:** A scientist installs one file, runs one command, and gets the whole
platform: coordinator, both databases, the userservice, and local workers —
no PostgreSQL, no Docker, no Python setup.
**Depends on:** CTX-17 (embedded migrations, SQLite, embedded userservice).
**Acceptance criteria:**
- `coordinator serve` provisions `~/.scimesh` (databases, secrets chmod 0600,
generated admin password printed once, artifacts dir) and serves the UI on
127.0.0.1:8080 by default; `--open` opens the browser;
- `--workers N` spawns N `coordinator agent` subprocesses that claim and
execute tasks locally; agents are stopped on shutdown;
- the embedded userservice listens on the loopback interface and shares the
JWT secret with the coordinator, so UI login/registration work unchanged;
- a managed scientific runtime venv (`~/.scimesh/venv`) is bootstrapped on
first start; `SCIMESH_PIP_PACKAGE` controls what gets installed (the PyPI
name is not ours), and `TASK_RUNNER` points at the venv python;
- `install.sh` / `install.ps1` detect the platform, download the release
binary, and print the start command; both are release assets;
- the PostgreSQL engine and the standalone userservice stay fully supported;
- `coordinator serve` passes the full local E2E without any external service:
health, login, upload, claim, compute, reduction, byte-exact result.
---
## 10. Suggested assignment bundles
These bundles minimize overlap. Do not run tasks from the same bundle in
@@ -1166,12 +1195,11 @@ Do not start these before CTX-12 is accepted.
- Add shard caching and content-addressed input deduplication.
- Add job priority and fair scheduling.
- Add a CLI for submitting and monitoring remote jobs.
- Implement CTX-17 step 3: a fully embedded userservice
(`coordinator userservice` subcommand) so one binary can serve the whole
platform without containers.
- Replace PostgreSQL with an embedded SQLite backend for fully self-contained
single-binary deployments (large storage-layer change; postgres row locks,
transactions, and integration tests must be re-derived).
- Publish the scimesh Python package to PyPI so the managed venv bootstrap
(`SCIMESH_PIP_PACKAGE`) works out of the box on a scientist's machine.
- Bundle a Python runtime (python-build-standalone) into the release so local
workers need no system Python at all.
- Native installers (.msi/.dmg/.deb) built by the release workflow.
---
README.md
+27 -19
View File
@@ -43,13 +43,34 @@ pip install -e .
Every `v*` tag pushes a GitHub Release with static binaries for `coordinator`
and `worker-agent` on linux/darwin/windows × amd64/arm64 (plus SHA-256
checksums) and the `coordinator` image on GHCR:
checksums), the installer scripts, and the `coordinator` image on GHCR:
```bash
docker pull ghcr.io/emil28092005/SciMesh/coordinator:latest
```
Download and run a release binary:
For scientists: one command downloads the right binary and prints the start
instructions:
```bash
# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash
coordinator serve --open
# Windows (PowerShell)
powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex"
coordinator serve --open
```
`coordinator serve` is the single-binary mode: it embeds SQLite (coordinator +
userservice databases), the userservice itself, and local worker agents
(`--workers N`, default 1). On first start it generates secrets and the admin
password under `~/.scimesh`, prints the login, and opens the UI. No
PostgreSQL, no Docker, no environment variables. The scientific runtime is a
managed venv (`~/.scimesh/venv`); point `SCIMESH_PIP_PACKAGE` at your scimesh
wheel to install it automatically.
Manual download and run of a release binary:
```bash
curl -L -o coordinator https://github.com/emil28092005/SciMesh/releases/latest/download/coordinator-linux-amd64
@@ -57,23 +78,10 @@ chmod +x coordinator
./coordinator --version
```
- **worker-agent** runs anywhere with Python: it spawns
`python -m scimesh.worker.task`, so the machine needs the `scimesh` package
in a venv (`pip install scimesh`) plus `COORDINATOR_URL`,
`WORKER_AUTH_TOKEN`, and `WORK_DIR`.
- **coordinator** needs PostgreSQL running. The binary applies its embedded
schema migrations itself on startup (`AUTO_MIGRATE=false` opts out), and the
interactive wizard provisions the rest — database creation when missing, a
generated `JWT_SECRET`, and a `.env` file:
```bash
./coordinator setup --yes --db 'postgres://user:pass@localhost:5432/scimesh?sslmode=disable'
ENV_FILE=.env ./coordinator
```
`coordinator setup --help` lists all options (`--admin-db`, `--env-file`,
`--force`, non-interactive `--yes`). The UI login additionally requires a
userservice (`USERSERVICE_URL`, see the `users/` service).
Cluster deployments keep the PostgreSQL engine (`SCIMESH_DB=postgres` with
`DATABASE_URL`, or `coordinator setup` to provision it) and the standalone
userservice (`users/`). `coordinator agent` runs a worker agent from the same
binary.
## Quick start
STATUS.md
+35 -23
View File
@@ -1,7 +1,7 @@
# SciMesh Status
**Updated:** 2026-08-01
**Branch baseline:** `main`; this revision adds the Workload SDK foundation.
**Updated:** 2026-08-02
**Branch baseline:** `main`; this revision adds the single-binary platform.
## Current state
@@ -17,11 +17,22 @@ the reference behaviour for future distributed execution:
The Go coordinator and its PostgreSQL-backed task lifecycle are implemented:
registration, atomic claiming, lease renewal, artifact storage, dataset
chunking, result/failure reporting, and job progress. The Python worker now
uses the live coordinator contract. Completed similarity-search shard results
are reduced once into a checksum-protected final CSV, which is downloadable
through the coordinator. The full Go checks (including a fresh migration and
real PostgreSQL smoke test) passed on 2026-07-24.
chunking, result/failure reporting, and job progress. The Go worker agent now
uses the live coordinator contract. Completed shard results are reduced once
into a checksum-protected final CSV, downloadable through the coordinator.
**Single-binary platform (`coordinator serve`)**: the coordinator now ships
an embedded SQLite storage backend (`SCIMESH_DB=sqlite`), an embedded
userservice, and `serve`/`agent` subcommands, so one downloaded binary runs
the whole platform — coordinator, both databases, UI logins, and local
workers — with no PostgreSQL, no Docker, and no environment variables. The
first start provisions `~/.scimesh` (secrets, admin password printed once,
managed scientific-runtime venv) and opens the UI. `install.sh` / `install.ps1`
download the right release binary in one command and are release assets. The
PostgreSQL engine, the `setup` wizard, and the standalone `users/` service
remain fully supported for cluster deployments. The full E2E passes with zero
external services: health, UI login, job upload, local agent compute,
reduction, and a byte-exact final CSV.
The User Service is merged into `main`. It owns user accounts, authentication,
roles, and verified-contributor status; the coordinator scopes user jobs and
@@ -37,24 +48,25 @@ the complete result-artifact SHA-256 before a task is accepted.
| --- | --- | --- |
| CTX-00 API and error contract | Implemented | Contract, OpenAPI, and request examples are in `docs/`. |
| CTX-01 Go coordinator bootstrap | Implemented | Go service and Docker runtime in `coordinator/`. |
| CTX-02 PostgreSQL migrations | Implemented | Applied by the Compose migration service. |
| CTX-03 Transactional queue | Implemented | Real-PostgreSQL integration tests cover atomic claims and concurrency. |
| CTX-02 PostgreSQL migrations | Implemented | Embedded into the binary (`AUTO_MIGRATE`); the CLI path is still available for managed databases. |
| CTX-03 Transactional queue | Implemented | Real-PostgreSQL integration tests cover atomic claims and concurrency; the SQLite backend mirrors the semantics. |
| CTX-04 Worker registry and HTTP API | Implemented | Registration, claim, heartbeat, result, failure, and status endpoints. |
| CTX-05 Artifact storage | Implemented | Coordinator-owned inputs/results, checksum verification, and upload flow. |
| CTX-06 Python Worker live-contract alignment | Superseded | The Python worker daemon was removed; the Go worker agent (`coordinator/internal/agent/` + `cmd/worker-agent`) now implements the lifecycle (register/claim/heartbeat/download/upload/submit/fail, token refresh, cleanup) and executes SDK workloads via the Python task entry `scimesh/worker/task.py`. E2E: `make smoke-two-worker` passes 4/4 shards with two agents. |
| CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering are in `scimesh/distributed/`. |
| CTX-08 Distributed similarity-search | Implemented | Python planner resolves `query_id` once, creates deterministic shard plans, worker adapter emits exact partial top-k CSVs/metrics, and reducer matches the local reference. |
| CTX-09 Reducer and final-result API | Implemented | Atomic `reducing` claim, deterministic coordinator-side top-k reducer, sanitized reducer failure, final artifact persistence, `result_uri`, and final CSV download. |
| CTX-10 Distributed similarity-graph | Not started | Local reference exists; the SDK-built local graph workload already enforces the pair-coverage invariant. |
| CTX-11 Dashboard/operator view | Implemented | Protected live control room: MkDocs documentation served at `/ui/docs/` (SCIMESH_DOCS_DIR; the demo mounts `site/` automatically), recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, bounded polling, a Workload library page rendering the embedded catalog from `scimesh workload export` (`/ui/workloads`, regenerated via `make workloads-export`). |
| CTX-12 Reliability, security, CI | In progress | Unit, race, PostgreSQL integration, and smoke checks exist; CI hardening remains. |
| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, self-service enrollment, and quorum-backed untrusted workers are merged; local Go/Python and Docker/PostgreSQL checks passed. |
| MkDocs documentation site | Implemented | A standalone documentation site (`mkdocs/`, `docs_dir: mkdocs`) covering the complete Workload SDK: guides (overview, authoring workloads, CLI, worker integration), the full auto-generated API reference for all 15 `scimesh.sdk` modules (mkdocstrings), and the writing rules (`mkdocs/approach.md`). Built with `make docs`, served inside the UI at `/ui/docs/`; the project's internal `docs/` directory is not part of the site. |
| CTX-16 Workload SDK foundation | Implemented | `scimesh.sdk` provides strict immutable manifests/plans/artifacts, digest/trust-pinned tasks, typed DAGs, compatibility negotiation, verifier primitives with owner/binding-safe quorum inputs, resource eligibility/local allocation, measured package discovery, a trusted local core-batch conformance harness, and strict package discovery. Enforcing coordinator/Worker profiles remain fail-closed. |
| SDK roadmap step 3: `descriptor-batch` | Implemented | The first SDK-built reference workload (`scimesh/workloads/descriptors/`): pinned 81-name RDKit 2D descriptor set, canonical one-row-per-input CSV, deterministic row-bounded shards, shard-index concatenation with one header, byte-identical local/distributed output, and a two-worker `untrusted_quorum` verifier test. |
| SDK-built `similarity-search` and `similarity-graph` | Implemented | Both workloads are SDK-built packages (`scimesh/workloads/search/`, `scimesh/workloads/graph/`) built on the `MapReduceWorkload` authoring scaffold (`scimesh/sdk/batch.py`); they reuse the local scientific cores and are byte-identical to the single-process references (search; graph for both threshold directions and any block size). The graph reducer enforces the CTX-10 pair-coverage invariant. `scimesh/workloads/library.py` composes the built-in registry/runtime. |
| SDK-built `molwt-filter` | Implemented | The minimal authoring example (`scimesh/workloads/molwt_filter/`): filters molecules by exact RDKit molecular weight with only one scientific hook, using the scaffold's new default sharding and concatenation hooks. Registered in the built-in library and as a `scimesh.workloads` entry point. |
| SDK authoring scaffold | Implemented | `MapReduceWorkload` (exported from `scimesh.sdk`) assembles manifest, map/reduce stages, workflow, and digest-pinned handlers from three scientific hooks (partition/compute/merge), with overridable hooks for domain validation, plan-time resolution, custom task planning, and partial-key policy. The generic `scimesh workload list|run` CLI and the worker's allowlist-driven loading (`SCIMESH_WORKLOAD_ALLOWLIST`, `SCIMESH_CAPABILITIES`) let new workloads run without touching other code. |
| CTX-06 Python Worker live-contract alignment | Superseded | The Python worker daemon was removed; the Go worker agent (`coordinator/internal/agent/` + `cmd/worker-agent`, or `coordinator agent`) implements the lifecycle and executes SDK workloads via `scimesh/worker/task.py`. |
| CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering. |
| CTX-08 Distributed similarity-search | Implemented | Planner/worker/reducer match the local reference byte-exactly. |
| CTX-09 Reducer and final-result API | Implemented | Atomic `reducing` claim, deterministic coordinator-side reducers (`top-k` and `ordered-concat`), sanitized failure, final artifact, `result_uri`. |
| CTX-10 Distributed similarity-graph | Not started | Local reference exists; the SDK-built local graph workload enforces the pair-coverage invariant. |
| CTX-11 Dashboard/operator view | Implemented | Protected live control room, workload library, workload-agnostic "New computation" form (SDK-declared `UIElement`s), MkDocs at `/ui/docs/`, final-result download. |
| CTX-12 Reliability, security, CI | In progress | vet, gofmt, race tests, golangci-lint (0 issues), PostgreSQL integration, and smoke checks exist. |
| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, quorum; also embeddable (`coordinator serve`). |
| CTX-16 Workload SDK foundation | Implemented | Strict immutable manifests/plans/artifacts, digest/trust-pinned tasks, negotiation, verifier primitives, conformance harness. |
| CTX-17 Self-provisioning + setup wizard | Implemented | Embedded migrations, `coordinator setup`, SQLite backend, embedded userservice, `serve` mode. |
| CTX-18 Single-binary platform | Implemented | `coordinator serve` (data dir, secrets, admin bootstrap, local agents, managed venv) + `install.sh`/`install.ps1`; full no-external-service E2E green. |
| SDK roadmap step 3: `descriptor-batch` | Implemented | Byte-identical local/distributed output, quorum verifier test. |
| SDK-built `similarity-search` and `similarity-graph` | Implemented | SDK-built packages, byte-identical to single-process references. |
| SDK-built `molwt-filter` | Implemented | Minimal authoring example; also the single-binary E2E workload. |
| SDK authoring scaffold | Implemented | `MapReduceWorkload` with `UIElement` declarations, `reduction`, `upload_ready`; generic `scimesh workload list|run|export|allowlist` CLI. |
## Next recommended assignment
mkdocs/index.md
+35 -22
View File
@@ -38,15 +38,32 @@ The two halves of the project:
## Quick start
The fastest path for a scientist: install the platform with one command and
start it. Everything — the coordinator, its databases, the userservice, and
local workers — is embedded in a single binary; no PostgreSQL, no Docker, no
Python setup.
```bash
# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash
coordinator serve --open
# Windows (PowerShell)
powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex"
coordinator serve --open
```
The first start prints the admin login (also stored under `~/.scimesh`), and
`--open` opens the UI in the browser. `coordinator serve --workers 2`
spawns two local workers; `SCIMESH_PIP_PACKAGE` points the managed venv at
your scimesh wheel so scientific workloads can run.
For development, install the Python SDK and run workloads locally:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
```
List the installed SDK workloads and run one locally:
```bash
scimesh workload list
scimesh workload run molwt-filter \
--input molecules.tsv \
@@ -54,12 +71,6 @@ scimesh workload run molwt-filter \
-o filtered.csv
```
Run the local scientific CLI workloads:
```bash
scimesh help
```
Start the full demo (PostgreSQL, coordinator, UI, two workers):
```bash
@@ -71,8 +82,8 @@ make demo-ui
Every `v*` tag pushes a GitHub Release with static binaries for
`coordinator` and `worker-agent` on linux/darwin/windows × amd64/arm64
(plus SHA-256 checksums) and the `coordinator` image on GHCR. Download and
run:
(plus SHA-256 checksums), the installer scripts above, and the `coordinator`
image on GHCR. Download and run:
```bash
curl -L -o coordinator https://github.com/emil28092005/SciMesh/releases/latest/download/coordinator-linux-amd64
@@ -82,15 +93,17 @@ chmod +x coordinator
- **worker-agent** runs anywhere with Python: it spawns
`python -m scimesh.worker.task`, so the machine needs the `scimesh`
package in a venv (`pip install scimesh`) and the usual environment:
`COORDINATOR_URL`, `WORKER_AUTH_TOKEN`, `WORK_DIR`.
- **coordinator** needs PostgreSQL running (`DATABASE_URL`,
`COORDINATOR_STORAGE_DIR`, `JWT_SECRET`); the binary applies its embedded
schema migrations itself on startup (`AUTO_MIGRATE=false` opts out), so no
separate migration step is needed. The interactive wizard provisions the
rest: it checks the database, creates it when missing, generates a
`JWT_SECRET`, and writes a `.env` file (`coordinator setup --help`, or
`make setup`; `--yes` for non-interactive runs). The UI login additionally
requires `USERSERVICE_URL`.
`COORDINATOR_URL`, `WORKER_AUTH_TOKEN`, `WORK_DIR`. The same binary can run
it via `coordinator agent`.
- **coordinator** needs no external services at all in its default mode:
`coordinator serve` embeds SQLite (both databases), the userservice, and
local workers. The `SCIMESH_DB=postgres` engine remains for cluster
deployments (`DATABASE_URL`, `COORDINATOR_STORAGE_DIR`, `JWT_SECRET`); the
binary applies its embedded schema migrations itself on startup
(`AUTO_MIGRATE=false` opts out), and `coordinator setup` provisions a
PostgreSQL deployment interactively. The UI login uses the userservice
(`USERSERVICE_URL`) — embedded by `serve`, or the standalone `users/`
service otherwise.
`coordinator --version` / `worker-agent --version` print the build tag.
Build and serve this documentation site: