SciMesh/mkdocs/index.md

SciMesh

SciMesh is a local-first platform for scientific computation on molecular datasets. It turns a scientific run into independent tasks, dispatches them to Python workers, and deterministically combines the partial results into a checksum-protected final artifact.

The two halves of the project:

• The Workload SDK (scimesh.sdk) — a strict Python framework for authoring scientific workloads. Workloads are ordinary user scripts built on the SDK; they run locally, in the conformance harness, and on claimed coordinator tasks without touching any other part of the program.
• The coordinator and worker — a Go/PostgreSQL coordinator with an operator UI and Python worker agents that execute SDK-built workloads over an HTTP contract.

What is implemented

• SDK-built workloads: similarity-search (exact top-k Tanimoto search), similarity-graph (exact sparse similarity graph with pair-coverage), descriptor-batch (pinned RDKit 2D descriptors), and molwt-filter (molecular-weight filter — the minimal authoring example).
• MapReduceWorkload: the primary authoring scaffold. A subclass declares identity, parameters, ports, and scientific hooks; the SDK assembles the manifest, map/reduce stages, the digest-pinned planner/runner/reducer, and the exact-artifact verifier.
• A local conformance runtime (LocalCoreBatchExecutor): a trusted, in-process harness that validates scientific parity, sealed outputs, provenance, and limits.
• A distributed worker that executes the same SDK workload handlers on tasks claimed from the coordinator, with digest-pinned TaskSpecs, resource reservation, and allowlist-driven workload discovery.
• An operator UI served by the coordinator: the admin console (/ui/admin) for cluster operators — system/storage/health, jobs, worker trust, users and worker keys, workload enable/disable, metrics and the worker token — plus a workload-agnostic "new computation" form whose controls come from each workload's own UIElement declarations, job detail pages, a workload library page, and this documentation site at /ui/docs/.

Quick start

The fastest path for a scientist: install the platform with one command and start it. Everything — the coordinator, its databases, the userservice, and local workers — is embedded in a single binary; no PostgreSQL, no Docker, no Python setup.

# Linux / macOS — installs and opens the admin console automatically
curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash

# Windows (PowerShell)
powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.ps1 | iex"

The installer starts the platform and opens the admin console in your browser (set SCIMESH_AUTO_START=0 to install only). The first start prints the admin login (also stored under ~/.scimesh). coordinator serve --workers 2 spawns two local workers; SCIMESH_PIP_PACKAGE points the managed venv at your scimesh wheel so scientific workloads can run.

For development, install the Python SDK and run workloads locally:

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
scimesh workload list
scimesh workload run molwt-filter \
  --input molecules.tsv \
  --params '{"min_molwt": 40.0}' \
  -o filtered.csv

Start the full demo (PostgreSQL, coordinator, UI, two workers):

make demo-ui
# open http://localhost:18080/ui  (root@scimesh.local / rootpassword)

Prebuilt binaries

Every v* tag pushes a GitHub Release with static binaries for coordinator and worker-agent on linux/darwin/windows × amd64/arm64 (plus SHA-256 checksums), the installer scripts above, and the coordinator image on GHCR. Download and run:

curl -L -o coordinator https://github.com/emil28092005/SciMesh/releases/latest/download/coordinator-linux-amd64
chmod +x coordinator

• worker-agent is installed separately and joins an existing coordinator. Point its local setup wizard at the cluster — no need to have the coordinator on this machine:

  curl -fsSL https://raw.githubusercontent.com/emil28092005/SciMesh/main/install.sh | bash -s worker
  # the installer opens the local wizard at http://127.0.0.1:12700 automatically
  

  The wizard collects the coordinator URL and token (or worker key), runs a preflight check, saves the configuration under ~/.scimesh-worker/ and starts the worker as a background process — with a live status page and log. Everything can also be done by hand:

  export COORDINATOR_URL=http://COORDINATOR_HOST:8080
  export WORKER_AUTH_TOKEN=<worker token from the coordinator>
  export WORK_DIR=~/scimesh-worker
  worker-agent
  

  It spawns python -m scimesh.worker.task, so the machine needs Python 3 with the scimesh package. The wizard installs it into its own venv; the package must come from your wheel, checkout or index — point SCIMESH_PIP_PACKAGE at it (the PyPI name scimesh belongs to an unrelated project). For a coordinator serve instance, the worker token is in ~/.scimesh/worker.token. On Windows set SCIMESH_COMPONENT=worker for install.ps1.

• coordinator needs no external services at all in its default mode: coordinator serve embeds SQLite (both databases), the userservice, and local workers. The SCIMESH_DB=postgres engine remains for cluster deployments (DATABASE_URL, COORDINATOR_STORAGE_DIR, JWT_SECRET); the binary applies its embedded schema migrations itself on startup (AUTO_MIGRATE=false opts out), and coordinator setup provisions a PostgreSQL deployment interactively. The UI login uses the userservice (USERSERVICE_URL) — embedded by serve, or the standalone users/ service otherwise. coordinator --version / worker-agent --version print the build tag.

Build and serve this documentation site:

make docs
make docs-serve      # http://localhost:8000

Where to go next

• SDK overview — what the SDK is and is not.
• Authoring workloads — write your first workload with MapReduceWorkload.
• Workload CLI — list, run, and export workloads from the command line.
• Worker integration — how the distributed worker executes SDK workloads.
• API reference — the complete scimesh.sdk API, generated from docstrings.
• Documentation approach — the rules this site is written by.