Filter release artifacts to binaries in the download step

.github/workflows/coordinator.yml

@@ -60,7 +60,7 @@ jobs:
         run: go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@v4.17.1
 
       - name: apply migrations
-        run: migrate -path migrations -database "$TEST_DATABASE_URL" up
+        run: migrate -path internal/storage/postgres/migrations -database "$TEST_DATABASE_URL" up
 
       - name: integration tests
         run: go test -tags=integration ./internal/storage/postgres/ -v

.github/workflows/release.yml

@@ -0,0 +1,107 @@
+name: release
+
+# Builds static coordinator and worker-agent binaries for every major
+# platform on a v* tag push, attaches them (plus SHA-256 checksums) to the
+# GitHub Release, and pushes the coordinator image to GHCR.
+#
+#   git tag v1.0.0 && git push origin v1.0.0
+on:
+  push:
+    tags: ["v*"]
+
+permissions:
+  contents: write
+  packages: write
+
+jobs:
+  binaries:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [linux, darwin, windows]
+        arch: [amd64, arm64]
+    defaults:
+      run:
+        working-directory: coordinator
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: coordinator/go.mod
+          cache-dependency-path: coordinator/go.sum
+
+      - name: vet
+        run: go vet ./...
+
+      - name: build coordinator and worker-agent
+        env:
+          VERSION: ${{ github.ref_name }}
+        run: |
+          mkdir -p dist
+          for cmd in coordinator worker-agent; do
+            CGO_ENABLED=0 GOOS=${{ matrix.os }} GOARCH=${{ matrix.arch }} \
+              go build -trimpath \
+                -ldflags="-s -w -X main.version=${VERSION#v}" \
+                -o "dist/${cmd}-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.os == 'windows' && '.exe' || '' }}" \
+                "./cmd/${cmd}"
+          done
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: binaries-${{ matrix.os }}-${{ matrix.arch }}
+          path: coordinator/dist/*
+          if-no-files-found: error
+
+  release:
+    needs: binaries
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          path: artifacts
+          # Only the binary artifacts: the image job also uploads a buildkit
+          # cache artifact (*.dockerbuild) that download-artifact cannot fetch.
+          pattern: binaries-*
+          merge-multiple: true
+
+      - name: checksums
+        working-directory: artifacts
+        run: sha256sum * > SHA256SUMS.txt
+
+      - uses: softprops/action-gh-release@v2
+        with:
+          files: artifacts/*
+          generate_release_notes: true
+
+  image:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: docker/setup-buildx-action@v3
+
+      - uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - uses: docker/metadata-action@v5
+        id: meta
+        with:
+          images: ghcr.io/${{ github.repository }}/coordinator
+          tags: |
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=raw,value=latest
+
+      - uses: docker/build-push-action@v6
+        with:
+          context: coordinator
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          build-args: |
+            VERSION=${{ github.ref_name }}

.gitignore

@@ -16,3 +16,5 @@ test_structures/
 worker-data*/
 scimesh-worker-data/
 coordinator/.demo/
+site/
+coordinator/bin/

.opencode/session-goal.md

@@ -0,0 +1,21 @@
+COMPLETED
+# Session Goal
+
+давай теперь почистим проект от линего кода
+
+## Plan
+
+1. Аудит: ruff/pyflakes — неиспользуемые импорты по scimesh/ и tests/; grep — неиспользуемые функции/модули (после рефакторингов могли остаться мёртвые экспорты, например в descriptors/search/graph core и sdk/_validation).
+2. Удаление мёртвого кода: неиспользуемые импорты, функции, дубли (например write_descriptor_shards/concatenate_descriptor_shards, если вытеснены дефолтами batch), устаревшие файлы-обёртки.
+3. Проверка, что ничего публичного/API не сломано: pyright 0 ошибок, pytest зелёный, go test/vet, mkdocs build.
+4. Финал: полный прогон, COMPLETED.
+
+## Progress
+
+Эта сессия (доп. задача): Go-агент доведён до паритета, Python-демон удалён.
+- [x] Go-агент: token provider (static + worker-key exchange + 401 refresh на API/download/upload), CLEANUP_AFTER_SECONDS (очистка attempt-директорий), тесты auth (exchange/cache/reject/select/401-retry).
+- [x] Python: удалены daemon.py, cli.py, config.py, coordinator.py, artifacts.py, auth.py, transport.py; `scimesh/worker/` = только task.py + runners.py (SDK-мост, allowlist из env) + models.py (ClaimedTask/RunResult); консольный скрипт scimesh-worker убран из pyproject.
+- [x] Тесты: удалены test_worker_daemon.py, test_worker_auth.py; 208 pytest зелёные.
+- [x] Демо/смок переведены на Go-агент (demo-ui.sh: build_agent + env; two-worker-smoke.sh: AGENT_BIN + TASK_RUNNER_JSON). `make smoke-two-worker` PASS: 4/4 шардов, worker-a=2, worker-b=2.
+- [x] Документация: README, AGENTS.md, STATUS, handoff, mkdocs worker-integration/cli.
+- [x] Верификация: ruff clean; 208 pytest; pyright 0 (scimesh+tests); go test 11 пакетов + vet; mkdocs 0 warnings.

AGENTS.md

@@ -5,8 +5,10 @@
 SciMesh is a Python package for molecular-similarity workloads. Source lives in
 `scimesh/`: `chemistry/` reads data and makes fingerprints, `workloads/`
 contains commands, and `core/` provides the workload protocol and registry.
-The worker daemon in `scimesh/worker/` is a coordinator client, not a database
-client. Tests are in `tests/`; specifications in `docs/`; roadmap: `PLAN.md`.
+The Go worker agent (`coordinator/internal/agent/`) is a coordinator client,
+not a database client; the Python side of a claimed task lives in
+`scimesh/worker/` (the per-task SDK execution entry). Tests are in `tests/`;
+specifications in `docs/`; roadmap: `PLAN.md`.
 
 For distributed work, read `.agents/`, `docs/api-contract.md`,
 and `STATUS.md`. Use one CTX task per pull request; local workloads are the

Makefile

@@ -1,19 +1,32 @@
 .DEFAULT_GOAL := help
 
-.PHONY: help demo-ui demo-down demo-logs
+.PHONY: help agent coordinator demo-ui demo-down demo-logs smoke-two-worker docs docs-serve
 
 help:
 	@printf '%s\n' \
 	  'SciMesh developer commands:' \
-	  '  make demo-ui                 Start the local UI pipeline demo with 2 workers.' \
-	  '  make demo-ui WORKERS=3       Start the demo with 3 local workers.' \
+	  '  make agent                   Build the Go worker agent (coordinator/bin/worker-agent).' \
+	  '  make coordinator             Build the coordinator server as a static binary' \
+	  '                              (coordinator/bin/coordinator).' \
+	  '  make demo-ui                 Start the local UI pipeline demo with 2 Go worker agents.' \
+	  '  make demo-ui WORKERS=3       Start the demo with 3 workers.' \
 	  '  make demo-logs               Follow coordinator logs for the demo.' \
 	  '  make demo-down               Stop demo containers and workers.' \
+	  '  make smoke-two-worker        E2E: two Go agents process 4 shards and the' \
+	  '                              result must match the local CLI reference.' \
+	  '  make docs                    Build the MkDocs site into site/.' \
+	  '  make docs-serve              Serve the MkDocs site at http://localhost:8000.' \
 	  '' \
 	  'After make demo-ui: open http://localhost:18080/ui (operator / demo-ui-secret).'
 
 # Convenient entry points from the repository root. Extra settings are passed
 # through, for example: make demo-ui WORKERS=3
+agent:
+	$(MAKE) -C coordinator agent
+
+coordinator:
+	$(MAKE) -C coordinator coordinator
+
 demo-ui:
 	$(MAKE) -C coordinator demo-ui
 
@@ -22,3 +35,12 @@ demo-down:
 
 demo-logs:
 	$(MAKE) -C coordinator demo-logs
+
+smoke-two-worker:
+	./scripts/two-worker-smoke.sh
+
+docs:
+	.venv/bin/mkdocs build
+
+docs-serve:
+	.venv/bin/mkdocs serve

PLAN.md

@@ -5,11 +5,13 @@
 > platform. It is intentionally detailed enough to split into independent task
 > briefs for developers or coding agents.
 >
-> **Planning baseline.** This branch starts from `Workers`: the Python package
-> has local `similarity-search` and `similarity-graph` workloads plus a Worker
-> Daemon client. The coordinator and PostgreSQL implementation do not yet
-> exist. The Worker contract and the Go/PostgreSQL design briefs in `docs/` are
-> part of this plan.
+> **Current planning baseline (2026-08-01).** The Go/PostgreSQL coordinator,
+> Python Worker Agent, versioned distributed-workload protocol, artifact-backed
+> task lifecycle, reducer orchestration, operator UI, User Service, and
+> distributed `similarity-search` are implemented on `main`. The evidence-based
+> completion tracker is [`STATUS.md`](STATUS.md); this document defines the
+> remaining direction and dependencies. Earlier descriptions of a missing
+> coordinator are historical context, not current work.
 
 ---
 
@@ -51,7 +53,7 @@ Coordinator reducer -> final artifact -> download/status API
 
 ### 2.1 In scope
 
-- Go 1.22+ coordinator service with PostgreSQL 15+;
+- Go 1.25+ coordinator service with PostgreSQL 15+;
 - Python Worker Daemon running existing SciMesh workloads locally;
 - durable job, task, worker, and artifact metadata;
 - local coordinator-managed artifact storage for the first deployment;
@@ -66,7 +68,8 @@ Coordinator reducer -> final artifact -> download/status API
 
 - cloud object storage, Kubernetes, autoscaling, and multi-region operation;
 - arbitrary shell commands sent by coordinator to workers;
-- user accounts, multi-tenancy, billing, or sophisticated authorization;
+- billing and sophisticated multi-tenant administration beyond the implemented
+  User Service and owner scoping;
 - GPU scheduling and multiprocessing inside a worker;
 - Docker as a required runtime dependency;
 - video/CV processing implementation;
@@ -490,14 +493,84 @@ brute-force graph for both `greater` and `less` threshold directions.
 
 ### 7.3 Future workload policy
 
-A new workload is accepted only when it supplies:
+A workload is more than a runner. It must define validation, planner and
+reducer behavior, worker allowlist/capabilities, input and output artifacts,
+UI/API parameters, reproducible execution environment, result verification,
+golden cross-worker fixtures, and applicable resource limits. The future public
+contract is described in [`docs/scimesh-sdk-roadmap.md`](docs/scimesh-sdk-roadmap.md).
+Its normative future interfaces and execution semantics are defined in the
+design-draft [`docs/scimesh-sdk-contract.md`](docs/scimesh-sdk-contract.md).
 
-- an input/parameter validator;
-- an explicit sharding strategy;
-- bounded-memory task execution;
-- deterministic reduction semantics;
-- fixture-based local and distributed correctness tests;
-- a `describe()` payload for UI/API discovery.
+Every workload declaration must classify its task decomposition and input/output
+artifact shapes, determinism, reduction semantics, verifier mode, supported
+trust modes, CPU/memory/accelerator needs, and maximum output growth. The
+initial profiles are:
+
+| Profile | Current acceptance policy |
+| --- | --- |
+| Byte-exact deterministic | Supported for untrusted quorum when whole artifacts have identical SHA-256. |
+| Canonical-exact deterministic | Deferred until the parser, schema, ordering, encoding, and serializer are versioned. |
+| Numeric deterministic with tolerance | Deferred until structured numeric comparison exists. |
+| Stochastic/search-based | Requires domain-specific evidence, repeated runs, or trusted execution. |
+| Trusted-only or domain-verified | May be planned only with an explicit trust policy and verifier. |
+
+The current untrusted quorum records one vote per owner and accepts a task only
+when distinct owners upload artifacts with the same complete-file SHA-256. It
+therefore supports only the byte-exact profile (or a workload that first makes
+its output byte-identical through a specified canonicalization step). Reducers
+must fail safely rather than silently merge inconsistent partial results.
+
+Before a workload is admitted to untrusted execution it needs a reproducibility
+gate: pinned environment/container digest and dependency versions; fixed locale,
+timezone, UTF-8/newline/CSV settings; explicit invalid-row and algorithm
+options; canonical representation and ordering; deterministic filenames/archive
+metadata; golden fixtures from two independently provisioned workers; local vs
+distributed parity; and retry/out-of-order completion tests. A loose dependency
+constraint is insufficient for byte-exact quorum.
+
+Near-term critical path:
+
+```text
+distributed similarity-graph
+  -> reliability, security, and cross-language CI
+    -> stable first release
+      -> SDK foundation and descriptor-batch
+        -> additional deterministic workloads
+```
+
+Initial deterministic-workload backlog: `descriptor-batch` (the first SDK
+reference workload), molecule standardization, SMARTS screening, fingerprint
+export, fixed-template SMIRKS enumeration with strict caps, and reaction
+validation/descriptors. `similarity-graph` remains ahead of this backlog.
+Bounded combinatorial libraries and seeded conformers need specialized controls.
+ML, retrosynthesis, docking, QM, molecular dynamics, and GPU workloads are
+deferred until verifier/trust and reproducibility requirements are met.
+
+### 7.4 Future verification, concurrency, and accelerators
+
+Verification is a future versioned workload capability, not permanent
+whole-file-SHA logic. Planned modes are `ExactArtifactVerifier`,
+`CanonicalRecordVerifier`, `NumericToleranceVerifier`, `DomainSpecificVerifier`,
+and `TrustedWorkerPolicy`. Exact SHA-256 remains the first and safest mode;
+canonical and numeric modes must compare bounded structured data and publish
+sanitized evidence and failure reasons.
+
+Worker concurrency remains **1** until implemented and tested. Its target model
+is one physical machine running one Worker Agent with `N` execution slots and
+one isolated subprocess per active Task, rather than one registered worker per
+CPU core. `max_concurrency` must be separate from `cpu_count`; each task keeps
+its own heartbeat, attempt directory, lease lifecycle, resource request, and
+graceful-drain behavior. CPU-bound scientific code should use processes and
+avoid nested oversubscription.
+
+Accelerator support is also deferred. The coordinator matches generic resource
+requirements; the Worker Agent discovers and isolates devices (including
+`CUDA_VISIBLE_DEVICES`) and owns process/accounting lifecycle; the Python
+workload owns batching, memory strategy, deterministic output, and scientific
+validation; reducers/verifiers define CPU/GPU-independent semantics. CUDA and
+scientific kernels do not belong in the Go coordinator. GPU work follows stable
+CPU slices, generic resource requirements, pinned worker images, and tested
+CPU/GPU or domain-valid equivalence.
 
 ---
 
@@ -820,6 +893,138 @@ CTX-09 enables final result downloads.
 - failure/retry scenarios have automated coverage;
 - README contains architecture diagram, security caveat, and troubleshooting.
 
+### CTX-13 — In-worker CPU parallelism
+
+**Goal:** Allow a worker to use a bounded, configured number of CPU threads or
+processes while preserving the existing one-task-per-lease coordinator model.
+
+**Depends on:** CTX-12.
+
+**Acceptance criteria:**
+
+- worker concurrency is an explicit configuration value with a safe default of
+  one;
+- a task's internal parallel execution has bounded memory and does not build a
+  dense N×N similarity matrix;
+- CPU-parallel `similarity-search` and `similarity-graph` outputs match the
+  single-threaded local reference byte-for-byte where ordering is observable;
+- result ordering is deterministic across worker counts and block sizes;
+- cancellation, lease loss, and worker failure stop child work safely and do
+  not report a successful result;
+- benchmarks and tests cover one-worker and multi-worker configurations.
+
+### CTX-14 — GPU-accelerated workload execution
+
+**Goal:** Add an optional GPU execution backend for supported molecular
+workloads, while retaining the validated CPU implementation as the reference
+and fallback.
+
+**Depends on:** CTX-13.
+
+**Acceptance criteria:**
+
+- GPU capability and backend version are advertised explicitly by a worker;
+- the coordinator schedules GPU work only to compatible workers and CPU-only
+  workers continue to claim CPU tasks;
+- unsupported hardware, unavailable drivers, and GPU execution errors produce
+  sanitized failures or a documented CPU fallback;
+- GPU results match the CPU reference within a documented, tested numerical
+  tolerance and preserve deterministic output ordering;
+- GPU memory use is bounded and no dense N×N similarity matrix is created;
+- CPU-only CI verifies backend selection and contract behavior, with GPU
+  integration tests documented for compatible runners.
+
+### CTX-15 — User Service and access control
+
+**Goal:** Introduce a dedicated User Service for user identity and access
+control, without coupling workers to user credentials or moving scientific
+workload logic into the service.
+
+**Depends on:** CTX-12.
+
+**Acceptance criteria:**
+
+- the service has a versioned, documented API in
+  [`docs/user-service-api-contract.md`](docs/user-service-api-contract.md) and
+  owns user identity data;
+- credentials and authentication tokens are stored and handled securely; they
+  are never logged or exposed to workers;
+- authenticated identity is propagated to coordinator requests through an
+  explicit, validated boundary;
+- authorization restricts access to jobs and artifacts to the intended user or
+  project;
+- unauthenticated, expired-token, and cross-user access attempts have
+  automated failure tests;
+- the existing single-operator demo remains usable through a documented local
+  development configuration.
+
+### CTX-16 — Workload SDK foundation
+
+**Goal:** Provide a strict Python authoring SDK for installed, allowlisted
+scientific workloads while retaining the CTX-07 distributed protocol as a
+compatible wire profile.
+
+**Depends on:** CTX-07 and CTX-08. Coordinator-backed generalized scheduling
+also depends on CTX-10 through CTX-14, but the Python contract and local
+conformance runtime can land independently and must fail closed for unavailable
+features.
+
+**Acceptance criteria:**
+
+- public manifest, workflow, task, artifact, resource, execution, provenance,
+  and verifier value objects are immutable, typed, JSON-safe, versioned, and
+  strict about unknown fields;
+- installed workload discovery requires an administrator allowlist plus exact
+  workload version and package digest; job parameters cannot select code;
+- compatibility negotiation covers SDK/protocol/profile/feature/environment
+  versions and occurs before planner invocation;
+- plans/tasks pin package and manifest digests plus selected trust mode, and
+  quorum candidates carry coordinator-owned candidate/owner and scientific
+  binding identities;
+- `core-batch-v1` has a trusted local conformance executor with atomic resource
+  reservation, sealed-output/provenance validation, declared verifier
+  invocation, and golden scientific parity;
+- exact, canonical-record, and structured numeric-tolerance verifier
+  primitives return bounded sanitized decisions;
+- the existing distributed `similarity-search` is available through an adapter
+  without changing its wire schema, worker alias boundary, or scientific
+  result, and parity is tested;
+- advanced dynamic, stream, accelerator, gang, and side-effect profiles are
+  rejected unless an enforcing runtime advertises their required features;
+- an author guide documents package entry points, security boundaries,
+  conformance tests, and current coordinator/Worker limitations.
+
+---
+
+### CTX-17 — Self-provisioning coordinator and setup wizard
+
+**Goal:** A downloaded coordinator binary should bring up a working platform
+with as little manual configuration as possible: it provisions its own
+schema, and a `setup` command walks the operator through the remaining
+environment (database creation, secrets, admin account).
+
+**Depends on:** the Go coordinator and the release build pipeline. Step 1
+(embedded migrations, `AUTO_MIGRATE`) and step 2 (the `coordinator setup`
+wizard: database reachability and creation, schema migration, `.env` with a
+generated `JWT_SECRET`, readiness summary) are implemented; step 3 — a fully
+embedded userservice — is the remaining work.
+
+**Acceptance criteria:**
+
+- the binary embeds the migrations and applies pending ones on startup by
+  default (`AUTO_MIGRATE=false` opts out for managed databases); applying is
+  idempotent and safe under concurrent starts;
+- `coordinator setup` (interactive, then non-interactive with `--yes`) checks
+  database reachability, offers to create the role/database when credentials
+  allow it, applies migrations, writes a `.env` with a generated `JWT_SECRET`
+  and storage path, and prints exact next steps;
+- `coordinator --version` and the setup output agree on the release build;
+- the wizard explains what it cannot do itself: running PostgreSQL and the
+  userservice, with concrete commands (docker compose, systemd) to finish;
+- the Docker image keeps working without the separate migrate step, and the
+  release workflow publishes the binaries that support `setup`;
+- setup fails closed on non-interactive input and never logs secrets.
+
 ---
 
 ## 10. Suggested assignment bundles
@@ -835,6 +1040,7 @@ parallel unless one engineer owns integration.
 | Distributed computation | CTX-07, CTX-08, CTX-10 | Scientific Python engineer |
 | Product surface | CTX-09, CTX-11 | Full-stack/backend engineer |
 | Quality gate | CTX-12 | DevOps/QA engineer |
+| Workload SDK | CTX-16 | Scientific Python/platform engineer |
 
 Suggested order for a small team:
 
@@ -948,15 +1154,24 @@ Before merging a task, reviewer checks:
 Do not start these before CTX-12 is accepted.
 
 - Replace local artifact storage with S3/MinIO behind an `ArtifactStore` API.
-- Add worker labels/capacity-aware scheduling and concurrency > 1.
+- Add worker labels and capacity-aware scheduling.
+- Implement CTX-13 for bounded in-worker CPU parallelism.
+- Implement CTX-14 for optional GPU-accelerated workload execution.
+- Implement CTX-15 for the User Service and authenticated user/project access.
 - Add cancellation propagation to workers.
 - Add image outputs and final PDF reporting to job artifacts.
 - Add CV/video workloads using the same planner/runner/reducer contract.
 - Add observability export (Prometheus/OpenTelemetry).
-- Add per-user/project authorization and signed artifact URLs.
+- Add signed artifact URLs.
 - Add shard caching and content-addressed input deduplication.
 - Add job priority and fair scheduling.
 - Add a CLI for submitting and monitoring remote jobs.
+- Implement CTX-17 step 3: a fully embedded userservice
+  (`coordinator userservice` subcommand) so one binary can serve the whole
+  platform without containers.
+- Replace PostgreSQL with an embedded SQLite backend for fully self-contained
+  single-binary deployments (large storage-layer change; postgres row locks,
+  transactions, and integration tests must be re-derived).
 
 ---
 

README.md

@@ -3,7 +3,8 @@
 SciMesh is a scientific-workload framework for molecular datasets. Its public CLI
 runs exact similarity search and sparse similarity-graph construction locally in
 one Python process; it creates no dense similarity matrix. The Go/PostgreSQL
-coordinator and Python worker can run a shard-based `similarity-search`
+coordinator and Go worker agents (which execute SDK workloads in a Python
+subprocess) can run a shard-based `similarity-search`
 pipeline locally. After every shard succeeds, the coordinator deterministically
 merges its candidates into one final global top-k CSV. See
 [`STATUS.md`](STATUS.md).
@@ -56,12 +57,24 @@ python3 -m venv .venv
 make demo-ui
 ```
 
+The MkDocs documentation site is served inside the UI at `/ui/docs/`
+(`make docs` builds it from `mkdocs/`; the demo mounts `site/`
+automatically, or set `SCIMESH_DOCS_DIR` for a manual coordinator). The site
+covers the complete Workload SDK: guides (`mkdocs/sdk/`), the full
+auto-generated API reference for `scimesh.sdk` (`mkdocs/api/`), and the
+documentation rules the site is written by (`mkdocs/approach.md`).
+
 Open `http://localhost:18080/ui` and sign in with username `operator` and
 password `demo-ui-secret`. The command starts PostgreSQL, the coordinator, and
-two local reference workers. Upload a small ChEMBL TSV, then use the job page
+two Go worker agents (built by `make agent`; each executes the SDK workload
+in a Python subprocess). Upload a small ChEMBL TSV, then use the job page
 to follow shard progress, inspect bounded **Preview CSV** results, and see a
-live processing-speed chart in shards per minute. To change the worker count,
-run `make demo-ui WORKERS=3`; stop everything with `make demo-down`.
+live processing-speed chart in shards per minute. The **Workloads** page shows
+the installed SDK workload library (descriptions, parameters, and artifact
+schemas) from the embedded catalog; regenerate it with
+`make workloads-export` (or `scimesh workload export`) whenever workloads
+change. To change the worker count, run `make demo-ui WORKERS=3`; stop
+everything with `make demo-down`.
 
 Run `make help` to display these commands in the terminal.
 
@@ -133,9 +146,39 @@ pytest
 
 The package separates common dataset parsing and fingerprints from independent workloads. Add future workloads through the workload registry without changing the main CLI.
 
+## Workload SDK
+
+`scimesh.sdk` is the framework only: strict and immutable workload manifests,
+typed artifact ports, static map/reduce plans, resource eligibility and local
+reservations, exact/canonical/numeric verifier primitives, installed-package
+allowlisting, and a local conformance executor. It contains no scientific
+workload code. Workloads are user scripts built on the SDK: the built-in
+`similarity-search`, `similarity-graph`, and `descriptor-batch` live in
+`scimesh/workloads/` (each a small package with `core.py` + `definition.py`),
+composed by `scimesh/workloads/library.py` and registered through
+`scimesh.workloads` entry points. The Worker Agent executes those SDK-built
+workloads directly (see `scimesh/worker/runners.py`), so the same scientific
+handlers run locally, in conformance, and on claimed coordinator tasks.
+`scimesh workload list` and `scimesh workload run` run any SDK workload from
+the command line. See the
+[SDK author guide](docs/workload-sdk.md), [contract](docs/scimesh-sdk-contract.md),
+and [delivery roadmap](docs/scimesh-sdk-roadmap.md).
+
+Dynamic workflows, real Worker concurrency, coordinator-backed GPU allocation,
+streaming, and gang execution remain fail-closed until their versioned runtime
+features are implemented; declaring those profiles does not silently enable
+them.
+
+The included `LocalCoreBatchExecutor` is a trusted, single-threaded in-process
+conformance harness. It validates scientific parity, sealed outputs, provenance,
+and limits, but intentionally refuses profiles that claim network/process
+isolation, secrets, accelerators, gangs, checkpoints, or retries; those require
+the future enforcing Agent runtime.
+
 ## Team
 
 - [Emil](https://github.com/emil28092005) — Project Lead
 - [Kristina](https://github.com/kristtma) — Tech Lead
+- [Veniamin](https://t.me/Veniamin_Kt) — Scientific Lead
 - [Arkhip](https://github.com/hIpa-ussr) — Programmer
-- [Makar](https://github.com/RERAN4K) — Programmer
+- [Reranchik](https://github.com/RERAN4K) — Programmer

STATUS.md

@@ -1,7 +1,7 @@
 # SciMesh Status
 
-**Updated:** 2026-07-24
-**Branch baseline:** `main` at `6e67daa` (distributed similarity-search)
+**Updated:** 2026-08-01
+**Branch baseline:** `main`; this revision adds the Workload SDK foundation.
 
 ## Current state
 
@@ -23,6 +23,14 @@ are reduced once into a checksum-protected final CSV, which is downloadable
 through the coordinator. The full Go checks (including a fresh migration and
 real PostgreSQL smoke test) passed on 2026-07-24.
 
+The User Service is merged into `main`. It owns user accounts, authentication,
+roles, and verified-contributor status; the coordinator scopes user jobs and
+worker operations to the authenticated owner. Its documented v1 contract is in
+[`docs/user-service-api-contract.md`](docs/user-service-api-contract.md).
+Users can create and revoke worker keys for self-service Worker Agent
+enrollment. Untrusted workers require quorum agreement from distinct owners on
+the complete result-artifact SHA-256 before a task is accepted.
+
 ## Milestone tracker
 
 | CTX | Status | Notes |
@@ -33,13 +41,20 @@ real PostgreSQL smoke test) passed on 2026-07-24.
 | CTX-03 Transactional queue | Implemented | Real-PostgreSQL integration tests cover atomic claims and concurrency. |
 | CTX-04 Worker registry and HTTP API | Implemented | Registration, claim, heartbeat, result, failure, and status endpoints. |
 | CTX-05 Artifact storage | Implemented | Coordinator-owned inputs/results, checksum verification, and upload flow. |
-| CTX-06 Python Worker live-contract alignment | Implemented | Worker completed a real uploaded shard via HTTP on 2026-07-23. |
+| CTX-06 Python Worker live-contract alignment | Superseded | The Python worker daemon was removed; the Go worker agent (`coordinator/internal/agent/` + `cmd/worker-agent`) now implements the lifecycle (register/claim/heartbeat/download/upload/submit/fail, token refresh, cleanup) and executes SDK workloads via the Python task entry `scimesh/worker/task.py`. E2E: `make smoke-two-worker` passes 4/4 shards with two agents. |
 | CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering are in `scimesh/distributed/`. |
 | CTX-08 Distributed similarity-search | Implemented | Python planner resolves `query_id` once, creates deterministic shard plans, worker adapter emits exact partial top-k CSVs/metrics, and reducer matches the local reference. |
 | CTX-09 Reducer and final-result API | Implemented | Atomic `reducing` claim, deterministic coordinator-side top-k reducer, sanitized reducer failure, final artifact persistence, `result_uri`, and final CSV download. |
-| CTX-10 Distributed similarity-graph | Not started | Local reference exists. |
-| CTX-11 Dashboard/operator view | Implemented | Protected live control room: recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, and bounded polling. |
+| CTX-10 Distributed similarity-graph | Not started | Local reference exists; the SDK-built local graph workload already enforces the pair-coverage invariant. |
+| CTX-11 Dashboard/operator view | Implemented | Protected live control room: MkDocs documentation served at `/ui/docs/` (SCIMESH_DOCS_DIR; the demo mounts `site/` automatically), recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, bounded polling, a Workload library page rendering the embedded catalog from `scimesh workload export` (`/ui/workloads`, regenerated via `make workloads-export`). |
 | CTX-12 Reliability, security, CI | In progress | Unit, race, PostgreSQL integration, and smoke checks exist; CI hardening remains. |
+| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, self-service enrollment, and quorum-backed untrusted workers are merged; local Go/Python and Docker/PostgreSQL checks passed. |
+| MkDocs documentation site | Implemented | A standalone documentation site (`mkdocs/`, `docs_dir: mkdocs`) covering the complete Workload SDK: guides (overview, authoring workloads, CLI, worker integration), the full auto-generated API reference for all 15 `scimesh.sdk` modules (mkdocstrings), and the writing rules (`mkdocs/approach.md`). Built with `make docs`, served inside the UI at `/ui/docs/`; the project's internal `docs/` directory is not part of the site. |
+| CTX-16 Workload SDK foundation | Implemented | `scimesh.sdk` provides strict immutable manifests/plans/artifacts, digest/trust-pinned tasks, typed DAGs, compatibility negotiation, verifier primitives with owner/binding-safe quorum inputs, resource eligibility/local allocation, measured package discovery, a trusted local core-batch conformance harness, and strict package discovery. Enforcing coordinator/Worker profiles remain fail-closed. |
+| SDK roadmap step 3: `descriptor-batch` | Implemented | The first SDK-built reference workload (`scimesh/workloads/descriptors/`): pinned 81-name RDKit 2D descriptor set, canonical one-row-per-input CSV, deterministic row-bounded shards, shard-index concatenation with one header, byte-identical local/distributed output, and a two-worker `untrusted_quorum` verifier test. |
+| SDK-built `similarity-search` and `similarity-graph` | Implemented | Both workloads are SDK-built packages (`scimesh/workloads/search/`, `scimesh/workloads/graph/`) built on the `MapReduceWorkload` authoring scaffold (`scimesh/sdk/batch.py`); they reuse the local scientific cores and are byte-identical to the single-process references (search; graph for both threshold directions and any block size). The graph reducer enforces the CTX-10 pair-coverage invariant. `scimesh/workloads/library.py` composes the built-in registry/runtime. |
+| SDK-built `molwt-filter` | Implemented | The minimal authoring example (`scimesh/workloads/molwt_filter/`): filters molecules by exact RDKit molecular weight with only one scientific hook, using the scaffold's new default sharding and concatenation hooks. Registered in the built-in library and as a `scimesh.workloads` entry point. |
+| SDK authoring scaffold | Implemented | `MapReduceWorkload` (exported from `scimesh.sdk`) assembles manifest, map/reduce stages, workflow, and digest-pinned handlers from three scientific hooks (partition/compute/merge), with overridable hooks for domain validation, plan-time resolution, custom task planning, and partial-key policy. The generic `scimesh workload list|run` CLI and the worker's allowlist-driven loading (`SCIMESH_WORKLOAD_ALLOWLIST`, `SCIMESH_CAPABILITIES`) let new workloads run without touching other code. |
 
 ## Next recommended assignment
 
@@ -48,14 +63,27 @@ block-pair planning and reduction for `similarity-graph`.
 
 ## Known constraints
 
-- The worker/coordinator flow currently accepts both underscore API workload
-  names and hyphenated CLI names while the contract is consolidated.
+- The worker/coordinator flow accepts both underscore API workload names and
+  hyphenated names at the runner boundary; the runner normalizes them.
+- The worker executes SDK-built workloads through `scimesh/worker/runners.py`
+  (a workload-generic v1-wire bridge over `TaskSpec`/`LocalTaskContext`);
+  `query_id` resolution and parameter validation live in the workload itself.
+  `max_rows` is a plan-time option and is rejected per task by the stage
+  projection.
 - A real-stack worker test uses a small `query_smiles` shard. The Python
   planner resolves `query_id` once and shares `query_smiles`; the upload UI
   currently accepts `query_smiles` only.
 - The coordinator accepts uploaded distributed jobs only for
   `similarity-search` with `query_smiles`. It rejects `similarity-graph` until
   CTX-10 supplies cross-shard pair planning.
+- The SDK can execute `core-batch-v1` locally, but the protocol-v1 coordinator
+  still has flat single-input/single-result tasks and no package/resource
+  leases. General DAG, concurrent-Agent, GPU, stream, and gang execution needs
+  a versioned coordinator/Worker rollout; unsupported features fail before
+  planner invocation.
+- The local SDK executor is intentionally trusted and in-process. It does not
+  enforce process/network/timeout/credential isolation and rejects declarations
+  that would require those guarantees.
 
 ## Update rule
 

coordinator/Dockerfile

@@ -16,6 +16,9 @@ RUN --mount=type=cache,target=/go/pkg/mod go mod download
 
 COPY . .
 
+# Release builds inject the tag via build-arg; local builds stay "dev".
+ARG VERSION=dev
+
 # The cache mounts persist the module cache and the compiler's build cache
 # *across* builds, so a rebuild after a code edit recompiles only what changed
 # instead of the whole dependency tree.
@@ -25,7 +28,7 @@ COPY . .
 RUN --mount=type=cache,target=/go/pkg/mod \
     --mount=type=cache,target=/root/.cache/go-build \
     CGO_ENABLED=0 GOOS=linux go build \
-        -trimpath -ldflags="-s -w" \
+        -trimpath -ldflags="-s -w -X main.version=${VERSION#v}" \
         -o /out/coordinator ./cmd/coordinator
 
 # --- runtime stage --------------------------------------------------------

coordinator/Makefile

@@ -1,6 +1,6 @@
 .DEFAULT_GOAL := help
 
-.PHONY: help build run test test-integration vet lint tidy check migrate-up migrate-down up down down-clean logs ps rebuild psql smoke demo-ui demo-down demo-reset demo-logs
+.PHONY: help build run test test-integration vet lint tidy check migrate-up migrate-down up down down-clean logs ps rebuild psql smoke agent coordinator setup workloads-export demo-ui demo-down demo-reset demo-logs
 
 # `check` deliberately uses its own Compose project and host ports. This keeps
 # it from connecting to or replacing a developer's local PostgreSQL instance.
@@ -26,6 +26,40 @@ DEMO_WORKERS ?= 2
 WORKERS ?= $(DEMO_WORKERS)
 DEMO_DIR ?= .demo
 
+# workloads.json is the UI workload catalog, generated from the Python SDK
+# workload library. It is checked in so the binary embeds it; regenerate it
+# whenever workloads or their manifests change (requires the Python venv).
+WORKLOADS_JSON := internal/workloads/workloads.json
+
+# Version injected into the binaries via -ldflags; falls back to "dev".
+VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
+LDFLAGS := -s -w -X main.version=$(VERSION)
+
+# The Go worker agent: a static coordinator client that executes SDK
+# workloads in a Python subprocess per claimed task.
+agent:
+	CGO_ENABLED=0 go build -trimpath -ldflags="$(LDFLAGS)" -o bin/worker-agent ./cmd/worker-agent
+	@printf '%s\n' 	  'Built bin/worker-agent. Configure via environment:' 	  '  COORDINATOR_URL, WORKER_AUTH_TOKEN, WORK_DIR, CPU_COUNT, MEMORY_MB,' 	  '  POLL_INTERVAL, REQUEST_TIMEOUT, HEARTBEAT_INTERVAL, CAPABILITIES,' 	  '  TASK_RUNNER, MAX_TASKS, EXIT_WHEN_IDLE, WORKER_NAME, WORKER_ID'
+
+# The coordinator server as a static binary, the same way the Docker image
+# builds it (CGO_ENABLED=0, trimmed). Requires PostgreSQL at runtime.
+coordinator:
+	CGO_ENABLED=0 go build -trimpath -ldflags="$(LDFLAGS)" -o bin/coordinator ./cmd/coordinator
+	@printf '%s\n' \
+	  'Built bin/coordinator. Configure via environment:' \
+	  '  DATABASE_URL, COORDINATOR_ADDR, COORDINATOR_TOKEN, UI_AUTH_TOKEN,' \
+	  '  COORDINATOR_STORAGE_DIR, SCIMESH_DOCS_DIR, JWT_SECRET, USERSERVICE_URL' \
+	  '  (embedded schema migrations run on startup; AUTO_MIGRATE=false disables)'
+
+# Interactive wizard: checks the database, creates it when missing (via
+# POSTGRES_ADMIN_URL or --admin-db), applies the embedded schema, generates a
+# JWT_SECRET, and writes a .env file. Non-interactive: SETUP_ARGS=--yes.
+setup: coordinator
+	./bin/coordinator setup $(SETUP_ARGS)
+
+workloads-export:
+	cd .. && .venv/bin/scimesh workload export -o coordinator/$(WORKLOADS_JSON)
+
 help:
 	@printf '%s\n' \
 	  'SciMesh coordinator commands:' \
@@ -34,6 +68,8 @@ help:
 	  '  make demo-logs               Follow coordinator logs for the UI demo.' \
 	  '  make demo-down               Stop the demo services and workers.' \
 	  '  make demo-reset              Stop the demo and wipe its data volumes.' \
+	  '  make workloads-export        Regenerate the embedded UI workload catalog.' \
+  '  make setup                   Interactive one-shot provisioning wizard.' \
 	  '  make test / make vet         Run Go verification.' \
 	  '' \
 	  'Demo UI: http://localhost:18080/ui (login page; admin root@scimesh.local / rootpassword).'
@@ -127,10 +163,10 @@ tidy:
 # DATABASE_URL must be set, e.g.:
 #   export DATABASE_URL='postgres://scimesh:scimesh@localhost:5432/scimesh?sslmode=disable'
 migrate-up:
-	migrate -path migrations -database "$(DATABASE_URL)" up
+	migrate -path internal/storage/postgres/migrations -database "$(DATABASE_URL)" up
 
 migrate-down:
-	migrate -path migrations -database "$(DATABASE_URL)" down 1
+	migrate -path internal/storage/postgres/migrations -database "$(DATABASE_URL)" down 1
 
 # --- docker --------------------------------------------------------------
 # `up` starts Postgres, applies migrations, then launches the coordinator.

coordinator/cmd/coordinator/main.go

@@ -2,6 +2,8 @@ package main
 
 import (
 	"context"
+	"flag"
+	"fmt"
 	"log/slog"
 	"os"
 	"os/signal"
@@ -14,9 +16,27 @@ import (
 	"github.com/emil28092005/SciMesh/coordinator/internal/storage/postgres"
 	httptransport "github.com/emil28092005/SciMesh/coordinator/internal/transport/http"
 	"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
+// version is injected at build time (-ldflags "-X main.version=...") and
+// reported by --version. "dev" marks a local build.
+var version = "dev"
+
 func main() {
+	args := os.Args[1:]
+	if len(args) > 0 && args[0] == "setup" {
+		if err := runSetup(args[1:]); err != nil {
+			os.Exit(1)
+		}
+		return
+	}
+	showVersion := flag.Bool("version", false, "print the build version and exit")
+	flag.Parse()
+	if *showVersion {
+		fmt.Println("coordinator " + version)
+		return
+	}
 	// All work happens in run() so its defers (pool.Close, log flush, signal
 	// stop) still execute: os.Exit skips deferred calls entirely.
 	if err := run(); err != nil {
@@ -53,6 +73,15 @@ func run() error {
 	}
 	defer pool.Close()
 
+	// A downloaded binary provisions its own schema; AUTO_MIGRATE=false keeps
+	// out-of-band migration workflows (the migrate CLI, CI, managed databases).
+	if cfg.AutoMigrate {
+		if err := postgres.Migrate(ctx, cfg.DatabaseURL, log); err != nil {
+			log.Error("apply migrations", "err", err)
+			return err
+		}
+	}
+
 	blobStore, err := blob.NewFSStore(cfg.StorageDir)
 	if err != nil {
 		log.Error("init blob storage", "err", err)
@@ -70,29 +99,35 @@ func run() error {
 		taskResultRepo = postgres.NewTaskResultRepo(pool)
 	)
 
+	catalog, err := workloads.Load()
+	if err != nil {
+		log.Error("load workload catalog", "err", err)
+		return err
+	}
+
 	useCases := httptransport.UseCases{
 		RegisterWorker:   usecase.NewRegisterWorker(workerRepo, clk),
 		CreateJob:        usecase.NewCreateJob(jobRepo, taskRepo, tx, clk),
-		SubmitDataset:    usecase.NewSubmitDataset(blobStore, artifactRepo, jobRepo, taskRepo, tx, clk, cfg.DefaultMaxAttempts),
-		ClaimTask:        usecase.NewClaimTask(taskRepo, jobRepo, workerRepo, tx, clk, cfg.LeaseDuration),
+		SubmitDataset:    usecase.NewSubmitDataset(blobStore, artifactRepo, jobRepo, taskRepo, tx, clk, cfg.DefaultMaxAttempts, catalog),
+		ClaimTask:        usecase.NewClaimTask(taskRepo, jobRepo, workerRepo, tx, clk, cfg.LeaseDuration, catalog),
 		RenewLease:       usecase.NewRenewLease(taskRepo, workerRepo, tx, clk, cfg.LeaseDuration),
-		CompleteTask:     usecase.NewCompleteTask(taskRepo, jobRepo, artifactRepo, workerRepo, taskResultRepo, tx, clk, cfg.QuorumSize),
-		ReduceJob:        usecase.NewReduceJob(jobRepo, taskRepo, artifactRepo, blobStore, tx, clk),
-		FailTask:         usecase.NewFailTask(taskRepo, jobRepo, tx, clk),
+		CompleteTask:     usecase.NewCompleteTask(taskRepo, jobRepo, artifactRepo, workerRepo, taskResultRepo, tx, clk, cfg.QuorumSize, catalog),
+		ReduceJob:        usecase.NewReduceJob(jobRepo, taskRepo, artifactRepo, blobStore, tx, clk, catalog),
+		FailTask:         usecase.NewFailTask(taskRepo, jobRepo, workerRepo, tx, clk, catalog),
 		GetJobStatus:     usecase.NewGetJobStatus(jobRepo, taskRepo),
 		CancelJob:        usecase.NewCancelJob(jobRepo, taskRepo, tx, clk),
-		UploadArtifact:   usecase.NewUploadArtifact(taskRepo, artifactRepo, blobStore, tx, clk),
+		UploadArtifact:   usecase.NewUploadArtifact(taskRepo, workerRepo, artifactRepo, blobStore, tx, clk),
 		DownloadArtifact: usecase.NewDownloadArtifact(artifactRepo, blobStore),
 		GetJobResult:     usecase.NewGetJobResult(jobRepo, usecase.NewDownloadArtifact(artifactRepo, blobStore)),
 		GetTaskInput:     usecase.NewGetTaskInput(taskRepo, artifactRepo, blobStore),
-		Dashboard:        usecase.NewDashboard(uiReadRepo),
+		Dashboard:        usecase.NewDashboard(uiReadRepo, catalog),
 		PreviewArtifact:  usecase.NewPreviewArtifact(uiReadRepo, blobStore),
 	}
 
 	// Background reapers are tracked so shutdown can wait for them. Without this
 	// the process would exit mid-UPDATE, and the deferred pool.Close() would pull
 	// connections out from under them.
-	expireLeases := usecase.NewExpireLeases(taskRepo, jobRepo, tx, clk)
+	expireLeases := usecase.NewExpireLeases(taskRepo, jobRepo, tx, clk, catalog)
 	markOffline := usecase.NewMarkWorkersOffline(workerRepo, clk, cfg.WorkerOfflineAfter)
 
 	var wg sync.WaitGroup
@@ -121,7 +156,7 @@ func run() error {
 
 	// pool.Ping backs /health: readiness means the database answers, not just
 	// that the process is alive.
-	api := httptransport.NewServer(useCases, log, cfg.RequestTimeout, cfg.HeartbeatInterval, cfg.MaxUploadBytes, cfg.JWTSecret, cfg.UserserviceURL, m, pool.Ping, cfg.PublicCoordinatorURL, cfg.PublicUserserviceURL)
+	api := httptransport.NewServer(useCases, log, cfg.RequestTimeout, cfg.HeartbeatInterval, cfg.MaxUploadBytes, cfg.JWTSecret, cfg.UserserviceURL, m, pool.Ping, cfg.PublicCoordinatorURL, cfg.PublicUserserviceURL, cfg.DocsDir)
 	err = infra.RunServer(ctx, log, cfg.Addr, api.Handler(cfg.Token, cfg.UIToken))
 
 	// Shutdown order matters, and defers alone cannot express it (they run

coordinator/cmd/coordinator/setup_cmd.go

@@ -0,0 +1,74 @@
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"log/slog"
+	"os"
+	"time"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/setup"
+)
+
+// runSetup implements `coordinator setup` with non-interactive flags and an
+// interactive fallback for anything still missing.
+func runSetup(args []string) error {
+	flags := flag.NewFlagSet("setup", flag.ContinueOnError)
+	flags.Usage = func() {
+		_, _ = fmt.Fprintf(flags.Output(), "usage: coordinator setup [options]\n")
+		_, _ = fmt.Fprintf(flags.Output(), "Provisions the coordinator database, schema, and local .env settings.\n\n")
+		flags.PrintDefaults()
+	}
+	var (
+		databaseURL = flags.String("db", "", "coordinator database URL (default: DATABASE_URL)")
+		adminURL    = flags.String("admin-db", "", "maintenance URL to create a missing database (default: same host, 'postgres' db)")
+		envFile     = flags.String("env-file", "", "settings file to write (default: .env)")
+		force       = flags.Bool("force", false, "overwrite an existing settings file")
+		yes         = flags.Bool("yes", false, "non-interactive: use defaults, fail on anything missing")
+	)
+	if err := flags.Parse(args); err != nil {
+		return err
+	}
+	if flags.NArg() > 0 {
+		return fmt.Errorf("setup takes no positional arguments")
+	}
+
+	databaseURLValue := *databaseURL
+	if databaseURLValue == "" {
+		databaseURLValue = os.Getenv("DATABASE_URL")
+	}
+	envFileValue := *envFile
+	if envFileValue == "" {
+		envFileValue = os.Getenv("ENV_FILE")
+	}
+	adminURLValue := *adminURL
+	if adminURLValue == "" {
+		adminURLValue = os.Getenv("POSTGRES_ADMIN_URL")
+	}
+
+	options := setup.Options{
+		DatabaseURL:      databaseURLValue,
+		AdminDatabaseURL: adminURLValue,
+		EnvFile:          envFileValue,
+		Force:            *force,
+		Yes:              *yes,
+		ConnectTimeout:   10 * time.Second,
+		Out:              os.Stdout,
+		In:               os.Stdin,
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	log := slog.New(slog.NewTextHandler(os.Stderr, nil))
+	log.Info("setup started", "db", setup.SanitizeDatabaseURL(databaseURLValue), "env_file", envFileValue)
+
+	summary, err := setup.Run(ctx, options)
+	if err != nil {
+		log.Error("setup failed", "err", err)
+		return err
+	}
+	_, _ = fmt.Fprint(os.Stdout, summary)
+	log.Info("setup complete")
+	return nil
+}

coordinator/cmd/worker-agent/main.go

@@ -0,0 +1,44 @@
+// Command worker-agent is the Go worker agent: a coordinator client that
+// executes SDK workloads in a Python subprocess per claimed task.
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log/slog"
+	"os"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/agent"
+)
+
+// version is injected at build time (-ldflags "-X main.version=...") and
+// reported by --version. "dev" marks a local build.
+var version = "dev"
+
+func main() {
+	showVersion := flag.Bool("version", false, "print the build version and exit")
+	flag.Parse()
+	if *showVersion {
+		fmt.Println("worker-agent " + version)
+		return
+	}
+	config, err := agent.LoadConfig()
+	if err != nil {
+		slog.Error("invalid configuration", "error", err)
+		os.Exit(2)
+	}
+	logger := slog.New(slog.NewTextHandler(os.Stderr, nil))
+	tokens := agent.NewTokenProvider(
+		config.WorkerKey,
+		config.UserserviceURL,
+		config.Token,
+		config.RequestTimeout,
+	)
+	client := agent.NewClient(config.CoordinatorURL, tokens, config.RequestTimeout)
+	runner := agent.NewTaskRunner(config.TaskRunner)
+	daemon := agent.NewDaemon(config, client, runner, logger)
+	if err := daemon.RunForever(); err != nil {
+		logger.Error("agent stopped", "error", err)
+		os.Exit(1)
+	}
+}

coordinator/docker-compose.yml

@@ -20,20 +20,8 @@ services:
       retries: 10
       start_period: 5s
 
-  # One-shot: applies migrations, then exits. Schema changes stay an explicit
-  # deployment step — the coordinator binary never migrates on startup.
-  migrate:
-    image: migrate/migrate:v4.17.1
-    depends_on:
-      postgres:
-        condition: service_healthy
-    volumes:
-      - ./migrations:/migrations:ro
-    command:
-      - -path=/migrations
-      - -database=postgres://${POSTGRES_USER:-scimesh}:${POSTGRES_PASSWORD:-scimesh}@postgres:5432/${POSTGRES_DB:-scimesh}?sslmode=disable
-      - up
-    restart: on-failure
+  # The coordinator applies its embedded schema migrations on startup
+  # (AUTO_MIGRATE, on by default), so no separate migration step is needed.
 
   coordinator:
     build:
@@ -41,9 +29,6 @@ services:
     depends_on:
       postgres:
         condition: service_healthy
-      # Start only once the schema exists, otherwise the first query fails.
-      migrate:
-        condition: service_completed_successfully
     environment:
       COORDINATOR_ADDR: ":8080"
       # Host is the service name: compose resolves it on the project network.
@@ -51,6 +36,9 @@ services:
       WORKER_AUTH_TOKEN: ${WORKER_AUTH_TOKEN:-dev-token}
       # Empty disables /ui. Set this separately from the worker token.
       UI_AUTH_TOKEN: ${UI_AUTH_TOKEN:-}
+      # Directory of the built MkDocs site served at /ui/docs/ (empty disables
+      # the docs route; the demo mounts ./site automatically).
+      SCIMESH_DOCS_DIR: ${SCIMESH_DOCS_DIR:-}
       DB_MAX_CONNS: "10"
       REQUEST_TIMEOUT: "15s"
       LEASE_DURATION: "2m"

coordinator/internal/agent/auth.go

@@ -0,0 +1,121 @@
+package agent
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"sync"
+	"time"
+)
+
+// TokenProvider supplies the current bearer token. A static token is served
+// forever; a worker key is exchanged at the userservice for short-lived JWTs
+// and refreshed before they expire (mirroring the former Python worker).
+type TokenProvider interface {
+	Token() (string, error)
+	Refresh() error
+}
+
+// StaticToken serves a fixed token forever; empty means no Authorization.
+type StaticToken struct{ token string }
+
+func (s *StaticToken) Token() (string, error) { return s.token, nil }
+func (s *StaticToken) Refresh() error         { return nil }
+
+// WorkerKeyToken exchanges a long-lived worker key for short-lived JWTs.
+type WorkerKeyToken struct {
+	userserviceURL string
+	workerKey      string
+	timeout        time.Duration
+	leeway         float64
+	mu             sync.Mutex
+	token          string
+	refreshAt      time.Time
+}
+
+func NewWorkerKeyToken(userserviceURL, workerKey string, timeout time.Duration) *WorkerKeyToken {
+	return &WorkerKeyToken{
+		userserviceURL: strings.TrimRight(userserviceURL, "/"),
+		workerKey:      workerKey,
+		timeout:        timeout,
+		leeway:         0.2,
+	}
+}
+
+// Token returns the current token, exchanging first when missing or stale.
+func (p *WorkerKeyToken) Token() (string, error) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if p.token == "" || time.Now().After(p.refreshAt) {
+		if err := p.exchangeLocked(); err != nil {
+			return "", err
+		}
+	}
+	return p.token, nil
+}
+
+// Refresh forces an immediate exchange.
+func (p *WorkerKeyToken) Refresh() error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	return p.exchangeLocked()
+}
+
+func (p *WorkerKeyToken) exchangeLocked() error {
+	payload, err := json.Marshal(map[string]string{"key": p.workerKey})
+	if err != nil {
+		return err
+	}
+	request, err := http.NewRequestWithContext(context.Background(), http.MethodPost, p.userserviceURL+"/worker-tokens/exchange", bytes.NewReader(payload))
+	if err != nil {
+		return err
+	}
+	request.Header.Set("Content-Type", "application/json")
+	client := &http.Client{Timeout: p.timeout}
+	response, err := client.Do(request)
+	if err != nil {
+		return fmt.Errorf("worker key exchange request failed")
+	}
+	defer func() { _ = response.Body.Close() }()
+	if response.StatusCode != http.StatusOK {
+		return fmt.Errorf("worker key exchange rejected with status %d", response.StatusCode)
+	}
+	raw, err := io.ReadAll(io.LimitReader(response.Body, 1<<20))
+	if err != nil {
+		return fmt.Errorf("worker key exchange request failed")
+	}
+	var data map[string]any
+	if err := json.Unmarshal(raw, &data); err != nil {
+		return fmt.Errorf("worker key exchange response is invalid")
+	}
+	token, _ := data["token"].(string)
+	if token == "" {
+		return fmt.Errorf("worker key exchange response is missing a token")
+	}
+	var ttl time.Duration
+	switch value := data["expires_in"].(type) {
+	case float64:
+		ttl = time.Duration(value * float64(time.Second))
+	case int:
+		ttl = time.Duration(value) * time.Second
+	}
+	p.token = token
+	p.refreshAt = time.Time{}
+	if ttl > 0 {
+		p.refreshAt = time.Now().Add(time.Duration(float64(ttl) * (1.0 - p.leeway)))
+	}
+	return nil
+}
+
+// NewTokenProvider picks the strategy: a worker key (with userservice) wins
+// over a static bearer token.
+func NewTokenProvider(workerKey, userserviceURL, bearerToken string, timeout time.Duration) TokenProvider {
+	if workerKey != "" && userserviceURL != "" {
+		return NewWorkerKeyToken(userserviceURL, workerKey, timeout)
+	}
+	return &StaticToken{token: bearerToken}
+}

coordinator/internal/agent/auth_test.go

@@ -0,0 +1,99 @@
+package agent
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+func TestWorkerKeyTokenExchangesAndCaches(t *testing.T) {
+	var exchanges atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/worker-tokens/exchange" {
+			http.NotFound(w, r)
+			return
+		}
+		var payload map[string]string
+		if err := json.NewDecoder(r.Body).Decode(&payload); err != nil || payload["key"] != "scimesh_wk_live_x" {
+			http.Error(w, "bad key", http.StatusUnauthorized)
+			return
+		}
+		exchanges.Add(1)
+		writeJSON(w, http.StatusOK, map[string]any{
+			"token":      "jwt-1",
+			"expires_in": 100,
+		})
+	}))
+	defer server.Close()
+
+	provider := NewWorkerKeyToken(server.URL, "scimesh_wk_live_x", 5*time.Second)
+	token, err := provider.Token()
+	if err != nil || token != "jwt-1" {
+		t.Fatalf("token = %q, err = %v", token, err)
+	}
+	// The second call within the TTL reuses the cache.
+	again, err := provider.Token()
+	if err != nil || again != "jwt-1" {
+		t.Fatalf("cached token = %q, err = %v", again, err)
+	}
+	if exchanges.Load() != 1 {
+		t.Errorf("exchanges = %d, want 1", exchanges.Load())
+	}
+	// An explicit refresh re-exchanges.
+	if err := provider.Refresh(); err != nil {
+		t.Fatalf("Refresh: %v", err)
+	}
+	if exchanges.Load() != 2 {
+		t.Errorf("exchanges after refresh = %d, want 2", exchanges.Load())
+	}
+}
+
+func TestWorkerKeyTokenRejectsBadKey(t *testing.T) {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusUnauthorized)
+	}))
+	defer server.Close()
+
+	provider := NewWorkerKeyToken(server.URL, "bad", 5*time.Second)
+	if _, err := provider.Token(); err == nil {
+		t.Error("expected exchange failure for a rejected key")
+	}
+}
+
+func TestNewTokenProviderSelectsStrategy(t *testing.T) {
+	if _, ok := NewTokenProvider("", "", "static", time.Second).(*StaticToken); !ok {
+		t.Error("expected a static token provider")
+	}
+	if _, ok := NewTokenProvider("key", "http://users", "", time.Second).(*WorkerKeyToken); !ok {
+		t.Error("expected a worker-key provider")
+	}
+}
+
+func TestClientRefreshesTokenOnceOn401(t *testing.T) {
+	var attempts atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		attempts.Add(1)
+		if attempts.Load() == 1 {
+			w.WriteHeader(http.StatusUnauthorized)
+			return
+		}
+		writeJSON(w, http.StatusOK, map[string]any{})
+	}))
+	defer server.Close()
+
+	provider := &StaticToken{token: "t"}
+	client := NewClient(server.URL, provider, 5*time.Second)
+	status, _, err := client.requestJSON(http.MethodGet, "/ok", map[string]any{})
+	if err != nil {
+		t.Fatalf("request: %v", err)
+	}
+	if status != http.StatusOK {
+		t.Errorf("status = %d", status)
+	}
+	if attempts.Load() != 2 {
+		t.Errorf("attempts = %d, want 2 (401 then retry)", attempts.Load())
+	}
+}

coordinator/internal/agent/client.go

@@ -0,0 +1,374 @@
+package agent
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// CoordinatorError is a non-retriable coordinator response.
+type CoordinatorError struct{ msg string }
+
+func (e *CoordinatorError) Error() string { return e.msg }
+
+// TransientError is a timeout, connection error, or 5xx response.
+type TransientError struct{ msg string }
+
+func (e *TransientError) Error() string { return e.msg }
+
+// ConflictError means the worker no longer owns the task lease.
+type ConflictError struct{ msg string }
+
+func (e *ConflictError) Error() string { return e.msg }
+
+// Client speaks the v1 worker contract over HTTP with a token provider.
+//
+// API calls never follow redirects (a redirect is a contract violation); the
+// artifact download follows redirects but strips the Authorization header on
+// cross-origin hops, matching the Python worker's SameOriginAuthRedirectHandler.
+// A 401 response refreshes the token exactly once and retries, so a lapsed JWT
+// does not fail an in-flight task.
+type Client struct {
+	baseURL   string
+	tokens    TokenProvider
+	timeout   time.Duration
+	apiClient *http.Client
+	dlClient  *http.Client
+}
+
+func NewClient(baseURL string, tokens TokenProvider, timeout time.Duration) *Client {
+	return &Client{
+		baseURL: strings.TrimRight(baseURL, "/"),
+		tokens:  tokens,
+		timeout: timeout,
+		apiClient: &http.Client{
+			Timeout:       timeout,
+			CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
+		},
+		dlClient: &http.Client{
+			Timeout: timeout,
+			CheckRedirect: func(req *http.Request, via []*http.Request) error {
+				if len(via) >= 10 {
+					return fmt.Errorf("too many redirects")
+				}
+				// Go strips Authorization on cross-host redirects by default;
+				// strip it explicitly on any origin change to be safe.
+				if len(via) > 0 && origin(req.URL) != origin(via[0].URL) {
+					req.Header.Del("Authorization")
+				}
+				return nil
+			},
+		},
+	}
+}
+
+func origin(u *url.URL) string {
+	return u.Scheme + "://" + u.Host
+}
+
+func (c *Client) authHeaders() (map[string]string, error) {
+	token, err := c.tokens.Token()
+	if err != nil {
+		return nil, &CoordinatorError{msg: "token refresh failed: " + err.Error()}
+	}
+	if token == "" {
+		return map[string]string{}, nil
+	}
+	return map[string]string{"Authorization": "Bearer " + token}, nil
+}
+
+func (c *Client) refreshAndRetry() bool {
+	return c.tokens.Refresh() == nil
+}
+
+// Register advertises the worker and returns its identity and heartbeat policy.
+func (c *Client) Register(name string, capabilities []string, cpuCount int, memoryMB int) (*RegisteredWorker, error) {
+	payload := map[string]any{
+		"name":         name,
+		"capabilities": capabilities,
+		"cpu_count":    cpuCount,
+	}
+	if memoryMB > 0 {
+		payload["memory_mb"] = memoryMB
+	}
+	status, body, err := c.requestJSON("POST", "/workers/register", payload)
+	if err != nil {
+		return nil, err
+	}
+	if status != http.StatusCreated {
+		return nil, &CoordinatorError{msg: fmt.Sprintf("worker registration rejected with status %d", status)}
+	}
+	return ParseRegistered(body)
+}
+
+// Claim leases one compatible task, or returns nil when the queue is empty.
+func (c *Client) Claim(workerID string, capabilities []string) (*Task, error) {
+	status, body, err := c.requestJSON("POST", "/tasks/claim", map[string]any{
+		"worker_id":       workerID,
+		"capabilities":    capabilities,
+		"max_concurrency": 1,
+	})
+	if err != nil {
+		return nil, err
+	}
+	if status == http.StatusNoContent {
+		return nil, nil
+	}
+	if status != http.StatusOK {
+		return nil, &CoordinatorError{msg: fmt.Sprintf("unexpected claim status %d", status)}
+	}
+	return ParseTask(body)
+}
+
+// Heartbeat renews the lease and returns the new deadline.
+func (c *Client) Heartbeat(task *Task, workerID string) (time.Time, error) {
+	status, body, err := c.requestJSON("POST", "/tasks/"+task.TaskID+"/heartbeat", map[string]any{
+		"worker_id": workerID,
+		"attempt":   task.Attempt,
+	})
+	if err != nil {
+		return time.Time{}, err
+	}
+	if status != http.StatusOK {
+		if status == http.StatusConflict {
+			return time.Time{}, &ConflictError{msg: "heartbeat rejected because the task lease was lost"}
+		}
+		return time.Time{}, &CoordinatorError{msg: fmt.Sprintf("heartbeat rejected with status %d", status)}
+	}
+	raw, ok := body["lease_expires_at"].(string)
+	if !ok {
+		return time.Time{}, &CoordinatorError{msg: "heartbeat response is missing lease_expires_at"}
+	}
+	lease, err := time.Parse(time.RFC3339, raw)
+	if err != nil {
+		return time.Time{}, &CoordinatorError{msg: "heartbeat returned an invalid lease_expires_at"}
+	}
+	task.LeaseExpiresAt = lease
+	task.leaseExpiresRaw = raw
+	return lease, nil
+}
+
+// Submit completes a task with the uploaded coordinator-owned artifact.
+func (c *Client) Submit(task *Task, workerID string, uploaded *Uploaded, metrics map[string]any) error {
+	status, _, err := c.requestJSON("POST", "/tasks/"+task.TaskID+"/result", map[string]any{
+		"worker_id": workerID,
+		"attempt":   task.Attempt,
+		"result":    map[string]any{"artifact_id": uploaded.ArtifactID},
+		"metrics":   metrics,
+	})
+	if err != nil {
+		return err
+	}
+	if status != http.StatusOK && status != http.StatusCreated && status != http.StatusAccepted {
+		if status == http.StatusConflict {
+			return &ConflictError{msg: "result rejected because the task lease was lost"}
+		}
+		return &CoordinatorError{msg: fmt.Sprintf("result rejected with status %d", status)}
+	}
+	return nil
+}
+
+// Fail reports a sanitized failure.
+func (c *Client) Fail(task *Task, workerID string, code, message string, retryable bool) error {
+	status, _, err := c.requestJSON("POST", "/tasks/"+task.TaskID+"/failure", map[string]any{
+		"worker_id":     workerID,
+		"attempt":       task.Attempt,
+		"error_code":    code,
+		"error_message": message,
+		"retryable":     retryable,
+	})
+	if err != nil {
+		return err
+	}
+	if status != http.StatusOK && status != http.StatusCreated && status != http.StatusAccepted {
+		if status == http.StatusConflict {
+			return &ConflictError{msg: "failure rejected because the task lease was lost"}
+		}
+		return &CoordinatorError{msg: fmt.Sprintf("failure report rejected with status %d", status)}
+	}
+	return nil
+}
+
+// Download streams the task input to destination and returns its SHA-256.
+func (c *Client) Download(uri, destination string) (string, error) {
+	resolved, err := url.Parse(uri)
+	if err != nil {
+		return "", fmt.Errorf("invalid input URI: %w", err)
+	}
+	if !resolved.IsAbs() {
+		base, parseErr := url.Parse(c.baseURL)
+		if parseErr != nil {
+			return "", fmt.Errorf("invalid coordinator URL")
+		}
+		resolved = base.ResolveReference(resolved)
+	}
+	if resolved.Scheme != "http" && resolved.Scheme != "https" {
+		return "", fmt.Errorf("input URI must be an HTTP(S) URL")
+	}
+	request, err := http.NewRequestWithContext(context.Background(), http.MethodGet, resolved.String(), nil)
+	if err != nil {
+		return "", err
+	}
+	headers, err := c.authHeaders()
+	if err != nil {
+		return "", err
+	}
+	for name, value := range headers {
+		request.Header.Set(name, value)
+	}
+	response, err := c.dlClient.Do(request)
+	if err != nil {
+		return "", &TransientError{msg: "input download failed"}
+	}
+	defer func() { _ = response.Body.Close() }()
+	if response.StatusCode == http.StatusUnauthorized && c.refreshAndRetry() {
+		return c.Download(uri, destination)
+	}
+	if response.StatusCode != http.StatusOK {
+		return "", &CoordinatorError{msg: fmt.Sprintf("input download rejected with status %d", response.StatusCode)}
+	}
+	if err := os.MkdirAll(filepath.Dir(destination), 0o750); err != nil {
+		return "", err
+	}
+	// #nosec G304 -- destination is the worker's own attempt directory file.
+	target, err := os.OpenFile(destination, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
+	if err != nil {
+		return "", err
+	}
+	digest := sha256.New()
+	_, copyErr := io.Copy(io.MultiWriter(target, digest), response.Body)
+	closeErr := target.Close()
+	if copyErr != nil {
+		_ = os.Remove(destination)
+		return "", &TransientError{msg: "input download interrupted"}
+	}
+	if closeErr != nil {
+		return "", closeErr
+	}
+	return hex.EncodeToString(digest.Sum(nil)), nil
+}
+
+// Upload streams a partial artifact and verifies the returned metadata.
+func (c *Client) Upload(task *Task, workerID string, path, contentType string) (*Uploaded, error) {
+	// #nosec G304 -- the upload path is this worker's own artifact file.
+	file, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	info, err := file.Stat()
+	if err != nil {
+		_ = file.Close()
+		return nil, err
+	}
+	digest := sha256.New()
+	if _, err := io.Copy(digest, file); err != nil {
+		_ = file.Close()
+		return nil, err
+	}
+	if _, err := file.Seek(0, io.SeekStart); err != nil {
+		_ = file.Close()
+		return nil, err
+	}
+	localSHA := hex.EncodeToString(digest.Sum(nil))
+	uploadURL := c.baseURL + "/tasks/" + url.PathEscape(task.TaskID) + "/artifacts/" + url.PathEscape(filepath.Base(path))
+	request, err := http.NewRequestWithContext(context.Background(), http.MethodPut, uploadURL, file)
+	if err != nil {
+		_ = file.Close()
+		return nil, err
+	}
+	request.ContentLength = info.Size()
+	request.Header.Set("Content-Type", contentType)
+	request.Header.Set("X-Worker-ID", workerID)
+	request.Header.Set("X-Task-Attempt", strconv.Itoa(task.Attempt))
+	headers, err := c.authHeaders()
+	if err != nil {
+		_ = file.Close()
+		return nil, err
+	}
+	for name, value := range headers {
+		request.Header.Set(name, value)
+	}
+	response, err := c.apiClient.Do(request)
+	_ = file.Close()
+	if err != nil {
+		return nil, &TransientError{msg: "artifact upload failed"}
+	}
+	defer func() { _ = response.Body.Close() }()
+	raw, err := io.ReadAll(io.LimitReader(response.Body, 1<<20))
+	if err != nil {
+		return nil, &TransientError{msg: "artifact upload interrupted"}
+	}
+	if response.StatusCode == http.StatusConflict {
+		return nil, &ConflictError{msg: "artifact upload rejected because the task lease was lost"}
+	}
+	if response.StatusCode == http.StatusUnauthorized && c.refreshAndRetry() {
+		return c.Upload(task, workerID, path, contentType)
+	}
+	if response.StatusCode != http.StatusOK {
+		return nil, &CoordinatorError{msg: fmt.Sprintf("artifact upload rejected with status %d", response.StatusCode)}
+	}
+	var payload map[string]any
+	if err := json.Unmarshal(raw, &payload); err != nil {
+		return nil, &CoordinatorError{msg: "artifact upload returned invalid metadata"}
+	}
+	uploaded, err := ParseUploaded(payload)
+	if err != nil {
+		return nil, &CoordinatorError{msg: "artifact upload returned invalid metadata"}
+	}
+	if uploaded.SHA256 != localSHA || uploaded.SizeBytes != info.Size() {
+		return nil, &CoordinatorError{msg: "artifact upload metadata does not match local artifact"}
+	}
+	return uploaded, nil
+}
+
+func (c *Client) requestJSON(method, path string, payload any) (int, map[string]any, error) {
+	body, err := json.Marshal(payload)
+	if err != nil {
+		return 0, nil, err
+	}
+	request, err := http.NewRequestWithContext(context.Background(), method, c.baseURL+path, bytes.NewReader(body))
+	if err != nil {
+		return 0, nil, err
+	}
+	request.Header.Set("Content-Type", "application/json")
+	headers, err := c.authHeaders()
+	if err != nil {
+		return 0, nil, err
+	}
+	for name, value := range headers {
+		request.Header.Set(name, value)
+	}
+	response, err := c.apiClient.Do(request)
+	if err != nil {
+		return 0, nil, &TransientError{msg: "coordinator request failed"}
+	}
+	defer func() { _ = response.Body.Close() }()
+	raw, err := io.ReadAll(io.LimitReader(response.Body, 1<<20))
+	if err != nil {
+		return 0, nil, &TransientError{msg: "coordinator request interrupted"}
+	}
+	if response.StatusCode == http.StatusUnauthorized && c.refreshAndRetry() {
+		return c.requestJSON(method, path, payload)
+	}
+	if response.StatusCode >= 500 {
+		return response.StatusCode, nil, &TransientError{msg: fmt.Sprintf("coordinator returned %d", response.StatusCode)}
+	}
+	var decoded map[string]any
+	if len(raw) > 0 {
+		if err := json.Unmarshal(raw, &decoded); err != nil {
+			return response.StatusCode, nil, &CoordinatorError{msg: "coordinator returned invalid JSON"}
+		}
+	}
+	return response.StatusCode, decoded, nil
+}

coordinator/internal/agent/client_test.go

@@ -0,0 +1,225 @@
+package agent
+
+import (
+	"crypto/sha256"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func newTestClient(t *testing.T, server *httptest.Server) *Client {
+	t.Helper()
+	return NewClient(server.URL, &StaticToken{token: "test-token"}, 5*time.Second)
+}
+
+func TestClientRegisterClaimHeartbeat(t *testing.T) {
+	var registered, claimed, heartbeated bool
+	var server *httptest.Server //nolint:staticcheck // the handler closure references server
+	server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Header.Get("Authorization") != "Bearer test-token" {
+			http.Error(w, "missing token", http.StatusUnauthorized)
+			return
+		}
+		switch {
+		case r.Method == http.MethodPost && r.URL.Path == "/workers/register":
+			registered = true
+			writeJSON(w, http.StatusCreated, map[string]any{
+				"worker_id":                  "22222222-2222-4222-8222-222222222222",
+				"heartbeat_interval_seconds": 15,
+			})
+		case r.Method == http.MethodPost && r.URL.Path == "/tasks/claim":
+			claimed = true
+			writeJSON(w, http.StatusOK, validTaskPayload())
+		case r.Method == http.MethodPost && r.URL.Path == "/tasks/11111111-1111-4111-8111-111111111111/heartbeat":
+			heartbeated = true
+			writeJSON(w, http.StatusOK, map[string]any{
+				"lease_expires_at": time.Now().Add(2 * time.Minute).UTC().Format(time.RFC3339),
+			})
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer server.Close()
+	client := newTestClient(t, server)
+
+	registeredWorker, err := client.Register("test-worker", []string{"similarity-search"}, 2, 1024)
+	if err != nil {
+		t.Fatalf("Register: %v", err)
+	}
+	if registeredWorker.WorkerID != "22222222-2222-4222-8222-222222222222" {
+		t.Errorf("worker id = %q", registeredWorker.WorkerID)
+	}
+
+	task, err := client.Claim("22222222-2222-4222-8222-222222222222", []string{"similarity-search"})
+	if err != nil {
+		t.Fatalf("Claim: %v", err)
+	}
+	if task == nil || task.Workload != "similarity-search" {
+		t.Fatalf("claim = %+v", task)
+	}
+
+	renewed, err := client.Heartbeat(task, "22222222-2222-4222-8222-222222222222")
+	if err != nil {
+		t.Fatalf("Heartbeat: %v", err)
+	}
+	if renewed.Before(time.Now()) {
+		t.Error("renewed lease is in the past")
+	}
+	if !registered || !claimed || !heartbeated {
+		t.Error("some endpoints were not hit")
+	}
+}
+
+func TestClientClaimEmptyAndConflict(t *testing.T) {
+	var server *httptest.Server //nolint:staticcheck // the handler closure references server
+	server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/tasks/claim":
+			w.WriteHeader(http.StatusNoContent)
+		case "/tasks/11111111-1111-4111-8111-111111111111/heartbeat":
+			w.WriteHeader(http.StatusConflict)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer server.Close()
+	client := newTestClient(t, server)
+
+	task, err := client.Claim("worker", []string{"similarity-search"})
+	if err != nil {
+		t.Fatalf("Claim: %v", err)
+	}
+	if task != nil {
+		t.Error("expected no task for 204")
+	}
+
+	claimed, err := ParseTask(validTaskPayload())
+	if err != nil {
+		t.Fatalf("ParseTask: %v", err)
+	}
+	if _, err := client.Heartbeat(claimed, "worker"); err == nil {
+		t.Error("expected conflict error")
+	} else if !errors.As(err, &conflictError) {
+		t.Errorf("error type = %T", err)
+	}
+}
+
+func TestClientUploadSubmitFail(t *testing.T) {
+	var uploadedPath string
+	var server *httptest.Server //nolint:staticcheck // the handler closure references server
+	server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.Method == http.MethodPut && strings.HasPrefix(r.URL.Path, "/tasks/11111111-1111-4111-8111-111111111111/artifacts/"):
+			if r.Header.Get("X-Worker-ID") != "worker" || r.Header.Get("X-Task-Attempt") != "1" {
+				t.Errorf("missing identity headers: %+v", r.Header)
+			}
+			uploadedPath = r.URL.Path
+			writeJSON(w, http.StatusOK, map[string]any{
+				"artifact_id": "33333333-3333-4333-8333-333333333333",
+				"uri":         server.URL + "/artifacts/333/download",
+				"sha256":      sha256Of(t, "partial body"),
+				"size_bytes":  int64(len("partial body")),
+			})
+		case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/result"):
+			writeJSON(w, http.StatusAccepted, map[string]any{})
+		case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/failure"):
+			writeJSON(w, http.StatusAccepted, map[string]any{})
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer server.Close()
+	client := newTestClient(t, server)
+	task, _ := ParseTask(validTaskPayload())
+
+	dir := t.TempDir()
+	partial := filepath.Join(dir, "result.csv")
+	if err := os.WriteFile(partial, []byte("partial body"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	uploaded, err := client.Upload(task, "worker", partial, "text/csv")
+	if err != nil {
+		t.Fatalf("Upload: %v", err)
+	}
+	if uploaded.SizeBytes != int64(len("partial body")) {
+		t.Errorf("size = %d", uploaded.SizeBytes)
+	}
+	if !strings.Contains(uploadedPath, "result.csv") {
+		t.Errorf("upload path = %q", uploadedPath)
+	}
+
+	if err := client.Submit(task, "worker", uploaded, map[string]any{"rows": 1}); err != nil {
+		t.Fatalf("Submit: %v", err)
+	}
+	if err := client.Fail(task, "worker", "ValueError", "bad input", false); err != nil {
+		t.Fatalf("Fail: %v", err)
+	}
+}
+
+func TestClientDownloadVerifiesChecksumAndStripsAuthOnRedirect(t *testing.T) {
+	var redirectedAuth string
+	bucket := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		redirectedAuth = r.Header.Get("Authorization")
+		_, _ = w.Write([]byte("input bytes"))
+	}))
+	defer bucket.Close()
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/tasks/11111111-1111-4111-8111-111111111111/input" {
+			http.Redirect(w, r, bucket.URL+"/presigned", http.StatusFound)
+			return
+		}
+		http.NotFound(w, r)
+	}))
+	defer server.Close()
+	client := newTestClient(t, server)
+
+	destination := filepath.Join(t.TempDir(), "input")
+	digest, err := client.Download(server.URL+"/tasks/11111111-1111-4111-8111-111111111111/input", destination)
+	if err != nil {
+		t.Fatalf("Download: %v", err)
+	}
+	if digest != sha256Of(t, "input bytes") {
+		t.Errorf("digest = %q", digest)
+	}
+	if redirectedAuth != "" {
+		t.Error("Authorization must be stripped on the redirected download")
+	}
+}
+
+func TestSanitizeErrorMessageRedactsPaths(t *testing.T) {
+	message := SanitizeErrorMessage(
+		"failed at /home/alice/work/attempts/1/input and /private/secret.txt",
+		"/home/alice/work",
+	)
+	for _, forbidden := range []string{"/home/alice", "/private/secret.txt"} {
+		if strings.Contains(message, forbidden) {
+			t.Errorf("message leaks %q: %q", forbidden, message)
+		}
+	}
+	if !strings.Contains(message, "<worker-dir>") {
+		t.Errorf("work dir not redacted: %q", message)
+	}
+	long := SanitizeErrorMessage(strings.Repeat("x", 500), "/tmp")
+	if len(long) != 300 {
+		t.Errorf("truncated length = %d", len(long))
+	}
+}
+
+func writeJSON(w http.ResponseWriter, status int, payload any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	_ = json.NewEncoder(w).Encode(payload)
+}
+
+func sha256Of(t *testing.T, value string) string {
+	t.Helper()
+	digest := sha256.Sum256([]byte(value))
+	return fmt.Sprintf("%x", digest)
+}

coordinator/internal/agent/config.go

@@ -0,0 +1,177 @@
+package agent
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
+)
+
+// Config is read only from the environment, mirroring the former Python
+// worker's configuration surface.
+type Config struct {
+	CoordinatorURL string
+	Token          string
+	WorkerKey      string
+	UserserviceURL string
+	WorkerName     string
+	WorkerID       string // set after registration; overridable for tests
+	WorkDir        string
+	CPUCount       int
+	MemoryMB       int // 0 = not advertised
+	PollInterval   time.Duration
+	RequestTimeout time.Duration
+	Heartbeat      time.Duration
+	CleanupAfter   time.Duration // 0 = keep attempt directories
+	Capabilities   []string
+	TaskRunner     []string // command + args; defaults to python -m scimesh.worker.task
+	MaxTasks       int      // 0 = unlimited
+	ExitWhenIdle   bool
+}
+
+func envList(name string) ([]string, error) {
+	raw := os.Getenv(name)
+	if raw == "" {
+		return nil, nil
+	}
+	var items []string
+	if err := json.Unmarshal([]byte(raw), &items); err != nil {
+		return nil, fmt.Errorf("%s must be a JSON array", name)
+	}
+	for _, item := range items {
+		if strings.TrimSpace(item) == "" {
+			return nil, fmt.Errorf("%s must not contain empty entries", name)
+		}
+	}
+	return items, nil
+}
+
+// LoadConfig validates the environment and fails fast on invalid values.
+func LoadConfig() (*Config, error) {
+	url := os.Getenv("COORDINATOR_URL")
+	if url == "" {
+		return nil, fmt.Errorf("COORDINATOR_URL is required")
+	}
+	if !strings.HasPrefix(url, "http://") && !strings.HasPrefix(url, "https://") {
+		return nil, fmt.Errorf("COORDINATOR_URL must be an absolute HTTP(S) URL")
+	}
+	workDir := os.Getenv("WORK_DIR")
+	if workDir == "" {
+		workDir = "./scimesh-agent-data"
+	}
+	cpu := 1
+	if raw := os.Getenv("CPU_COUNT"); raw != "" {
+		parsed, err := strconv.Atoi(raw)
+		if err != nil || parsed < 1 {
+			return nil, fmt.Errorf("CPU_COUNT must be a positive integer")
+		}
+		cpu = parsed
+	}
+	memoryMB := 0
+	if raw := os.Getenv("MEMORY_MB"); raw != "" {
+		parsed, err := strconv.Atoi(raw)
+		if err != nil || parsed < 1 {
+			return nil, fmt.Errorf("MEMORY_MB must be a positive integer")
+		}
+		memoryMB = parsed
+	}
+	poll, err := durationEnv("POLL_INTERVAL", 2*time.Second)
+	if err != nil {
+		return nil, err
+	}
+	timeout, err := durationEnv("REQUEST_TIMEOUT", 30*time.Second)
+	if err != nil {
+		return nil, err
+	}
+	heartbeat, err := durationEnv("HEARTBEAT_INTERVAL", 15*time.Second)
+	if err != nil {
+		return nil, err
+	}
+	cleanup, err := durationEnv("CLEANUP_AFTER_SECONDS", 0)
+	if err != nil {
+		return nil, err
+	}
+	capabilities, err := envList("CAPABILITIES")
+	if err != nil {
+		return nil, err
+	}
+	if len(capabilities) == 0 {
+		capabilities = defaultCapabilities()
+	}
+	runner, err := envList("TASK_RUNNER")
+	if err != nil {
+		return nil, err
+	}
+	if len(runner) == 0 {
+		runner = []string{"python", "-m", "scimesh.worker.task"}
+	}
+	maxTasks := 0
+	if raw := os.Getenv("MAX_TASKS"); raw != "" {
+		parsed, err := strconv.Atoi(raw)
+		if err != nil || parsed < 1 {
+			return nil, fmt.Errorf("MAX_TASKS must be a positive integer")
+		}
+		maxTasks = parsed
+	}
+	name := os.Getenv("WORKER_NAME")
+	if name == "" {
+		host, _ := os.Hostname()
+		name = host
+	}
+	absWorkDir, err := filepath.Abs(workDir)
+	if err != nil {
+		return nil, fmt.Errorf("WORK_DIR must be an absolute path")
+	}
+	return &Config{
+		CoordinatorURL: strings.TrimRight(url, "/"),
+		Token:          os.Getenv("WORKER_AUTH_TOKEN"),
+		WorkerKey:      os.Getenv("WORKER_KEY"),
+		UserserviceURL: strings.TrimRight(os.Getenv("USERSERVICE_URL"), "/"),
+		WorkerName:     name,
+		WorkerID:       os.Getenv("WORKER_ID"),
+		WorkDir:        absWorkDir,
+		CPUCount:       cpu,
+		MemoryMB:       memoryMB,
+		PollInterval:   poll,
+		RequestTimeout: timeout,
+		Heartbeat:      heartbeat,
+		CleanupAfter:   cleanup,
+		Capabilities:   capabilities,
+		TaskRunner:     runner,
+		MaxTasks:       maxTasks,
+		ExitWhenIdle:   os.Getenv("EXIT_WHEN_IDLE") == "1",
+	}, nil
+}
+
+func durationEnv(name string, fallback time.Duration) (time.Duration, error) {
+	raw := os.Getenv(name)
+	if raw == "" {
+		return fallback, nil
+	}
+	parsed, err := time.ParseDuration(raw)
+	if err != nil || parsed < 0 {
+		return 0, fmt.Errorf("%s must be a non-negative duration", name)
+	}
+	return parsed, nil
+}
+
+// defaultCapabilities derives the worker's advertised capabilities from the
+// embedded workload catalog, so an agent is workload-agnostic out of the box:
+// it claims whatever enabled workloads the coordinator library declares.
+// Explicit CAPABILITIES still overrides this for operators who want a subset.
+func defaultCapabilities() []string {
+	catalog, err := workloads.Load()
+	if err != nil {
+		return []string{"similarity-search"}
+	}
+	names := make([]string, 0, len(catalog.Enabled()))
+	for _, workload := range catalog.Enabled() {
+		names = append(names, workload.Name)
+	}
+	return names
+}

coordinator/internal/agent/daemon.go

@@ -0,0 +1,336 @@
+package agent
+
+import (
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Outcome reports whether a claim was made and whether it completed.
+type Outcome struct {
+	Claimed   bool
+	Completed bool
+}
+
+// Daemon is the agent state machine: register, claim, execute via the Python
+// task runner, upload, and submit — mirroring the Python worker's lifecycle.
+// conflictError is the errors.As target for lease conflicts.
+var conflictError *ConflictError
+
+type Daemon struct {
+	config     *Config
+	client     *Client
+	runner     *TaskRunner
+	log        *slog.Logger
+	workerID   string
+	registered bool
+	completed  int
+	mu         sync.Mutex
+}
+
+func NewDaemon(config *Config, client *Client, runner *TaskRunner, log *slog.Logger) *Daemon {
+	return &Daemon{config: config, client: client, runner: runner, log: log}
+}
+
+// RunForever loops until interrupted, idle-exit, or max tasks.
+func (d *Daemon) RunForever() error {
+	failures := 0
+	for {
+		if !d.registered {
+			if err := d.register(); err != nil {
+				return err
+			}
+		}
+		d.cleanupExpiredDirectories()
+		outcome, err := d.runOnce()
+		if err != nil {
+			failures++
+			d.log.Warn("agent cycle failed", "error", err)
+			backoff := d.config.PollInterval
+			for i := 0; i < failures && i < 6; i++ {
+				backoff *= 2
+			}
+			if backoff > 60*time.Second {
+				backoff = 60 * time.Second
+			}
+			time.Sleep(backoff)
+			continue
+		}
+		failures = 0
+		if outcome.Claimed && outcome.Completed {
+			d.completed++
+			if d.config.MaxTasks > 0 && d.completed >= d.config.MaxTasks {
+				d.log.Info("max tasks reached")
+				return nil
+			}
+		}
+		if !outcome.Claimed && d.config.ExitWhenIdle {
+			d.log.Info("queue empty, exiting")
+			return nil
+		}
+		if outcome.Claimed && d.config.ExitWhenIdle {
+			d.log.Info("one claim processed, exiting")
+			return nil
+		}
+		if !outcome.Claimed {
+			time.Sleep(d.config.PollInterval)
+		}
+	}
+}
+
+func (d *Daemon) register() error {
+	registered, err := d.client.Register(
+		d.config.WorkerName,
+		d.config.Capabilities,
+		d.config.CPUCount,
+		d.config.MemoryMB,
+	)
+	if err != nil {
+		return err
+	}
+	d.mu.Lock()
+	if d.config.WorkerID != "" {
+		d.workerID = d.config.WorkerID
+	} else {
+		d.workerID = registered.WorkerID
+	}
+	d.registered = true
+	d.mu.Unlock()
+	d.log.Info("registered", "worker_id", d.workerID)
+	return nil
+}
+
+func (d *Daemon) workerIDOrEmpty() string {
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	return d.workerID
+}
+
+// cleanupExpiredDirectories removes task attempt directories older than the
+// configured retention, mirroring the former Python worker's cleanup.
+func (d *Daemon) cleanupExpiredDirectories() {
+	if d.config.CleanupAfter <= 0 {
+		return
+	}
+	cutoff := time.Now().Add(-d.config.CleanupAfter)
+	tasks, err := os.ReadDir(d.config.WorkDir)
+	if err != nil {
+		return
+	}
+	for _, taskEntry := range tasks {
+		if !taskEntry.IsDir() {
+			continue
+		}
+		taskDir := filepath.Join(d.config.WorkDir, taskEntry.Name())
+		attempts, err := os.ReadDir(taskDir)
+		if err != nil {
+			continue
+		}
+		for _, attemptEntry := range attempts {
+			if !attemptEntry.IsDir() {
+				continue
+			}
+			info, err := attemptEntry.Info()
+			if err == nil && info.ModTime().Before(cutoff) {
+				_ = os.RemoveAll(filepath.Join(taskDir, attemptEntry.Name()))
+			}
+		}
+		if entries, err := os.ReadDir(taskDir); err == nil && len(entries) == 0 {
+			_ = os.Remove(taskDir)
+		}
+	}
+}
+
+func (d *Daemon) runOnce() (Outcome, error) {
+	workerID := d.workerIDOrEmpty()
+	if workerID == "" {
+		return Outcome{}, fmt.Errorf("agent is not registered")
+	}
+	task, err := d.client.Claim(workerID, d.config.Capabilities)
+	if err != nil {
+		return Outcome{}, err
+	}
+	if task == nil {
+		return Outcome{Claimed: false}, nil
+	}
+	started := time.Now()
+	taskDir := filepath.Join(d.config.WorkDir, task.TaskID, fmt.Sprint(task.Attempt))
+	if err := os.MkdirAll(taskDir, 0o750); err != nil {
+		return Outcome{Claimed: true}, err
+	}
+	heartbeat := newLeaseHeartbeat(task, workerID, d.client, d.config.Heartbeat)
+	completed := false
+	err = heartbeat.Start()
+	if err != nil {
+		if errors.As(err, &conflictError) {
+			d.log.Warn("lease lost", "task_id", task.TaskID)
+			return Outcome{Claimed: true}, nil
+		}
+		return Outcome{Claimed: true}, err
+	}
+	defer heartbeat.Stop()
+
+	// Attempt directory cleanup is deliberately minimal in the prototype:
+	// attempt directories are retained under the work directory.
+	failure := d.executeTask(task, workerID, taskDir, started, heartbeat)
+	if failure != nil {
+		if errors.As(failure, &conflictError) {
+			d.log.Warn("lease lost", "task_id", task.TaskID)
+			return Outcome{Claimed: true}, nil
+		}
+		if err := heartbeat.RaiseIfFailed(); err != nil {
+			return Outcome{Claimed: true}, nil
+		}
+		d.reportFailure(task, workerID, failure)
+		return Outcome{Claimed: true}, nil
+	}
+	if err := heartbeat.RaiseIfFailed(); err != nil {
+		return Outcome{Claimed: true}, nil
+	}
+	completed = true
+	d.log.Info("task completed", "task_id", task.TaskID, "elapsed_seconds", time.Since(started).Seconds())
+	return Outcome{Claimed: true, Completed: completed}, nil
+}
+
+// executeTask returns nil on success or a classified failure.
+func (d *Daemon) executeTask(task *Task, workerID, taskDir string, started time.Time, heartbeat *leaseHeartbeat) error {
+	inputPath := filepath.Join(taskDir, "input")
+	// Downloads use the coordinator-provided URI verbatim; a relative path is
+	// resolved against the coordinator by the client.
+	actualSHA, err := d.client.Download(task.Input.URI, inputPath)
+	if err != nil {
+		return err
+	}
+	if !strings.EqualFold(actualSHA, task.Input.SHA256) {
+		return &CoordinatorError{msg: "input checksum mismatch"}
+	}
+	if err := heartbeat.RaiseIfFailed(); err != nil {
+		return err
+	}
+	manifestPath := filepath.Join(taskDir, "manifest.json")
+	manifest, err := d.runner.Run(task, taskDir, manifestPath, nil)
+	if err != nil {
+		return err
+	}
+	if err := heartbeat.RaiseIfFailed(); err != nil {
+		return err
+	}
+	uploaded, err := d.client.Upload(task, workerID, manifest.ArtifactPath, manifest.ContentType)
+	if err != nil {
+		return err
+	}
+	if err := heartbeat.RaiseIfFailed(); err != nil {
+		return err
+	}
+	metrics := map[string]any{"elapsed_seconds": roundSeconds(time.Since(started).Seconds())}
+	for name, value := range manifest.Metrics {
+		metrics[name] = value
+	}
+	return d.client.Submit(task, workerID, uploaded, metrics)
+}
+
+func (d *Daemon) reportFailure(task *Task, workerID string, failure error) {
+	var code string
+	var coordinatorErr *CoordinatorError
+	if errors.As(failure, &coordinatorErr) {
+		code = "ValueError"
+	} else {
+		code = "TaskRunnerFailed"
+	}
+	retryable := IsRetryableError(failure)
+	message := SanitizeErrorMessage(failure.Error(), d.config.WorkDir)
+	d.log.Warn("task failed", "task_id", task.TaskID, "error_code", code, "retryable", retryable)
+	if err := d.client.Fail(task, workerID, code, message, retryable); err != nil {
+		if errors.As(err, &conflictError) {
+			d.log.Warn("lease lost while reporting failure", "task_id", task.TaskID)
+			return
+		}
+		d.log.Warn("failure report rejected", "task_id", task.TaskID, "error", err)
+	}
+}
+
+// leaseHeartbeat renews the lease from the returned deadline at less than
+// half of the remaining TTL, mirroring the Python worker.
+type leaseHeartbeat struct {
+	task     *Task
+	workerID string
+	client   *Client
+	interval time.Duration
+	stop     chan struct{}
+	once     sync.Once
+	mu       sync.Mutex
+	lease    time.Time
+	failed   error
+}
+
+func newLeaseHeartbeat(task *Task, workerID string, client *Client, interval time.Duration) *leaseHeartbeat {
+	return &leaseHeartbeat{
+		task:     task,
+		workerID: workerID,
+		client:   client,
+		interval: interval,
+		stop:     make(chan struct{}),
+		lease:    task.LeaseExpiresAt,
+	}
+}
+
+func (h *leaseHeartbeat) Start() error {
+	if _, err := h.client.Heartbeat(h.task, h.workerID); err != nil {
+		return err
+	}
+	go h.loop()
+	return nil
+}
+
+func (h *leaseHeartbeat) Stop() {
+	h.once.Do(func() { close(h.stop) })
+}
+
+func (h *leaseHeartbeat) RaiseIfFailed() error {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	return h.failed
+}
+
+func (h *leaseHeartbeat) loop() {
+	for {
+		delay := h.nextDelay()
+		select {
+		case <-h.stop:
+			return
+		case <-time.After(delay):
+		}
+		renewed, err := h.client.Heartbeat(h.task, h.workerID)
+		h.mu.Lock()
+		if err != nil {
+			h.failed = err
+			h.mu.Unlock()
+			return
+		}
+		h.lease = renewed
+		h.mu.Unlock()
+	}
+}
+
+func (h *leaseHeartbeat) nextDelay() time.Duration {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	remaining := time.Until(h.lease)
+	if remaining <= 0 {
+		return 0
+	}
+	half := remaining / 2
+	if h.interval < half {
+		return h.interval
+	}
+	return half
+}
+
+func roundSeconds(seconds float64) float64 {
+	return float64(int64(seconds*1000)) / 1000
+}

coordinator/internal/agent/daemon_test.go

@@ -0,0 +1,296 @@
+package agent
+
+import (
+	"crypto/sha256"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func sha256HexOf(value string) string {
+	sum := sha256.Sum256([]byte(value))
+	return fmt.Sprintf("%x", sum)
+}
+
+// fakeRunnerScript writes a result manifest for --output and exits with the
+// given code.
+func fakeRunnerScript(t *testing.T, dir string, exitCode int) string {
+	t.Helper()
+	script := filepath.Join(dir, "fake-runner.sh")
+	content := `#!/bin/sh
+out=""
+task_dir=""
+while [ "$#" -gt 0 ]; do
+  case "$1" in
+    --output) out="$2"; shift 2;;
+    --task-dir) task_dir="$2"; shift 2;;
+    *) shift;;
+  esac
+done
+printf 'id,score\n' > "$task_dir/result.csv"
+printf '{"artifact_path":"%s/result.csv","content_type":"text/csv","metrics":{"rows":1}}' "$task_dir" > "$out"
+exit ` + fmt.Sprint(exitCode) + "\n"
+	if err := os.WriteFile(script, []byte(content), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	return script
+}
+
+// fakeCoordinator implements the v1 contract over HTTP and records calls.
+type fakeCoordinator struct {
+	server     *httptest.Server
+	task       map[string]any
+	submits    []map[string]any
+	failures   []map[string]any
+	heartbeats int
+	uploadSHA  string
+	uploadSize int64
+	inputBytes []byte
+	conflict   bool // 409 on heartbeat/upload/result
+}
+
+func newFakeCoordinator(t *testing.T, task map[string]any) *fakeCoordinator {
+	t.Helper()
+	fake := &fakeCoordinator{task: task, inputBytes: []byte("input fixture")}
+	fake.uploadSHA = sha256HexOf(string(fake.inputBytes))
+	fake.uploadSize = int64(len(fake.inputBytes))
+	var server *httptest.Server
+	server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.Method == http.MethodPost && r.URL.Path == "/workers/register":
+			writeJSON(w, http.StatusCreated, map[string]any{
+				"worker_id":                  "22222222-2222-4222-8222-222222222222",
+				"heartbeat_interval_seconds": 15.0,
+			})
+		case r.Method == http.MethodPost && r.URL.Path == "/tasks/claim":
+			if fake.task == nil {
+				w.WriteHeader(http.StatusNoContent)
+				return
+			}
+			writeJSON(w, http.StatusOK, fake.task)
+		case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/input"):
+			_, _ = w.Write(fake.inputBytes)
+		case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/heartbeat"):
+			fake.heartbeats++
+			if fake.conflict {
+				w.WriteHeader(http.StatusConflict)
+				return
+			}
+			writeJSON(w, http.StatusOK, map[string]any{
+				"lease_expires_at": time.Now().Add(2 * time.Minute).UTC().Format(time.RFC3339),
+			})
+		case r.Method == http.MethodPut && strings.Contains(r.URL.Path, "/artifacts/"):
+			raw, _ := io.ReadAll(r.Body)
+			fake.uploadSize = int64(len(raw))
+			fake.uploadSHA = fmt.Sprintf("%x", sha256.Sum256(raw))
+			if fake.conflict {
+				w.WriteHeader(http.StatusConflict)
+				return
+			}
+			writeJSON(w, http.StatusOK, map[string]any{
+				"artifact_id": "33333333-3333-4333-8333-333333333333",
+				"uri":         server.URL + "/artifacts/333/download",
+				"sha256":      fake.uploadSHA,
+				"size_bytes":  fake.uploadSize,
+			})
+		case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/result"):
+			var payload map[string]any
+			_ = json.NewDecoder(r.Body).Decode(&payload)
+			fake.submits = append(fake.submits, payload)
+			if fake.conflict {
+				w.WriteHeader(http.StatusConflict)
+				return
+			}
+			w.WriteHeader(http.StatusAccepted)
+		case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/failure"):
+			var payload map[string]any
+			_ = json.NewDecoder(r.Body).Decode(&payload)
+			fake.failures = append(fake.failures, payload)
+			w.WriteHeader(http.StatusAccepted)
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	fake.server = server
+	return fake
+}
+
+func (f *fakeCoordinator) close() { f.server.Close() }
+
+func testDaemon(t *testing.T, fake *fakeCoordinator, script string) *Daemon {
+	t.Helper()
+	config := &Config{
+		CoordinatorURL: fake.server.URL,
+		WorkerName:     "test-worker",
+		WorkerID:       "22222222-2222-4222-8222-222222222222",
+		WorkDir:        t.TempDir(),
+		CPUCount:       1,
+		PollInterval:   time.Millisecond,
+		RequestTimeout: 5 * time.Second,
+		Heartbeat:      15 * time.Second,
+		Capabilities:   []string{"similarity-search"},
+		TaskRunner:     []string{script},
+	}
+	client := NewClient(fake.server.URL, &StaticToken{token: "test-token"}, 5*time.Second)
+	runner := NewTaskRunner(config.TaskRunner)
+	logger := slog.New(slog.NewTextHandler(io.Discard, nil))
+	daemon := NewDaemon(config, client, runner, logger)
+	if err := daemon.register(); err != nil {
+		t.Fatalf("register: %v", err)
+	}
+	return daemon
+}
+
+func validClaimedTaskPayload() map[string]any {
+	return map[string]any{
+		"task_id":          "11111111-1111-4111-8111-111111111111",
+		"attempt":          1.0,
+		"lease_expires_at": time.Now().Add(time.Minute).UTC().Format(time.RFC3339),
+		"workload":         "similarity-search",
+		"input": map[string]any{
+			"uri":    "/tasks/11111111-1111-4111-8111-111111111111/input",
+			"sha256": sha256HexOf("input fixture"),
+		},
+		"parameters": map[string]any{"query_smiles": "CCO"},
+	}
+}
+
+func TestDaemonCompletesAClaimedTask(t *testing.T) {
+	fake := newFakeCoordinator(t, validClaimedTaskPayload())
+	defer fake.close()
+	daemon := testDaemon(t, fake, fakeRunnerScript(t, t.TempDir(), 0))
+
+	outcome, err := daemon.runOnce()
+	if err != nil {
+		t.Fatalf("runOnce: %v", err)
+	}
+	if !outcome.Claimed || !outcome.Completed {
+		t.Fatalf("outcome = %+v", outcome)
+	}
+	if len(fake.submits) != 1 {
+		t.Fatalf("submits = %d", len(fake.submits))
+	}
+	result := fake.submits[0]["result"].(map[string]any)
+	if result["artifact_id"] != "33333333-3333-4333-8333-333333333333" {
+		t.Errorf("result artifact = %v", result)
+	}
+	metrics := fake.submits[0]["metrics"].(map[string]any)
+	if metrics["rows"] != float64(1) {
+		t.Errorf("metrics = %v", metrics)
+	}
+	if _, ok := metrics["elapsed_seconds"].(float64); !ok {
+		t.Errorf("missing elapsed_seconds: %v", metrics)
+	}
+	if fake.heartbeats < 1 {
+		t.Error("expected at least one heartbeat")
+	}
+	if len(fake.failures) != 0 {
+		t.Errorf("unexpected failures: %v", fake.failures)
+	}
+}
+
+func TestDaemonReportsChecksumMismatchAsPermanentFailure(t *testing.T) {
+	payload := validClaimedTaskPayload()
+	payload["input"].(map[string]any)["sha256"] = strings.Repeat("b", 64)
+	fake := newFakeCoordinator(t, payload)
+	defer fake.close()
+	daemon := testDaemon(t, fake, fakeRunnerScript(t, t.TempDir(), 0))
+
+	outcome, err := daemon.runOnce()
+	if err != nil {
+		t.Fatalf("runOnce: %v", err)
+	}
+	if outcome.Completed {
+		t.Fatal("task must not complete on checksum mismatch")
+	}
+	if len(fake.failures) != 1 {
+		t.Fatalf("failures = %d", len(fake.failures))
+	}
+	failure := fake.failures[0]
+	if failure["error_code"] != "ValueError" || failure["retryable"] != false {
+		t.Errorf("failure = %v", failure)
+	}
+	if !strings.Contains(failure["error_message"].(string), "checksum") {
+		t.Errorf("message = %v", failure["error_message"])
+	}
+	if len(fake.submits) != 0 {
+		t.Error("no submission expected")
+	}
+}
+
+func TestDaemonReportsPermanentRunnerFailure(t *testing.T) {
+	fake := newFakeCoordinator(t, validClaimedTaskPayload())
+	defer fake.close()
+	daemon := testDaemon(t, fake, fakeRunnerScript(t, t.TempDir(), ExitPermanent))
+
+	outcome, err := daemon.runOnce()
+	if err != nil {
+		t.Fatalf("runOnce: %v", err)
+	}
+	if outcome.Completed {
+		t.Fatal("task must not complete")
+	}
+	if len(fake.failures) != 1 || fake.failures[0]["retryable"] != false {
+		t.Fatalf("failures = %v", fake.failures)
+	}
+}
+
+func TestDaemonReportsRetryableRunnerFailure(t *testing.T) {
+	fake := newFakeCoordinator(t, validClaimedTaskPayload())
+	defer fake.close()
+	daemon := testDaemon(t, fake, fakeRunnerScript(t, t.TempDir(), 1))
+
+	outcome, err := daemon.runOnce()
+	if err != nil {
+		t.Fatalf("runOnce: %v", err)
+	}
+	if outcome.Completed {
+		t.Fatal("task must not complete")
+	}
+	if len(fake.failures) != 1 || fake.failures[0]["retryable"] != true {
+		t.Fatalf("failures = %v", fake.failures)
+	}
+}
+
+func TestDaemonLeaseConflictStopsWithoutFailureReport(t *testing.T) {
+	fake := newFakeCoordinator(t, validClaimedTaskPayload())
+	fake.conflict = true
+	defer fake.close()
+	daemon := testDaemon(t, fake, fakeRunnerScript(t, t.TempDir(), 0))
+
+	outcome, err := daemon.runOnce()
+	if err != nil {
+		t.Fatalf("runOnce: %v", err)
+	}
+	if !outcome.Claimed {
+		t.Fatal("task was claimed")
+	}
+	if len(fake.failures) != 0 {
+		t.Errorf("no failure report expected after lease loss: %v", fake.failures)
+	}
+	if len(fake.submits) != 0 {
+		t.Errorf("no submission expected after lease loss: %v", fake.submits)
+	}
+}
+
+func TestDaemonIdleClaimIsNotCompleted(t *testing.T) {
+	fake := newFakeCoordinator(t, nil)
+	defer fake.close()
+	daemon := testDaemon(t, fake, fakeRunnerScript(t, t.TempDir(), 0))
+
+	outcome, err := daemon.runOnce()
+	if err != nil {
+		t.Fatalf("runOnce: %v", err)
+	}
+	if outcome.Claimed || outcome.Completed {
+		t.Fatalf("outcome = %+v", outcome)
+	}
+}

coordinator/internal/agent/models.go

@@ -0,0 +1,205 @@
+// Package agent implements a Go worker agent: a coordinator client and
+// task-lifecycle supervisor that executes SDK workloads in a Python
+// subprocess. It mirrors the Python worker's v1 wire contract exactly; the
+// Python worker remains the reference implementation.
+package agent
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"regexp"
+	"strings"
+	"time"
+)
+
+var (
+	uuidPattern     = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
+	sha256Pattern   = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
+	workloadPattern = regexp.MustCompile(`^[a-z][a-z0-9]*(?:[_-][a-z0-9]+)*$`)
+)
+
+// RegisteredWorker is the coordinator's answer to /workers/register.
+type RegisteredWorker struct {
+	WorkerID                 string
+	HeartbeatIntervalSeconds float64
+}
+
+// Input is the claimed task's input artifact.
+type Input struct {
+	URI    string
+	SHA256 string
+}
+
+// Task is one claimed, leased task.
+type Task struct {
+	TaskID          string
+	Attempt         int
+	LeaseExpiresAt  time.Time
+	Workload        string
+	Input           Input
+	Parameters      map[string]any
+	leaseExpiresRaw string
+}
+
+// Uploaded is the coordinator-owned metadata returned after artifact upload.
+type Uploaded struct {
+	ArtifactID string
+	URI        string
+	SHA256     string
+	SizeBytes  int64
+}
+
+func requireString(value any, field string) (string, error) {
+	text, ok := value.(string)
+	if !ok || strings.TrimSpace(text) == "" {
+		return "", fmt.Errorf("%s must be a non-empty string", field)
+	}
+	return text, nil
+}
+
+func safeCoordinatorURI(value any, field string) (string, error) {
+	uri, err := requireString(value, field)
+	if err != nil {
+		return "", err
+	}
+	if strings.HasPrefix(uri, "/") {
+		// A network-path reference (//host/path) or dot segments would
+		// resolve to another origin; reject both.
+		if strings.HasPrefix(uri, "//") {
+			return "", fmt.Errorf("%s must be a safe coordinator path", field)
+		}
+		for _, segment := range strings.Split(uri, "/") {
+			if segment == ".." {
+				return "", fmt.Errorf("%s must be a safe coordinator path", field)
+			}
+		}
+		return uri, nil
+	}
+	parsed, err := url.Parse(uri)
+	if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" {
+		return "", fmt.Errorf("%s must be an absolute HTTP(S) URL or coordinator path", field)
+	}
+	return uri, nil
+}
+
+func sha256Hex(value any, field string) (string, error) {
+	digest, err := requireString(value, field)
+	if err != nil {
+		return "", err
+	}
+	digest = strings.ToLower(digest)
+	if !sha256Pattern.MatchString(digest) {
+		return "", fmt.Errorf("%s must be a SHA-256 hex digest", field)
+	}
+	return digest, nil
+}
+
+func uuid(value any, field string) (string, error) {
+	text, err := requireString(value, field)
+	if err != nil {
+		return "", err
+	}
+	if !uuidPattern.MatchString(text) {
+		return "", fmt.Errorf("%s must be a UUID", field)
+	}
+	return strings.ToLower(text), nil
+}
+
+// ParseTask validates a claimed-task response with the same strictness as the
+// Python worker's ClaimedTask.from_json.
+func ParseTask(payload map[string]any) (*Task, error) {
+	rawInput, ok := payload["input"].(map[string]any)
+	if !ok {
+		return nil, fmt.Errorf("input must be an object")
+	}
+	rawAttempt, ok := payload["attempt"].(float64)
+	if !ok || rawAttempt < 1 || rawAttempt != float64(int(rawAttempt)) {
+		return nil, fmt.Errorf("attempt must be a positive integer")
+	}
+	taskID, err := uuid(payload["task_id"], "task_id")
+	if err != nil {
+		return nil, fmt.Errorf("invalid claimed-task response: %w", err)
+	}
+	rawLease, err := requireString(payload["lease_expires_at"], "lease_expires_at")
+	if err != nil {
+		return nil, fmt.Errorf("invalid claimed-task response: %w", err)
+	}
+	lease, err := time.Parse(time.RFC3339, rawLease)
+	if err != nil || lease.Location() == nil {
+		return nil, fmt.Errorf("lease_expires_at must include a timezone")
+	}
+	workload, err := requireString(payload["workload"], "workload")
+	if err != nil || !workloadPattern.MatchString(workload) {
+		return nil, fmt.Errorf("workload must be a canonical name")
+	}
+	uri, err := safeCoordinatorURI(rawInput["uri"], "input.uri")
+	if err != nil {
+		return nil, fmt.Errorf("invalid claimed-task response: %w", err)
+	}
+	digest, err := sha256Hex(rawInput["sha256"], "input.sha256")
+	if err != nil {
+		return nil, fmt.Errorf("invalid claimed-task response: %w", err)
+	}
+	parameters, ok := payload["parameters"].(map[string]any)
+	if !ok {
+		parameters = map[string]any{}
+	}
+	return &Task{
+		TaskID:          taskID,
+		Attempt:         int(rawAttempt),
+		LeaseExpiresAt:  lease,
+		leaseExpiresRaw: rawLease,
+		Workload:        workload,
+		Input:           Input{URI: uri, SHA256: digest},
+		Parameters:      parameters,
+	}, nil
+}
+
+// LeaseExpiresRaw returns the original lease timestamp string for
+// round-tripping in heartbeat deadlines.
+func (t *Task) LeaseExpiresRaw() string { return t.leaseExpiresRaw }
+
+// ParseRegistered validates a registration response.
+func ParseRegistered(payload map[string]any) (*RegisteredWorker, error) {
+	workerID, err := uuid(payload["worker_id"], "worker_id")
+	if err != nil {
+		return nil, fmt.Errorf("invalid worker registration response: %w", err)
+	}
+	interval, ok := payload["heartbeat_interval_seconds"].(float64)
+	if !ok || interval <= 0 {
+		return nil, fmt.Errorf("heartbeat_interval_seconds must be positive")
+	}
+	return &RegisteredWorker{WorkerID: workerID, HeartbeatIntervalSeconds: interval}, nil
+}
+
+// ParseUploaded validates an artifact upload response.
+func ParseUploaded(payload map[string]any) (*Uploaded, error) {
+	artifactID, err := uuid(payload["artifact_id"], "artifact_id")
+	if err != nil {
+		return nil, fmt.Errorf("invalid artifact upload response: %w", err)
+	}
+	uri, err := safeCoordinatorURI(payload["uri"], "uri")
+	if err != nil {
+		return nil, fmt.Errorf("invalid artifact upload response: %w", err)
+	}
+	digest, err := sha256Hex(payload["sha256"], "sha256")
+	if err != nil {
+		return nil, fmt.Errorf("invalid artifact upload response: %w", err)
+	}
+	rawSize, ok := payload["size_bytes"].(float64)
+	if !ok || rawSize < 0 || rawSize != float64(int64(rawSize)) {
+		return nil, fmt.Errorf("artifact size_bytes must be a non-negative integer")
+	}
+	return &Uploaded{ArtifactID: artifactID, URI: uri, SHA256: digest, SizeBytes: int64(rawSize)}, nil
+}
+
+// TaskRunnerManifest is what the Python task entry writes on success.
+type TaskRunnerManifest struct {
+	ArtifactPath string         `json:"artifact_path"`
+	ContentType  string         `json:"content_type"`
+	Metrics      map[string]any `json:"metrics"`
+}
+
+// Encode serializes a claim payload for /tasks/claim.
+func Encode(v any) ([]byte, error) { return json.Marshal(v) }

coordinator/internal/agent/models_test.go

@@ -0,0 +1,111 @@
+package agent
+
+import (
+	"testing"
+	"time"
+)
+
+func validTaskPayload() map[string]any {
+	return map[string]any{
+		"task_id":          "11111111-1111-4111-8111-111111111111",
+		"attempt":          1.0,
+		"lease_expires_at": "2026-08-02T00:00:00Z",
+		"workload":         "similarity-search",
+		"input": map[string]any{
+			"uri":    "/tasks/11111111-1111-4111-8111-111111111111/input",
+			"sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+		},
+		"parameters": map[string]any{"query_smiles": "CCO"},
+	}
+}
+
+func TestParseTaskAcceptsValidPayload(t *testing.T) {
+	task, err := ParseTask(validTaskPayload())
+	if err != nil {
+		t.Fatalf("ParseTask: %v", err)
+	}
+	if task.TaskID != "11111111-1111-4111-8111-111111111111" {
+		t.Errorf("task id = %q", task.TaskID)
+	}
+	if task.Attempt != 1 || task.Workload != "similarity-search" {
+		t.Errorf("attempt/workload = %d/%q", task.Attempt, task.Workload)
+	}
+	if task.Input.SHA256 != "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" {
+		t.Errorf("sha256 = %q", task.Input.SHA256)
+	}
+	if task.LeaseExpiresAt.IsZero() {
+		t.Error("lease must parse")
+	}
+}
+
+func TestParseTaskRejectsInvalidPayloads(t *testing.T) {
+	tests := []struct {
+		name   string
+		mutate func(map[string]any)
+	}{
+		{"non-uuid task id", func(p map[string]any) { p["task_id"] = "../outside" }},
+		{"zero attempt", func(p map[string]any) { p["attempt"] = 0 }},
+		{"naive lease", func(p map[string]any) { p["lease_expires_at"] = "2026-08-02T00:00:00" }},
+		{"network-path uri", func(p map[string]any) {
+			p["input"].(map[string]any)["uri"] = "//outside.example/input"
+		}},
+		{"dot-segment uri", func(p map[string]any) {
+			p["input"].(map[string]any)["uri"] = "/tasks/../outside/input"
+		}},
+		{"short sha256", func(p map[string]any) {
+			p["input"].(map[string]any)["sha256"] = "abc"
+		}},
+	}
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			payload := validTaskPayload()
+			test.mutate(payload)
+			if _, err := ParseTask(payload); err == nil {
+				t.Error("expected ParseTask to reject the payload")
+			}
+		})
+	}
+}
+
+func TestParseRegisteredAndUploaded(t *testing.T) {
+	registered, err := ParseRegistered(map[string]any{
+		"worker_id":                  "22222222-2222-4222-8222-222222222222",
+		"heartbeat_interval_seconds": 15.0,
+	})
+	if err != nil {
+		t.Fatalf("ParseRegistered: %v", err)
+	}
+	if registered.HeartbeatIntervalSeconds != 15 {
+		t.Errorf("interval = %v", registered.HeartbeatIntervalSeconds)
+	}
+
+	uploaded, err := ParseUploaded(map[string]any{
+		"artifact_id": "33333333-3333-4333-8333-333333333333",
+		"uri":         "https://coordinator.example/artifacts/333/download",
+		"sha256":      "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+		"size_bytes":  12.0,
+	})
+	if err != nil {
+		t.Fatalf("ParseUploaded: %v", err)
+	}
+	if uploaded.SizeBytes != 12 {
+		t.Errorf("size = %d", uploaded.SizeBytes)
+	}
+
+	if _, err := ParseUploaded(map[string]any{"artifact_id": "missing"}); err == nil {
+		t.Error("expected invalid upload metadata to fail")
+	}
+}
+
+func TestLeaseHeartbeatDelayIsBelowHalfTTL(t *testing.T) {
+	task, err := ParseTask(validTaskPayload())
+	if err != nil {
+		t.Fatal(err)
+	}
+	task.LeaseExpiresAt = time.Now().Add(60 * time.Second)
+	heartbeat := newLeaseHeartbeat(task, "worker", nil, 15*time.Second)
+	delay := heartbeat.nextDelay()
+	if delay > 30*time.Second || delay <= 0 {
+		t.Errorf("delay = %v, want < 30s", delay)
+	}
+}

coordinator/internal/agent/sanitize.go

@@ -0,0 +1,58 @@
+package agent
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"regexp"
+	"strings"
+)
+
+var (
+	// Go's regexp (RE2) has no lookbehind, so these patterns conservatively
+	// anchor on the characters that typically precede a local path:
+	// whitespace, quotes, parens, brackets, or the start of the message.
+	windowsPathPattern = regexp.MustCompile(`[A-Za-z]:\\[^\s'"\],)]+`)
+	posixPathPattern   = regexp.MustCompile(`(^|[\s'"(\[=])/(?:[^\s'"\],)]+)`)
+)
+
+// SanitizeErrorMessage keeps coordinator-visible failures useful without
+// exposing local paths. It mirrors the Python worker's sanitizer: local work
+// directories and absolute paths are redacted, and the message is truncated
+// to 300 characters.
+func SanitizeErrorMessage(message string, workDir string) string {
+	message = strings.ReplaceAll(message, workDir, "<worker-dir>")
+	message = windowsPathPattern.ReplaceAllString(message, "<path>")
+	message = posixPathPattern.ReplaceAllString(message, "${1}<path>")
+	if len(message) > 300 {
+		message = message[:300]
+	}
+	return message
+}
+
+// IsRetryableError classifies failures for the coordinator. Invalid scientific
+// input and missing local tools are permanent; everything else (transient
+// transport errors, unexpected runner failures) may be retried.
+func IsRetryableError(err error) bool {
+	if err == nil {
+		return false
+	}
+	var coordinatorErr *CoordinatorError
+	var pathErr *os.PathError
+	if errors.As(err, &coordinatorErr) || errors.As(err, &pathErr) {
+		return false
+	}
+	return true
+}
+
+// TaskRunnerExit classifies subprocess exits.
+const (
+	ExitPermanent = 3 // runner classified the failure as invalid input
+)
+
+func runnerExitError(exit int, stderr string) error {
+	if exit == ExitPermanent {
+		return &CoordinatorError{msg: stderr}
+	}
+	return fmt.Errorf("task runner failed with exit code %d: %s", exit, stderr)
+}

coordinator/internal/agent/taskrunner.go

@@ -0,0 +1,86 @@
+package agent
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+)
+
+// TaskRunner spawns the Python task entry and returns the sealed partial
+// artifact manifest it produced.
+type TaskRunner struct {
+	command []string
+}
+
+func NewTaskRunner(command []string) *TaskRunner {
+	return &TaskRunner{command: command}
+}
+
+// Run executes one task: the task payload is written as JSON into the attempt
+// directory, the Python entry computes and seals the partial, and the written
+// manifest is parsed back. stderr is captured for failure reporting.
+func (r *TaskRunner) Run(task *Task, taskDir string, manifestPath string, extraEnv []string) (*TaskRunnerManifest, error) {
+	if err := os.MkdirAll(taskDir, 0o750); err != nil {
+		return nil, err
+	}
+	payload := map[string]any{
+		"task_id":          task.TaskID,
+		"attempt":          task.Attempt,
+		"lease_expires_at": task.leaseExpiresRaw,
+		"workload":         task.Workload,
+		"input": map[string]any{
+			"uri":    task.Input.URI,
+			"sha256": task.Input.SHA256,
+		},
+		"parameters": task.Parameters,
+	}
+	payloadBytes, err := json.Marshal(payload)
+	if err != nil {
+		return nil, err
+	}
+	taskJSONPath := filepath.Join(taskDir, "task.json")
+	if err := os.WriteFile(taskJSONPath, payloadBytes, 0o600); err != nil {
+		return nil, err
+	}
+	args := append([]string{}, r.command[1:]...)
+	args = append(args,
+		"--task-json", taskJSONPath,
+		"--task-dir", taskDir,
+		"--output", manifestPath,
+	)
+	// #nosec G204 -- the command comes from the operator-configured TASK_RUNNER.
+	command := exec.CommandContext(context.Background(), r.command[0], args...)
+	command.Dir = taskDir
+	command.Env = append(os.Environ(), extraEnv...)
+	var stderr bytes.Buffer
+	command.Stderr = &stderr
+	if err := command.Run(); err != nil {
+		var exitErr *exec.ExitError
+		if errors.As(err, &exitErr) {
+			return nil, runnerExitError(exitErr.ExitCode(), stderr.String())
+		}
+		return nil, fmt.Errorf("task runner could not be started: %w", err)
+	}
+	// #nosec G304 -- the manifest path is inside the worker's own task directory.
+	raw, err := os.ReadFile(manifestPath)
+	if err != nil {
+		return nil, fmt.Errorf("task runner produced no result manifest")
+	}
+	var manifest TaskRunnerManifest
+	if err := json.Unmarshal(raw, &manifest); err != nil {
+		return nil, fmt.Errorf("task runner produced an invalid result manifest")
+	}
+	if manifest.ArtifactPath == "" || manifest.ContentType == "" {
+		return nil, fmt.Errorf("task runner produced an incomplete result manifest")
+	}
+	info, err := os.Stat(manifest.ArtifactPath)
+	if err != nil || !info.Mode().IsRegular() {
+		return nil, fmt.Errorf("task runner produced no artifact file")
+	}
+	return &manifest, nil
+}

coordinator/internal/infra/config.go

@@ -51,6 +51,9 @@ type Config struct {
 	LogFile string
 	// Directory where artifact bytes are stored.
 	StorageDir string
+	// Directory of the built MkDocs site (site/) served at /ui/docs/. Empty
+	// disables the docs route; the UI shows a hint page instead.
+	DocsDir string
 	// Upper bound on an uploaded dataset or artifact body, in bytes.
 	MaxUploadBytes int64
 
@@ -75,6 +78,10 @@ type Config struct {
 	ReaperInterval time.Duration
 	// A worker silent for longer than this is marked offline by the reaper.
 	WorkerOfflineAfter time.Duration
+	// Whether the binary applies its embedded schema migrations on startup.
+	// On by default so a downloaded binary provisions its own database; set
+	// AUTO_MIGRATE=false when an operator manages migrations out of band.
+	AutoMigrate bool
 }
 
 // Load reads the environment and fails fast on anything required-but-missing
@@ -108,6 +115,7 @@ func LoadConfig() (Config, error) {
 		LogLevel:             getEnv("LOG_LEVEL", "info"),
 		LogFile:              os.Getenv("LOG_FILE"),
 		StorageDir:           getEnv("COORDINATOR_STORAGE_DIR", "./data"),
+		DocsDir:              os.Getenv("SCIMESH_DOCS_DIR"),
 		MaxUploadBytes:       1 << 30, // 1 GiB
 		DBMaxConns:           10,
 		DBConnectTimeout:     30 * time.Second,
@@ -169,6 +177,14 @@ func LoadConfig() (Config, error) {
 	if cfg.DefaultMaxAttempts < 1 {
 		return Config{}, fmt.Errorf("DEFAULT_MAX_ATTEMPTS must be positive")
 	}
+	cfg.AutoMigrate = true
+	if raw := os.Getenv("AUTO_MIGRATE"); raw != "" {
+		parsed, err := strconv.ParseBool(raw)
+		if err != nil {
+			return Config{}, fmt.Errorf("AUTO_MIGRATE must be true or false")
+		}
+		cfg.AutoMigrate = parsed
+	}
 
 	return cfg, nil
 }

coordinator/internal/reducer/concat.go

@@ -0,0 +1,57 @@
+// Package reducer contains deterministic, coordinator-side result reductions.
+package reducer
+
+import (
+	"bytes"
+	"encoding/csv"
+	"errors"
+	"fmt"
+	"io"
+)
+
+// ReduceOrderedConcat concatenates worker partial tables in shard order into a
+// single table with one header. Every partial must carry the same header as the
+// first partial and rows of the same width; anything else fails the job closed.
+func ReduceOrderedConcat(partials []io.Reader) ([]byte, error) {
+	var out bytes.Buffer
+	writer := csv.NewWriter(&out)
+	var firstHeader []string
+	for _, partial := range partials {
+		reader := csv.NewReader(partial)
+		header, err := reader.Read()
+		if err != nil {
+			return nil, fmt.Errorf("read partial header: %w", err)
+		}
+		if len(header) == 0 {
+			return nil, fmt.Errorf("partial result has an empty header")
+		}
+		if firstHeader == nil {
+			firstHeader = header
+			if err := writer.Write(header); err != nil {
+				return nil, err
+			}
+		} else if !equalStrings(header, firstHeader) {
+			return nil, fmt.Errorf("partial result has an inconsistent header")
+		}
+		for {
+			row, err := reader.Read()
+			if errors.Is(err, io.EOF) {
+				break
+			}
+			if err != nil {
+				return nil, fmt.Errorf("read partial row: %w", err)
+			}
+			if len(row) != len(header) {
+				return nil, fmt.Errorf("partial result has a row with an inconsistent width")
+			}
+			if err := writer.Write(row); err != nil {
+				return nil, err
+			}
+		}
+	}
+	writer.Flush()
+	if err := writer.Error(); err != nil {
+		return nil, err
+	}
+	return out.Bytes(), nil
+}

coordinator/internal/reducer/concat_test.go

@@ -0,0 +1,64 @@
+package reducer
+
+import (
+	"io"
+	"strings"
+	"testing"
+)
+
+func TestReduceOrderedConcatJoinsPartialsInOrderWithOneHeader(t *testing.T) {
+	first := strings.NewReader("chembl_id,canonical_smiles\nA,CC\nB,CCC\n")
+	second := strings.NewReader("chembl_id,canonical_smiles\nC,CCCC\n")
+
+	output, err := ReduceOrderedConcat([]io.Reader{first, second})
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := "chembl_id,canonical_smiles\nA,CC\nB,CCC\nC,CCCC\n"
+	if string(output) != want {
+		t.Fatalf("output = %q, want %q", output, want)
+	}
+}
+
+func TestReduceOrderedConcatIsDeterministicAcrossInputOrder(t *testing.T) {
+	left := strings.NewReader("id,rows\nA,1\nB,2\n")
+	right := strings.NewReader("id,rows\nC,3\n")
+
+	first, err := ReduceOrderedConcat([]io.Reader{left, right})
+	if err != nil {
+		t.Fatal(err)
+	}
+	left, right = strings.NewReader("id,rows\nA,1\nB,2\n"), strings.NewReader("id,rows\nC,3\n")
+	second, err := ReduceOrderedConcat([]io.Reader{left, right})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if string(first) != string(second) {
+		t.Fatalf("concat is not deterministic: %q != %q", first, second)
+	}
+}
+
+func TestReduceOrderedConcatRejectsInconsistentHeaders(t *testing.T) {
+	first := strings.NewReader("a,b\n1,2\n")
+	second := strings.NewReader("a,c\n1,2\n")
+	if _, err := ReduceOrderedConcat([]io.Reader{first, second}); err == nil {
+		t.Fatal("inconsistent headers must fail")
+	}
+}
+
+func TestReduceOrderedConcatRejectsRaggedRows(t *testing.T) {
+	partial := strings.NewReader("a,b\n1,2,3\n")
+	if _, err := ReduceOrderedConcat([]io.Reader{partial}); err == nil {
+		t.Fatal("ragged rows must fail")
+	}
+}
+
+func TestReduceOrderedConcatEmptyPartials(t *testing.T) {
+	output, err := ReduceOrderedConcat(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(output) != 0 {
+		t.Fatalf("empty input must produce empty output, got %q", output)
+	}
+}

coordinator/internal/setup/setup.go

@@ -0,0 +1,239 @@
+// Package setup implements the `coordinator setup` wizard: database reachability
+// and creation, embedded schema migration, secret generation, and .env writing.
+// The wizard never logs or echoes secrets.
+package setup
+
+import (
+	"bufio"
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"github.com/jackc/pgx/v5"
+	"github.com/jackc/pgx/v5/pgconn"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/storage/postgres"
+)
+
+// Options configures one wizard run.
+type Options struct {
+	// DatabaseURL is the target coordinator database (pgx/libpq URL).
+	DatabaseURL string
+	// AdminDatabaseURL, when set, is used to create a missing target database.
+	// Defaults to the target URL with the database name replaced by "postgres".
+	AdminDatabaseURL string
+	// EnvFile is where the generated settings are written (default ".env").
+	EnvFile string
+	// Force overwrites an existing EnvFile.
+	Force bool
+	// Yes disables interactive prompts; missing values fail instead.
+	Yes bool
+	// ConnectTimeout bounds the reachability check.
+	ConnectTimeout time.Duration
+	// Out receives progress and summary output; In feeds interactive answers.
+	Out io.Writer
+	In  io.Reader
+}
+
+// Run executes the wizard and returns a summary of what was done.
+func Run(ctx context.Context, options Options) (string, error) {
+	if options.DatabaseURL == "" {
+		return "", fmt.Errorf("DATABASE_URL is required (or pass --db)")
+	}
+	if options.EnvFile == "" {
+		options.EnvFile = ".env"
+	}
+	if options.ConnectTimeout <= 0 {
+		options.ConnectTimeout = 5 * time.Second
+	}
+	if options.Out == nil {
+		options.Out = os.Stdout
+	}
+
+	report := func(format string, args ...any) {
+		_, _ = fmt.Fprintf(options.Out, format+"\n", args...)
+	}
+
+	report("SciMesh coordinator setup")
+	report("")
+
+	// 1. Reachability, with optional database creation.
+	target, err := pgx.ParseConfig(options.DatabaseURL)
+	if err != nil {
+		return "", fmt.Errorf("DATABASE_URL is not a valid postgres URL: %w", err)
+	}
+	if err := probeDatabase(ctx, target, options.ConnectTimeout); err != nil {
+		var pgErr *pgconn.PgError
+		if !errors.As(err, &pgErr) || pgErr.Code != "3D000" {
+			return "", fmt.Errorf("cannot reach the coordinator database: %w", err)
+		}
+		report("database %q does not exist yet", target.Database)
+		admin, err := resolveAdminConfig(options, target)
+		if err != nil {
+			return "", err
+		}
+		if err := createDatabase(ctx, admin, target.Database, options.ConnectTimeout); err != nil {
+			return "", fmt.Errorf("cannot create database %q: %w", target.Database, err)
+		}
+		report("created database %q", target.Database)
+	}
+	report("database %q is reachable", target.Database)
+
+	// 2. Apply the embedded schema migrations (idempotent).
+	if err := postgres.Migrate(ctx, options.DatabaseURL, nil); err != nil {
+		return "", fmt.Errorf("apply schema migrations: %w", err)
+	}
+	report("schema migrations applied")
+
+	// 3. JWT secret: reuse the environment value when strong, else generate.
+	secret := os.Getenv("JWT_SECRET")
+	if secret != "" && len(secret) < 32 {
+		return "", fmt.Errorf("JWT_SECRET must be at least 32 bytes")
+	}
+	if secret == "" {
+		generated, err := generateSecret()
+		if err != nil {
+			return "", fmt.Errorf("generate JWT_SECRET: %w", err)
+		}
+		secret = generated
+		report("generated a fresh JWT_SECRET")
+	}
+
+	// 4. Write the .env file.
+	storageDir := os.Getenv("COORDINATOR_STORAGE_DIR")
+	if storageDir == "" {
+		storageDir = "./data"
+	}
+	if err := writeEnvFile(options, secret, storageDir); err != nil {
+		return "", err
+	}
+
+	// 5. Summary.
+	var summary strings.Builder
+	fmt.Fprintf(&summary, "Setup complete.\n\n")
+	fmt.Fprintf(&summary, "Ready:\n")
+	fmt.Fprintf(&summary, "  - database %s is reachable and migrated\n", target.Database)
+	fmt.Fprintf(&summary, "  - settings written to %s (chmod 0600)\n", options.EnvFile)
+	fmt.Fprintf(&summary, "\nStart the coordinator:\n")
+	fmt.Fprintf(&summary, "  ENV_FILE=%s ./coordinator\n", options.EnvFile)
+	fmt.Fprintf(&summary, "\nOptional — userservice for UI logins (must share JWT_SECRET):\n")
+	fmt.Fprintf(&summary, "  cd users && JWT_SECRET=%q docker compose up -d\n", secret)
+	fmt.Fprintf(&summary, "  then set USERSERVICE_URL=http://localhost:8081 and BOOTSTRAP_ADMIN_EMAIL/PASSWORD\n")
+	fmt.Fprintf(&summary, "\nThe wizard cannot run PostgreSQL or the userservice for you; the\n")
+	fmt.Fprintf(&summary, "commands above are the supported way to start them.\n")
+	return summary.String(), nil
+}
+
+// probeDatabase verifies the target database accepts connections.
+func probeDatabase(ctx context.Context, config *pgx.ConnConfig, timeout time.Duration) error {
+	ctx, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+	conn, err := pgx.ConnectConfig(ctx, config)
+	if err != nil {
+		return err
+	}
+	return conn.Close(ctx)
+}
+
+// resolveAdminConfig picks the maintenance connection used to create
+// databases. pgx's ConnConfig.ConnString() caches the original URL, so the
+// config itself (not a re-rendered string) is what the caller connects with.
+func resolveAdminConfig(options Options, target *pgx.ConnConfig) (*pgx.ConnConfig, error) {
+	if options.AdminDatabaseURL != "" {
+		config, err := pgx.ParseConfig(options.AdminDatabaseURL)
+		if err != nil {
+			return nil, fmt.Errorf("--admin-db is not a valid postgres URL: %w", err)
+		}
+		return config, nil
+	}
+	admin := *target
+	admin.Database = "postgres"
+	return &admin, nil
+}
+
+// createDatabase creates the named database through the maintenance connection.
+func createDatabase(ctx context.Context, admin *pgx.ConnConfig, name string, timeout time.Duration) error {
+	ctx, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+	conn, err := pgx.ConnectConfig(ctx, admin)
+	if err != nil {
+		return err
+	}
+	defer func() { _ = conn.Close(ctx) }()
+	quoted := `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
+	if _, err := conn.Exec(ctx, "CREATE DATABASE "+quoted); err != nil {
+		return err
+	}
+	return nil
+}
+
+// generateSecret returns 32 random bytes as lowercase hex.
+func generateSecret() (string, error) {
+	buffer := make([]byte, 32)
+	if _, err := rand.Read(buffer); err != nil {
+		return "", err
+	}
+	return hex.EncodeToString(buffer), nil
+}
+
+// writeEnvFile writes the settings, refusing to clobber without --force.
+func writeEnvFile(options Options, secret, storageDir string) error {
+	path := filepath.Clean(options.EnvFile)
+	if _, err := os.Stat(path); err == nil && !options.Force {
+		return fmt.Errorf("%s already exists (use --force to overwrite)", path)
+	}
+	content := strings.Join([]string{
+		"DATABASE_URL=" + options.DatabaseURL,
+		"JWT_SECRET=" + secret,
+		"COORDINATOR_STORAGE_DIR=" + storageDir,
+		"", // trailing newline
+	}, "\n")
+	// #nosec G703 -- the env file path is operator-supplied (--env-file / ENV_FILE).
+	if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
+		return fmt.Errorf("write %s: %w", path, err)
+	}
+	if err := os.Chmod(path, 0o600); err != nil {
+		return fmt.Errorf("chmod %s: %w", path, err)
+	}
+	return nil
+}
+
+// SanitizeDatabaseURL hides the password for logging.
+func SanitizeDatabaseURL(raw string) string {
+	at := strings.LastIndex(raw, "@")
+	if at < 0 {
+		return raw
+	}
+	start := 0
+	if strings.HasPrefix(raw, "postgres://") || strings.HasPrefix(raw, "postgresql://") {
+		start = len("postgres://")
+	}
+	colon := strings.Index(raw[start:at], ":")
+	if colon < 0 {
+		return raw
+	}
+	colon += start
+	return raw[:colon] + ":***@" + raw[at+1:]
+}
+
+// prompt asks a question and returns the trimmed answer ("" on EOF).
+func prompt(options Options, question, fallback string) string {
+	_, _ = fmt.Fprintf(options.Out, "%s [%s]: ", question, fallback)
+	reader := bufio.NewReader(options.In)
+	line, err := reader.ReadString('\n')
+	if err != nil && !errors.Is(err, io.EOF) {
+		return fallback
+	}
+	answer := strings.TrimSpace(line)
+	if answer == "" {
+		return fallback
+	}
+	return answer
+}

coordinator/internal/setup/setup_integration_test.go

@@ -0,0 +1,91 @@
+//go:build integration
+
+package setup
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/jackc/pgx/v5"
+)
+
+func TestRunProvisionsDatabaseSchemaAndEnvFile(t *testing.T) {
+	ctx := context.Background()
+	base := os.Getenv("TEST_DATABASE_URL")
+	if base == "" {
+		t.Skip("TEST_DATABASE_URL is not set")
+	}
+	// The wizard must create a *missing* database through the admin URL.
+	slash := strings.LastIndex(base, "/")
+	target := base[:slash+1] + "scimesh_setup_test"
+	admin := base[:slash+1] + "postgres"
+
+	cleanup := func() {
+		conn, err := pgx.Connect(ctx, admin)
+		if err != nil {
+			return
+		}
+		defer func() { _ = conn.Close(ctx) }()
+		_, _ = conn.Exec(ctx, `DROP DATABASE IF EXISTS "scimesh_setup_test"`)
+	}
+	cleanup()
+	t.Cleanup(cleanup)
+
+	envPath := filepath.Join(t.TempDir(), ".env")
+	var output strings.Builder
+	options := Options{
+		DatabaseURL:      target,
+		AdminDatabaseURL: admin,
+		EnvFile:          envPath,
+		Force:            true,
+		Yes:              true,
+		ConnectTimeout:   10 * time.Second,
+		Out:              &output,
+		In:               strings.NewReader(""),
+	}
+	summary, err := Run(ctx, options)
+	if err != nil {
+		t.Fatalf("setup run: %v", err)
+	}
+	for _, expected := range []string{"created database", "schema migrations applied"} {
+		if !strings.Contains(output.String(), expected) {
+			t.Errorf("progress output is missing %q:\n%s", expected, output.String())
+		}
+	}
+	for _, expected := range []string{"Setup complete", "Start the coordinator"} {
+		if !strings.Contains(summary, expected) {
+			t.Errorf("summary is missing %q:\n%s", expected, summary)
+		}
+	}
+
+	// The database now exists, is migrated, and the env file is written.
+	conn, err := pgx.Connect(ctx, target)
+	if err != nil {
+		t.Fatalf("connect to provisioned database: %v", err)
+	}
+	defer func() { _ = conn.Close(ctx) }()
+	var watermark int64
+	if err := conn.QueryRow(ctx, "SELECT COALESCE(MAX(version), 0) FROM schema_migrations").Scan(&watermark); err != nil {
+		t.Fatalf("read schema_migrations: %v", err)
+	}
+	if watermark < 13 {
+		t.Errorf("schema watermark = %d, want >= 13", watermark)
+	}
+	envContent, err := os.ReadFile(envPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	env := string(envContent)
+	if !strings.Contains(env, "DATABASE_URL="+target) || !strings.Contains(env, "JWT_SECRET=") {
+		t.Errorf("env file is incomplete:\n%s", env)
+	}
+
+	// A second run is idempotent: no create-database error, same outcome.
+	if _, err := Run(ctx, options); err != nil {
+		t.Fatalf("second setup run: %v", err)
+	}
+}

coordinator/internal/setup/setup_test.go

@@ -0,0 +1,96 @@
+package setup
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestGenerateSecretIsRandomAndStrong(t *testing.T) {
+	first, err := generateSecret()
+	if err != nil {
+		t.Fatal(err)
+	}
+	second, err := generateSecret()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(first) != 64 || len(second) != 64 {
+		t.Fatalf("secrets must be 32 random bytes as hex, got %d and %d", len(first), len(second))
+	}
+	if first == second {
+		t.Fatal("two generated secrets must differ")
+	}
+}
+
+func TestWriteEnvFileContentsAndPermissions(t *testing.T) {
+	path := filepath.Join(t.TempDir(), ".env")
+	if err := writeEnvFile(Options{
+		EnvFile:     path,
+		DatabaseURL: "postgres://scimesh@localhost/scimesh",
+	}, "s3cr3t", "./data"); err != nil {
+		t.Fatal(err)
+	}
+	content, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := "DATABASE_URL=postgres://scimesh@localhost/scimesh\nJWT_SECRET=s3cr3t\nCOORDINATOR_STORAGE_DIR=./data\n"
+	if got := string(content); got != want {
+		t.Fatalf("env file = %q, want %q", got, want)
+	}
+	info, err := os.Stat(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if info.Mode().Perm() != 0o600 {
+		t.Errorf("env file mode = %o, want 0600", info.Mode().Perm())
+	}
+}
+
+func TestWriteEnvFileRefusesWithoutForce(t *testing.T) {
+	path := filepath.Join(t.TempDir(), ".env")
+	if err := writeEnvFile(Options{EnvFile: path, DatabaseURL: "postgres://x@localhost/a"}, "a", "./data"); err != nil {
+		t.Fatal(err)
+	}
+	if err := writeEnvFile(Options{EnvFile: path, DatabaseURL: "postgres://x@localhost/a"}, "b", "./data"); err == nil {
+		t.Fatal("second write without --force must fail")
+	}
+	if err := writeEnvFile(Options{EnvFile: path, Force: true, DatabaseURL: "postgres://x@localhost/a"}, "b", "./data"); err != nil {
+		t.Fatalf("write with --force: %v", err)
+	}
+}
+
+func TestPromptReadsAnswer(t *testing.T) {
+	var out bytes.Buffer
+	answer := prompt(Options{Out: &out, In: strings.NewReader("postgres://custom\n")}, "Database URL", "default")
+	if answer != "postgres://custom" {
+		t.Fatalf("answer = %q, want the typed value", answer)
+	}
+	if !strings.Contains(out.String(), "Database URL [default]:") {
+		t.Fatalf("prompt output = %q", out.String())
+	}
+	fallback := prompt(Options{Out: &out, In: strings.NewReader("\n")}, "Question", "fb")
+	if fallback != "fb" {
+		t.Fatalf("empty answer must fall back, got %q", fallback)
+	}
+}
+
+func TestSanitizeDatabaseURL(t *testing.T) {
+	cases := map[string]string{
+		"postgres://scimesh:hunter2@localhost:5432/scimesh?sslmode=disable": "postgres://scimesh:***@localhost:5432/scimesh?sslmode=disable",
+		"postgresql://scimesh@localhost/scimesh":                            "postgresql://scimesh@localhost/scimesh",
+		"not-a-url":                                                         "not-a-url",
+	}
+	for raw, want := range cases {
+		got := SanitizeDatabaseURL(raw)
+		if got != want {
+			t.Errorf("sanitize(%q) = %q, want %q", raw, got, want)
+		}
+		if strings.Contains(got, "hunter2") {
+			t.Errorf("sanitize(%q) leaked the password", raw)
+		}
+	}
+}

coordinator/internal/storage/postgres/integration_test.go

@@ -24,6 +24,7 @@ import (
 
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
 	"github.com/emil28092005/SciMesh/coordinator/internal/usecase"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 func testPool(t *testing.T) *pgxpool.Pool {
@@ -408,7 +409,7 @@ func TestCompleteTaskReplayIsIdempotent(t *testing.T) {
 	tasks, jobs, artifacts, tx := NewTaskRepo(pool), NewJobRepo(pool), NewArtifactRepo(pool), NewTxManager(pool)
 	workers, results := NewWorkerRepo(pool), NewTaskResultRepo(pool)
 	clk := fixedClock{now: time.Now().UTC()}
-	uc := usecase.NewCompleteTask(tasks, jobs, artifacts, workers, results, tx, clk, 2)
+	uc := usecase.NewCompleteTask(tasks, jobs, artifacts, workers, results, tx, clk, 2, integrationCatalog())
 
 	claimed, err := tasks.ClaimNext(ctx, usecase.ClaimFilter{
 		Owner: "worker-1", Now: clk.now, LeaseUntil: clk.now.Add(time.Minute),
@@ -657,3 +658,46 @@ func TestExpireLeasesRequeuesElapsedTasks(t *testing.T) {
 		t.Errorf("pending = %d, want 1 — a dead worker must not strand its task", counts[domain.TaskPending])
 	}
 }
+
+func TestMigrateProvisionsAndIsIdempotent(t *testing.T) {
+	ctx := context.Background()
+	url := os.Getenv("TEST_DATABASE_URL")
+	if url == "" {
+		t.Skip("TEST_DATABASE_URL is not set")
+	}
+	if err := Migrate(ctx, url, nil); err != nil {
+		t.Fatalf("first migrate: %v", err)
+	}
+	if err := Migrate(ctx, url, nil); err != nil {
+		t.Fatalf("second migrate (idempotent): %v", err)
+	}
+	pool := testPool(t)
+	migrations, err := listMigrations()
+	if err != nil {
+		t.Fatal(err)
+	}
+	var watermark int64
+	if err := pool.QueryRow(ctx, "SELECT COALESCE(MAX(version), 0) FROM schema_migrations").Scan(&watermark); err != nil {
+		t.Fatalf("read schema_migrations: %v", err)
+	}
+	if watermark != int64(len(migrations)) {
+		t.Errorf("schema watermark = %d, want %d", watermark, len(migrations))
+	}
+	var hasJobs bool
+	if err := pool.QueryRow(ctx,
+		"SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'jobs')",
+	).Scan(&hasJobs); err != nil {
+		t.Fatal(err)
+	}
+	if !hasJobs {
+		t.Error("jobs table was not created by the embedded migrations")
+	}
+}
+
+func integrationCatalog() *workloads.Catalog {
+	catalog, err := workloads.Load()
+	if err != nil {
+		panic(err)
+	}
+	return catalog
+}

coordinator/internal/storage/postgres/migrate.go

@@ -0,0 +1,144 @@
+package postgres
+
+import (
+	"context"
+	"embed"
+	"fmt"
+	"log/slog"
+	"regexp"
+	"sort"
+	"strconv"
+
+	"github.com/jackc/pgx/v5"
+)
+
+//go:embed migrations/*.sql
+var migrationFiles embed.FS
+
+var migrationNamePattern = regexp.MustCompile(`^([0-9]+)_[a-z0-9_]+\.(up|down)\.sql$`)
+
+// migration is one parsed embedded migration file.
+type migration struct {
+	version int
+	name    string
+	sql     string
+}
+
+// listMigrations parses and orders the embedded .up.sql files by version.
+func listMigrations() ([]migration, error) {
+	entries, err := migrationFiles.ReadDir("migrations")
+	if err != nil {
+		return nil, fmt.Errorf("read embedded migrations: %w", err)
+	}
+	up := map[int]migration{}
+	for _, entry := range entries {
+		match := migrationNamePattern.FindStringSubmatch(entry.Name())
+		if match == nil {
+			continue
+		}
+		if match[2] != "up" {
+			continue
+		}
+		version, err := strconv.Atoi(match[1])
+		if err != nil {
+			return nil, fmt.Errorf("migration %q has an invalid version: %w", entry.Name(), err)
+		}
+		if _, duplicate := up[version]; duplicate {
+			return nil, fmt.Errorf("migration version %d is duplicated", version)
+		}
+		body, err := migrationFiles.ReadFile("migrations/" + entry.Name())
+		if err != nil {
+			return nil, fmt.Errorf("read migration %q: %w", entry.Name(), err)
+		}
+		up[version] = migration{version: version, name: entry.Name(), sql: string(body)}
+	}
+	if len(up) == 0 {
+		return nil, fmt.Errorf("no .up.sql migrations are embedded")
+	}
+	versions := make([]int, 0, len(up))
+	for version := range up {
+		versions = append(versions, version)
+	}
+	sort.Ints(versions)
+	migrations := make([]migration, 0, len(versions))
+	for _, version := range versions {
+		migrations = append(migrations, up[version])
+	}
+	for index, item := range migrations {
+		if item.version != index+1 {
+			return nil, fmt.Errorf("embedded migrations are not contiguous: version %d at position %d", item.version, index+1)
+		}
+	}
+	return migrations, nil
+}
+
+// Migrate applies every embedded migration above the recorded schema version,
+// so the binary provisions its own schema. It is idempotent and interoperates
+// with the golang-migrate CLI: both tools use the same schema_migrations
+// watermark table (single row: version + dirty flag), and a PostgreSQL
+// advisory lock serializes concurrent migrators. Each migration file runs as
+// its own transaction (the files carry explicit BEGIN/COMMIT, matching the
+// golang-migrate format the CLI and CI still use).
+func Migrate(ctx context.Context, databaseURL string, log *slog.Logger) error {
+	migrations, err := listMigrations()
+	if err != nil {
+		return err
+	}
+	connConfig, err := pgx.ParseConfig(databaseURL)
+	if err != nil {
+		return fmt.Errorf("parse database url: %w", err)
+	}
+	// Migration files contain multiple statements (BEGIN...COMMIT), which the
+	// extended query protocol rejects; run them with the simple protocol.
+	connConfig.DefaultQueryExecMode = pgx.QueryExecModeSimpleProtocol
+	conn, err := pgx.ConnectConfig(ctx, connConfig)
+	if err != nil {
+		return fmt.Errorf("connect for migration: %w", err)
+	}
+	defer func() { _ = conn.Close(ctx) }()
+
+	if _, err := conn.Exec(ctx, "SELECT pg_advisory_lock(82473911)"); err != nil {
+		return fmt.Errorf("acquire migration lock: %w", err)
+	}
+	defer func() { _, _ = conn.Exec(ctx, "SELECT pg_advisory_unlock(82473911)") }()
+
+	if _, err := conn.Exec(ctx,
+		"CREATE TABLE IF NOT EXISTS schema_migrations (version bigint PRIMARY KEY, dirty boolean NOT NULL DEFAULT false)",
+	); err != nil {
+		return fmt.Errorf("ensure schema_migrations: %w", err)
+	}
+
+	var applied int64
+	if err := conn.QueryRow(ctx,
+		"SELECT COALESCE(MAX(version), 0) FROM schema_migrations",
+	).Scan(&applied); err != nil {
+		return fmt.Errorf("read applied schema version: %w", err)
+	}
+
+	for _, item := range migrations {
+		if int64(item.version) <= applied {
+			continue
+		}
+		if log != nil {
+			log.Info("applying migration", "version", item.version, "file", item.name)
+		}
+		if _, err := conn.Exec(ctx, item.sql); err != nil {
+			return fmt.Errorf("apply migration %s: %w", item.name, err)
+		}
+		// Advance the watermark to the single-row golang-migrate layout.
+		tag, err := conn.Exec(ctx,
+			"UPDATE schema_migrations SET version = $1, dirty = false", item.version)
+		if err != nil {
+			return fmt.Errorf("record migration %s: %w", item.name, err)
+		}
+		if tag.RowsAffected() == 0 {
+			if _, err := conn.Exec(ctx,
+				"INSERT INTO schema_migrations (version, dirty) VALUES ($1, false)",
+				item.version,
+			); err != nil {
+				return fmt.Errorf("record migration %s: %w", item.name, err)
+			}
+		}
+	}
+	return nil
+}

coordinator/internal/storage/postgres/migrate_test.go

@@ -0,0 +1,60 @@
+package postgres
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestListMigrationsParsesAndOrdersEmbeddedFiles(t *testing.T) {
+	migrations, err := listMigrations()
+	if err != nil {
+		t.Fatalf("list migrations: %v", err)
+	}
+	if len(migrations) == 0 {
+		t.Fatal("no embedded migrations")
+	}
+	for index, item := range migrations {
+		if item.version != index+1 {
+			t.Errorf("migration %d has version %d, want contiguous ordering", index, item.version)
+		}
+		if item.name != expectedMigrationName(item.version) {
+			t.Errorf("migration %d file is %q, want %q", item.version, item.name, expectedMigrationName(item.version))
+		}
+		if strings.TrimSpace(item.sql) == "" {
+			t.Errorf("migration %d is empty", item.version)
+		}
+	}
+}
+
+func expectedMigrationName(version int) string {
+	switch version {
+	case 1:
+		return "0001_init.up.sql"
+	case 2:
+		return "0002_workers.up.sql"
+	case 3:
+		return "0003_artifacts.up.sql"
+	case 4:
+		return "0004_result_artifact.up.sql"
+	case 5:
+		return "0005_uploaded_input.up.sql"
+	case 6:
+		return "0006_task_running_enum.up.sql"
+	case 7:
+		return "0007_task_running_lease.up.sql"
+	case 8:
+		return "0008_artifact_attempt.up.sql"
+	case 9:
+		return "0009_unique_partial_result_attempt.up.sql"
+	case 10:
+		return "0010_job_reduction.up.sql"
+	case 11:
+		return "0011_job_owner.up.sql"
+	case 12:
+		return "0012_worker_trust.up.sql"
+	case 13:
+		return "0013_task_results.up.sql"
+	default:
+		return ""
+	}
+}

coordinator/internal/transport/http/server.go

@@ -56,6 +56,8 @@ type Server struct {
 	publicUserserviceURL string
 	// httpClient makes the login/register calls to the userservice.
 	httpClient *http.Client
+	// docsDir serves the built MkDocs site at /ui/docs/. Empty disables it.
+	docsDir string
 	// metrics holds the Prometheus registry and HTTP instrumentation.
 	metrics *metrics.Metrics
 	// ready probes downstream dependencies (the database) for /health. Kept as
@@ -78,6 +80,10 @@ func NewServer(uc UseCases, log *slog.Logger, requestTimeout, heartbeatInterval
 	if len(publicURLs) > 1 {
 		publicUserserviceURL = strings.TrimRight(publicURLs[1], "/")
 	}
+	docsDir := ""
+	if len(publicURLs) > 2 {
+		docsDir = publicURLs[2]
+	}
 	return &Server{
 		uc:                   uc,
 		log:                  log,
@@ -88,6 +94,7 @@ func NewServer(uc UseCases, log *slog.Logger, requestTimeout, heartbeatInterval
 		userserviceURL:       strings.TrimRight(userserviceURL, "/"),
 		publicCoordinatorURL: publicCoordinatorURL,
 		publicUserserviceURL: publicUserserviceURL,
+		docsDir:              docsDir,
 		httpClient:           &http.Client{Timeout: 10 * time.Second},
 		metrics:              m,
 		ready:                ready,
@@ -135,6 +142,9 @@ func (s *Server) Handler(token string, uiToken ...string) http.Handler {
 		}{
 			{"GET /ui", s.handleUIHome},
 			{"GET /ui/jobs/new", s.handleUINewJob},
+			{"GET /ui/workloads", s.handleUIWorkloads},
+			{"GET /ui/docs", s.handleUIDocsIndex},
+			{"GET /ui/docs/{path...}", s.handleUIDocs},
 			{"GET /ui/jobs/{job_id}", s.handleUIJob},
 			{"GET /ui/api/overview", s.handleUIOverviewJSON},
 			{"GET /ui/api/jobs/{job_id}", s.handleUIJobJSON},

coordinator/internal/transport/http/server_test.go

@@ -47,19 +47,19 @@ func newEnvWithUIToken(t *testing.T, ready func(context.Context) error, configur
 	uc := coordhttp.UseCases{
 		RegisterWorker:   usecase.NewRegisterWorker(work, clk),
 		CreateJob:        usecase.NewCreateJob(jobs, tasks, tx, clk),
-		SubmitDataset:    usecase.NewSubmitDataset(blobs, arts, jobs, tasks, tx, clk, 3),
-		ClaimTask:        usecase.NewClaimTask(tasks, jobs, work, tx, clk, lease),
+		SubmitDataset:    usecase.NewSubmitDataset(blobs, arts, jobs, tasks, tx, clk, 3, testCatalog()),
+		ClaimTask:        usecase.NewClaimTask(tasks, jobs, work, tx, clk, lease, testCatalog()),
 		RenewLease:       usecase.NewRenewLease(tasks, work, tx, clk, lease),
-		CompleteTask:     usecase.NewCompleteTask(tasks, jobs, arts, work, memstore.NewTaskResultRepo(), tx, clk, 2),
-		ReduceJob:        usecase.NewReduceJob(jobs, tasks, arts, blobs, tx, clk),
-		FailTask:         usecase.NewFailTask(tasks, jobs, tx, clk),
+		CompleteTask:     usecase.NewCompleteTask(tasks, jobs, arts, work, memstore.NewTaskResultRepo(), tx, clk, 2, testCatalog()),
+		ReduceJob:        usecase.NewReduceJob(jobs, tasks, arts, blobs, tx, clk, testCatalog()),
+		FailTask:         usecase.NewFailTask(tasks, jobs, work, tx, clk, testCatalog()),
 		GetJobStatus:     usecase.NewGetJobStatus(jobs, tasks),
 		GetJobResult:     usecase.NewGetJobResult(jobs, downloadArtifact),
 		CancelJob:        usecase.NewCancelJob(jobs, tasks, tx, clk),
-		UploadArtifact:   usecase.NewUploadArtifact(tasks, arts, blobs, tx, clk),
+		UploadArtifact:   usecase.NewUploadArtifact(tasks, work, arts, blobs, tx, clk),
 		DownloadArtifact: downloadArtifact,
 		GetTaskInput:     usecase.NewGetTaskInput(tasks, arts, blobs),
-		Dashboard:        usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, work, arts)),
+		Dashboard:        usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, work, arts), testCatalog()),
 		PreviewArtifact:  usecase.NewPreviewArtifact(memstore.NewUIReadRepo(jobs, tasks, work, arts), blobs),
 	}
 	worker, err := uc.RegisterWorker.Execute(context.Background(), usecase.RegisterWorkerInput{

coordinator/internal/transport/http/templates/add-worker.html

@@ -33,7 +33,7 @@
         </ol>
         <h2 style="margin-top:24px">Will my results count?</h2>
         <p>Your worker is <strong>untrusted</strong> by default: its results are cross-checked and accepted once a second independent worker computes the same answer (quorum), or once an admin marks your account <strong>verified</strong> — then your workers are trusted and results count immediately.</p>
-        <p><span class="cap">similarity-search</span> is the only workload a volunteer worker runs today.</p>
+        <p><span class="cap">similarity-search</span> and other SDK workloads from the library run on volunteer workers.</p>
       </aside>
     </div>
   </main>

coordinator/internal/transport/http/templates/dashboard.html

@@ -13,7 +13,7 @@
   <main class="page">
     <header class="top">
       <div><p class="eyebrow">Local scientific compute</p><h1 class="title">SciMesh control room</h1><p class="lead">Follow the real path from a molecular TSV to a globally reduced similarity result—without reading coordinator logs.</p><div class="live"><i class="pulse"></i><span id="refresh-state">Live overview · refreshes every 2 seconds</span></div></div>
-      <div style="display:flex;gap:10px;align-items:center;flex-wrap:wrap">{{if .Session}}<span class="live" style="margin-top:0">Signed in · {{.Session.Role}}</span>{{end}}{{if .Session}}<a class="button" href="/ui/profile" style="background:#23344d;color:#dce8ff;box-shadow:none">Profile</a>{{end}}{{if and .Session (eq .Session.Role "admin")}}<a class="button" href="/ui/admin" style="background:#23344d;color:#dce8ff;box-shadow:none">Admin</a>{{end}}{{if .Session}}<a class="button" href="/ui/workers/new" style="background:#23344d;color:#dce8ff;box-shadow:none">🖥 Add your machine</a>{{end}}<a class="button" href="/ui/jobs/new">＋ New similarity search</a>{{if .Session}}<form method="post" action="/ui/logout" style="margin:0"><button class="button" type="submit" style="background:#23344d;color:#dce8ff;box-shadow:none">Log out</button></form>{{end}}</div>
+      <div style="display:flex;gap:10px;align-items:center;flex-wrap:wrap">{{if .Session}}<span class="live" style="margin-top:0">Signed in · {{.Session.Role}}</span>{{end}}{{if .Session}}<a class="button" href="/ui/profile" style="background:#23344d;color:#dce8ff;box-shadow:none">Profile</a>{{end}}{{if and .Session (eq .Session.Role "admin")}}<a class="button" href="/ui/admin" style="background:#23344d;color:#dce8ff;box-shadow:none">Admin</a>{{end}}{{if .Session}}<a class="button" href="/ui/workloads" style="background:#23344d;color:#dce8ff;box-shadow:none">Workloads</a>{{end}}{{if .Session}}<a class="button" href="/ui/docs/" style="background:#23344d;color:#dce8ff;box-shadow:none">Docs</a>{{end}}{{if .Session}}<a class="button" href="/ui/workers/new" style="background:#23344d;color:#dce8ff;box-shadow:none">🖥 Add your machine</a>{{end}}<a class="button" href="/ui/jobs/new">＋ New computation</a>{{if .Session}}<form method="post" action="/ui/logout" style="margin:0"><button class="button" type="submit" style="background:#23344d;color:#dce8ff;box-shadow:none">Log out</button></form>{{end}}</div>
     </header>
     <section class="summary" aria-label="Pipeline summary">
       <div class="panel"><strong>How a search becomes a result</strong><div class="pipeline"><div class="flow-step"><span><i class="dot"></i>01</span><b>Upload TSV</b><small>The coordinator validates and slices the dataset.</small></div><div class="flow-step"><span><i class="dot"></i>02</span><b>Run shards</b><small>Workers fingerprint molecules and return shard top-k CSVs.</small></div><div class="flow-step"><span><i class="dot"></i>03</span><b>Merge exactly</b><small>The coordinator ranks retained candidates deterministically.</small></div><div class="flow-step"><span><i class="dot"></i>04</span><b>Download CSV</b><small>A checksum-protected global result is ready.</small></div></div></div>
@@ -22,7 +22,7 @@
       <div class="metric"><span>Finished runs</span><b id="finished-jobs">{{.FinishedJobs}}</b><small>in the latest 20</small></div>
     </section>
 
-    <section><div class="section-head"><h2>Recent computations</h2><p id="job-count">{{len .Jobs}} shown · newest first</p></div><div id="jobs" class="jobs">{{range .Jobs}}<a class="job" href="/ui/jobs/{{.ID}}"><div><div class="job-name">{{workloadLabel .Workload}}</div><div class="job-id">{{.ID}}</div></div><div><span class="badge badge-{{statusClass .Status}}">{{statusLabel .Status}}</span><div class="job-id">{{statusHint .Status}}</div></div><div><div class="numbers"><b>{{.Completed}}</b> / {{.Total}} shards complete{{if gt .Failed 0}} · <span class="offline">{{.Failed}} failed</span>{{end}}</div><div class="bar"><span style="width:{{progressPercent .Completed .Failed .Cancelled .Total}}%"></span></div></div><span class="arrow" aria-hidden="true">→</span></a>{{else}}<div class="empty"><strong>No computations yet.</strong><br>Start a small similarity search, then keep one or more workers running to watch this dashboard come alive.</div>{{end}}</div></section>
+    <section><div class="section-head"><h2>Recent computations</h2><p id="job-count">{{len .Jobs}} shown · newest first</p></div><div id="jobs" class="jobs">{{range .Jobs}}<a class="job" href="/ui/jobs/{{.ID}}"><div><div class="job-name">{{workloadLabel .Workload}}</div><div class="job-id">{{.ID}}</div></div><div><span class="badge badge-{{statusClass .Status}}">{{statusLabel .Status}}</span><div class="job-id">{{statusHint .Status}}</div></div><div><div class="numbers"><b>{{.Completed}}</b> / {{.Total}} shards complete{{if gt .Failed 0}} · <span class="offline">{{.Failed}} failed</span>{{end}}</div><div class="bar"><span style="width:{{progressPercent .Completed .Failed .Cancelled .Total}}%"></span></div></div><span class="arrow" aria-hidden="true">→</span></a>{{else}}<div class="empty"><strong>No computations yet.</strong><br>Start a computation, then keep one or more workers running to watch this dashboard come alive.</div>{{end}}</div></section>
     {{if and .Session (ne .Session.Role "admin")}}<section><div class="section-head"><h2>My machines</h2><p>Workers you registered. <a href="/ui/workers/new" style="color:#79aaff">Add your machine →</a></p></div><div id="my-workers" class="workers">{{range .MyWorkers}}<article class="worker"><div class="worker-head"><strong>{{.Name}}</strong><span class="badge badge-{{workerStatusClass .Status}}">{{workerStatusLabel .Status}}</span></div><small>{{.ID}}</small><p>{{range .Capabilities}}<span class="cap">{{.}}</span>{{end}}</p><p>Last signal · {{time .LastHeartbeatAt}}</p></article>{{else}}<div class="empty"><strong>No machine of yours is connected.</strong><br><a href="/ui/workers/new" style="color:#79aaff">Turn this computer into a worker →</a></div>{{end}}</div></section>{{end}}
     <section><div class="section-head"><h2>Worker fleet</h2><p>Workers register themselves; this page never controls their processes.</p></div><div id="workers" class="workers">{{range .Workers}}<article class="worker"><div class="worker-head"><strong>{{.Name}}</strong><span class="badge badge-{{workerStatusClass .Status}}">{{workerStatusLabel .Status}}</span></div><small>{{.ID}}</small><p>{{range .Capabilities}}<span class="cap">{{.}}</span>{{end}}</p><p>Last signal · {{time .LastHeartbeatAt}}</p></article>{{else}}<div class="empty"><strong>No worker is registered.</strong><br>Start <code>scimesh-worker</code> in another terminal, then return here.</div>{{end}}</div></section>
   </main>
@@ -30,7 +30,7 @@
     const statusInfo={pending:['Waiting for a worker','waiting'],leased:['Assigned to a worker','active'],running:['Running','active'],reducing:['Merging results','active'],completed:['Completed','success'],failed:['Needs attention','danger'],cancelled:['Stopped','waiting']};
     const pct=j=>j.total?Math.min(100,Math.floor((j.completed+j.failed+j.cancelled)*100/j.total)):0;
     const node=(tag,text,cls)=>{const n=document.createElement(tag);if(text!==undefined)n.textContent=text;if(cls)n.className=cls;return n};
-    const renderJobs=jobs=>{const box=document.querySelector('#jobs');box.replaceChildren();if(!jobs.length){const empty=node('div',undefined,'empty');empty.append(node('strong','No computations yet.'),document.createElement('br'),document.createTextNode('Start a small similarity search, then keep one or more workers running to watch this dashboard come alive.'));box.append(empty);return}for(const job of jobs){const info=statusInfo[job.status]||[job.status,'waiting'],link=node('a',undefined,'job');link.href='/ui/jobs/'+encodeURIComponent(job.id);const intro=node('div');intro.append(node('div',job.workload==='similarity-search'?'Molecule similarity search':job.workload,'job-name'),node('div',job.id,'job-id'));const state=node('div');state.append(node('span',info[0],'badge badge-'+info[1]),node('div',job.status==='reducing'?'Every shard is complete; coordinator is ranking the global top-k.':'Live coordinator state','job-id'));const progress=node('div'),numbers=node('div',undefined,'numbers');numbers.append(node('b',String(job.completed)),document.createTextNode(' / '+job.total+' shards complete'));if(job.failed){numbers.append(document.createTextNode(' · '),node('span',job.failed+' failed','offline'))}const bar=node('div',undefined,'bar'),fill=node('span');fill.style.width=pct(job)+'%';bar.append(fill);progress.append(numbers,bar);link.append(intro,state,progress,node('span','→','arrow'));box.append(link)}};
+    const renderJobs=jobs=>{const box=document.querySelector('#jobs');box.replaceChildren();if(!jobs.length){const empty=node('div',undefined,'empty');empty.append(node('strong','No computations yet.'),document.createElement('br'),document.createTextNode('Start a computation, then keep one or more workers running to watch this dashboard come alive.'));box.append(empty);return}for(const job of jobs){const info=statusInfo[job.status]||[job.status,'waiting'],link=node('a',undefined,'job');link.href='/ui/jobs/'+encodeURIComponent(job.id);const intro=node('div');intro.append(node('div',job.workload==='similarity-search'?'Molecule similarity search':job.workload,'job-name'),node('div',job.id,'job-id'));const state=node('div');state.append(node('span',info[0],'badge badge-'+info[1]),node('div',job.status==='reducing'?'Every shard is complete; coordinator is ranking the global top-k.':'Live coordinator state','job-id'));const progress=node('div'),numbers=node('div',undefined,'numbers');numbers.append(node('b',String(job.completed)),document.createTextNode(' / '+job.total+' shards complete'));if(job.failed){numbers.append(document.createTextNode(' · '),node('span',job.failed+' failed','offline'))}const bar=node('div',undefined,'bar'),fill=node('span');fill.style.width=pct(job)+'%';bar.append(fill);progress.append(numbers,bar);link.append(intro,state,progress,node('span','→','arrow'));box.append(link)}};
     const workerCard=worker=>{const card=node('article',undefined,'worker'),head=node('div',undefined,'worker-head'),left=node('div'),workerInfo=worker.status==='online'?['Available','success']:worker.status==='busy'?['Busy','active']:['Offline','waiting'];left.append(node('strong',worker.name),node('small',worker.id));head.append(left,node('span',workerInfo[0],'badge badge-'+workerInfo[1]));const caps=node('p');for(const capability of worker.capabilities||[])caps.append(node('span',capability,'cap'));card.append(head,caps,node('p','Last signal · '+new Date(worker.last_heartbeat_at).toLocaleString()));return card};
     const renderWorkers=workers=>{const box=document.querySelector('#workers');box.replaceChildren();if(!workers.length){const empty=node('div',undefined,'empty');empty.append(node('strong','No worker is registered.'),document.createElement('br'),document.createTextNode('Start scimesh-worker in another terminal, then return here.'));box.append(empty);return}for(const worker of workers)box.append(workerCard(worker))};
     const renderMyWorkers=workers=>{const box=document.querySelector('#my-workers');if(!box)return;box.replaceChildren();if(!workers.length){const empty=node('div',undefined,'empty'),link=node('a','Turn this computer into a worker →');link.href='/ui/workers/new';link.style.color='#79aaff';empty.append(node('strong','No machine of yours is connected.'),document.createElement('br'),link);box.append(empty);return}for(const worker of workers)box.append(workerCard(worker))};

coordinator/internal/transport/http/templates/docs-unavailable.html

@@ -0,0 +1,24 @@
+{{define "docs-unavailable.html"}}
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Documentation · SciMesh</title>
+  <style>
+    :root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 12% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:760px;margin:auto;padding:80px 22px}a{color:#94bdff}.eyebrow{margin:0 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3rem);letter-spacing:-.055em}.card{margin-top:26px;border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021;padding:22px}.card p{color:#b9c9e2}code{background:#0a1626;border:1px solid #2b4a6b;border-radius:6px;padding:2px 7px;color:#b5d3f5;font-size:.88em}.back{display:inline-block;margin-top:22px;text-decoration:none}
+  </style>
+</head>
+<body>
+  <main class="page">
+    <p class="eyebrow">MkDocs site</p>
+    <h1>Documentation is not available</h1>
+    <div class="card">
+      <p>The documentation site has not been built or the coordinator has not been pointed at it. From the repository root, run:</p>
+      <p><code>make docs</code> &nbsp;then restart the coordinator with <code>SCIMESH_DOCS_DIR</code> set to the generated <code>site/</code> directory (the <code>make demo-ui</code> demo does this automatically).</p>
+      <a class="back" href="/ui">← Back to the control room</a>
+    </div>
+  </main>
+</body>
+</html>
+{{end}}

coordinator/internal/transport/http/templates/job.html

@@ -20,7 +20,7 @@
 
     <section><div class="section-head"><h2>Pipeline stages</h2><p>Each stage reflects coordinator state, not a simulated progress bar.</p></div><div id="stages" class="pipeline"><article class="stage stage-done"><span class="index">1</span><b>TSV accepted</b><p>The coordinator stored the source and created shard tasks.</p></article><article class="stage {{if or (eq .Status "running") (eq .Status "leased") (eq .Status "reducing") (eq .Status "completed")}}stage-active{{else}}stage-waiting{{end}}"><span class="index">2</span><b>Shards execute</b><p id="stage-shards">{{.Completed}} of {{.Total}} candidate partitions are complete.</p></article><article class="stage {{if or (eq .Status "running") (eq .Status "leased")}}stage-active{{else if or (eq .Status "reducing") (eq .Status "completed")}}stage-done{{else}}stage-waiting{{end}}"><span class="index">3</span><b>Workers return CSVs</b><p id="stage-workers">Workers upload a checked partial result for every completed shard.</p></article><article class="stage {{if eq .Status "reducing"}}stage-active{{else if eq .Status "completed"}}stage-done{{else}}stage-waiting{{end}}"><span class="index">4</span><b>Global reduction</b><p id="stage-reducer">The coordinator waits until all shards are complete.</p></article><article class="stage {{if .FinalResultAvailable}}stage-done{{else}}stage-waiting{{end}}"><span class="index">5</span><b>Final CSV</b><p id="stage-final">Available only after deterministic reduction succeeds.</p></article></div></section>
 
-    <section class="two-col"><div><div class="section-head"><h2>Run configuration</h2><p>Allowlisted scientific parameters.</p></div><article class="run-note"><h3>What is being computed?</h3><div id="parameters" class="parameter-list">{{range .Parameters}}<div class="parameter"><span>{{.Label}}</span><code>{{.Value}}</code></div>{{else}}<p>No displayable parameters were supplied.</p>{{end}}</div></article></div><div><div class="section-head"><h2>Result status</h2><p>Safe operator guidance.</p></div><article id="result-card" class="run-note {{if .FinalResultAvailable}}result{{else if eq .Status "failed"}}alert{{end}}">{{if .FinalResultAvailable}}<h3>Final result ready</h3><p>The coordinator merged shard candidates with exact scores and stored the global top-k CSV.</p>{{range .Artifacts}}{{if and (eq .Kind "final_result") .Downloadable}}<a class="download" href="/ui/jobs/{{$.ID}}/artifacts/{{.ID}}/preview">Preview CSV</a><a class="download" href="/ui/jobs/{{$.ID}}/artifacts/{{.ID}}">Download final CSV</a>{{end}}{{end}}{{else if eq .Status "reducing"}}<h3>Merging completed shards</h3><p>The final candidate heap is being ranked now. This page will update when the CSV is stored.</p>{{else if eq .Status "failed"}}<h3>Run needs attention</h3><p>{{if .ErrorMessage}}{{.ErrorMessage}}{{else}}One or more shards could not produce a final result. Review the task table below.{{end}}</p>{{else}}<h3>Waiting for the final result</h3><p>Partial CSVs are diagnostics. They become one global result only after every shard completes and reduction succeeds.</p>{{end}}</article></div></section>
+    <section class="two-col"><div><div class="section-head"><h2>Run configuration</h2><p>Allowlisted scientific parameters.</p></div><article class="run-note"><h3>What is being computed?</h3><div id="parameters" class="parameter-list">{{range .Parameters}}<div class="parameter"><span>{{.Label}}</span><code>{{.Value}}</code></div>{{else}}<p>No displayable parameters were supplied.</p>{{end}}</div></article></div><div><div class="section-head"><h2>Result status</h2><p>Safe operator guidance.</p></div><article id="result-card" class="run-note {{if .FinalResultAvailable}}result{{else if eq .Status "failed"}}alert{{end}}">{{if .FinalResultAvailable}}<h3>Final result ready</h3><p>The coordinator reduced every completed shard into one checksum-protected result file.</p>{{range .Artifacts}}{{if and (eq .Kind "final_result") .Downloadable}}<a class="download" href="/ui/jobs/{{$.ID}}/artifacts/{{.ID}}/preview">Preview CSV</a><a class="download" href="/ui/jobs/{{$.ID}}/artifacts/{{.ID}}">Download final CSV</a>{{end}}{{end}}{{else if eq .Status "reducing"}}<h3>Merging completed shards</h3><p>Every shard is complete; the coordinator is reducing the partial results now. This page will update when the result file is stored.</p>{{else if eq .Status "failed"}}<h3>Run needs attention</h3><p>{{if .ErrorMessage}}{{.ErrorMessage}}{{else}}One or more shards could not produce a final result. Review the task table below.{{end}}</p>{{else}}<h3>Waiting for the final result</h3><p>Partial CSVs are diagnostics. They become one final result only after every shard completes and reduction succeeds.</p>{{end}}</article></div></section>
 
     <section><div class="section-head"><h2>Shard activity</h2><p id="task-caption">Every task is one input partition. The table refreshes while work is in progress.</p></div><div class="table-wrap"><table><thead><tr><th>Shard</th><th>State</th><th>Attempt</th><th>Worker / lease</th><th>Outcome</th></tr></thead><tbody id="tasks">{{range .Tasks}}<tr><td>#{{.ChunkIndex}}</td><td><span class="badge badge-{{statusClass .Status}}">{{statusLabel .Status}}</span></td><td>{{.Attempt}} / {{.MaxAttempts}}</td><td>{{if .LeaseOwner}}<strong>{{.LeaseOwner}}</strong>{{if .LeaseExpiresAt}}<br><small class="muted">lease until {{time .LeaseExpiresAt}}</small>{{end}}{{else}}<span class="muted">—</span>{{end}}</td><td class="error">{{if .ErrorCode}}<strong>{{taskErrorLabel .ErrorCode}}</strong><br><small>{{taskErrorHint .ErrorCode}}</small>{{else if eq .Status "completed"}}<span class="muted">Partial CSV uploaded</span>{{else}}<span class="muted">—</span>{{end}}</td></tr>{{else}}<tr><td colspan="5" class="empty">No shard tasks are present yet.</td></tr>{{end}}</tbody></table></div></section>
 
@@ -31,9 +31,9 @@
     const id={{printf "%q" .ID}},statusInfo={pending:['Waiting for a worker','waiting','A compatible worker has not claimed a shard yet.'],leased:['Assigned to a worker','active','A worker has a shard lease and should begin shortly.'],running:['Running','active','Workers are calculating fingerprints and returning shard-level candidates.'],reducing:['Merging results','active','All shards are complete. The coordinator is ranking the global top-k.'],completed:['Completed','success','The final result is stored and ready to download.'],failed:['Needs attention','danger','The job cannot produce a final result. Review the safe diagnosis below.'],cancelled:['Stopped','waiting','The operator stopped unfinished shards.']},terminal=new Set(['completed','failed','cancelled']);
     const text=(tag,value,cls)=>{const n=document.createElement(tag);if(value!==undefined)n.textContent=value;if(cls)n.className=cls;return n},pct=j=>j.total?Math.min(100,Math.floor((j.completed+j.failed+j.cancelled)*100/j.total)):0,fmtTime=value=>value?new Date(value).toLocaleString():'—',fmtBytes=n=>n<1024?n+' B':n<1024*1024?(n/1024).toFixed(1)+' KiB':(n/(1024*1024)).toFixed(1)+' MiB',taskError={CalledProcessError:['Local calculation failed','The local SciMesh command stopped before it could upload a result. Check the worker terminal for the original error.'],ValueError:['Task input could not be processed','The shard or its parameters did not meet the worker validation rules.'],CoordinatorTransientError:['Coordinator connection was interrupted','The worker will retry when the coordinator is available again.'],CoordinatorConflictError:['Worker lease was no longer valid','Another worker or a lease timeout changed this shard before completion.'],FileNotFoundError:['Local task file is missing','Restart the worker with an absolute --work-dir.']};
     const stage=(index,title,description,state)=>{const card=text('article',undefined,'stage stage-'+state);card.append(text('span',String(index),'index'),text('b',title),text('p',description));return card};
-    const renderStages=job=>{const holder=document.querySelector('#stages'),allDone=job.completed===job.total&&job.total>0,reducerFailed=job.status==='failed'&&job.error_code==='reducer_failed',shardFailed=job.status==='failed'&&!reducerFailed;holder.replaceChildren(stage(1,'TSV accepted','The coordinator stored the source and created shard tasks.','done'),stage(2,'Shards execute',job.completed+' of '+job.total+' candidate partitions are complete.',shardFailed?'failed':allDone?'done':(job.running||job.leased)?'active':'waiting'),stage(3,'Workers return CSVs',allDone?'Every completed shard has a coordinator-owned partial CSV.':(job.running||job.leased)?'Workers are actively claiming and processing partitions.':'Waiting for a worker to claim a shard.',shardFailed?'failed':allDone?'done':(job.running||job.leased)?'active':'waiting'),stage(4,'Global reduction',reducerFailed?'The coordinator could not safely reduce partial results.':job.status==='reducing'?'The coordinator is merging exact candidate scores.':job.status==='completed'?'The global top-k has been merged deterministically.':'Reduction begins only after every shard completes.',reducerFailed?'failed':job.status==='reducing'?'active':job.status==='completed'?'done':'waiting'),stage(5,'Final CSV',job.final_result_available?'The checksum-protected global result is ready.':'Available only after deterministic reduction succeeds.',job.final_result_available?'done':reducerFailed?'failed':'waiting'))};
+    const renderStages=job=>{const holder=document.querySelector('#stages'),allDone=job.completed===job.total&&job.total>0,reducerFailed=job.status==='failed'&&job.error_code==='reducer_failed',shardFailed=job.status==='failed'&&!reducerFailed;holder.replaceChildren(stage(1,'TSV accepted','The coordinator stored the source and created shard tasks.','done'),stage(2,'Shards execute',job.completed+' of '+job.total+' candidate partitions are complete.',shardFailed?'failed':allDone?'done':(job.running||job.leased)?'active':'waiting'),stage(3,'Workers return CSVs',allDone?'Every completed shard has a coordinator-owned partial CSV.':(job.running||job.leased)?'Workers are actively claiming and processing partitions.':'Waiting for a worker to claim a shard.',shardFailed?'failed':allDone?'done':(job.running||job.leased)?'active':'waiting'),stage(4,'Global reduction',reducerFailed?'The coordinator could not safely reduce partial results.':job.status==='reducing'?'The coordinator is merging exact candidate scores.':job.status==='completed'?'The partial results have been reduced deterministically.':'Reduction begins only after every shard completes.',reducerFailed?'failed':job.status==='reducing'?'active':job.status==='completed'?'done':'waiting'),stage(5,'Final CSV',job.final_result_available?'The checksum-protected global result is ready.':'Available only after deterministic reduction succeeds.',job.final_result_available?'done':reducerFailed?'failed':'waiting'))};
     const renderParameters=parameters=>{const holder=document.querySelector('#parameters');holder.replaceChildren();if(!parameters.length){holder.append(text('p','No displayable parameters were supplied.'));return}for(const parameter of parameters){const row=text('div',undefined,'parameter');row.append(text('span',parameter.label),text('code',parameter.value));holder.append(row)}};
-    const renderResult=job=>{const card=document.querySelector('#result-card');card.className='run-note';card.replaceChildren();if(job.final_result_available){card.classList.add('result');card.append(text('h3','Final result ready'),text('p','The coordinator merged shard candidates with exact scores and stored the global top-k CSV.'));const final=(job.artifacts||[]).find(a=>a.kind==='final_result'&&a.downloadable);if(final){const preview=text('a','Preview CSV','download');preview.href='/ui/jobs/'+encodeURIComponent(job.id)+'/artifacts/'+encodeURIComponent(final.id)+'/preview';const download=text('a','Download final CSV','download');download.href='/ui/jobs/'+encodeURIComponent(job.id)+'/artifacts/'+encodeURIComponent(final.id);card.append(preview,download)}}else if(job.status==='reducing'){card.append(text('h3','Merging completed shards'),text('p','The final candidate heap is being ranked now. This page will update when the CSV is stored.'))}else if(job.status==='failed'){card.classList.add('alert');card.append(text('h3','Run needs attention'),text('p',job.error_message||'One or more shards could not produce a final result. Review the task table below.'))}else{card.append(text('h3','Waiting for the final result'),text('p','Partial CSVs are diagnostics. They become one global result only after every shard completes and reduction succeeds.'))}};
+    const renderResult=job=>{const card=document.querySelector('#result-card');card.className='run-note';card.replaceChildren();if(job.final_result_available){card.classList.add('result');card.append(text('h3','Final result ready'),text('p','The coordinator reduced every completed shard into one checksum-protected result file.'));const final=(job.artifacts||[]).find(a=>a.kind==='final_result'&&a.downloadable);if(final){const preview=text('a','Preview CSV','download');preview.href='/ui/jobs/'+encodeURIComponent(job.id)+'/artifacts/'+encodeURIComponent(final.id)+'/preview';const download=text('a','Download final CSV','download');download.href='/ui/jobs/'+encodeURIComponent(job.id)+'/artifacts/'+encodeURIComponent(final.id);card.append(preview,download)}}else if(job.status==='reducing'){card.append(text('h3','Merging completed shards'),text('p','Every shard is complete; the coordinator is reducing the partial results now. This page will update when the result file is stored.'))}else if(job.status==='failed'){card.classList.add('alert');card.append(text('h3','Run needs attention'),text('p',job.error_message||'One or more shards could not produce a final result. Review the task table below.'))}else{card.append(text('h3','Waiting for the final result'),text('p','Partial CSVs are diagnostics. They become one final result only after every shard completes and reduction succeeds.'))}};
     const renderTasks=tasks=>{const holder=document.querySelector('#tasks');holder.replaceChildren();if(!tasks.length){const row=document.createElement('tr'),cell=text('td','No shard tasks are present yet.','empty');cell.colSpan=5;row.append(cell);holder.append(row);return}for(const task of tasks){const row=document.createElement('tr'),info=statusInfo[task.status]||[task.status,'waiting',''];row.append(text('td','#'+task.chunk_index));const state=text('td'),badge=text('span',info[0],'badge badge-'+info[1]);state.append(badge);row.append(state,text('td',task.attempt+' / '+task.max_attempts));const worker=text('td');if(task.lease_owner){worker.append(text('strong',task.lease_owner));if(task.lease_expires_at){worker.append(document.createElement('br'),text('small','lease until '+fmtTime(task.lease_expires_at),'muted'))}}else worker.append(text('span','—','muted'));row.append(worker);const outcome=text('td',undefined,'error');if(task.error_code){const explanation=taskError[task.error_code]||['Task needs attention','Check the worker terminal for the original error.'];outcome.append(text('strong',explanation[0]),document.createElement('br'),text('small',explanation[1]))}else if(task.status==='completed')outcome.append(text('span','Partial CSV uploaded','muted'));else outcome.append(text('span','—','muted'));row.append(outcome);holder.append(row)}};
     const renderArtifacts=job=>{const holder=document.querySelector('#artifacts');holder.replaceChildren();const artifacts=job.artifacts||[];if(!artifacts.length){holder.append(text('div','Artifacts appear here as the coordinator stores input, shards, partial results, and the final CSV.','empty'));return}for(const artifact of artifacts){const card=text('article',undefined,'artifact'+(artifact.kind==='final_result'?' artifact-final':''));card.append(text('div',artifact.diagnostic?'Partial result · diagnostic':artifact.kind,'artifact-type'),text('strong',artifact.filename),text('span',fmtBytes(artifact.size_bytes),'muted'),text('code','SHA-256 '+artifact.sha256));if(artifact.downloadable){const preview=text('a','Preview CSV');preview.href='/ui/jobs/'+encodeURIComponent(job.id)+'/artifacts/'+encodeURIComponent(artifact.id)+'/preview';const download=text('a',artifact.kind==='final_result'?'Download final CSV':'Download CSV');download.href='/ui/jobs/'+encodeURIComponent(job.id)+'/artifacts/'+encodeURIComponent(artifact.id);card.append(preview,download)}holder.append(card)}};
     const speedHistory=[{at:Date.now(),completed:Number({{.Completed}})}],speedWindow=15,speedLimit=90,svgNS='http://www.w3.org/2000/svg',speedSVG=(tag,attrs)=>{const node=document.createElementNS(svgNS,tag);for(const [key,value] of Object.entries(attrs))node.setAttribute(key,String(value));return node},rateLabel=rate=>rate.toFixed(1)+' shards/min';

coordinator/internal/transport/http/templates/new-job.html

@@ -4,20 +4,32 @@
 <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1">
-  <title>New similarity search · SciMesh</title>
+  <title>New computation · SciMesh</title>
   <style>
-    :root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:980px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.back{text-decoration:none}.eyebrow{margin:28px 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3.25rem);letter-spacing:-.055em}.lead{max-width:720px;margin:10px 0 0;color:#aabed9;font-size:1.06rem}.layout{display:grid;grid-template-columns:1.45fr .8fr;gap:15px;margin-top:28px}.card,.aside,.notice{border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021}.card{padding:22px}.aside,.notice{padding:18px}.aside h2,.notice h2{margin:0;color:#f1f6ff;font-size:1rem}.aside p,.notice p{color:#9fb3cf}.aside ol{margin:13px 0 0;padding-left:20px;color:#aebfda}.aside li{margin:10px 0}label{display:block;margin:18px 0 5px;color:#eaf2ff;font-weight:750}input,select{width:100%;border:1px solid #42617f;border-radius:9px;padding:10px 11px;background:#0a1626;color:#e5efff;font:inherit}input:focus,select:focus{outline:2px solid #5d97f5;outline-offset:1px}input[type=file]{padding:8px}.hint{margin:5px 0;color:#92a9c6;font-size:.87rem}.split{display:grid;grid-template-columns:1fr 1fr;gap:12px}.run-preview{margin-top:18px;border:1px solid #2c8c70;border-radius:11px;padding:13px;background:#0c2b2a;color:#a8f1d0}.run-preview strong{color:#e6fff4}.button{display:inline-flex;margin-top:22px;border:0;border-radius:10px;padding:12px 16px;background:#67e3b8;color:#062018;font:inherit;font-weight:850;cursor:pointer}.button:disabled{opacity:.6;cursor:wait}.working{margin:14px 0 0;color:#9fc5ff}.error{margin:12px 0 0;color:#ffacba}.hidden{display:none}.cap{display:inline-block;margin:4px 5px 0 0;border:1px solid #375978;border-radius:5px;padding:2px 6px;color:#b5d3f5;font:.76rem ui-monospace,SFMono-Regular,monospace}@media(max-width:720px){.layout,.split{grid-template-columns:1fr}.page{padding:22px 14px}}
+    :root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 10% -8%,#183f77 0,transparent 32rem),#08111f}.page{max-width:980px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.back{text-decoration:none}.eyebrow{margin:28px 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3.25rem);letter-spacing:-.055em}.lead{max-width:720px;margin:10px 0 0;color:#aabed9;font-size:1.06rem}.layout{display:grid;grid-template-columns:1.45fr .8fr;gap:15px;margin-top:28px}.card,.aside,.notice{border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021}.card{padding:22px}.aside,.notice{padding:18px}.aside h2,.notice h2{margin:0;color:#f1f6ff;font-size:1rem}.aside p,.notice p{color:#9fb3cf}.aside ol{margin:13px 0 0;padding-left:20px;color:#aebfda}.aside li{margin:10px 0}label{display:block;margin:18px 0 5px;color:#eaf2ff;font-weight:750}input,select{width:100%;border:1px solid #42617f;border-radius:9px;padding:10px 11px;background:#0a1626;color:#e5efff;font:inherit}input:focus,select:focus{outline:2px solid #5d97f5;outline-offset:1px}input[type=file]{padding:8px}.checkbox-row{display:flex;align-items:flex-start;gap:10px;margin-top:16px}.checkbox-row input[type=checkbox]{width:18px;height:18px;margin-top:4px;accent-color:#67e3b8}.checkbox-row label{margin:0}.checkbox-row .hint{margin:0}.hint{margin:5px 0;color:#92a9c6;font-size:.87rem}.split{display:grid;grid-template-columns:1fr 1fr;gap:12px}.run-preview{margin-top:18px;border:1px solid #2c8c70;border-radius:11px;padding:13px;background:#0c2b2a;color:#a8f1d0}.run-preview strong{color:#e6fff4}.button{display:inline-flex;margin-top:22px;border:0;border-radius:10px;padding:12px 16px;background:#67e3b8;color:#062018;font:inherit;font-weight:850;cursor:pointer}.button:disabled{opacity:.6;cursor:wait}.working{margin:14px 0 0;color:#9fc5ff}.error{margin:12px 0 0;color:#ffacba}.hidden{display:none}.cap{display:inline-block;margin:4px 5px 0 0;border:1px solid #375978;border-radius:5px;padding:2px 6px;color:#bcd2f0;font-size:.75rem;font-weight:700}.req{color:#ffb4c0}.workload-meta{margin:6px 0 0;color:#8fa7c8;font-size:.9rem}
   </style>
 </head>
 <body>
   <main class="page">
-    <a class="back" href="/ui">← Back to control room</a><p class="eyebrow">New computation</p><h1>Similarity search, end to end</h1><p class="lead">Upload a ChEMBL-style TSV. Workers calculate shard candidates; after every shard succeeds, SciMesh merges the exact global top-k into one final CSV.</p>
-    <div class="layout"><form id="run" class="card" novalidate><label for="file">ChEMBL TSV</label><input id="file" type="file" name="file" required accept=".tsv,.txt,text/tab-separated-values"><p class="hint">Required columns: <code>chembl_id</code> and <code>canonical_smiles</code>.</p><label for="query-smiles">Target molecule (SMILES)</label><input id="query-smiles" name="query_smiles" required maxlength="200" value="CCO" autocomplete="off"><p class="hint">Use a valid SMILES. The coordinator shares this exact query with every shard.</p><div class="split"><div><label for="top-k">Global top-k</label><input id="top-k" name="top_k" type="number" min="1" max="100000" value="20" required><p class="hint">How many final molecules to retain.</p></div><div><label for="chunk-rows">Rows per shard</label><input id="chunk-rows" name="chunk_rows" type="number" min="1" max="100000" value="1000" required><p class="hint">Smaller shards make more visible tasks.</p></div></div><div class="split"><div><label for="threshold">Similarity threshold <small>(optional)</small></label><input id="threshold" name="threshold" type="number" min="0" max="1" step="0.01" placeholder="For example: 0.70"><p class="hint">Leave blank to rank every valid candidate.</p></div><div><label for="direction">Keep molecules</label><select id="direction" name="threshold_direction"><option value="greater">more similar (≥ threshold)</option><option value="less">less similar (≤ threshold)</option></select><p class="hint">“Less” helps explore dissimilar molecules.</p></div></div><label for="max-rows">Maximum dataset rows <small>(optional quick run)</small></label><input id="max-rows" name="max_rows" type="number" min="1" max="10000000" placeholder="For example: 500"><p class="hint">Only the first N data rows become shards; the original upload remains stored by the coordinator.</p><div id="preview" class="run-preview"><strong>Ready to plan a run.</strong><br>Select a TSV to see the file that will be sent to the coordinator.</div><button class="button" id="submit" type="submit">Create pipeline run →</button><p id="working" class="working hidden" aria-live="polite">Uploading TSV and creating coordinator-owned shards…</p><p id="error" class="error" role="alert"></p></form><aside class="aside"><h2>What you will observe</h2><ol><li><strong>Input accepted</strong><br>TSV is validated and split into durable shard artifacts.</li><li><strong>Workers claim tasks</strong><br>Each worker downloads one shard, fingerprints it, and uploads a partial CSV.</li><li><strong>Global reduction</strong><br>The coordinator compares exact scores from all partial results.</li><li><strong>Final download</strong><br>The result page exposes a checksum-protected global CSV.</li></ol><h2 style="margin-top:26px">Before you submit</h2><p>Keep at least one worker running in another terminal. The browser cannot start or control worker processes.</p><p><span class="cap">similarity-search</span> is currently the only distributed workload available here.</p></aside></div>
+    <a class="back" href="/ui">← Back to control room</a><p class="eyebrow">New computation</p><h1>Any workload, end to end</h1><p class="lead">Choose a workload from the installed library. Upload its dataset; workers compute shards; the coordinator reduces the partials into one final artifact.</p>
+    <div class="layout"><form id="run" class="card" novalidate><label for="workload">Workload</label><select id="workload" name="workload"></select><p id="workload-meta" class="workload-meta hidden"></p><div id="params"></div><div class="split"><div><label for="chunk-rows">Rows per shard</label><input id="chunk-rows" name="chunk_rows" type="number" min="1" max="100000" value="1000" required><p class="hint">Smaller shards make more visible tasks.</p></div><div><label for="max-rows">Maximum dataset rows <small>(optional)</small></label><input id="max-rows" name="max_rows" type="number" min="1" max="10000000" placeholder="For example: 500"><p class="hint">Only the first N data rows become shards; the upload stays stored.</p></div></div><label for="file">Dataset file</label><input id="file" type="file" name="file" required accept=".tsv,.txt,.csv,text/tab-separated-values,text/csv"><p class="hint">A delimited table with a header row. The workload defines the required columns.</p><div id="preview" class="run-preview"><strong>Ready to plan a run.</strong><br>Select a dataset to see the file that will be sent to the coordinator.</div><button class="button" id="submit" type="submit">Create pipeline run →</button><p id="working" class="working hidden" aria-live="polite">Uploading dataset and creating coordinator-owned shards…</p><p id="error" class="error" role="alert"></p></form><aside class="aside"><h2>What you will observe</h2><ol><li><strong>Input accepted</strong><br>Dataset is validated and split into durable shard artifacts.</li><li><strong>Workers claim tasks</strong><br>Each worker downloads one shard, computes, and uploads a partial result.</li><li><strong>Global reduction</strong><br>The coordinator reduces all partials into one final artifact.</li><li><strong>Final download</strong><br>The result page exposes a checksum-protected result file.</li></ol><h2 style="margin-top:26px">Before you submit</h2><p>Keep at least one worker running in another terminal. The browser cannot start or control worker processes.</p><p>The form controls come from the workload's own declarations in the SDK library.</p></aside></div>
   </main>
   <script>
-    const form=document.querySelector('#run'),button=document.querySelector('#submit'),working=document.querySelector('#working'),error=document.querySelector('#error'),preview=document.querySelector('#preview'),fileInput=document.querySelector('#file');
-    const bytes=size=>size<1024?size+' B':size<1024*1024?(size/1024).toFixed(1)+' KiB':(size/(1024*1024)).toFixed(1)+' MiB';fileInput.addEventListener('change',()=>{const file=fileInput.files&&fileInput.files[0];preview.replaceChildren();if(!file){preview.append(document.createTextNode('Select a TSV to see the file that will be sent to the coordinator.'));return}const strong=document.createElement('strong');strong.textContent='Source ready: '+file.name;preview.append(strong,document.createElement('br'),document.createTextNode(bytes(file.size)+' · the coordinator will validate its header before creating tasks.'))});
-    form.addEventListener('submit',async event=>{event.preventDefault();error.textContent='';const fields=new FormData(form),file=fields.get('file'),query=String(fields.get('query_smiles')||'').trim(),topK=Number(fields.get('top_k')),chunkRows=Number(fields.get('chunk_rows')),threshold=String(fields.get('threshold')||'').trim(),maxRows=String(fields.get('max_rows')||'').trim();if(!(file instanceof File)||file.size===0){error.textContent='Choose a non-empty TSV file.';return}if(!query||query.length>200||!Number.isInteger(topK)||topK<1||!Number.isInteger(chunkRows)||chunkRows<1){error.textContent='Enter a target SMILES, a positive global top-k, and a positive rows-per-shard value.';return}if(threshold&&(Number.isNaN(Number(threshold))||Number(threshold)<0||Number(threshold)>1)){error.textContent='Similarity threshold must be between 0 and 1.';return}const parameters={query_smiles:query,top_k:topK,threshold_direction:fields.get('threshold_direction'),progress_every:0};if(threshold)parameters.threshold=Number(threshold);const upload=new FormData();upload.append('workload','similarity-search');upload.append('parameters',JSON.stringify(parameters));upload.append('chunk_rows',String(chunkRows));if(maxRows)upload.append('max_rows',maxRows);upload.append('file',file,file.name);button.disabled=true;working.classList.remove('hidden');try{const response=await fetch('/ui/api/jobs/upload',{method:'POST',body:upload}),data=await response.json();if(!response.ok)throw Error(data.error||'Unable to create the job.');location.href='/ui/jobs/'+encodeURIComponent(data.job_id)}catch(err){error.textContent=err.message==='invalid input'?'The coordinator could not accept this run. Check the TSV columns and form values.':err.message;button.disabled=false;working.classList.add('hidden')}});
+    const DATA={{.Payload}};
+    const form=document.querySelector('#run'),button=document.querySelector('#submit'),working=document.querySelector('#working'),error=document.querySelector('#error'),preview=document.querySelector('#preview'),fileInput=document.querySelector('#file'),workloadSelect=document.querySelector('#workload'),workloadMeta=document.querySelector('#workload-meta'),paramsBox=document.querySelector('#params');
+    const bytes=size=>size<1024?size+' B':size<1024*1024?(size/1024).toFixed(1)+' KiB':(size/(1024*1024)).toFixed(1)+' MiB';
+    const uploadable=DATA.workloads.filter(w=>w.upload_ready);
+    for(const w of uploadable){const option=document.createElement('option');option.value=w.name;option.textContent=w.name;workloadSelect.append(option)}
+    const schemaOf=w=>{const properties=w.schema.properties||{};return name=>properties[name]||{}};
+    const inputType=prop=>prop.type==='number'||prop.type==='integer'?'number':'text';
+    const makeField=(w,element)=>{const prop=schemaOf(w)(element.field),box=document.createElement('div');const label=document.createElement('label');label.textContent=element.label||element.field;if(element.required||(w.one_of||[]).some(group=>group.includes(element.field))){const star=document.createElement('span');star.className='req';star.textContent=' *';label.append(star)}if(element.widget==='checkbox'){const row=document.createElement('div');row.className='checkbox-row';const input=document.createElement('input');input.type='checkbox';input.id='param-'+element.field;input.checked=element.default===true||(element.default==null&&w.defaults[element.field]===true);input.name=element.field;row.append(input,label);if(element.help){const hint=document.createElement('p');hint.className='hint';hint.textContent=element.help;row.append(hint)}box.append(row);return box}let input;if(element.widget==='select'){input=document.createElement('select');input.name=element.field;input.id='param-'+element.field;for(const option of element.options){const node=document.createElement('option');node.value=option;node.textContent=option;input.append(node)}const defaultValue=element.default!=null?element.default:w.defaults[element.field];if(defaultValue!=null)input.value=String(defaultValue)}else if(element.widget==='textarea'){input=document.createElement('textarea');input.name=element.field;input.id='param-'+element.field;input.rows=2;if(element.default!=null)input.value=String(element.default)}else{input=document.createElement('input');input.type=inputType(prop);input.name=element.field;input.id='param-'+element.field;input.autocomplete='off';if(prop.minLength!=null)input.maxLength=prop.maxLength;if(prop.type==='number'||prop.type==='integer'){if(prop.minimum!=null)input.min=prop.minimum;if(prop.maximum!=null)input.max=prop.maximum;input.step=prop.type==='integer'?1:'any'}if(element.placeholder)input.placeholder=element.placeholder;const defaultValue=element.default!=null?element.default:w.defaults[element.field];if(defaultValue!=null)input.value=String(defaultValue)}label.htmlFor=input.id;box.append(label,input);if(element.help){const hint=document.createElement('p');hint.className='hint';hint.textContent=element.help;box.append(hint)}return box};
+    const fieldsFor=w=>{const elements=[...(w.ui||[])];const declared=new Set(elements.map(e=>e.field));const properties=w.schema.properties||{};const fallback=Object.entries(properties).filter(([name])=>!declared.has(name)&&name!=='max_rows').map(([name,prop])=>({field:name,widget:prop.enum?'select':prop.type==='boolean'?'checkbox':prop.type==='string'?'text':(prop.type==='number'||prop.type==='integer')?'number':'text',label:name,help:prop.description||'',options:prop.enum||[],default:prop.default!=null?prop.default:null,placeholder:'',order:100,required:(w.required||[]).includes(name)}));return [...elements,...fallback].sort((a,b)=>a.order-b.order)};
+    const oneOfHint=w=>{const group=(w.one_of||[]).find(g=>g.length>1);if(!group)return null;return 'Exactly one required: '+group.map(f=>{const element=(w.ui||[]).find(e=>e.field===f);return element&&element.label?element.label:f}).join(' or ')+'.'};
+    const render=()=>{paramsBox.replaceChildren();const w=DATA.workloads.find(x=>x.name===workloadSelect.value);if(!w)return;workloadMeta.classList.remove('hidden');workloadMeta.textContent=w.description;const hint=oneOfHint(w);if(hint){const p=document.createElement('p');p.className='hint';p.style.marginTop='18px';p.textContent=hint;paramsBox.append(p)}for(const element of fieldsFor(w)){paramsBox.append(makeField(w,element))}};
+    workloadSelect.addEventListener('change',render);render();
+    fileInput.addEventListener('change',()=>{const file=fileInput.files&&fileInput.files[0];preview.replaceChildren();if(!file){preview.append(document.createTextNode('Select a dataset to see the file that will be sent to the coordinator.'));return}const strong=document.createElement('strong');strong.textContent='Source ready: '+file.name;preview.append(strong,document.createElement('br'),document.createTextNode(bytes(file.size)+' · the coordinator will validate the dataset before creating tasks.'))});
+    const values=()=>{const w=DATA.workloads.find(x=>x.name===workloadSelect.value),props=w.schema.properties||{},out={};for(const element of fieldsFor(w)){const prop=props[element.field]||{},node=document.querySelector('#param-'+CSS.escape(element.field));if(!node)continue;if(element.widget==='checkbox'){out[element.field]=node.checked;continue}const raw=String(node.value||'').trim();if(element.widget==='select'){out[element.field]=raw;continue}if(!raw){if(element.required)throw new Error((element.label||element.field)+' is required.');continue}if(prop.type==='integer'){if(!/^-?\d+$/.test(raw))throw new Error((element.label||element.field)+' must be an integer.');out[element.field]=parseInt(raw,10)}else if(prop.type==='number'){const number=Number(raw);if(Number.isNaN(number))throw new Error((element.label||element.field)+' must be a number.');out[element.field]=number}else{out[element.field]=raw}}for(const group of w.one_of||[]){if(group.length<2)continue;const filled=group.filter(field=>out[field]!==undefined&&out[field]!==null&&out[field]!==false&&String(out[field]).trim()!=='');if(filled.length!==1)throw new Error('Exactly one of '+group.join(', ')+' must be provided.')}return out};
+    form.addEventListener('submit',async event=>{event.preventDefault();error.textContent='';const w=DATA.workloads.find(x=>x.name===workloadSelect.value);if(!w){error.textContent='Choose a workload.';return}const file=fileInput.files&&fileInput.files[0];const chunkRows=Number(document.querySelector('#chunk-rows').value),maxRows=String(document.querySelector('#max-rows').value||'').trim();if(!(file instanceof File)||file.size===0){error.textContent='Choose a non-empty dataset file.';return}if(!Number.isInteger(chunkRows)||chunkRows<1){error.textContent='Rows per shard must be a positive integer.';return}if(maxRows&&(!Number.isInteger(Number(maxRows))||Number(maxRows)<1)){error.textContent='Maximum dataset rows must be a positive integer.';return}let parameters;try{parameters=values()}catch(err){error.textContent=err.message;return}const upload=new FormData();upload.append('workload',w.name);upload.append('parameters',JSON.stringify(parameters));upload.append('chunk_rows',String(chunkRows));if(maxRows)upload.append('max_rows',maxRows);upload.append('file',file,file.name);button.disabled=true;working.classList.remove('hidden');try{const response=await fetch('/ui/api/jobs/upload',{method:'POST',body:upload}),data=await response.json();if(!response.ok)throw Error(data.error||'Unable to create the job.');location.href='/ui/jobs/'+encodeURIComponent(data.job_id)}catch(err){error.textContent=err.message==='invalid input'?'The coordinator could not accept this run. Check the dataset and form values.':err.message;button.disabled=false;working.classList.add('hidden')}});
   </script>
 </body>
 </html>

coordinator/internal/transport/http/templates/workloads.html

@@ -0,0 +1,55 @@
+{{define "workloads.html"}}
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Workload library · SciMesh</title>
+  <style>
+    :root{color:#e5efff;background:#08111f;font:16px/1.5 Inter,ui-sans-serif,system-ui,sans-serif;color-scheme:dark}*{box-sizing:border-box}body{margin:0;background:radial-gradient(circle at 12% -8%,#183f77 0,transparent 32rem),radial-gradient(circle at 95% 5%,#123e39 0,transparent 29rem),#08111f}.page{max-width:1240px;margin:auto;padding:28px 22px 64px}a{color:#94bdff}.back{text-decoration:none}.eyebrow{margin:28px 0 4px;color:#7baaff;font-size:.78rem;font-weight:800;letter-spacing:.14em;text-transform:uppercase}h1{margin:0;color:#f4f8ff;font-size:clamp(2rem,5vw,3.25rem);letter-spacing:-.055em}.lead{max-width:760px;margin:10px 0 0;color:#aabed9;font-size:1.06rem}.library{display:grid;gap:14px;margin-top:30px}.workload{border:1px solid #294662;border-radius:15px;background:#0d1a2cdd;box-shadow:0 20px 45px #00000021;padding:20px 22px}.workload-head{display:flex;align-items:baseline;justify-content:space-between;gap:14px;flex-wrap:wrap}.workload-head h2{margin:0;color:#f2f7ff;font-size:1.22rem;letter-spacing:-.02em}.version{margin:0;color:#7d93b2;font:0.82rem ui-monospace,SFMono-Regular,monospace}.description{margin:8px 0 0;color:#b9c9e2;max-width:860px}.cap{display:inline-block;margin:12px 5px 0 0;border:1px solid #375978;border-radius:5px;padding:2px 7px;color:#b5d3f5;font:.76rem ui-monospace,SFMono-Regular,monospace}.badge{display:inline-flex;align-items:center;border-radius:999px;padding:4px 10px;font-size:.78rem;font-weight:800}.badge-success{background:#123f34;color:#76efb5}.badge-waiting{background:#23344d;color:#b9cce9}.meta{display:flex;gap:9px;flex-wrap:wrap;margin-top:14px}.meta span{border:1px solid #2b4a6b;border-radius:7px;padding:3px 8px;color:#a9c3e2;font-size:.8rem}.meta b{color:#dbe9fb;font-weight:750}.schema-grid{display:grid;grid-template-columns:1fr 1fr;gap:13px;margin-top:16px}.schema{border:1px solid #233e5c;border-radius:11px;background:#091627;padding:13px}.schema h3{margin:0 0 8px;color:#cfe1f7;font-size:.86rem;letter-spacing:.04em;text-transform:uppercase}.schema pre{margin:0;overflow:auto;max-height:300px;color:#9fc1e8;font:.76rem ui-monospace,SFMono-Regular,monospace;white-space:pre-wrap;word-break:break-word}.params{padding:13px}.params h3{margin:0 0 8px;color:#cfe1f7;font-size:.86rem;letter-spacing:.04em;text-transform:uppercase}.param{margin:0;padding:6px 0;border-bottom:1px dashed #223a56;color:#b9c9e2;font-size:.9rem}.param:last-child{border-bottom:0}.param b{color:#e8f2ff}.param small{display:block;margin-top:2px;color:#7f96b5}.empty{padding:30px;border:1px dashed #35516f;border-radius:14px;color:#9ab0cb;text-align:center}@media(max-width:820px){.schema-grid{grid-template-columns:1fr}}
+  </style>
+</head>
+<body>
+  <main class="page">
+    <p class="eyebrow">Installed packages</p>
+    <h1>Workload library</h1>
+    <p class="lead">Every SDK-built workload installed in this SciMesh deployment, as declared by <code>scimesh workload export</code>. The coordinator stores this catalog as presentation metadata only; it never executes workload code.</p>
+    <div class="library">
+      {{range .Workloads}}
+      <article class="workload">
+        <div class="workload-head">
+          <h2>{{.Name}}</h2>
+          <p class="version">{{.Version}}</p>
+          {{if .Enabled}}<span class="badge badge-success">enabled</span>{{else}}<span class="badge badge-waiting">disabled</span>{{end}}
+        </div>
+        <p class="description">{{.Description}}</p>
+        <div>{{range .Capabilities}}<span class="cap">{{.}}</span>{{end}}</div>
+        <div class="meta">
+          <span>determinism <b>{{.Determinism}}</b></span>
+          <span>verifier <b>{{.Verifier}}</b></span>
+          <span>trust <b>{{range $i, $mode := .TrustModes}}{{if $i}}, {{end}}{{$mode}}{{end}}</b></span>
+        </div>
+        <div class="schema-grid">
+          <div class="schema params">
+            <h3>Parameters</h3>
+            <pre>{{.Parameters}}</pre>
+          </div>
+          <div>
+            {{range .Inputs}}
+            <div class="schema"><h3>Input · {{.Name}}</h3><pre>{{.Schema}}</pre></div>
+            {{end}}
+            {{range .Outputs}}
+            <div class="schema"><h3>Output · {{.Name}}</h3><pre>{{.Schema}}</pre></div>
+            {{end}}
+          </div>
+        </div>
+      </article>
+      {{else}}
+      <div class="empty"><strong>No workloads are installed.</strong><br>Install an SDK workload package and run <code>scimesh workload export</code> to republish this catalog.</div>
+      {{end}}
+    </div>
+    <p class="lead" style="margin-top:26px"><a class="back" href="/ui">← Back to the control room</a></p>
+  </main>
+</body>
+</html>
+{{end}}

coordinator/internal/transport/http/testcatalog_test.go

@@ -0,0 +1,15 @@
+package http_test
+
+import (
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
+)
+
+// testCatalog loads the embedded workload catalog for http tests, the same
+// catalog the server binary loads at startup.
+func testCatalog() *workloads.Catalog {
+	catalog, err := workloads.Load()
+	if err != nil {
+		panic(err)
+	}
+	return catalog
+}

coordinator/internal/transport/http/ui.go

@@ -2,17 +2,20 @@ package http
 
 import (
 	"embed"
+	"encoding/json"
 	"fmt"
 	"html/template"
 	"io"
 	"mime"
 	"net/http"
+	"sort"
 	"strconv"
 	"time"
 
 	"github.com/google/uuid"
 
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 //go:embed templates/*.html
@@ -165,6 +168,10 @@ func uiWorkloadLabel(workload string) string {
 		return "Molecule similarity search"
 	case "similarity-graph", "similarity_graph":
 		return "Molecular similarity graph"
+	case "molwt-filter", "molwt_filter":
+		return "Molecular weight filter"
+	case "descriptor-batch", "descriptor_batch":
+		return "Descriptor batch"
 	default:
 		return workload
 	}
@@ -236,7 +243,144 @@ func (s *Server) handleUIOverviewJSON(w http.ResponseWriter, r *http.Request) {
 }
 
 func (s *Server) handleUINewJob(w http.ResponseWriter, r *http.Request) {
-	s.renderUI(w, "new-job.html", nil)
+	catalog, err := workloads.Load()
+	if err != nil {
+		s.log.Error("load workload catalog", "err", err)
+		http.Error(w, "internal error", http.StatusInternalServerError)
+		return
+	}
+	s.renderUI(w, "new-job.html", uiNewJobView{Payload: newJobPayload(catalog)})
+}
+
+// uiNewJobWorkloadJSON is the per-workload data handed to the page script. It
+// is presentation metadata from the embedded catalog; the coordinator re-
+// validates everything server-side on upload.
+type uiNewJobWorkloadJSON struct {
+	Name        string                `json:"name"`
+	Description string                `json:"description"`
+	Schema      map[string]any        `json:"schema"`
+	UI          []uiNewJobElementJSON `json:"ui"`
+	Required    []string              `json:"required"`
+	OneOf       [][]string            `json:"one_of"`
+	UploadReady bool                  `json:"upload_ready"`
+	Reduction   string                `json:"reduction"`
+	Defaults    map[string]any        `json:"defaults"`
+	InputMedia  map[string]string     `json:"input_media"`
+}
+
+type uiNewJobElementJSON struct {
+	Field       string   `json:"field"`
+	Widget      string   `json:"widget"`
+	Label       string   `json:"label"`
+	Help        string   `json:"help"`
+	Placeholder string   `json:"placeholder"`
+	Options     []string `json:"options"`
+	Default     any      `json:"default"`
+	Order       int      `json:"order"`
+	Required    bool     `json:"required"`
+}
+
+type uiNewJobView struct {
+	Payload template.JS
+}
+
+func newJobPayload(catalog *workloads.Catalog) template.JS {
+	payload := struct {
+		Workloads []uiNewJobWorkloadJSON `json:"workloads"`
+	}{Workloads: make([]uiNewJobWorkloadJSON, 0, len(catalog.Enabled()))}
+	for _, workload := range catalog.Enabled() {
+		required := map[string]bool{}
+		if entries, ok := workload.Parameters["required"].([]any); ok {
+			for _, entry := range entries {
+				if name, ok := entry.(string); ok {
+					required[name] = true
+				}
+			}
+		}
+		elementViews := make([]uiNewJobElementJSON, 0, len(workload.UIElements))
+		for _, element := range workload.UIElements {
+			elementViews = append(elementViews, uiNewJobElementJSON{
+				Field:       element.Field,
+				Widget:      element.Widget,
+				Label:       element.Label,
+				Help:        element.Help,
+				Placeholder: element.Placeholder,
+				Options:     element.Options,
+				Default:     element.Default,
+				Order:       element.Order,
+				Required:    required[element.Field],
+			})
+		}
+		payload.Workloads = append(payload.Workloads, uiNewJobWorkloadJSON{
+			Name:        workload.Name,
+			Description: workload.Description,
+			Schema:      workload.Parameters,
+			UI:          elementViews,
+			Required:    sortedKeys(required),
+			OneOf:       oneOfGroups(workload.Parameters),
+			UploadReady: workload.UploadReady,
+			Reduction:   workload.Reduction,
+			Defaults:    catalog.ParameterDefaults(workload.Name),
+			InputMedia:  inputMediaViews(catalog, workload.Name),
+		})
+	}
+	encoded, err := json.Marshal(payload)
+	if err != nil {
+		return template.JS("null")
+	}
+	// #nosec G203 -- the payload is marshaled JSON from the embedded workload catalog, injected as script data.
+	return template.JS(encoded)
+}
+
+func sortedKeys(values map[string]bool) []string {
+	keys := make([]string, 0, len(values))
+	for key := range values {
+		keys = append(keys, key)
+	}
+	sort.Strings(keys)
+	return keys
+}
+
+func oneOfGroups(schema map[string]any) [][]string {
+	rawOneOf, ok := schema["oneOf"].([]any)
+	if !ok {
+		return nil
+	}
+	// The clean shape is "exactly one of these single fields" (e.g. the search
+	// query id vs SMILES choice): every branch requires exactly one distinct
+	// field. Anything more complex is left to server-side validation.
+	var fields []string
+	seen := map[string]bool{}
+	for _, rawOption := range rawOneOf {
+		option, ok := rawOption.(map[string]any)
+		if !ok {
+			return nil
+		}
+		required, _ := option["required"].([]any)
+		if len(required) != 1 {
+			return nil
+		}
+		field, ok := required[0].(string)
+		if !ok || seen[field] {
+			return nil
+		}
+		seen[field] = true
+		fields = append(fields, field)
+	}
+	if len(fields) != len(rawOneOf) {
+		return nil
+	}
+	return [][]string{fields}
+}
+
+func inputMediaViews(catalog *workloads.Catalog, name string) map[string]string {
+	views := map[string]string{}
+	for _, port := range catalog.InputPortNames(name) {
+		if mediaType := catalog.InputMediaType(name, port); mediaType != "" {
+			views[port] = mediaType
+		}
+	}
+	return views
 }
 
 func (s *Server) uiJobID(w http.ResponseWriter, r *http.Request) (uuid.UUID, bool) {

coordinator/internal/transport/http/ui_docs.go

@@ -0,0 +1,53 @@
+package http
+
+import (
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// handleUIDocsIndex redirects /ui/docs to the trailing-slash form so the
+// wildcard route below can resolve index.html.
+func (s *Server) handleUIDocsIndex(w http.ResponseWriter, r *http.Request) {
+	http.Redirect(w, r, "/ui/docs/", http.StatusPermanentRedirect)
+}
+
+// handleUIDocs serves the built MkDocs site (site/) as static files. The
+// configured docs directory is an operator-supplied path, never derived from
+// a request; path traversal is rejected by joining against the cleaned root
+// and checking the result stays inside it.
+func (s *Server) handleUIDocs(w http.ResponseWriter, r *http.Request) {
+	if s.docsDir == "" {
+		s.renderUI(w, "docs-unavailable.html", nil)
+		return
+	}
+	root, err := filepath.Abs(s.docsDir)
+	if err != nil {
+		s.renderUI(w, "docs-unavailable.html", nil)
+		return
+	}
+	clean := filepath.Clean(strings.TrimPrefix(r.URL.Path, "/ui/docs/"))
+	target, err := filepath.Abs(filepath.Join(root, clean))
+	if err != nil {
+		s.renderUI(w, "docs-unavailable.html", nil)
+		return
+	}
+	rel, err := filepath.Rel(root, target)
+	if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(os.PathSeparator)) {
+		http.NotFound(w, r)
+		return
+	}
+	info, err := os.Stat(target)
+	if err != nil || info.IsDir() {
+		if err == nil && info.IsDir() {
+			target = filepath.Join(target, "index.html")
+			info, err = os.Stat(target)
+		}
+		if err != nil || info.IsDir() {
+			s.renderUI(w, "docs-unavailable.html", nil)
+			return
+		}
+	}
+	http.ServeFile(w, r, target)
+}

coordinator/internal/transport/http/ui_docs_internal_test.go

@@ -0,0 +1,90 @@
+package http
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func docsTestServer(t *testing.T, docsDir string) *Server {
+	t.Helper()
+	return &Server{
+		log:     slog.New(slog.NewTextHandler(io.Discard, nil)),
+		docsDir: docsDir,
+	}
+}
+
+func TestUIDocsServesIndexAndNestedFiles(t *testing.T) {
+	root := t.TempDir()
+	if err := os.WriteFile(filepath.Join(root, "index.html"), []byte("<h1>Home</h1>"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	sub := filepath.Join(root, "api")
+	if err := os.Mkdir(sub, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(sub, "page.html"), []byte("<h1>API page</h1>"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	server := docsTestServer(t, root)
+
+	index := httptest.NewRecorder()
+	server.handleUIDocs(index, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/ui/docs/", nil))
+	if index.Code != http.StatusOK || !strings.Contains(index.Body.String(), "<h1>Home</h1>") {
+		t.Fatalf("index = %d %q", index.Code, index.Body.String())
+	}
+
+	page := httptest.NewRecorder()
+	server.handleUIDocs(page, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/ui/docs/api/page.html", nil))
+	if page.Code != http.StatusOK || !strings.Contains(page.Body.String(), "<h1>API page</h1>") {
+		t.Fatalf("nested page = %d %q", page.Code, page.Body.String())
+	}
+}
+
+func TestUIDocsRejectsPathTraversal(t *testing.T) {
+	root := t.TempDir()
+	secret := filepath.Join(root, "secret.txt")
+	if err := os.WriteFile(secret, []byte("private"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	server := docsTestServer(t, root)
+
+	request := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/ui/docs/../secret.txt", nil)
+	request.URL.Path = "/ui/docs/../secret.txt"
+	recorder := httptest.NewRecorder()
+	server.handleUIDocs(recorder, request)
+	if recorder.Code != http.StatusNotFound {
+		t.Fatalf("traversal status = %d, want 404", recorder.Code)
+	}
+}
+
+func TestUIDocsShowsBuildHintWhenDisabledOrMissing(t *testing.T) {
+	disabled := docsTestServer(t, "")
+	recorder := httptest.NewRecorder()
+	disabled.handleUIDocs(recorder, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/ui/docs/", nil))
+	if recorder.Code != http.StatusOK || !strings.Contains(recorder.Body.String(), "Documentation is not available") {
+		t.Fatalf("disabled docs = %d %q", recorder.Code, recorder.Body.String())
+	}
+
+	missing := docsTestServer(t, filepath.Join(t.TempDir(), "does-not-exist"))
+	recorder = httptest.NewRecorder()
+	missing.handleUIDocs(recorder, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/ui/docs/", nil))
+	if recorder.Code != http.StatusOK || !strings.Contains(recorder.Body.String(), "Documentation is not available") {
+		t.Fatalf("missing docs = %d %q", recorder.Code, recorder.Body.String())
+	}
+}
+
+func TestUIDocsIndexRedirectsToTrailingSlash(t *testing.T) {
+	server := docsTestServer(t, t.TempDir())
+	recorder := httptest.NewRecorder()
+	server.handleUIDocsIndex(recorder, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/ui/docs", nil))
+	if recorder.Code != http.StatusPermanentRedirect || recorder.Header().Get("Location") != "/ui/docs/" {
+		t.Fatalf("redirect = %d %q", recorder.Code, recorder.Header().Get("Location"))
+	}
+}

coordinator/internal/transport/http/ui_newjob_internal_test.go

@@ -0,0 +1,44 @@
+package http
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
+)
+
+func TestNewJobPageCarriesWorkloadCatalogPayload(t *testing.T) {
+	catalog, err := workloads.Load()
+	if err != nil {
+		t.Fatalf("load workload catalog: %v", err)
+	}
+	view := uiNewJobView{Payload: newJobPayload(catalog)}
+	var builder strings.Builder
+	if err := uiTemplates.ExecuteTemplate(&builder, "new-job.html", view); err != nil {
+		t.Fatalf("render new-job page: %v", err)
+	}
+	page := builder.String()
+	for _, expected := range []string{"const DATA=", "similarity-search", "molwt-filter", "descriptor-batch", "one_of", "query_id", "min_molwt", "skip_invalid", "upload_ready"} {
+		if !strings.Contains(page, expected) {
+			t.Errorf("new-job page is missing %q", expected)
+		}
+	}
+}
+
+func TestNewJobPayloadDeclaresUploadReadiness(t *testing.T) {
+	catalog, err := workloads.Load()
+	if err != nil {
+		t.Fatalf("load workload catalog: %v", err)
+	}
+	payload := newJobPayload(catalog)
+	text := string(payload)
+	if !strings.Contains(text, `"upload_ready":false`) {
+		t.Errorf("catalog payload must mark similarity-graph as not upload-ready")
+	}
+	if !strings.Contains(text, `"reduction":"top-k"`) {
+		t.Errorf("catalog payload is missing the top-k reduction for search")
+	}
+	if !strings.Contains(text, `"reduction":"ordered-concat"`) {
+		t.Errorf("catalog payload is missing the ordered-concat reduction for row workloads")
+	}
+}

coordinator/internal/transport/http/ui_workloads.go

@@ -0,0 +1,106 @@
+package http
+
+import (
+	"encoding/json"
+	"net/http"
+	"sort"
+	"sync"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
+)
+
+// uiPortView is one sorted input/output port of a workload, rendered as
+// pretty JSON on the library page.
+type uiPortView struct {
+	Name   string
+	Schema string
+}
+
+// uiWorkloadView is the library page's view of one catalog workload.
+type uiWorkloadView struct {
+	Name         string
+	Version      string
+	Description  string
+	Capabilities []string
+	TrustModes   []string
+	Determinism  string
+	Verifier     string
+	Enabled      bool
+	Reduction    string
+	Parameters   string
+	Inputs       []uiPortView
+	Outputs      []uiPortView
+}
+
+type uiWorkloadsView struct {
+	Workloads []uiWorkloadView
+}
+
+var (
+	workloadLibraryOnce sync.Once
+	workloadLibraryView uiWorkloadsView
+	workloadLibraryErr  error
+)
+
+func loadWorkloadLibrary() (uiWorkloadsView, error) {
+	workloadLibraryOnce.Do(func() {
+		catalog, err := workloads.Load()
+		if err != nil {
+			workloadLibraryErr = err
+			return
+		}
+		view := uiWorkloadsView{Workloads: make([]uiWorkloadView, 0, len(catalog.Enabled()))}
+		for _, item := range catalog.Enabled() {
+			view.Workloads = append(view.Workloads, uiWorkloadView{
+				Name:         item.Name,
+				Version:      item.Version,
+				Description:  item.Description,
+				Capabilities: item.Capabilities,
+				TrustModes:   item.TrustModes,
+				Determinism:  item.Determinism,
+				Verifier:     item.Verifier,
+				Enabled:      item.Enabled,
+				Reduction:    item.Reduction,
+				Parameters:   prettyJSON(item.Parameters),
+				Inputs:       portViews(item.Inputs),
+				Outputs:      portViews(item.Outputs),
+			})
+		}
+		workloadLibraryView = view
+	})
+	return workloadLibraryView, workloadLibraryErr
+}
+
+func prettyJSON(value any) string {
+	if value == nil {
+		return "{}"
+	}
+	encoded, err := json.MarshalIndent(value, "", "  ")
+	if err != nil {
+		return "{}"
+	}
+	return string(encoded)
+}
+
+func portViews(ports map[string]any) []uiPortView {
+	names := make([]string, 0, len(ports))
+	for name := range ports {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	views := make([]uiPortView, 0, len(names))
+	for _, name := range names {
+		views = append(views, uiPortView{Name: name, Schema: prettyJSON(ports[name])})
+	}
+	return views
+}
+
+func (s *Server) handleUIWorkloads(w http.ResponseWriter, r *http.Request) {
+	view, err := loadWorkloadLibrary()
+	if err != nil {
+		s.log.Error("load workload library", "err", err)
+		http.Error(w, "internal error", http.StatusInternalServerError)
+		return
+	}
+	s.renderUI(w, "workloads.html", view)
+}

coordinator/internal/transport/http/ui_workloads_internal_test.go

@@ -0,0 +1,51 @@
+package http
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestWorkloadLibraryLoadsAndListsEveryWorkload(t *testing.T) {
+	view, err := loadWorkloadLibrary()
+	if err != nil {
+		t.Fatalf("load workload library: %v", err)
+	}
+	names := make(map[string]bool)
+	for _, workload := range view.Workloads {
+		if workload.Name == "" || workload.Version == "" {
+			t.Errorf("workload with empty name or version: %+v", workload)
+		}
+		if workload.Description == "" {
+			t.Errorf("workload %s has no description", workload.Name)
+		}
+		if workload.Parameters == "" {
+			t.Errorf("workload %s has no parameter schema", workload.Name)
+		}
+		if len(workload.Inputs) == 0 || len(workload.Outputs) == 0 {
+			t.Errorf("workload %s has no input or output ports", workload.Name)
+		}
+		names[workload.Name] = true
+	}
+	for _, expected := range []string{"similarity-search", "similarity-graph", "descriptor-batch", "molwt-filter"} {
+		if !names[expected] {
+			t.Errorf("workload library is missing %s", expected)
+		}
+	}
+}
+
+func TestWorkloadLibraryPageRendersWorkloads(t *testing.T) {
+	view, err := loadWorkloadLibrary()
+	if err != nil {
+		t.Fatalf("load workload library: %v", err)
+	}
+	var builder strings.Builder
+	if err := uiTemplates.ExecuteTemplate(&builder, "workloads.html", view); err != nil {
+		t.Fatalf("render workloads page: %v", err)
+	}
+	page := builder.String()
+	for _, expected := range []string{"Workload library", "descriptor-batch", "molwt-filter", "byte_exact", "exact-artifact@1"} {
+		if !strings.Contains(page, expected) {
+			t.Errorf("workloads page is missing %q", expected)
+		}
+	}
+}

coordinator/internal/usecase/artifact.go

@@ -12,18 +12,22 @@ import (
 // UploadArtifact stores a worker's partial-result bytes and records the metadata.
 type UploadArtifact struct {
 	tasks     TaskRepository
+	workers   WorkerRepository
 	artifacts ArtifactRepository
 	blobs     BlobStore
 	tx        TxManager
 	clk       Clock
 }
 
-func NewUploadArtifact(tasks TaskRepository, artifacts ArtifactRepository,
+func NewUploadArtifact(tasks TaskRepository, workers WorkerRepository, artifacts ArtifactRepository,
 	blobs BlobStore, tx TxManager, clk Clock) *UploadArtifact {
-	return &UploadArtifact{tasks: tasks, artifacts: artifacts, blobs: blobs, tx: tx, clk: clk}
+	return &UploadArtifact{tasks: tasks, workers: workers, artifacts: artifacts, blobs: blobs, tx: tx, clk: clk}
 }
 
 func (uc *UploadArtifact) Execute(ctx context.Context, in UploadArtifactInput) (*domain.Artifact, error) {
+	if err := authorizeWorkerOwner(ctx, uc.workers, in.WorkerID); err != nil {
+		return nil, err
+	}
 	task, err := uc.tasks.Get(ctx, in.TaskID)
 	if err != nil {
 		return nil, err

coordinator/internal/usecase/job.go

@@ -7,6 +7,7 @@ import (
 	"github.com/google/uuid"
 
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 // Job operations: the submitter-facing lifecycle of a whole submission.
@@ -227,7 +228,7 @@ func progressFrom(job domain.Job, counts map[domain.TaskStatus]int) domain.JobPr
 // Shared by CompleteTask and FailTask so both close a job by the same rule —
 // the rule itself lives in domain.JobProgress.DeriveStatus.
 func syncJobStatus(ctx context.Context, jobs JobRepository, tasks TaskRepository,
-	jobID uuid.UUID, now time.Time) error {
+	catalog *workloads.Catalog, jobID uuid.UUID, now time.Time) error {
 
 	counts, err := tasks.CountByStatus(ctx, jobID)
 	if err != nil {
@@ -239,9 +240,11 @@ func syncJobStatus(ctx context.Context, jobs JobRepository, tasks TaskRepository
 	}
 	status := progressFrom(*job, counts).DeriveStatus()
 	// All worker shards being complete means scientific reduction is ready, not
-	// that the job's final artifact already exists. CTX-09 owns the transition
-	// from reducing to completed after it persists that artifact.
-	if status == domain.JobCompleted && job.Workload == "similarity-search" {
+	// that the job's final artifact already exists. Known catalog workloads
+	// transition to reducing so ReduceJob can produce the final artifact; the
+	// reducer then completes the job with the result. Unknown (URI-based) jobs
+	// complete without a coordinator-owned final artifact.
+	if status == domain.JobCompleted && catalog != nil && catalog.Reduction(job.Workload) != "" {
 		status = domain.JobReducing
 	}
 
@@ -253,14 +256,14 @@ func syncJobStatus(ctx context.Context, jobs JobRepository, tasks TaskRepository
 }
 
 func syncExpiredJobStatuses(ctx context.Context, jobs JobRepository, tasks TaskRepository,
-	jobIDs []uuid.UUID, now time.Time) error {
+	catalog *workloads.Catalog, jobIDs []uuid.UUID, now time.Time) error {
 	seen := make(map[uuid.UUID]struct{}, len(jobIDs))
 	for _, jobID := range jobIDs {
 		if _, duplicate := seen[jobID]; duplicate {
 			continue
 		}
 		seen[jobID] = struct{}{}
-		if err := syncJobStatus(ctx, jobs, tasks, jobID, now); err != nil {
+		if err := syncJobStatus(ctx, jobs, tasks, catalog, jobID, now); err != nil {
 			return err
 		}
 	}

coordinator/internal/usecase/reduce.go

@@ -9,6 +9,7 @@ import (
 
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
 	"github.com/emil28092005/SciMesh/coordinator/internal/reducer"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 // ReduceJob turns completed coordinator-owned partial artifacts into one final
@@ -20,11 +21,12 @@ type ReduceJob struct {
 	blobs     BlobStore
 	tx        TxManager
 	clock     Clock
+	catalog   *workloads.Catalog
 }
 
 func NewReduceJob(jobs JobRepository, tasks TaskRepository, artifacts ArtifactRepository,
-	blobs BlobStore, tx TxManager, clock Clock) *ReduceJob {
-	return &ReduceJob{jobs: jobs, tasks: tasks, artifacts: artifacts, blobs: blobs, tx: tx, clock: clock}
+	blobs BlobStore, tx TxManager, clock Clock, catalog *workloads.Catalog) *ReduceJob {
+	return &ReduceJob{jobs: jobs, tasks: tasks, artifacts: artifacts, blobs: blobs, tx: tx, clock: clock, catalog: catalog}
 }
 
 // Execute is idempotent for jobs that are not currently reducing. The worker
@@ -42,7 +44,11 @@ func (uc *ReduceJob) Execute(ctx context.Context, jobID uuid.UUID) error {
 	if job.Status != domain.JobReducing {
 		return nil
 	}
-	if job.Workload != "similarity-search" {
+	if uc.catalog == nil {
+		return uc.fail(ctx, jobID)
+	}
+	reduction := uc.catalog.Reduction(job.Workload)
+	if reduction == "" {
 		return uc.fail(ctx, jobID)
 	}
 
@@ -73,13 +79,13 @@ func (uc *ReduceJob) Execute(ctx context.Context, jobID uuid.UUID) error {
 		readers = append(readers, body)
 		closers = append(closers, body)
 	}
-	output, reduceErr := reducer.ReduceSimilaritySearch(readers, job.Parameters)
+	output, reduceErr := reducePartials(reduction, readers, job.Parameters)
 	closeAll(closers)
 	if reduceErr != nil {
 		return uc.fail(ctx, jobID)
 	}
 
-	final, err := domain.NewArtifact(jobID, nil, domain.ArtifactFinalResult, "similarity-search.csv", "text/csv", uc.clock.Now())
+	final, err := domain.NewArtifact(jobID, nil, domain.ArtifactFinalResult, job.Workload+".csv", "text/csv", uc.clock.Now())
 	if err != nil {
 		return uc.fail(ctx, jobID)
 	}
@@ -100,6 +106,17 @@ func (uc *ReduceJob) Execute(ctx context.Context, jobID uuid.UUID) error {
 	return nil
 }
 
+func reducePartials(reduction string, readers []io.Reader, parameters map[string]any) ([]byte, error) {
+	switch reduction {
+	case "top-k":
+		return reducer.ReduceSimilaritySearch(readers, parameters)
+	case "ordered-concat":
+		return reducer.ReduceOrderedConcat(readers)
+	default:
+		return nil, domain.ErrInvalidInput
+	}
+}
+
 func (uc *ReduceJob) fail(ctx context.Context, jobID uuid.UUID) error {
 	// The public state carries a stable sanitized failure, never parser/storage
 	// internals that may include local paths or implementation details.

coordinator/internal/usecase/task.go

@@ -7,8 +7,8 @@ import (
 
 	"github.com/google/uuid"
 
-	"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 // Task operations: the worker-facing lifecycle of a single chunk.
@@ -28,10 +28,11 @@ type ClaimTask struct {
 	tx            TxManager
 	clock         Clock
 	leaseDuration time.Duration
+	catalog       *workloads.Catalog
 }
 
-func NewClaimTask(tasks TaskRepository, jobs JobRepository, workers WorkerRepository, tx TxManager, clock Clock, leaseDuration time.Duration) *ClaimTask {
-	return &ClaimTask{tasks: tasks, jobs: jobs, workers: workers, tx: tx, clock: clock, leaseDuration: leaseDuration}
+func NewClaimTask(tasks TaskRepository, jobs JobRepository, workers WorkerRepository, tx TxManager, clock Clock, leaseDuration time.Duration, catalog *workloads.Catalog) *ClaimTask {
+	return &ClaimTask{tasks: tasks, jobs: jobs, workers: workers, tx: tx, clock: clock, leaseDuration: leaseDuration, catalog: catalog}
 }
 
 // Execute reclaims elapsed leases first, then hands out one task.
@@ -59,11 +60,8 @@ func (uc *ClaimTask) Execute(ctx context.Context, in ClaimTaskInput) (*domain.Cl
 		// tier would be read off a caller-supplied worker_id, letting anyone who
 		// knows a trusted worker's id claim as it. A shared-token caller (no
 		// requester) is a lab operator and may act as any worker.
-		if r, ok := authctx.From(ctx); ok {
-			if worker.OwnerID == nil || *worker.OwnerID != r.UserID {
-				// Don't disclose that another user's worker exists.
-				return nil, domain.ErrWorkerNotFound
-			}
+		if err := authorizeWorkerOwner(ctx, uc.workers, in.WorkerID); err != nil {
+			return nil, err
 		}
 		// An untrusted volunteer may claim, but never a chunk its owner has
 		// already voted on — so quorum needs genuinely independent computations.
@@ -81,7 +79,7 @@ func (uc *ClaimTask) Execute(ctx context.Context, in ClaimTaskInput) (*domain.Cl
 		if err != nil {
 			return err
 		}
-		if err := syncExpiredJobStatuses(ctx, uc.jobs, uc.tasks, affectedJobs, now); err != nil {
+		if err := syncExpiredJobStatuses(ctx, uc.jobs, uc.tasks, uc.catalog, affectedJobs, now); err != nil {
 			return err
 		}
 
@@ -126,6 +124,9 @@ func NewRenewLease(tasks TaskRepository, workers WorkerRepository, tx TxManager,
 // locked: two concurrent heartbeats must not interleave into a lost update.
 // Whether the caller may renew at all is decided by the entity, not here.
 func (uc *RenewLease) Execute(ctx context.Context, in RenewLeaseInput) (*domain.ClaimedTask, error) {
+	if err := authorizeWorkerOwner(ctx, uc.workers, in.WorkerID); err != nil {
+		return nil, err
+	}
 	var claimed domain.ClaimedTask
 
 	err := uc.tx.WithinTx(ctx, func(ctx context.Context) error {
@@ -160,6 +161,7 @@ func (uc *RenewLease) Execute(ctx context.Context, in RenewLeaseInput) (*domain.
 type CompleteTask struct {
 	tasks     TaskRepository
 	jobs      JobRepository
+	catalog   *workloads.Catalog
 	artifacts ArtifactRepository
 	workers   WorkerRepository
 	results   TaskResultRepository
@@ -171,12 +173,12 @@ type CompleteTask struct {
 }
 
 func NewCompleteTask(tasks TaskRepository, jobs JobRepository, artifacts ArtifactRepository,
-	workers WorkerRepository, results TaskResultRepository, tx TxManager, clock Clock, quorum int) *CompleteTask {
+	workers WorkerRepository, results TaskResultRepository, tx TxManager, clock Clock, quorum int, catalog *workloads.Catalog) *CompleteTask {
 	if quorum < 1 {
 		quorum = 2
 	}
 	return &CompleteTask{tasks: tasks, jobs: jobs, artifacts: artifacts, workers: workers,
-		results: results, tx: tx, clock: clock, quorum: quorum}
+		results: results, tx: tx, clock: clock, quorum: quorum, catalog: catalog}
 }
 
 // Execute applies the result and, when that was the job's last outstanding
@@ -186,6 +188,9 @@ func NewCompleteTask(tasks TaskRepository, jobs JobRepository, artifacts Artifac
 // Lease ownership, staleness, and idempotent replays are all decided by
 // Task.CompleteWith; this use case only orchestrates.
 func (uc *CompleteTask) Execute(ctx context.Context, in CompleteTaskInput) (*domain.Task, error) {
+	if err := authorizeWorkerOwner(ctx, uc.workers, in.WorkerID); err != nil {
+		return nil, err
+	}
 	var out *domain.Task
 
 	err := uc.tx.WithinTx(ctx, func(ctx context.Context) error {
@@ -227,7 +232,7 @@ func (uc *CompleteTask) Execute(ctx context.Context, in CompleteTaskInput) (*dom
 		if err := uc.tasks.Update(ctx, task); err != nil {
 			return err
 		}
-		return syncJobStatus(ctx, uc.jobs, uc.tasks, task.JobID, now)
+		return syncJobStatus(ctx, uc.jobs, uc.tasks, uc.catalog, task.JobID, now)
 	})
 	if err != nil {
 		return nil, err
@@ -266,7 +271,7 @@ func (uc *CompleteTask) recordVote(ctx context.Context, task *domain.Task, in Co
 	if err := uc.tasks.Update(ctx, task); err != nil {
 		return err
 	}
-	return syncJobStatus(ctx, uc.jobs, uc.tasks, task.JobID, now)
+	return syncJobStatus(ctx, uc.jobs, uc.tasks, uc.catalog, task.JobID, now)
 }
 
 // workerTrust reports whether the worker's results are accepted directly, and
@@ -317,19 +322,24 @@ func (uc *CompleteTask) verifyResultArtifact(ctx context.Context, taskID uuid.UU
 // --- FailTask ------------------------------------------------------------
 
 type FailTask struct {
-	tasks TaskRepository
-	jobs  JobRepository
-	tx    TxManager
-	clock Clock
+	tasks   TaskRepository
+	jobs    JobRepository
+	workers WorkerRepository
+	tx      TxManager
+	clock   Clock
+	catalog *workloads.Catalog
 }
 
-func NewFailTask(tasks TaskRepository, jobs JobRepository, tx TxManager, clock Clock) *FailTask {
-	return &FailTask{tasks: tasks, jobs: jobs, tx: tx, clock: clock}
+func NewFailTask(tasks TaskRepository, jobs JobRepository, workers WorkerRepository, tx TxManager, clock Clock, catalog *workloads.Catalog) *FailTask {
+	return &FailTask{tasks: tasks, jobs: jobs, workers: workers, tx: tx, clock: clock, catalog: catalog}
 }
 
 // Execute delegates the requeue-or-terminate decision to Task.Fail, then keeps
 // the parent job's status consistent in the same transaction.
 func (uc *FailTask) Execute(ctx context.Context, in FailTaskInput) (*domain.Task, error) {
+	if err := authorizeWorkerOwner(ctx, uc.workers, in.WorkerID); err != nil {
+		return nil, err
+	}
 	var out *domain.Task
 
 	err := uc.tx.WithinTx(ctx, func(ctx context.Context) error {
@@ -345,7 +355,7 @@ func (uc *FailTask) Execute(ctx context.Context, in FailTaskInput) (*domain.Task
 			return err
 		}
 		out = task
-		return syncJobStatus(ctx, uc.jobs, uc.tasks, task.JobID, now)
+		return syncJobStatus(ctx, uc.jobs, uc.tasks, uc.catalog, task.JobID, now)
 	})
 	if err != nil {
 		return nil, err
@@ -356,14 +366,15 @@ func (uc *FailTask) Execute(ctx context.Context, in FailTaskInput) (*domain.Task
 // --- ExpireLeases --------------------------------------------------------
 
 type ExpireLeases struct {
-	tasks TaskRepository
-	jobs  JobRepository
-	tx    TxManager
-	clock Clock
+	tasks   TaskRepository
+	jobs    JobRepository
+	tx      TxManager
+	clock   Clock
+	catalog *workloads.Catalog
 }
 
-func NewExpireLeases(tasks TaskRepository, jobs JobRepository, tx TxManager, clock Clock) *ExpireLeases {
-	return &ExpireLeases{tasks: tasks, jobs: jobs, tx: tx, clock: clock}
+func NewExpireLeases(tasks TaskRepository, jobs JobRepository, tx TxManager, clock Clock, catalog *workloads.Catalog) *ExpireLeases {
+	return &ExpireLeases{tasks: tasks, jobs: jobs, tx: tx, clock: clock, catalog: catalog}
 }
 
 // Execute reclaims elapsed tasks and persists the state of every affected job.
@@ -380,7 +391,7 @@ func (uc *ExpireLeases) Execute(ctx context.Context) (int64, error) {
 		if err != nil {
 			return err
 		}
-		return syncExpiredJobStatuses(ctx, uc.jobs, uc.tasks, affected, now)
+		return syncExpiredJobStatuses(ctx, uc.jobs, uc.tasks, uc.catalog, affected, now)
 	})
 	return int64(len(affected)), err
 }

coordinator/internal/usecase/testcatalog_test.go

@@ -0,0 +1,62 @@
+package usecase_test
+
+import (
+	"testing"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
+)
+
+// testCatalog loads the embedded workload catalog for usecase tests. The
+// catalog is checked in and generated from the SDK library, so tests exercise
+// the real validation contract.
+func testCatalog() *workloads.Catalog {
+	catalog, err := workloads.Load()
+	if err != nil {
+		panic(err)
+	}
+	return catalog
+}
+
+func TestEmbeddedCatalogLoadsAndValidatesSearchParameters(t *testing.T) {
+	catalog := testCatalog()
+	if err := catalog.ValidateParameters("similarity-search", map[string]any{
+		"query_smiles": "CCO", "top_k": 10, "threshold_direction": "greater",
+	}); err != nil {
+		t.Fatalf("valid search parameters rejected: %v", err)
+	}
+	if err := catalog.ValidateParameters("molwt-filter", map[string]any{
+		"min_molwt": 100, "max_molwt": 600, "skip_invalid": true,
+	}); err != nil {
+		t.Fatalf("valid molwt parameters rejected: %v", err)
+	}
+	if err := catalog.ValidateParameters("nope", map[string]any{}); err == nil {
+		t.Error("unknown workload accepted")
+	}
+	if err := catalog.ValidateParameters("similarity-graph", map[string]any{"threshold": 0.7}); err != nil {
+		t.Errorf("graph parameters rejected: %v", err)
+	}
+	if !catalog.UploadReady("molwt-filter") {
+		t.Error("molwt-filter must be upload-ready")
+	}
+	if catalog.UploadReady("similarity-graph") {
+		t.Error("similarity-graph must not be upload-ready")
+	}
+	if got := catalog.Reduction("similarity-search"); got != "top-k" {
+		t.Errorf("search reduction = %q, want top-k", got)
+	}
+	if got := catalog.Reduction("descriptor-batch"); got != "ordered-concat" {
+		t.Errorf("descriptor reduction = %q, want ordered-concat", got)
+	}
+	for name, parameters := range map[string]map[string]any{
+		"both query fields":    {"query_id": "CHEMBL1", "query_smiles": "CCO"},
+		"undeclared parameter": {"query_smiles": "CCO", "bogus": 1},
+		"bad top_k":            {"query_smiles": "CCO", "top_k": -1},
+		"bad enum":             {"query_smiles": "CCO", "threshold_direction": "sideways"},
+		"missing query":        {},
+		"non-integer top_k":    {"query_smiles": "CCO", "top_k": 1.5},
+	} {
+		if err := catalog.ValidateParameters("similarity-search", parameters); err == nil {
+			t.Errorf("%s accepted", name)
+		}
+	}
+}

coordinator/internal/usecase/ui.go

@@ -3,12 +3,15 @@ package usecase
 import (
 	"context"
 	"fmt"
+	"sort"
+	"strings"
 	"time"
 
 	"github.com/google/uuid"
 
 	"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 // UIReadRepository is a read-only projection source for the local operator UI.
@@ -125,9 +128,14 @@ type JobDetailView struct {
 	Session              *SessionView    `json:"-"`
 }
 
-type Dashboard struct{ read UIReadRepository }
+type Dashboard struct {
+	read    UIReadRepository
+	catalog *workloads.Catalog
+}
 
-func NewDashboard(read UIReadRepository) *Dashboard { return &Dashboard{read: read} }
+func NewDashboard(read UIReadRepository, catalog *workloads.Catalog) *Dashboard {
+	return &Dashboard{read: read, catalog: catalog}
+}
 
 func (d *Dashboard) Overview(ctx context.Context, limit int) (DashboardView, error) {
 	jobs, err := d.read.ListJobs(ctx, uiOwnerFilter(ctx), limit)
@@ -219,7 +227,7 @@ func (d *Dashboard) JobDetail(ctx context.Context, jobID uuid.UUID) (JobDetailVi
 		JobCard:    jobCard(*job, tasks),
 		Tasks:      make([]TaskCard, 0, len(tasks)),
 		Artifacts:  make([]ArtifactCard, 0, len(artifacts)),
-		Parameters: uiParameters(job.Parameters),
+		Parameters: uiParameters(job.Parameters, d.catalog, job.Workload),
 		Session:    sessionViewFrom(ctx),
 	}
 	for _, task := range tasks {
@@ -304,31 +312,67 @@ func jobCard(job domain.Job, tasks []domain.Task) JobCard {
 	return c
 }
 
-func uiParameters(parameters map[string]any) []ParameterCard {
-	keys := []struct {
-		key   string
-		label string
-	}{
-		{"query_smiles", "Target SMILES"},
-		{"query_id", "Target ChEMBL ID"},
-		{"top_k", "Global top-k"},
-		{"threshold", "Similarity threshold"},
-		{"threshold_direction", "Threshold direction"},
+func uiParameters(parameters map[string]any, catalog *workloads.Catalog, workload string) []ParameterCard {
+	labels := map[string]string{
+		"query_smiles":        "Target SMILES",
+		"query_id":            "Target ChEMBL ID",
+		"top_k":               "Global top-k",
+		"threshold":           "Similarity threshold",
+		"threshold_direction": "Threshold direction",
+		"min_molwt":           "Minimum molecular weight",
+		"max_molwt":           "Maximum molecular weight",
+		"skip_invalid":        "Skip invalid molecules",
+		"block_size":          "Block size",
 	}
+	keys := make([]string, 0, len(parameters))
+	declared := declaredParameterNames(catalog, workload)
+	for key := range parameters {
+		if declared != nil && !declared[key] {
+			// Only schema-declared scientific parameters may reach the browser;
+			// anything else could carry internal coordinator state.
+			continue
+		}
+		keys = append(keys, key)
+	}
+	sort.Strings(keys)
 	out := make([]ParameterCard, 0, len(keys))
-	for _, entry := range keys {
-		value, ok := parameters[entry.key]
+	for _, key := range keys {
+		value, ok := parameters[key]
 		if !ok {
 			continue
 		}
 		formatted, ok := formatUIParameter(value)
-		if ok {
-			out = append(out, ParameterCard{Label: entry.label, Value: formatted})
+		if !ok {
+			continue
 		}
+		label := labels[key]
+		if label == "" {
+			label = strings.ReplaceAll(key, "_", " ")
+		}
+		out = append(out, ParameterCard{Label: label, Value: formatted})
 	}
 	return out
 }
 
+func declaredParameterNames(catalog *workloads.Catalog, workload string) map[string]bool {
+	if catalog == nil || workload == "" {
+		return nil
+	}
+	item := catalog.ByName(workload)
+	if item == nil {
+		return nil
+	}
+	properties, ok := item.Parameters["properties"].(map[string]any)
+	if !ok {
+		return nil
+	}
+	declared := make(map[string]bool, len(properties))
+	for name := range properties {
+		declared[name] = true
+	}
+	return declared
+}
+
 func formatUIParameter(value any) (string, bool) {
 	switch typed := value.(type) {
 	case string:

coordinator/internal/usecase/ui_internal_test.go

@@ -1,14 +1,22 @@
 package usecase
 
-import "testing"
+import (
+	"testing"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
+)
 
 func TestUIParametersAreAllowlisted(t *testing.T) {
+	catalog, err := workloads.Load()
+	if err != nil {
+		t.Fatal(err)
+	}
 	parameters := uiParameters(map[string]any{
 		"query_smiles":         "CCO",
 		"top_k":                float64(20),
 		"internal_storage_key": "must-not-reach-browser",
 		"nested":               map[string]any{"secret": "no"},
-	})
+	}, catalog, "similarity-search")
 	if len(parameters) != 2 {
 		t.Fatalf("parameters = %#v, want only two allowlisted values", parameters)
 	}
@@ -17,3 +25,28 @@ func TestUIParametersAreAllowlisted(t *testing.T) {
 		t.Fatalf("parameters = %#v", parameters)
 	}
 }
+
+func TestUIParametersRenderEverySchemaDeclaredField(t *testing.T) {
+	catalog, err := workloads.Load()
+	if err != nil {
+		t.Fatal(err)
+	}
+	parameters := uiParameters(map[string]any{
+		"min_molwt":    100,
+		"max_molwt":    600,
+		"skip_invalid": true,
+		"secret_key":   "no",
+	}, catalog, "molwt-filter")
+	if len(parameters) != 3 {
+		t.Fatalf("parameters = %#v, want three declared values", parameters)
+	}
+	labels := map[string]bool{}
+	for _, card := range parameters {
+		labels[card.Label] = true
+	}
+	for _, expected := range []string{"Minimum molecular weight", "Maximum molecular weight", "Skip invalid molecules"} {
+		if !labels[expected] {
+			t.Errorf("missing parameter card %q in %#v", expected, parameters)
+		}
+	}
+}

coordinator/internal/usecase/ui_scope_test.go

@@ -19,7 +19,7 @@ func newDashboard() (*usecase.Dashboard, *memstore.JobRepo) {
 	tasks := memstore.NewTaskRepo()
 	workers := memstore.NewWorkerRepo()
 	artifacts := memstore.NewArtifactRepo()
-	return usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, workers, artifacts)), jobs
+	return usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, workers, artifacts), testCatalog()), jobs
 }
 
 func ownedJob(t *testing.T, jobs *memstore.JobRepo, owner uuid.UUID) uuid.UUID {

coordinator/internal/usecase/ui_workers_scope_test.go

@@ -17,7 +17,7 @@ func newDashboardWithWorkers() (*usecase.Dashboard, *memstore.WorkerRepo) {
 	tasks := memstore.NewTaskRepo()
 	workers := memstore.NewWorkerRepo()
 	artifacts := memstore.NewArtifactRepo()
-	return usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, workers, artifacts)), workers
+	return usecase.NewDashboard(memstore.NewUIReadRepo(jobs, tasks, workers, artifacts), testCatalog()), workers
 }
 
 func seedWorker(t *testing.T, workers *memstore.WorkerRepo, owner *uuid.UUID, name string) {

coordinator/internal/usecase/upload.go

@@ -4,12 +4,12 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"math"
 
 	"github.com/google/uuid"
 
 	"github.com/emil28092005/SciMesh/coordinator/internal/chunk"
 	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
+	"github.com/emil28092005/SciMesh/coordinator/internal/workloads"
 )
 
 // SubmitDataset accepts an uploaded dataset, splits it into shard artifacts, and
@@ -23,15 +23,16 @@ type SubmitDataset struct {
 	tx          TxManager
 	clk         Clock
 	maxAttempts int
+	catalog     *workloads.Catalog
 }
 
 func NewSubmitDataset(blobs BlobStore, artifacts ArtifactRepository, jobs JobRepository,
-	tasks TaskRepository, tx TxManager, clk Clock, maxAttempts int) *SubmitDataset {
-	return &SubmitDataset{blobs: blobs, artifacts: artifacts, jobs: jobs, tasks: tasks, tx: tx, clk: clk, maxAttempts: maxAttempts}
+	tasks TaskRepository, tx TxManager, clk Clock, maxAttempts int, catalog *workloads.Catalog) *SubmitDataset {
+	return &SubmitDataset{blobs: blobs, artifacts: artifacts, jobs: jobs, tasks: tasks, tx: tx, clk: clk, maxAttempts: maxAttempts, catalog: catalog}
 }
 
 func (uc *SubmitDataset) Execute(ctx context.Context, in SubmitDatasetInput) (SubmitDatasetResult, error) {
-	if err := validateUploadedWorkload(in.Workload, in.Parameters); err != nil {
+	if err := validateUploadedWorkload(uc.catalog, in.Workload, in.Parameters); err != nil {
 		return SubmitDatasetResult{}, err
 	}
 	if uc.maxAttempts < 1 {
@@ -86,7 +87,8 @@ func (uc *SubmitDataset) Execute(ctx context.Context, in SubmitDatasetInput) (Su
 		putKeys = append(putKeys, art.StorageKey)
 		art.SetContent(ssum, ssize)
 
-		task, err := domain.NewShardTask(job.ID, index, in.Workload, art.ID, ssum, in.Parameters, uc.maxAttempts, now)
+		task, err := domain.NewShardTask(job.ID, index, in.Workload, art.ID, ssum,
+			taskParameterSubset(in.Parameters), uc.maxAttempts, now)
 		if err != nil {
 			return err
 		}
@@ -128,74 +130,38 @@ func (uc *SubmitDataset) Execute(ctx context.Context, in SubmitDatasetInput) (Su
 	}, nil
 }
 
-// validateUploadedWorkload is deliberately narrow until CTX-07/08/10 adds a
-// typed distributed-workload registry. In particular, running similarity-graph
-// independently per TSV shard is scientifically wrong: cross-shard pairs would
-// be absent from the apparent graph.
-func validateUploadedWorkload(workload string, parameters map[string]any) error {
-	if workload != "similarity-search" {
+// validateUploadedWorkload checks the submitted workload against the embedded
+// catalog: it must be an enabled, upload-ready workload whose parameters
+// satisfy the declared JSON schema. Workloads that need planner-produced
+// inputs (such as the graph block-pair shards) declare upload_ready=false and
+// cannot be driven from a single uploaded dataset.
+func validateUploadedWorkload(catalog *workloads.Catalog, workload string, parameters map[string]any) error {
+	if catalog == nil {
 		return domain.ErrInvalidInput
 	}
-	allowed := map[string]struct{}{
-		"query_smiles": {}, "top_k": {}, "threshold": {},
-		"threshold_direction": {}, "progress_every": {},
-	}
-	for key := range parameters {
-		if _, ok := allowed[key]; !ok {
-			return domain.ErrInvalidInput
-		}
-	}
-	query, ok := parameters["query_smiles"].(string)
-	if !ok || query == "" || len(query) > 200 {
+	if err := catalog.ValidateParameters(workload, parameters); err != nil {
 		return domain.ErrInvalidInput
 	}
-	if value, ok := parameters["top_k"]; ok && !isPositiveJSONInteger(value) {
-		return domain.ErrInvalidInput
-	}
-	if value, ok := parameters["progress_every"]; ok && !isNonNegativeJSONInteger(value) {
-		return domain.ErrInvalidInput
-	}
-	if value, ok := parameters["threshold"]; ok && !isUnitIntervalNumber(value) {
-		return domain.ErrInvalidInput
-	}
-	if value, ok := parameters["threshold_direction"]; ok && value != "greater" && value != "less" {
+	if !catalog.UploadReady(workload) {
 		return domain.ErrInvalidInput
 	}
 	return nil
 }
 
-func isPositiveJSONInteger(value any) bool    { return isJSONInteger(value, false) }
-func isNonNegativeJSONInteger(value any) bool { return isJSONInteger(value, true) }
-
-func isJSONInteger(value any, allowZero bool) bool {
-	var n int64
-	switch v := value.(type) {
-	case int:
-		n = int64(v)
-	case int64:
-		n = v
-	case float64:
-		if math.Trunc(v) != v || v > math.MaxInt64 || v < math.MinInt64 {
-			return false
+// taskParameterSubset drops coordinator-level keys from the parameters that
+// are handed to workers. max_rows is a plan-time bound applied by the chunker
+// here; a worker would reject it as outside its stage projection.
+func taskParameterSubset(parameters map[string]any) map[string]any {
+	if _, present := parameters["max_rows"]; !present {
+		return parameters
+	}
+	subset := make(map[string]any, len(parameters)-1)
+	for key, value := range parameters {
+		if key != "max_rows" {
+			subset[key] = value
 		}
-		n = int64(v)
-	default:
-		return false
-	}
-	return n >= 0 && (allowZero || n > 0)
-}
-
-func isUnitIntervalNumber(value any) bool {
-	switch v := value.(type) {
-	case float64:
-		return !math.IsNaN(v) && !math.IsInf(v, 0) && v >= 0 && v <= 1
-	case int:
-		return v >= 0 && v <= 1
-	case int64:
-		return v >= 0 && v <= 1
-	default:
-		return false
 	}
+	return subset
 }
 
 // GetTaskInput resolves a task's input shard and opens it for streaming. The

coordinator/internal/usecase/usecase_test.go

@@ -73,20 +73,20 @@ func newHarness() *harness {
 	}
 	tx := memstore.Tx{}
 	h.createJob = usecase.NewCreateJob(h.jobs, h.tasks, tx, h.clk)
-	h.submit = usecase.NewSubmitDataset(h.blobs, h.arts, h.jobs, h.tasks, tx, h.clk, 3)
-	h.claim = usecase.NewClaimTask(h.tasks, h.jobs, h.work, tx, h.clk, lease)
+	h.submit = usecase.NewSubmitDataset(h.blobs, h.arts, h.jobs, h.tasks, tx, h.clk, 3, testCatalog())
+	h.claim = usecase.NewClaimTask(h.tasks, h.jobs, h.work, tx, h.clk, lease, testCatalog())
 	h.renew = usecase.NewRenewLease(h.tasks, h.work, tx, h.clk, lease)
-	h.complete = usecase.NewCompleteTask(h.tasks, h.jobs, h.arts, h.work, h.taskResults, tx, h.clk, 2)
-	h.fail = usecase.NewFailTask(h.tasks, h.jobs, tx, h.clk)
+	h.complete = usecase.NewCompleteTask(h.tasks, h.jobs, h.arts, h.work, h.taskResults, tx, h.clk, 2, testCatalog())
+	h.fail = usecase.NewFailTask(h.tasks, h.jobs, h.work, tx, h.clk, testCatalog())
 	h.status = usecase.NewGetJobStatus(h.jobs, h.tasks)
 	h.results = usecase.NewListResults(h.tasks)
 	h.register = usecase.NewRegisterWorker(h.work, h.clk)
-	h.uploadArt = usecase.NewUploadArtifact(h.tasks, h.arts, h.blobs, tx, h.clk)
+	h.uploadArt = usecase.NewUploadArtifact(h.tasks, h.work, h.arts, h.blobs, tx, h.clk)
 	h.downloadArt = usecase.NewDownloadArtifact(h.arts, h.blobs)
 	h.getInput = usecase.NewGetTaskInput(h.tasks, h.arts, h.blobs)
-	h.expire = usecase.NewExpireLeases(h.tasks, h.jobs, tx, h.clk)
+	h.expire = usecase.NewExpireLeases(h.tasks, h.jobs, tx, h.clk, testCatalog())
 	h.cancel = usecase.NewCancelJob(h.jobs, h.tasks, tx, h.clk)
-	h.reduce = usecase.NewReduceJob(h.jobs, h.tasks, h.arts, h.blobs, tx, h.clk)
+	h.reduce = usecase.NewReduceJob(h.jobs, h.tasks, h.arts, h.blobs, tx, h.clk, testCatalog())
 	h.jobResult = usecase.NewGetJobResult(h.jobs, h.downloadArt)
 	return h
 }
@@ -133,6 +133,44 @@ func TestSimilaritySearchReductionCreatesFinalArtifact(t *testing.T) {
 	}
 }
 
+func TestMolwtFilterReductionConcatenatesPartialsInOrder(t *testing.T) {
+	h := newHarness()
+	jobID := h.seedJob(t, "molwt-filter", 2)
+	if err := h.jobs.UpdateStatus(ctx, jobID, domain.JobRunning, nil); err != nil {
+		t.Fatal(err)
+	}
+	partials := []string{
+		"chembl_id,canonical_smiles\nA,CC\n",
+		"chembl_id,canonical_smiles\nB,CCCC\n",
+	}
+	for _, partial := range partials {
+		taskID, attempt := h.leaseOne(t, "w1", "molwt-filter")
+		art, err := h.uploadArt.Execute(ctx, usecase.UploadArtifactInput{TaskID: taskID, WorkerID: "w1", Attempt: attempt, Filename: "partial.csv", ContentType: "text/csv", Body: strings.NewReader(partial)})
+		if err != nil {
+			t.Fatal(err)
+		}
+		if _, err := h.complete.Execute(ctx, usecase.CompleteTaskInput{TaskID: taskID, WorkerID: "w1", Attempt: attempt, ResultArtifactID: art.ID}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if err := h.reduce.Execute(ctx, jobID); err != nil {
+		t.Fatal(err)
+	}
+	progress, err := h.status.Execute(ctx, jobID)
+	if err != nil || progress.Job.Status != domain.JobCompleted {
+		t.Fatalf("status=%s err=%v", progress.Job.Status, err)
+	}
+	art, body, err := h.jobResult.Execute(ctx, jobID)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer body.Close()
+	bytes, _ := io.ReadAll(body)
+	if art.Kind != domain.ArtifactFinalResult || string(bytes) != "chembl_id,canonical_smiles\nA,CC\nB,CCCC\n" {
+		t.Fatalf("unexpected final %q", bytes)
+	}
+}
+
 func TestSimilaritySearchReductionFailureIsSanitized(t *testing.T) {
 	h := newHarness()
 	jobID := h.seedJob(t, "similarity-search", 1)
@@ -318,6 +356,36 @@ func TestJWTCallerCannotClaimAsAnotherUsersWorker(t *testing.T) {
 	}
 }
 
+func TestJWTCallerCannotMutateAnotherUsersWorkerLease(t *testing.T) {
+	h := newHarness()
+	h.seedJob(t, "w", 1)
+	victimOwner := uuid.New()
+	victim, err := h.register.Execute(ctx, usecase.RegisterWorkerInput{
+		Name: "victim", Capabilities: []string{"w"}, OwnerID: &victimOwner, TrustLevel: domain.WorkerTrusted,
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	claimed, err := h.claim.Execute(ctx, usecase.ClaimTaskInput{WorkerID: victim.ID.String()})
+	if err != nil || claimed == nil {
+		t.Fatalf("claim = (%v, %v)", claimed, err)
+	}
+	attacker := authctx.With(ctx, authctx.Requester{UserID: uuid.New(), Role: "user"})
+
+	if _, err := h.renew.Execute(attacker, usecase.RenewLeaseInput{TaskID: claimed.TaskID, WorkerID: victim.ID.String(), Attempt: claimed.Attempt}); !errors.Is(err, domain.ErrWorkerNotFound) {
+		t.Errorf("foreign heartbeat err = %v, want ErrWorkerNotFound", err)
+	}
+	if _, err := h.fail.Execute(attacker, usecase.FailTaskInput{TaskID: claimed.TaskID, WorkerID: victim.ID.String(), Attempt: claimed.Attempt, ErrorCode: "x"}); !errors.Is(err, domain.ErrWorkerNotFound) {
+		t.Errorf("foreign failure err = %v, want ErrWorkerNotFound", err)
+	}
+	if _, err := h.uploadArt.Execute(attacker, usecase.UploadArtifactInput{TaskID: claimed.TaskID, WorkerID: victim.ID.String(), Attempt: claimed.Attempt, Filename: "x.csv", ContentType: "text/csv", Body: strings.NewReader("x")}); !errors.Is(err, domain.ErrWorkerNotFound) {
+		t.Errorf("foreign upload err = %v, want ErrWorkerNotFound", err)
+	}
+	if _, err := h.complete.Execute(attacker, usecase.CompleteTaskInput{TaskID: claimed.TaskID, WorkerID: victim.ID.String(), Attempt: claimed.Attempt, ResultArtifactID: uuid.New()}); !errors.Is(err, domain.ErrWorkerNotFound) {
+		t.Errorf("foreign result err = %v, want ErrWorkerNotFound", err)
+	}
+}
+
 func TestJWTCallerClaimsAsOwnTrustedWorker(t *testing.T) {
 	h := newHarness()
 	h.seedJob(t, "w", 1)
@@ -551,7 +619,7 @@ func TestUploadRejectsLeaseThatExpiresDuringStreaming(t *testing.T) {
 	h.seedJob(t, "w", 1)
 	taskID, attempt := h.leaseOne(t, "w1", "w")
 	h.uploadArt = usecase.NewUploadArtifact(
-		h.tasks, h.arts, expiringBlobStore{BlobStore: h.blobs, clock: h.clk}, memstore.Tx{}, h.clk,
+		h.tasks, h.work, h.arts, expiringBlobStore{BlobStore: h.blobs, clock: h.clk}, memstore.Tx{}, h.clk,
 	)
 
 	_, err := h.uploadArt.Execute(ctx, usecase.UploadArtifactInput{
@@ -787,8 +855,18 @@ func TestSubmitDatasetRejectsUnsupportedDistributedWorkloads(t *testing.T) {
 		Filename: "chembl.tsv", ContentType: "text/tab-separated-values",
 		Body: strings.NewReader("chembl_id\tcanonical_smiles\nA\tCC\n"),
 	})
+	if err != nil {
+		t.Errorf("query_id submission err = %v, want nil", err)
+	}
+	_, err = h.submit.Execute(ctx, usecase.SubmitDatasetInput{
+		Workload: "similarity-search", Parameters: map[string]any{
+			"query_id": "CHEMBL1", "query_smiles": "CCO",
+		}, RowsPerShard: 2,
+		Filename: "chembl.tsv", ContentType: "text/tab-separated-values",
+		Body: strings.NewReader("chembl_id\tcanonical_smiles\nA\tCC\n"),
+	})
 	if !errors.Is(err, domain.ErrInvalidInput) {
-		t.Errorf("query_id submission err = %v, want ErrInvalidInput", err)
+		t.Errorf("both query fields submission err = %v, want ErrInvalidInput", err)
 	}
 }
 

coordinator/internal/usecase/worker_authorization.go

@@ -0,0 +1,33 @@
+package usecase
+
+import (
+	"context"
+
+	"github.com/google/uuid"
+
+	"github.com/emil28092005/SciMesh/coordinator/internal/authctx"
+	"github.com/emil28092005/SciMesh/coordinator/internal/domain"
+)
+
+// authorizeWorkerOwner binds a JWT-authenticated requester to a worker. The
+// shared coordinator token intentionally has no requester and retains its
+// existing operator privileges.
+func authorizeWorkerOwner(ctx context.Context, workers WorkerRepository, workerID string) error {
+	requester, ok := authctx.From(ctx)
+	if !ok {
+		return nil
+	}
+	id, err := uuid.Parse(workerID)
+	if err != nil {
+		return domain.ErrWorkerNotFound
+	}
+	worker, err := workers.Get(ctx, id)
+	if err != nil {
+		return err
+	}
+	if worker.OwnerID == nil || *worker.OwnerID != requester.UserID {
+		// Mask ownership and existence from another user.
+		return domain.ErrWorkerNotFound
+	}
+	return nil
+}

coordinator/internal/workloads/catalog.go

@@ -0,0 +1,273 @@
+// Package workloads provides the coordinator-side view of the SDK workload
+// library. The catalog is generated by `scimesh workload export` and embedded
+// into the binary; it is presentation and orchestration metadata, never
+// executable code. Every field that drives coordinator behaviour (reduction
+// mode, parameter schema, required input columns) is validated at load time
+// so a bad export fails fast instead of misbehaving at runtime.
+package workloads
+
+import (
+	"embed"
+	"encoding/json"
+	"fmt"
+	"sort"
+)
+
+//go:embed workloads.json
+var catalogFile embed.FS
+
+// Catalog is the parsed and validated workload library.
+type Catalog struct {
+	workloads []*Workload
+}
+
+// Workload is one entry of the embedded catalog.
+type Workload struct {
+	Name              string            `json:"name"`
+	Version           string            `json:"version"`
+	Description       string            `json:"description"`
+	Capabilities      []string          `json:"capabilities"`
+	TrustModes        []string          `json:"trust_modes"`
+	Determinism       string            `json:"determinism"`
+	Verifier          string            `json:"verifier"`
+	Enabled           bool              `json:"enabled"`
+	Reduction         string            `json:"reduction"`
+	UploadReady       bool              `json:"upload_ready"`
+	Parameters        map[string]any    `json:"parameters_schema"`
+	UIElements        []UIElement       `json:"ui_elements"`
+	Inputs            map[string]any    `json:"inputs"`
+	Outputs           map[string]any    `json:"outputs"`
+	requiredColumns   map[string]bool   // derived from input validator configuration
+	inputMediaTypes   map[string]string // port name -> media type
+	parameterDefaults map[string]any    // derived from the schema
+}
+
+// UIElement is one workload-declared form control for the "new job" page.
+type UIElement struct {
+	Field       string   `json:"field"`
+	Widget      string   `json:"widget"`
+	Label       string   `json:"label"`
+	Help        string   `json:"help"`
+	Placeholder string   `json:"placeholder"`
+	Options     []string `json:"options"`
+	Default     any      `json:"default"`
+	Order       int      `json:"order"`
+	Group       string   `json:"group"`
+}
+
+type libraryFile struct {
+	SchemaVersion int         `json:"schema_version"`
+	GeneratedBy   string      `json:"generated_by"`
+	Workloads     []*Workload `json:"workloads"`
+}
+
+// Load reads and validates the embedded catalog.
+func Load() (*Catalog, error) {
+	raw, err := catalogFile.ReadFile("workloads.json")
+	if err != nil {
+		return nil, fmt.Errorf("read embedded workload catalog: %w", err)
+	}
+	return Parse(raw)
+}
+
+// Parse validates and builds a Catalog from catalog JSON bytes.
+func Parse(raw []byte) (*Catalog, error) {
+	var file libraryFile
+	if err := json.Unmarshal(raw, &file); err != nil {
+		return nil, fmt.Errorf("parse workload catalog: %w", err)
+	}
+	if file.SchemaVersion != 2 {
+		return nil, fmt.Errorf("workload catalog schema_version must be 2, got %d", file.SchemaVersion)
+	}
+	if len(file.Workloads) == 0 {
+		return nil, fmt.Errorf("workload catalog contains no workloads")
+	}
+	catalog := &Catalog{}
+	names := make(map[string]bool, len(file.Workloads))
+	for _, workload := range file.Workloads {
+		if err := validateWorkload(workload); err != nil {
+			return nil, err
+		}
+		if names[workload.Name] {
+			return nil, fmt.Errorf("workload catalog lists %q more than once", workload.Name)
+		}
+		names[workload.Name] = true
+		catalog.workloads = append(catalog.workloads, workload)
+	}
+	sort.Slice(catalog.workloads, func(i, j int) bool {
+		return catalog.workloads[i].Name < catalog.workloads[j].Name
+	})
+	return catalog, nil
+}
+
+func validateWorkload(workload *Workload) error {
+	if workload.Name == "" || workload.Version == "" {
+		return fmt.Errorf("workload catalog entry must have a name and version")
+	}
+	switch workload.Reduction {
+	case "top-k", "ordered-concat":
+	default:
+		return fmt.Errorf("workload %q declares unknown reduction %q", workload.Name, workload.Reduction)
+	}
+	if err := validateSchema(workload.Name, workload.Parameters); err != nil {
+		return err
+	}
+	workload.parameterDefaults = schemaDefaults(workload.Parameters)
+	workload.requiredColumns = map[string]bool{}
+	for portName, port := range workload.Inputs {
+		config, ok := port.(map[string]any)
+		if !ok {
+			continue
+		}
+		validator, _ := config["validator_configuration"].(map[string]any)
+		columns, _ := validator["required_columns"].([]any)
+		for _, column := range columns {
+			text, ok := column.(string)
+			if ok {
+				workload.requiredColumns[text] = true
+			}
+		}
+		if mediaType, ok := config["media_type"].(string); ok {
+			if workload.inputMediaTypes == nil {
+				workload.inputMediaTypes = map[string]string{}
+			}
+			workload.inputMediaTypes[portName] = mediaType
+		}
+	}
+	for _, element := range workload.UIElements {
+		if err := validateUIElement(workload, element); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func validateUIElement(workload *Workload, element UIElement) error {
+	switch element.Widget {
+	case "text", "textarea", "number", "select", "checkbox":
+	default:
+		return fmt.Errorf("workload %q ui element %q has unknown widget %q", workload.Name, element.Field, element.Widget)
+	}
+	if element.Widget == "select" && len(element.Options) == 0 {
+		return fmt.Errorf("workload %q ui element %q is a select without options", workload.Name, element.Field)
+	}
+	properties, ok := workload.Parameters["properties"].(map[string]any)
+	if !ok {
+		return nil
+	}
+	property, declared := properties[element.Field].(map[string]any)
+	if !declared {
+		return fmt.Errorf("workload %q ui element %q does not name a declared parameter", workload.Name, element.Field)
+	}
+	if schemaType(property) == "boolean" && element.Widget != "checkbox" {
+		return fmt.Errorf("workload %q ui element %q must use the checkbox widget for a boolean parameter", workload.Name, element.Field)
+	}
+	return nil
+}
+
+// Enabled returns the enabled workloads, sorted by name.
+func (c *Catalog) Enabled() []*Workload {
+	result := make([]*Workload, 0, len(c.workloads))
+	for _, workload := range c.workloads {
+		if workload.Enabled {
+			result = append(result, workload)
+		}
+	}
+	return result
+}
+
+// ByName returns the workload with the given name, or nil.
+func (c *Catalog) ByName(name string) *Workload {
+	for _, workload := range c.workloads {
+		if workload.Name == name {
+			return workload
+		}
+	}
+	return nil
+}
+
+// ValidateParameters checks job parameters against the workload schema. It
+// enforces the strict subset of JSON Schema used by the SDK manifests: object
+// shape, types, required, enum, numeric bounds, string lengths, and oneOf.
+func (c *Catalog) ValidateParameters(name string, parameters map[string]any) error {
+	workload := c.ByName(name)
+	if workload == nil {
+		return fmt.Errorf("unknown workload %q", name)
+	}
+	if !workload.Enabled {
+		return fmt.Errorf("workload %q is not enabled", name)
+	}
+	return validateParameters(name, workload.Parameters, parameters)
+}
+
+// UploadReady reports whether the workload can be driven from a single
+// uploaded dataset file. Workloads that need planner-produced inputs (such as
+// the graph block-pair shards) declare upload_ready=false.
+func (c *Catalog) UploadReady(name string) bool {
+	workload := c.ByName(name)
+	if workload == nil {
+		return false
+	}
+	return workload.UploadReady
+}
+
+// RequiredColumns reports every column the workload's input port requires.
+func (c *Catalog) RequiredColumns(name string) map[string]bool {
+	workload := c.ByName(name)
+	if workload == nil {
+		return nil
+	}
+	return workload.requiredColumns
+}
+
+// InputMediaType returns the declared media type of the named input port.
+func (c *Catalog) InputMediaType(name, port string) string {
+	workload := c.ByName(name)
+	if workload == nil {
+		return ""
+	}
+	return workload.inputMediaTypes[port]
+}
+
+// InputPortNames returns the sorted input port names of the workload.
+func (c *Catalog) InputPortNames(name string) []string {
+	workload := c.ByName(name)
+	if workload == nil {
+		return nil
+	}
+	names := make([]string, 0, len(workload.Inputs))
+	for port := range workload.Inputs {
+		names = append(names, port)
+	}
+	sort.Strings(names)
+	return names
+}
+
+// Reduction returns the reduction mode for the workload, or "" if unknown.
+func (c *Catalog) Reduction(name string) string {
+	workload := c.ByName(name)
+	if workload == nil {
+		return ""
+	}
+	return workload.Reduction
+}
+
+// ParameterDefaults returns the schema-declared defaults for the workload.
+func (c *Catalog) ParameterDefaults(name string) map[string]any {
+	workload := c.ByName(name)
+	if workload == nil {
+		return nil
+	}
+	return workload.parameterDefaults
+}
+
+// ParameterSchema returns the schema property for a workload parameter, and
+// whether the parameter exists.
+func (w *Workload) PropertySchema(field string) (map[string]any, bool) {
+	properties, ok := w.Parameters["properties"].(map[string]any)
+	if !ok {
+		return nil, false
+	}
+	property, ok := properties[field].(map[string]any)
+	return property, ok
+}

coordinator/internal/workloads/schema.go

@@ -0,0 +1,367 @@
+package workloads
+
+import (
+	"fmt"
+	"math"
+	"sort"
+)
+
+// validateSchema checks the strict JSON Schema subset used by SDK manifests.
+// It mirrors what the SDK registry enforces on the Python side: an object
+// schema with additionalProperties=false, typed properties, and the keyword
+// subset the coordinator understands (type, enum, required, minimum/maximum,
+// minLength/maxLength, oneOf, not).
+func validateSchema(workloadName string, schema map[string]any) error {
+	if schema == nil {
+		return fmt.Errorf("workload %q has no parameter schema", workloadName)
+	}
+	if err := validateSchemaNode(workloadName+".parameters_schema", schema); err != nil {
+		return err
+	}
+	if schemaType(schema) != "object" {
+		return fmt.Errorf("workload %q parameter schema must be an object schema", workloadName)
+	}
+	if additional, ok := schema["additionalProperties"].(bool); !ok || additional {
+		return fmt.Errorf("workload %q parameter schema must set additionalProperties=false", workloadName)
+	}
+	properties, ok := schema["properties"].(map[string]any)
+	if !ok {
+		return fmt.Errorf("workload %q parameter schema must declare properties", workloadName)
+	}
+	for name, property := range properties {
+		child, ok := property.(map[string]any)
+		if !ok {
+			return fmt.Errorf("workload %q parameter %q must be a schema object", workloadName, name)
+		}
+		if err := validateSchemaNode(name, child); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func validateSchemaNode(field string, node map[string]any) error {
+	for keyword := range node {
+		switch keyword {
+		case "type", "enum", "required", "minimum", "maximum", "exclusiveMinimum",
+			"exclusiveMaximum", "minLength", "maxLength", "properties",
+			"additionalProperties", "oneOf", "not", "default", "description",
+			"items", "minItems", "maxItems":
+		default:
+			return fmt.Errorf("%s uses unsupported JSON Schema keyword %q", field, keyword)
+		}
+	}
+	if rawType, ok := node["type"]; ok {
+		schemaType, ok := rawType.(string)
+		if !ok {
+			return fmt.Errorf("%s type must be a string", field)
+		}
+		switch schemaType {
+		case "string", "number", "integer", "boolean", "object", "array":
+		default:
+			return fmt.Errorf("%s has unknown type %q", field, schemaType)
+		}
+	}
+	for _, keyword := range []string{"minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum"} {
+		if value, ok := node[keyword]; ok {
+			if number, ok := value.(float64); !ok || math.IsNaN(number) || math.IsInf(number, 0) {
+				return fmt.Errorf("%s %s must be a finite number", field, keyword)
+			}
+		}
+	}
+	for _, keyword := range []string{"minLength", "maxLength", "minItems", "maxItems"} {
+		if value, ok := node[keyword]; ok {
+			if number, ok := value.(float64); !ok || number < 0 || number != math.Trunc(number) {
+				return fmt.Errorf("%s %s must be a non-negative integer", field, keyword)
+			}
+		}
+	}
+	if required, ok := node["required"]; ok {
+		entries, ok := required.([]any)
+		if !ok {
+			return fmt.Errorf("%s required must be an array of strings", field)
+		}
+		for _, entry := range entries {
+			if _, ok := entry.(string); !ok {
+				return fmt.Errorf("%s required must be an array of strings", field)
+			}
+		}
+	}
+	if enums, ok := node["enum"]; ok {
+		entries, ok := enums.([]any)
+		if !ok || len(entries) == 0 {
+			return fmt.Errorf("%s enum must be a non-empty array", field)
+		}
+	}
+	if oneOf, ok := node["oneOf"]; ok {
+		entries, ok := oneOf.([]any)
+		if !ok || len(entries) == 0 {
+			return fmt.Errorf("%s oneOf must be a non-empty array", field)
+		}
+		for index, entry := range entries {
+			child, ok := entry.(map[string]any)
+			if !ok {
+				return fmt.Errorf("%s oneOf[%d] must be a schema object", field, index)
+			}
+			if err := validateSchemaNode(fmt.Sprintf("%s.oneOf[%d]", field, index), child); err != nil {
+				return err
+			}
+		}
+	}
+	if child, ok := node["not"]; ok {
+		not, ok := child.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s not must be a schema object", field)
+		}
+		if err := validateSchemaNode(field+".not", not); err != nil {
+			return err
+		}
+	}
+	if items, ok := node["items"]; ok {
+		child, ok := items.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s items must be a schema object", field)
+		}
+		if err := validateSchemaNode(field+".items", child); err != nil {
+			return err
+		}
+	}
+	if properties, ok := node["properties"]; ok {
+		entries, ok := properties.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s properties must be an object", field)
+		}
+		for name, property := range entries {
+			child, ok := property.(map[string]any)
+			if !ok {
+				return fmt.Errorf("%s property %q must be a schema object", field, name)
+			}
+			if err := validateSchemaNode(field+"."+name, child); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func schemaType(node map[string]any) string {
+	rawType, _ := node["type"].(string)
+	return rawType
+}
+
+// validateParameters checks values against the strict schema subset.
+func validateParameters(workloadName string, schema map[string]any, parameters map[string]any) error {
+	properties, _ := schema["properties"].(map[string]any)
+	for name := range parameters {
+		if _, declared := properties[name]; !declared {
+			return fmt.Errorf("workload %q does not accept parameter %q", workloadName, name)
+		}
+	}
+	required, _ := schema["required"].([]any)
+	for _, name := range required {
+		field, _ := name.(string)
+		if _, present := parameters[field]; !present {
+			return fmt.Errorf("workload %q requires parameter %q", workloadName, field)
+		}
+	}
+	for name, value := range parameters {
+		property, declared := properties[name].(map[string]any)
+		if !declared {
+			continue
+		}
+		if err := validateProperty(workloadName+"."+name, property, value); err != nil {
+			return err
+		}
+	}
+	if oneOf, ok := schema["oneOf"].([]any); ok && len(oneOf) > 0 {
+		if err := validateOneOf(workloadName, oneOf, parameters); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func validateProperty(field string, property map[string]any, value any) error {
+	if enums, ok := property["enum"].([]any); ok {
+		for _, candidate := range enums {
+			if valuesEqual(candidate, value) {
+				return nil
+			}
+		}
+		return fmt.Errorf("%s must be one of the declared enum values", field)
+	}
+	switch schemaType(property) {
+	case "string":
+		text, ok := value.(string)
+		if !ok {
+			return fmt.Errorf("%s must be a string", field)
+		}
+		if minimum, ok := lengthBound(property["minLength"]); ok && len([]rune(text)) < minimum {
+			return fmt.Errorf("%s is shorter than the minimum length", field)
+		}
+		if maximum, ok := lengthBound(property["maxLength"]); ok && len([]rune(text)) > maximum {
+			return fmt.Errorf("%s exceeds the maximum length", field)
+		}
+	case "number", "integer":
+		number, ok := asFloat(value)
+		if !ok {
+			return fmt.Errorf("%s must be a number", field)
+		}
+		if schemaType(property) == "integer" && number != math.Trunc(number) {
+			return fmt.Errorf("%s must be an integer", field)
+		}
+		if minimum, ok := numberBound(property["minimum"]); ok && number < minimum {
+			return fmt.Errorf("%s is below the minimum", field)
+		}
+		if maximum, ok := numberBound(property["maximum"]); ok && number > maximum {
+			return fmt.Errorf("%s exceeds the maximum", field)
+		}
+	case "boolean":
+		if _, ok := value.(bool); !ok {
+			return fmt.Errorf("%s must be a boolean", field)
+		}
+	case "object":
+		child, ok := value.(map[string]any)
+		if !ok {
+			return fmt.Errorf("%s must be an object", field)
+		}
+		properties, _ := property["properties"].(map[string]any)
+		for name := range child {
+			if _, declared := properties[name]; !declared {
+				return fmt.Errorf("%s has undeclared field %q", field, name)
+			}
+		}
+	case "array":
+		items, ok := value.([]any)
+		if !ok {
+			return fmt.Errorf("%s must be an array", field)
+		}
+		if itemSchema, ok := property["items"].(map[string]any); ok {
+			for index, item := range items {
+				if err := validateProperty(fmt.Sprintf("%s[%d]", field, index), itemSchema, item); err != nil {
+					return err
+				}
+			}
+		}
+	case "":
+		// No type keyword: enum-only properties are handled above.
+		return fmt.Errorf("%s has no JSON Schema type", field)
+	}
+	return nil
+}
+
+func validateOneOf(workloadName string, oneOf []any, parameters map[string]any) error {
+	satisfied := 0
+	for _, candidate := range oneOf {
+		option, ok := candidate.(map[string]any)
+		if !ok {
+			continue
+		}
+		if optionSatisfied(option, parameters) {
+			satisfied++
+		}
+	}
+	if satisfied != 1 {
+		return fmt.Errorf("workload %q requires exactly one of the declared parameter alternatives", workloadName)
+	}
+	return nil
+}
+
+func optionSatisfied(option map[string]any, parameters map[string]any) bool {
+	if required, ok := option["required"].([]any); ok {
+		for _, name := range required {
+			field, _ := name.(string)
+			if _, present := parameters[field]; !present {
+				return false
+			}
+		}
+	}
+	if not, ok := option["not"].(map[string]any); ok {
+		if required, ok := not["required"].([]any); ok {
+			for _, name := range required {
+				field, _ := name.(string)
+				if _, present := parameters[field]; present {
+					return false
+				}
+			}
+		}
+	}
+	return true
+}
+
+func lengthBound(value any) (int, bool) {
+	number, ok := value.(float64)
+	if !ok || number != math.Trunc(number) {
+		return 0, false
+	}
+	return int(number), true
+}
+
+func numberBound(value any) (float64, bool) {
+	number, ok := asFloat(value)
+	if !ok || math.IsNaN(number) || math.IsInf(number, 0) {
+		return 0, false
+	}
+	return number, true
+}
+
+func asFloat(value any) (float64, bool) {
+	switch v := value.(type) {
+	case float64:
+		return v, true
+	case float32:
+		return float64(v), true
+	case int:
+		return float64(v), true
+	case int32:
+		return float64(v), true
+	case int64:
+		return float64(v), true
+	}
+	return 0, false
+}
+
+func valuesEqual(left, right any) bool {
+	switch l := left.(type) {
+	case float64:
+		r, ok := right.(float64)
+		return ok && l == r
+	case string:
+		r, ok := right.(string)
+		return ok && l == r
+	case bool:
+		r, ok := right.(bool)
+		return ok && l == r
+	case nil:
+		return right == nil
+	}
+	return false
+}
+
+// schemaDefaults collects the declared default for each property. The UI uses
+// these to pre-fill controls that have no workload-declared UI default.
+func schemaDefaults(schema map[string]any) map[string]any {
+	properties, _ := schema["properties"].(map[string]any)
+	defaults := map[string]any{}
+	for name, property := range properties {
+		child, ok := property.(map[string]any)
+		if !ok {
+			continue
+		}
+		if value, present := child["default"]; present {
+			defaults[name] = value
+		}
+	}
+	return defaults
+}
+
+// SortedFields returns the sorted declared parameter names.
+func SortedFields(schema map[string]any) []string {
+	properties, _ := schema["properties"].(map[string]any)
+	names := make([]string, 0, len(properties))
+	for name := range properties {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	return names
+}

coordinator/internal/workloads/workloads.json

@@ -0,0 +1,602 @@
+{
+  "generated_by": "scimesh workload export",
+  "schema_version": 2,
+  "workloads": [
+    {
+      "capabilities": [
+        "descriptor-batch"
+      ],
+      "description": "Compute a pinned set of RDKit 2D descriptors, one canonical CSV row per input molecule, in deterministic input order.",
+      "determinism": "byte_exact",
+      "enabled": true,
+      "inputs": {
+        "input": {
+          "allow_nested_collections": false,
+          "canonicalizer": "scimesh-tsv-v1",
+          "encoding": "utf-8",
+          "max_bytes": 10737418240,
+          "max_dimensions": [],
+          "max_records": 100000000,
+          "media_type": "text/tab-separated-values",
+          "privacy_class": "project",
+          "ref": "molecule-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "required_columns": [
+              "canonical_smiles",
+              "chembl_id"
+            ]
+          }
+        }
+      },
+      "name": "descriptor-batch",
+      "outputs": {
+        "result": {
+          "allow_nested_collections": false,
+          "canonicalizer": "descriptor-table-v1",
+          "encoding": "utf-8",
+          "max_bytes": 107374182400,
+          "max_dimensions": [],
+          "max_records": 100000000,
+          "media_type": "text/csv",
+          "privacy_class": "project",
+          "ref": "descriptor-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "columns": [
+              "chembl_id",
+              "canonical_smiles",
+              "ExactMolWt",
+              "MolWt",
+              "HeavyAtomMolWt",
+              "HeavyAtomCount",
+              "NumHDonors",
+              "NumHAcceptors",
+              "NumRotatableBonds",
+              "NumHeteroatoms",
+              "NumRadicalElectrons",
+              "NumValenceElectrons",
+              "FractionCSP3",
+              "RingCount",
+              "NumAromaticRings",
+              "NumSaturatedRings",
+              "NumAliphaticRings",
+              "NumAromaticHeterocycles",
+              "NumSaturatedHeterocycles",
+              "NumAliphaticHeterocycles",
+              "NumAromaticCarbocycles",
+              "NumSaturatedCarbocycles",
+              "NumAliphaticCarbocycles",
+              "TPSA",
+              "LabuteASA",
+              "MolLogP",
+              "MolMR",
+              "BalabanJ",
+              "BertzCT",
+              "HallKierAlpha",
+              "Kappa1",
+              "Kappa2",
+              "Kappa3",
+              "Chi0",
+              "Chi1",
+              "Chi0n",
+              "Chi1n",
+              "Chi2n",
+              "Chi3n",
+              "Chi4n",
+              "Chi0v",
+              "Chi1v",
+              "Chi2v",
+              "Chi3v",
+              "Chi4v",
+              "PEOE_VSA1",
+              "PEOE_VSA2",
+              "PEOE_VSA3",
+              "PEOE_VSA4",
+              "PEOE_VSA5",
+              "PEOE_VSA6",
+              "PEOE_VSA7",
+              "PEOE_VSA8",
+              "PEOE_VSA9",
+              "PEOE_VSA10",
+              "PEOE_VSA11",
+              "PEOE_VSA12",
+              "PEOE_VSA13",
+              "PEOE_VSA14",
+              "SMR_VSA1",
+              "SMR_VSA2",
+              "SMR_VSA3",
+              "SMR_VSA4",
+              "SMR_VSA5",
+              "SMR_VSA6",
+              "SMR_VSA7",
+              "SMR_VSA8",
+              "SMR_VSA9",
+              "SMR_VSA10",
+              "SlogP_VSA1",
+              "SlogP_VSA2",
+              "SlogP_VSA3",
+              "SlogP_VSA4",
+              "SlogP_VSA5",
+              "SlogP_VSA6",
+              "SlogP_VSA7",
+              "SlogP_VSA8",
+              "SlogP_VSA9",
+              "SlogP_VSA10",
+              "SlogP_VSA11",
+              "SlogP_VSA12",
+              "NHOHCount",
+              "NOCount"
+            ]
+          }
+        }
+      },
+      "parameters_schema": {
+        "additionalProperties": false,
+        "properties": {
+          "skip_invalid": {
+            "default": true,
+            "description": "Skip rows with invalid SMILES instead of failing",
+            "type": "boolean"
+          }
+        },
+        "type": "object"
+      },
+      "reduction": "ordered-concat",
+      "trust_modes": [
+        "trusted",
+        "untrusted_quorum"
+      ],
+      "ui_elements": [
+        {
+          "default": true,
+          "field": "skip_invalid",
+          "group": "",
+          "help": "Skip rows with invalid SMILES instead of failing the shard.",
+          "label": "Skip invalid molecules",
+          "options": [],
+          "order": 1,
+          "placeholder": "",
+          "widget": "checkbox"
+        }
+      ],
+      "upload_ready": true,
+      "verifier": "exact-artifact@1",
+      "version": "1.0.0"
+    },
+    {
+      "capabilities": [
+        "molwt-filter"
+      ],
+      "description": "Filter molecules by exact RDKit molecular weight, one canonical CSV row per kept input molecule, in deterministic input order.",
+      "determinism": "byte_exact",
+      "enabled": true,
+      "inputs": {
+        "input": {
+          "allow_nested_collections": false,
+          "canonicalizer": "scimesh-tsv-v1",
+          "encoding": "utf-8",
+          "max_bytes": 10737418240,
+          "max_dimensions": [],
+          "max_records": 100000000,
+          "media_type": "text/tab-separated-values",
+          "privacy_class": "project",
+          "ref": "molecule-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "required_columns": [
+              "canonical_smiles",
+              "chembl_id"
+            ]
+          }
+        }
+      },
+      "name": "molwt-filter",
+      "outputs": {
+        "result": {
+          "allow_nested_collections": false,
+          "canonicalizer": "molwt-filtered-table-v1",
+          "encoding": "utf-8",
+          "max_bytes": 107374182400,
+          "max_dimensions": [],
+          "max_records": 100000000,
+          "media_type": "text/csv",
+          "privacy_class": "project",
+          "ref": "molwt-filtered-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "columns": [
+              "chembl_id",
+              "canonical_smiles",
+              "molwt"
+            ]
+          }
+        }
+      },
+      "parameters_schema": {
+        "additionalProperties": false,
+        "properties": {
+          "max_molwt": {
+            "description": "Keep molecules with MolWt <= this value",
+            "minimum": 0,
+            "type": "number"
+          },
+          "min_molwt": {
+            "description": "Keep molecules with MolWt >= this value",
+            "minimum": 0,
+            "type": "number"
+          },
+          "skip_invalid": {
+            "default": true,
+            "description": "Skip rows with invalid SMILES instead of failing",
+            "type": "boolean"
+          }
+        },
+        "type": "object"
+      },
+      "reduction": "ordered-concat",
+      "trust_modes": [
+        "trusted",
+        "untrusted_quorum"
+      ],
+      "ui_elements": [
+        {
+          "default": null,
+          "field": "min_molwt",
+          "group": "",
+          "help": "Keep molecules with MolWt at least this value. Optional.",
+          "label": "Minimum molecular weight",
+          "options": [],
+          "order": 1,
+          "placeholder": "e.g. 100",
+          "widget": "number"
+        },
+        {
+          "default": null,
+          "field": "max_molwt",
+          "group": "",
+          "help": "Keep molecules with MolWt at most this value. Optional.",
+          "label": "Maximum molecular weight",
+          "options": [],
+          "order": 2,
+          "placeholder": "e.g. 600",
+          "widget": "number"
+        },
+        {
+          "default": true,
+          "field": "skip_invalid",
+          "group": "",
+          "help": "Skip rows with invalid SMILES instead of failing the shard.",
+          "label": "Skip invalid molecules",
+          "options": [],
+          "order": 3,
+          "placeholder": "",
+          "widget": "checkbox"
+        }
+      ],
+      "upload_ready": true,
+      "verifier": "exact-artifact@1",
+      "version": "1.0.0"
+    },
+    {
+      "capabilities": [
+        "similarity-graph"
+      ],
+      "description": "Exact sparse Tanimoto similarity graph over deterministic block pairs with a duplicate-safe, coverage-checked merge.",
+      "determinism": "byte_exact",
+      "enabled": true,
+      "inputs": {
+        "input": {
+          "allow_nested_collections": false,
+          "canonicalizer": "scimesh-tsv-v1",
+          "encoding": "utf-8",
+          "max_bytes": 10737418240,
+          "max_dimensions": [],
+          "max_records": 100000000,
+          "media_type": "text/tab-separated-values",
+          "privacy_class": "project",
+          "ref": "molecule-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "required_columns": [
+              "canonical_smiles",
+              "chembl_id"
+            ]
+          }
+        }
+      },
+      "name": "similarity-graph",
+      "outputs": {
+        "result": {
+          "allow_nested_collections": false,
+          "canonicalizer": "similarity-edge-table-v1",
+          "encoding": "utf-8",
+          "max_bytes": 107374182400,
+          "max_dimensions": [],
+          "max_records": 1000000000,
+          "media_type": "text/csv",
+          "privacy_class": "project",
+          "ref": "similarity-edge-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "columns": [
+              "source_id",
+              "target_id",
+              "similarity"
+            ]
+          }
+        }
+      },
+      "parameters_schema": {
+        "additionalProperties": false,
+        "properties": {
+          "block_size": {
+            "minimum": 1,
+            "type": "integer"
+          },
+          "max_rows": {
+            "minimum": 1,
+            "type": "integer"
+          },
+          "threshold": {
+            "maximum": 1,
+            "minimum": 0,
+            "type": "number"
+          },
+          "threshold_direction": {
+            "enum": [
+              "greater",
+              "less"
+            ]
+          }
+        },
+        "required": [
+          "threshold"
+        ],
+        "type": "object"
+      },
+      "reduction": "ordered-concat",
+      "trust_modes": [
+        "trusted",
+        "untrusted_quorum"
+      ],
+      "ui_elements": [
+        {
+          "default": null,
+          "field": "threshold",
+          "group": "",
+          "help": "Minimum (greater) or maximum (less) edge similarity. Required.",
+          "label": "Similarity threshold",
+          "options": [],
+          "order": 1,
+          "placeholder": "",
+          "widget": "number"
+        },
+        {
+          "default": "greater",
+          "field": "threshold_direction",
+          "group": "",
+          "help": "Whether to keep edges above (greater) or below (less) the threshold.",
+          "label": "Direction",
+          "options": [
+            "greater",
+            "less"
+          ],
+          "order": 2,
+          "placeholder": "",
+          "widget": "select"
+        },
+        {
+          "default": 100,
+          "field": "block_size",
+          "group": "",
+          "help": "Deterministic block size for pair sharding.",
+          "label": "Block size",
+          "options": [],
+          "order": 3,
+          "placeholder": "",
+          "widget": "number"
+        }
+      ],
+      "upload_ready": false,
+      "verifier": "exact-artifact@1",
+      "version": "1.0.0"
+    },
+    {
+      "capabilities": [
+        "similarity-search"
+      ],
+      "description": "Exact top-k Tanimoto molecular similarity search over deterministic TSV shards with a bounded merge.",
+      "determinism": "byte_exact",
+      "enabled": true,
+      "inputs": {
+        "input": {
+          "allow_nested_collections": false,
+          "canonicalizer": "scimesh-tsv-v1",
+          "encoding": "utf-8",
+          "max_bytes": 10737418240,
+          "max_dimensions": [],
+          "max_records": 100000000,
+          "media_type": "text/tab-separated-values",
+          "privacy_class": "project",
+          "ref": "molecule-table@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "required_columns": [
+              "canonical_smiles",
+              "chembl_id"
+            ]
+          }
+        }
+      },
+      "name": "similarity-search",
+      "outputs": {
+        "result": {
+          "allow_nested_collections": false,
+          "canonicalizer": "scimesh-search-result-v1",
+          "encoding": "utf-8",
+          "max_bytes": 1073741824,
+          "max_dimensions": [],
+          "max_records": 100000,
+          "media_type": "text/csv",
+          "privacy_class": "project",
+          "ref": "similarity-search-result@1",
+          "retention_class": "durable",
+          "streaming": false,
+          "validator": "delimited-table@1",
+          "validator_configuration": {
+            "columns": [
+              "rank",
+              "chembl_id",
+              "canonical_smiles",
+              "similarity"
+            ]
+          }
+        }
+      },
+      "parameters_schema": {
+        "additionalProperties": false,
+        "oneOf": [
+          {
+            "not": {
+              "required": [
+                "query_smiles"
+              ]
+            },
+            "required": [
+              "query_id"
+            ]
+          },
+          {
+            "not": {
+              "required": [
+                "query_id"
+              ]
+            },
+            "required": [
+              "query_smiles"
+            ]
+          }
+        ],
+        "properties": {
+          "max_rows": {
+            "minimum": 1,
+            "type": "integer"
+          },
+          "progress_every": {
+            "minimum": 0,
+            "type": "integer"
+          },
+          "query_id": {
+            "maxLength": 200,
+            "minLength": 1,
+            "type": "string"
+          },
+          "query_smiles": {
+            "maxLength": 200,
+            "minLength": 1,
+            "type": "string"
+          },
+          "threshold": {
+            "maximum": 1,
+            "minimum": 0,
+            "type": "number"
+          },
+          "threshold_direction": {
+            "enum": [
+              "greater",
+              "less"
+            ]
+          },
+          "top_k": {
+            "minimum": 1,
+            "type": "integer"
+          }
+        },
+        "type": "object"
+      },
+      "reduction": "top-k",
+      "trust_modes": [
+        "trusted",
+        "untrusted_quorum"
+      ],
+      "ui_elements": [
+        {
+          "default": null,
+          "field": "query_id",
+          "group": "",
+          "help": "ChEMBL id of the query molecule. Provide exactly one of id or SMILES.",
+          "label": "Query molecule id",
+          "options": [],
+          "order": 1,
+          "placeholder": "",
+          "widget": "text"
+        },
+        {
+          "default": null,
+          "field": "query_smiles",
+          "group": "",
+          "help": "SMILES of the query molecule. Provide exactly one of id or SMILES.",
+          "label": "Query molecule SMILES",
+          "options": [],
+          "order": 2,
+          "placeholder": "",
+          "widget": "text"
+        },
+        {
+          "default": 20,
+          "field": "top_k",
+          "group": "",
+          "help": "Number of most similar molecules to keep per shard (global merge keeps the best of these).",
+          "label": "Top k",
+          "options": [],
+          "order": 3,
+          "placeholder": "",
+          "widget": "number"
+        },
+        {
+          "default": "greater",
+          "field": "threshold_direction",
+          "group": "",
+          "help": "Keep molecules with similarity greater or less than the threshold.",
+          "label": "Direction",
+          "options": [
+            "greater",
+            "less"
+          ],
+          "order": 4,
+          "placeholder": "",
+          "widget": "select"
+        },
+        {
+          "default": null,
+          "field": "threshold",
+          "group": "",
+          "help": "Optional similarity bound: results are filtered to this direction.",
+          "label": "Similarity threshold",
+          "options": [],
+          "order": 5,
+          "placeholder": "e.g. 0.8",
+          "widget": "number"
+        }
+      ],
+      "upload_ready": true,
+      "verifier": "exact-artifact@1",
+      "version": "1.0.0"
+    }
+  ]
+}

coordinator/scripts/demo-ui.sh

@@ -28,10 +28,33 @@ case "$demo_dir" in
   /*) ;;
   *) demo_dir="$coordinator_dir/$demo_dir" ;;
 esac
-worker_bin=${SCIMESH_WORKER_BIN:-"$repo_dir/.venv/bin/scimesh-worker"}
+agent_bin=${SCIMESH_AGENT_BIN:-"$coordinator_dir/bin/worker-agent"}
 pid_file="$demo_dir/workers.pids"
 logs_dir="$demo_dir/logs"
 
+# The built MkDocs site is mounted into the demo coordinator so the UI can
+# serve it at /ui/docs/. When site/ is missing (make docs), the docs route
+# shows a build hint instead.
+docs_compose_file="$demo_dir/docker-compose.docs.yml"
+docs_compose_files=""
+
+prepare_docs_override() {
+  mkdir -p "$demo_dir"
+  if [[ -d "$repo_dir/site" ]]; then
+    cat > "$docs_compose_file" <<DOCS_OVERRIDE_EOF
+services:
+  coordinator:
+    volumes:
+      - $repo_dir/site:/site:ro
+    environment:
+      SCIMESH_DOCS_DIR: /site
+DOCS_OVERRIDE_EOF
+    docs_compose_files="-f $docs_compose_file"
+  else
+    docs_compose_files=""
+  fi
+}
+
 compose() {
   POSTGRES_PORT="$postgres_port" \
   COORDINATOR_PORT="$coordinator_port" \
@@ -46,7 +69,8 @@ compose() {
   docker compose -p "$project" \
     -f "$coordinator_dir/docker-compose.yml" \
     -f "$coordinator_dir/docker-compose.users.yml" \
-    -f "$coordinator_dir/docker-compose.monitoring.yml" "$@"
+    -f "$coordinator_dir/docker-compose.monitoring.yml" \
+    $docs_compose_files "$@"
 }
 
 stop_workers() {
@@ -55,13 +79,23 @@ stop_workers() {
     [[ "$pid" =~ ^[0-9]+$ ]] || continue
     command_line=$(ps -p "$pid" -o args= 2>/dev/null || true)
     # Never kill a recycled PID or a worker launched outside this demo.
-    if [[ "$command_line" == *"$demo_dir/worker-"* ]]; then
+    if [[ "$command_line" == *"worker-agent"* ]]; then
       kill "$pid" 2>/dev/null || true
     fi
   done < "$pid_file"
   rm -f "$pid_file"
 }
 
+build_agent() {
+  if [[ ! -x "$agent_bin" ]]; then
+    echo "Building the Go worker agent..." >&2
+    make -C "$coordinator_dir" agent >&2 || {
+      echo "Failed to build the Go worker agent." >&2
+      exit 2
+    }
+  fi
+}
+
 wait_for_coordinator() {
   local attempt=0
   until curl --fail --silent --show-error "http://localhost:$coordinator_port/health" >/dev/null; do
@@ -116,15 +150,12 @@ wait_for_workers() {
 }
 
 start() {
+  prepare_docs_override
   if ! [[ "$workers" =~ ^[1-9][0-9]*$ ]]; then
     echo "DEMO_WORKERS must be a positive integer (got $workers)." >&2
     exit 2
   fi
-  if [[ ! -x "$worker_bin" ]]; then
-    echo "Reference worker not found: $worker_bin" >&2
-    echo "Create it first from the repository root: python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'" >&2
-    exit 2
-  fi
+  build_agent
   command -v docker >/dev/null || { echo "Docker is required." >&2; exit 2; }
   command -v curl >/dev/null || { echo "curl is required." >&2; exit 2; }
 
@@ -137,14 +168,21 @@ start() {
   wait_for_userservice
 
   : > "$pid_file"
+  task_runner="[\"$repo_dir/.venv/bin/python\",\"-m\",\"scimesh.worker.task\"]"
   for index in $(seq 1 "$workers"); do
     work_dir="$demo_dir/worker-$index"
     mkdir -p "$work_dir"
-    SCIMESH_COORDINATOR_URL="http://localhost:$coordinator_port" \
-    SCIMESH_BEARER_TOKEN="$worker_token" \
-    "$worker_bin" \
-      --worker-name "demo-worker-$index" \
-      --work-dir "$work_dir" \
+    COORDINATOR_URL="http://localhost:$coordinator_port" \
+    WORKER_AUTH_TOKEN="$worker_token" \
+    WORKER_NAME="demo-worker-$index" \
+    WORK_DIR="$work_dir" \
+    CPU_COUNT=1 \
+    MEMORY_MB=1024 \
+    POLL_INTERVAL=0.5s \
+    REQUEST_TIMEOUT=15s \
+    HEARTBEAT_INTERVAL=15s \
+    TASK_RUNNER="$task_runner" \
+    "$agent_bin" \
       >"$logs_dir/worker-$index.log" 2>&1 &
     echo "$!" >> "$pid_file"
   done
@@ -159,11 +197,11 @@ SciMesh manual demo is ready.
   Userservice: http://localhost:$userservice_port
   Grafana:     http://localhost:$grafana_port   (anonymous view; admin/${GRAFANA_PASSWORD:-admin} to edit)
   Prometheus:  http://localhost:$prometheus_port
-  Workers:     $workers local reference workers
+  Workers:     $workers Go worker agents (Python task execution)
 
 Sign in with the admin above, or register a new account from the login page.
 The admin sees every job; a plain user sees only their own. Upload a small
-ChEMBL TSV through “New similarity search”, then watch the job page update.
+ChEMBL TSV through the "New computation" form, then watch the job page update.
 Worker logs are in $logs_dir. Stop everything with:
 
   make demo-down

docs/openapi.yaml

@@ -96,10 +96,13 @@ paths:
       description: >
         multipart/form-data. The text fields (`workload`, `parameters`,
         `chunk_rows`, `max_rows`) MUST precede the `file` part: the file is streamed, not
-        buffered, so the fields have to be parsed before it arrives. Currently
-        only `similarity-search` with `parameters.query_smiles` is accepted.
-        When every shard succeeds, the coordinator merges their candidates into
-        one final CSV; distributed graph planning is not implemented.
+        buffered, so the fields have to be parsed before it arrives. The
+        workload must be an enabled, upload-ready entry of the embedded SDK
+        workload catalog, and `parameters` must satisfy its declared JSON
+        schema. When every shard succeeds, the coordinator reduces the partial
+        results (`top-k` workloads merge exactly; `ordered-concat` workloads
+        concatenate in shard order) into one final CSV; distributed graph
+        planning is not implemented.
       requestBody:
         required: true
         content:

docs/scimesh-sdk-contract.md

@@ -0,0 +1,556 @@
+# SciMesh Workload SDK contract
+
+**Status:** contract `0.1`. The Python `core-batch-v1` foundation is implemented
+in `scimesh.sdk`; dynamic, streaming, accelerator, gang, side-effect, and
+coordinator protocol-v2 behavior remains a normative target. Normative words
+**MUST**, **MUST NOT**, **SHOULD**, and **MAY** apply to an implementation only
+when it advertises the affected profile or feature.
+
+This document defines the compatibility boundary for approved SciMesh workload
+packages. The sequencing and unresolved product decisions remain in
+[`scimesh-sdk-roadmap.md`](scimesh-sdk-roadmap.md). The production coordinator
+wire compatibility profile remains
+[`ctx-07-distributed-workload-protocol.md`](ctx-07-distributed-workload-protocol.md).
+
+## 1. Scope and invariants
+
+The SDK must express current molecular workloads and future batch, iterative,
+streaming, optimization, simulation, ML, image, engineering, and accelerator
+workloads through installed, allowlisted packages. "Universal" means that a
+workload can declare its dataflow, resources, execution semantics, and result
+validation; it does not mean users may submit arbitrary executable code.
+
+Every implementation MUST preserve these invariants:
+
+1. The coordinator owns Jobs, workflow state, Tasks, Attempts, Leases, resource
+   assignments, and durable Artifact metadata.
+2. A Worker Agent executes only an installed package and entry point whose
+   digest is enabled by an administrator.
+3. Scientific code never receives PostgreSQL credentials, worker identity
+   credentials, or unrestricted coordinator credentials.
+4. Every input and output crosses a typed artifact port. Worker-local paths are
+   attempt-scoped implementation details and never become durable references.
+5. A Task reports success only after all declared outputs are durable and its
+   verifier policy has produced an admissible decision.
+6. Resource allocation is explicit and lease-fenced. A worker MUST NOT execute
+   a Task without reserving its complete declared resource set.
+7. Unknown fields, unsupported versions, undeclared outputs, non-finite numeric
+   values, and limit violations fail closed.
+8. No verifier, reducer, or trust policy may silently downgrade to a weaker
+   mode.
+
+## 2. Version and identity model
+
+The following versions are independent and MUST be recorded in each Job and
+output provenance manifest:
+
+| Identifier | Meaning |
+| --- | --- |
+| `sdk_api_version` | Python authoring API compatibility. |
+| `protocol_version` | Coordinator/Worker wire contract. |
+| `workload.name` + `workload.version` | Scientific behavior and planner contract. |
+| `manifest_schema_version` | Shape of the workload manifest. |
+| `workflow_schema_version` | Shape of workflow/stage/task plans. |
+| `artifact_schema.name` + `version` | Logical data format. |
+| `verifier.name` + `version` | Result-acceptance semantics. |
+| `environment.digest` | Exact executable environment or image. |
+
+Versions use explicit compatibility ranges. The coordinator enables a workload
+only when there is a non-empty intersection among coordinator protocol, Worker
+runtime, SDK API, workload package, and verifier versions. A missing or unknown
+version is never interpreted as "latest". Jobs remain pinned to the resolved
+versions even after an administrator upgrades the installed package.
+
+### 2.1 Feature negotiation and conformance profiles
+
+Universality does not require every deployment to enable every execution mode.
+The protocol negotiates explicit feature IDs; a workload declares required and
+optional features, and coordinator, Worker, and verifier runtimes advertise
+supported versions. Planning fails before Job creation when a required feature
+is absent.
+
+| Profile | Required feature set |
+| --- | --- |
+| `core-batch-v1` | Typed parameters/artifacts, static map/reduce DAG, subprocess runner, exact verifier, CPU/memory/scratch reservations. |
+| `dynamic-v1` | Transactional expansion manifests and bounded loop controllers. |
+| `stream-v1` | Partition offsets, windows, checkpoints, backpressure, and declared delivery guarantees. |
+| `accelerator-v1` | Generic device inventory, fenced allocation, visibility/isolation, and device failure codes. |
+| `gang-v1` | Atomic multi-Agent reservation, rendezvous, group lease, and fail-all semantics. |
+| `side-effect-v1` | Idempotency/audit/compensation and scoped external credentials. |
+
+Example feature IDs include `artifact-collections@1`, `dynamic-expansion@1`,
+`bounded-loops@1`, `stream-checkpoints@1`, `gpu-exclusive@1`, `gpu-mig@1`,
+`gang-leases@1`, and `numeric-verifier@1`. Optional features may select a
+manifest-declared fallback such as CPU execution; they MUST NOT change output
+schema or verification semantics unless the fallback is a separately versioned
+workflow variant.
+
+## 3. Workload package and manifest
+
+An SDK workload is an installed Python distribution containing:
+
+- one or more versioned workload manifests;
+- explicit Python entry points for planner, runners, reducers, and verifiers;
+- artifact and parameter schemas;
+- pinned environment/image metadata;
+- golden fixtures and conformance tests;
+- provenance and license metadata.
+
+Discovery MUST use configured package entry points and an administrator
+allowlist. It MUST NOT import modules from job parameters, uploaded archives, or
+user-provided filesystem paths.
+
+Illustrative manifest shape:
+
+```yaml
+manifest_schema_version: 1
+sdk_api: ">=1.0,<2.0"
+protocol: ">=2,<3"
+workload:
+  name: descriptor-batch
+  version: 1.0.0
+  description: Pinned RDKit 2D descriptors
+package:
+  distribution: scimesh-descriptors
+  digest: sha256:...
+  signature: cosign-or-project-signature-reference
+environment:
+  kind: oci
+  digest: sha256:...
+parameters_schema: schemas/parameters-v1.json
+workflow: workflows/default-v1.yaml
+inputs:
+  molecules: {schema: molecule-table@1, cardinality: one}
+outputs:
+  descriptors: {schema: descriptor-table@1, cardinality: one}
+determinism: byte_exact
+trust_modes: [trusted, verified, untrusted_quorum]
+verifier: {name: exact-artifact, version: 1}
+limits:
+  max_input_bytes: 10737418240
+  max_tasks: 10000
+  max_output_bytes: 10737418240
+capabilities: [descriptor-batch]
+```
+
+The manifest MUST declare canonical hyphenated workload names, parameter
+schema, external ports, workflow, determinism, supported trust modes, verifier,
+resource bounds, output-growth bounds, environment, and capabilities.
+
+## 4. Workflow model
+
+### 4.1 Workflow graph
+
+`WorkflowSpec` defines versioned stages and artifact edges. Its persisted form
+is a DAG. Iteration is represented by a bounded controller that materializes a
+new DAG segment for each iteration; persisted task dependencies never contain a
+cycle.
+
+```yaml
+workflow_schema_version: 1
+id: default
+inputs: [molecules]
+stages:
+  - id: partition
+    kind: plan
+    runner: descriptor.partition:v1
+  - id: calculate
+    kind: map
+    needs: [partition]
+    runner: descriptor.calculate:v1
+  - id: combine
+    kind: reduce
+    needs: [calculate]
+    reducer: descriptor.combine:v1
+  - id: verify
+    kind: verify
+    needs: [combine]
+outputs: [descriptors]
+failure_policy: fail_fast
+```
+
+Stage IDs are stable within a workflow version. Valid stage kinds are initially
+`plan`, `map`, `reduce`, `verify`, `loop-controller`, `stream`, `service`, and
+`side-effect`. Runtimes MAY add kinds only through a new workflow schema
+version.
+
+### 4.2 Stage contract
+
+A `StageSpec` MUST declare:
+
+- stable ID, kind, entry-point identity, and dependencies;
+- named input/output ports and their artifact schemas/cardinality;
+- parameter projection from immutable Job parameters;
+- resource and execution profiles;
+- retry, timeout, checkpoint, cancellation, and failure policies;
+- fan-out/fan-in bounds and ordering semantics;
+- verifier and trust requirements where stage outputs affect acceptance;
+- cacheability, side effects, and network/secrets policy.
+
+Dynamic fan-out requires an accepted `ExpansionManifest` containing stable child
+keys, bounded child count, TaskSpecs, artifact bindings, and a digest. The
+coordinator validates and persists the entire expansion transactionally. A
+retry producing a different expansion digest is a conflict, not a replacement.
+
+### 4.3 Bounded iteration
+
+`LoopSpec` supports training epochs, optimization, adaptive sampling, MD
+segments, and convergence algorithms:
+
+```yaml
+loop:
+  state_schema: optimizer-state@1
+  max_iterations: 100
+  max_wall_seconds: 86400
+  body_workflow: optimize-step@2
+  continue_when: verifier-entry-point-reference
+  checkpoint_every: 5
+  on_limit: fail   # fail | accept-best | return-inconclusive
+```
+
+The loop controller is trusted orchestration code from the workload package.
+It consumes immutable prior-state and evaluation artifacts and emits an
+immutable next-iteration expansion. It MUST NOT mutate completed Tasks or reuse
+an Attempt directory. Termination is bounded by iterations, wall time, cost,
+and output growth. The condition and best-result selection are versioned and
+auditable.
+
+### 4.4 Streaming profile
+
+Streaming is an explicit profile rather than an indefinitely running batch
+Task. A `StreamSpec` declares source identity, partitioning, offset/checkpoint
+schema, event-time or processing-time windows, watermark behavior, backpressure
+limit, delivery guarantee, idle/terminal condition, and output compaction.
+
+Supported guarantees are `at_least_once` initially and, only where the source
+and sink support transactional offsets, `exactly_once`. A checkpoint commits
+source offsets only after corresponding output artifacts are durable. Stream
+processors MUST be restartable from a sealed checkpoint artifact. An unbounded
+stream produces versioned window/result artifacts and does not hold one Task
+lease forever.
+
+### 4.5 Side effects and human interaction
+
+Stages controlling instruments or writing external systems are trusted-only.
+They require an idempotency key, declared target, credential scope, audit event,
+timeout, and compensation/manual-recovery policy. Retries are disabled unless
+the stage proves idempotency. Human approval is modeled as a coordinator state
+transition with an authenticated decision record, never as a worker waiting
+indefinitely while holding resources.
+
+## 5. Task and artifact contracts
+
+`TaskSpec` is the concrete unit leased to a Worker Agent:
+
+```yaml
+schema_version: 1
+workload: descriptor-batch@1.2.0
+package_digest: sha256:...
+manifest_digest: ...
+trust_mode: trusted
+task_key: calculate/shard-000042
+stage_id: calculate
+parameters: {...validated JSON...}
+inputs:
+  molecules:
+    collection: ordered
+    artifacts: [{artifact_id: uuid, sha256: ..., schema: molecule-table@1}]
+expected_outputs:
+  descriptors: {schema: descriptor-table@1, cardinality: one}
+resources: {profile: cpu-medium@1}
+execution: {profile: python-process@1}
+```
+
+`task_key` is deterministic within a workflow expansion. The coordinator adds
+its durable Task ID, Attempt number, Lease, and generated download/upload URLs.
+Plans contain artifact IDs and checksums, never external credentials or local
+paths.
+
+### 5.1 Artifact schemas and collections
+
+Each port references an `ArtifactSchema` defining logical type, schema version,
+media type, encoding, cardinality, maximum bytes/records/dimensions, streaming
+support, canonicalizer, validation entry point, and privacy/retention class.
+
+Collections are `single`, `ordered`, `keyed`, or `set`:
+
+- `ordered` preserves declared order and is included in the collection digest;
+- `keyed` requires unique canonical string keys;
+- `set` canonicalizes by artifact identity and forbids duplicates;
+- nested collections require explicit schema permission and depth limits.
+
+Protocol v1 compatibility uses one immutable composite-manifest artifact to
+represent a collection. A later protocol may persist collection edges directly.
+
+### 5.2 Output and provenance manifest
+
+Before completion, a runner uploads an `OutputManifest` listing every declared
+output artifact, checksum, schema, size, record/dimension summary, metrics, and
+provenance. Provenance includes resolved versions, package/environment and
+manifest digests, Worker runtime, allocated resource IDs, parameters digest,
+input collection digest, timestamps, random seed where applicable, and
+checkpoint lineage.
+
+Unexpected ports, missing required outputs, extra artifacts, schema failures,
+or limit violations reject the Attempt. Logs and checkpoints are separate
+artifact kinds and never satisfy scientific output ports.
+
+### 5.3 Locality and cache
+
+Artifacts remain coordinator-owned even when cached. Worker Agents MAY maintain
+a content-addressed read cache verified by checksum. Cache entries carry size,
+last-use, schema, environment sensitivity, and retention class; eviction never
+deletes the durable coordinator copy.
+
+Task matching MAY score data locality after eligibility and fairness checks.
+It MUST NOT weaken trust, resource, lease, or ownership constraints. Large input
+staging occurs before execution timeout starts, with a bounded staging lease.
+Cache hits are verified before use; private artifacts are isolated by tenant or
+encrypted policy.
+
+## 6. Resource and execution contracts
+
+### 6.1 Resource inventory and requests
+
+A Worker Agent advertises versioned, periodically refreshed inventory:
+
+```yaml
+agent:
+  cpu:
+    logical_cores: 32
+    allocatable_cores: 28
+    architecture: x86_64
+  memory_mb: 131072
+  scratch_mb: 1000000
+  accelerators:
+    - kind: gpu
+      vendor: nvidia
+      device_uuid: GPU-...
+      model: A100
+      memory_mb: 81920
+      compute_capability: "8.0"
+      partitioning: [exclusive, mig]
+      topology_group: nvlink-0
+  runtime:
+    os: linux
+    container_runtime: ...
+    driver_versions: {...}
+    environment_digests: [sha256:...]
+```
+
+A `ResourceRequirements` request distinguishes minimums, preferred values, and
+hard constraints. Core fields are CPU cores, memory, scratch, accelerator count
+and kind, device memory, architecture/capability, exclusivity, topology,
+network/interconnect class, environment digest, estimated input/output bytes,
+and maximum duration.
+
+Allocation is atomic and lease-fenced. A Task cannot start until the Agent has
+confirmed the reservation token. Resources are released only after the process
+group exits and attempt cleanup completes. Device IDs and secrets are not part
+of scientific parameters.
+
+### 6.2 CPU concurrency
+
+One machine runs one Worker Agent with `max_concurrency` execution slots. Each
+Task separately requests `cpu_cores`; the sum of reservations cannot exceed
+allocatable capacity. `ExecutionProfile` declares:
+
+- `process_model`: `single`, `process_pool`, `thread_pool`, or `external_runtime`;
+- maximum worker processes and threads per process;
+- OpenMP/BLAS/native-library thread limits;
+- affinity/NUMA preference when required;
+- whether nested parallelism is prohibited (default) or explicitly bounded.
+
+CPU-bound Python SHOULD use isolated processes. Threads remain valid for I/O or
+native extensions that release the GIL. Independent task heartbeat supervisors
+remain outside scientific subprocesses. Draining stops claims first, maintains
+active leases, then checkpoints/cancels at the declared deadline.
+
+### 6.3 GPU and accelerator allocation
+
+GPU requests can specify `exclusive_device`, `fractional`, or `partition`
+(including MIG-like partitions); runtimes MUST advertise which modes they can
+enforce. Requests may require multiple devices in one topology group. The
+coordinator performs generic eligibility and gang selection; the Agent owns
+device isolation and sets backend-specific visibility variables.
+
+The Agent MUST fence allocation by device UUID/partition ID, validate driver,
+runtime and environment compatibility, prevent incompatible sharing, monitor
+device health and memory, and terminate the whole process group on lease loss.
+OOM, device reset, ECC failure, and unavailable-device errors have distinct
+sanitized codes and workload-declared retry policies. Preemptible GPU Tasks need
+an explicit compatible checkpoint contract.
+
+The workload owns CPU/GPU implementation, batching, mixed precision, seeds,
+algorithm determinism, memory strategy, and scientific parity tests. The
+coordinator contains no CUDA calls or domain formulas. A GPU result follows the
+same output schema and verifier semantics as any CPU result.
+
+### 6.4 Multi-node and gang execution
+
+`GangSpec` expresses MPI, multi-node training, and tightly coupled simulations:
+
+```yaml
+gang:
+  replicas: 4
+  per_replica_resources: {cpu_cores: 8, gpu_count: 1, memory_mb: 32768}
+  topology: {same_fabric: true, min_bandwidth_class: infiniband}
+  rendezvous: worker-agent-managed
+  failure_mode: fail_all
+```
+
+The coordinator atomically reserves all replicas or none, then issues one gang
+lease and per-replica fenced assignments. Worker Agents establish a scoped
+rendezvous channel without exposing general coordinator credentials. A failed,
+expired, or cancelled replica invalidates the gang according to `failure_mode`;
+partial success cannot complete the stage. Gang retries use a new Attempt and
+new rendezvous credentials.
+
+## 7. Verification and trust
+
+Verifier decisions are `accepted`, `rejected`, or `inconclusive`. Only
+`accepted` can satisfy a stage. Evidence is a bounded, sanitized artifact tied
+to input/output and verifier digests.
+
+| Verifier | Intended semantics |
+| --- | --- |
+| `ExactArtifactVerifier` | Whole-file or declared collection digest equality. |
+| `CanonicalRecordVerifier` | Parse, validate, normalize, order, and serialize through a versioned canonicalizer. |
+| `NumericToleranceVerifier` | Structured element/aggregate comparison with declared absolute, relative, ULP, NaN, and shape policies. |
+| `StatisticalVerifier` | Repeated seeded evidence and versioned statistical acceptance criteria. |
+| `DomainSpecificVerifier` | Workload-owned invariants, constraints, objective bounds, or reference checks. |
+| `TrustedWorkerPolicy` | Accept only from allowed trust/environment attestations; still validate schema and bounds. |
+
+The current whole-artifact SHA quorum maps only to `ExactArtifactVerifier`.
+Canonical, numeric, stochastic, optimization, and side-effecting workloads MUST
+declare an appropriate verifier/trust combination. Reducers consume only
+accepted partial outputs and MUST detect missing, duplicate, conflicting, or
+inconclusive inputs.
+
+Quorum candidates MUST be coordinator-authenticated envelopes with unique
+Attempt/candidate identity and an owner identity. A verifier counts at most one
+vote per owner. It also receives a coordinator-owned binding for workload,
+task, package/manifest/environment, parameters, and input-collection digests;
+outputs from another job or code pin are invalid even when their result bytes
+match.
+
+## 8. Failure, retry, cancellation, and checkpoint semantics
+
+Every failure has a stable sanitized code, category (`input`, `scientific`,
+`resource`, `infrastructure`, `lease`, `verification`, or `policy`), retryability,
+and optional bounded evidence reference. Raw tracebacks and private paths remain
+local.
+
+- Retries create a new Attempt directory and resource lease; they never mutate
+  prior artifacts.
+- Idempotent completion accepts the identical output manifest for the same
+  Attempt; conflicting manifests are rejected.
+- Speculative execution, when enabled, creates multiple Attempts but commits at
+  most one accepted result and cancels the rest.
+- Lease loss immediately fences upload/completion and terminates execution.
+- Cancellation propagates to process groups/gangs and disposes uncommitted
+  staging artifacts.
+- Retry budgets may be per Task, Stage, Loop, and Job; the strictest exhausted
+  budget wins.
+- Checkpoint resume requires matching workload, schema, environment, and
+  checkpoint compatibility versions. Otherwise execution restarts cleanly.
+- Side-effecting retries require an idempotency record or explicit operator
+  recovery.
+
+Workflow failure policies are `fail_fast`, `continue_independent`,
+`allow_partial` (only with an output schema/verifier that defines partial
+results), and `compensate`. A failed reducer/verifier never leaves a Job marked
+completed.
+
+## 9. Package security and permissions
+
+Installation and job submission are separate authorities. Only administrators
+or managed policy may install, sign, approve, enable, upgrade, or revoke a
+workload package. A job references an enabled immutable package digest.
+
+Package policy MUST cover signature trust roots, dependency/image scanning,
+license/provenance records, supported platforms, vulnerability/revocation
+status, and reproducible build evidence. Upgrade does not rewrite running or
+historical Jobs.
+
+Each stage declares least-privilege permissions:
+
+- network: none, coordinator-artifacts-only, allowlisted egress, or trusted;
+- filesystem: read-only inputs, attempt scratch, declared outputs;
+- secrets: named scoped handles, never raw values in Task parameters;
+- subprocess: denied by default except the installed runner/runtime contract;
+- devices and host interfaces: only allocated resources.
+
+Worker Agents enforce permissions through the strongest available OS/container
+sandbox and report the enforcement profile. A workload requiring unavailable
+isolation is ineligible rather than silently unsandboxed.
+
+## 10. SDK interfaces and conformance
+
+The Python API exposes protocols equivalent to:
+
+```python
+class Planner(Protocol):
+    def validate(self, request: JobRequest) -> ValidatedJob: ...
+    def plan(self, job: ValidatedJob, context: PlanningContext) -> WorkflowPlan: ...
+
+class Runner(Protocol):
+    def run(self, context: TaskContext) -> OutputManifest: ...
+
+class Reducer(Protocol):
+    def reduce(self, context: ReduceContext) -> OutputManifest: ...
+
+class Verifier(Protocol):
+    def verify(self, context: VerifyContext, candidates: CandidateOutputs) -> VerificationDecision: ...
+```
+
+Concrete public value objects are immutable, typed, JSON-safe, strict about
+unknown fields, and canonically serialized inside versioned wire contracts.
+Scientific cores SHOULD remain callable without a coordinator so the same
+implementation powers local and distributed adapters.
+
+The shipped `LocalCoreBatchExecutor` is a trusted in-process conformance
+harness, not the `core-batch-v1` production isolation boundary. It rejects
+restricted-network, parallel-process/thread, accelerator, secret, checkpoint,
+retry, gang, and advanced-stage declarations. Subprocess isolation, hard
+timeouts, leases, and credential enforcement remain requirements for an Agent
+runtime that advertises those guarantees.
+
+An SDK conformance suite MUST test manifest/schema validation, deterministic
+planning, no local-path/URI leakage, output bounds, local/distributed parity,
+retry and completion-order invariance, lease-loss cleanup, verifier behavior,
+resource eligibility, and package permission declarations. Profile-specific
+suites add two-worker byte equality, numeric tolerance, stochastic evidence,
+stream recovery, loop limits, gang failure, or GPU parity as applicable.
+
+## 11. Existing workload compatibility
+
+- Local and distributed `similarity-search` map to a bounded shard-map and
+  top-k reducer workflow without duplicating the scientific algorithm.
+- Local `similarity-graph` and future CTX-10 distribution map to triangular
+  block-pair expansion plus duplicate-safe edge reduction and pair-coverage
+  verification.
+- Existing `DistributedPlan`, single input artifact, single partial result, and
+  `chunk_index` become the SDK compatibility profile `map-reduce-v1`.
+- `descriptor-batch` is the first new reference implementation for the full
+  manifest, exact verifier, golden fixtures, and local/distributed parity.
+
+No current workload is removed or renamed by adopting the SDK. Migration is an
+adapter and manifest exercise first; protocol/database generalization occurs in
+versioned later phases.
+
+## 12. Deferred decisions
+
+The roadmap must resolve these before implementation reaches the affected
+phase:
+
+1. SDK package ownership and independent release cadence.
+2. Go-to-Python planner/reducer/verifier bridge and isolation boundary.
+3. Verifier execution placement and environment attestation.
+4. Streaming source/sink integrations and exactly-once scope.
+5. Multi-node rendezvous, network identity, and gang scheduling persistence.
+6. Accelerator sharing/MIG portability and accounting.
+7. Workload signing technology, trust roots, and revocation distribution.
+8. Tenant quotas, costs, priorities, fairness, and data-retention policy.
+9. First-class artifact collections versus composite manifests.
+10. Compatibility negotiation and deprecation support windows.

docs/scimesh-sdk-roadmap.md

@@ -0,0 +1,292 @@
+# SciMesh Workload SDK roadmap
+
+**Status:** active delivery roadmap. The Python `scimesh.sdk` package now
+implements the `core-batch-v1` foundation, verifier primitives, resource
+eligibility/local allocation, installed-package registry, local conformance
+runtime, and legacy similarity-search adapter. Coordinator-backed generalized
+DAG execution, Worker concurrency, accelerators, streaming, gang execution,
+and authoring CLI commands remain future phases.
+
+The normative future API, workflow, execution, resource, security, and failure
+semantics are specified in the design-draft
+[`scimesh-sdk-contract.md`](scimesh-sdk-contract.md). This roadmap controls
+delivery order and does not override that contract.
+
+## Purpose and boundaries
+
+The SDK should let a scientific developer add an allowlisted workload without
+learning coordinator internals or writing SQL, while keeping one scientific
+implementation usable locally and in distributed execution:
+
+```text
+scientific implementation -> local adapter -> planner/tasks -> reducer -> verifier
+```
+
+It must not execute arbitrary code or shell commands supplied by a coordinator.
+SDK v1 is not a public marketplace, generic container/job runner, cross-language
+SDK, automatic correctness-proof system, or immediate route to stochastic ML or
+GPU workloads.
+
+The current foundation is the Python `DistributedWorkload` protocol and registry
+under `scimesh/distributed/`, the Worker Agent under `scimesh/worker/`, and the
+coordinator API contract. See [CTX-07](ctx-07-distributed-workload-protocol.md),
+[worker-building guide](building-workers.md), and [API contract](api-contract.md).
+
+## Proposed public concepts
+
+The detailed schemas and invariants are defined in the SDK contract; this table
+is the roadmap-level responsibility map.
+
+| Concept | Responsibility |
+| --- | --- |
+| `WorkloadDefinition` / `WorkloadManifest` | Name, versions, schemas, execution and verification metadata. |
+| `ParameterSchema`, `InputSpec`, `OutputSpec`, `ArtifactRef` | Typed public inputs and durable artifact shapes. |
+| `TaskPlan`, `Planner`, `Runner`, `Reducer` | Validate, split, execute, and deterministically combine work. |
+| `Verifier` | Accept or reject result evidence; never silently downgrade checks. |
+| `ResourceRequirements`, `ExecutionProfile`, `ReproducibilityProfile` | Bounded resource needs and pinned execution assumptions. |
+
+A manifest should include workload/version and SDK compatibility versions,
+description, parameter/input/output schemas, planner/runner/reducer/verifier
+types, determinism and trust profiles, resource/output limits, worker
+capabilities, and environment or image digest. Compatibility must be explicit
+among SDK, coordinator protocol, worker runtime, workload, output schema, and
+verifier versions.
+
+## Contracts
+
+**Planner:** validates before durable Job/Task creation; produces versioned,
+JSON-serializable plans that refer only to durable artifacts; gives stable task
+order and expected resources/outputs; fails transactionally without a partial
+task graph.
+
+**Runner:** receives typed parameters and owned artifact references; runs only
+allowlisted SDK code; writes to its attempt directory; produces output manifest,
+metrics, and sanitized failures; respects cancellation/lease loss when platform
+support exists; never uses `shell=True` or unnecessarily exposes credentials.
+
+**Reducer:** consumes only accepted partial artifacts in stable order; is
+idempotent or coordinator-state protected; creates a versioned final manifest;
+defines missing, duplicate, and malformed-shard failures.
+
+**Verifier:** is versioned with the workload and states one of byte-exact,
+canonical, numeric, domain-specific, or trust-policy comparison. It processes
+structured manifests and bounded streams where practical, records sanitized
+evidence, and rejects inconsistent results.
+
+Current untrusted quorum is only `ExactArtifactVerifier`: coordinator-created
+candidate envelopes from distinct owners must share the exact workload, task,
+package/manifest/environment, parameters, and input binding and produce whole
+files with identical SHA-256. Future modes are
+`CanonicalRecordVerifier`, `NumericToleranceVerifier`,
+`DomainSpecificVerifier`, and `TrustedWorkerPolicy`. Canonical mode requires a
+specified parser/schema/order/encoding/serialization; numeric mode compares
+structured values, not CSV text.
+
+## Resources and reproducibility
+
+The extensible requirement model is `cpu_cores`, `memory_mb`, `scratch_mb`,
+`gpu_count`, `gpu_memory_mb`, `accelerator_kind`, `exclusive_device`, and
+`estimated_output_bytes`. It must not imply one Task equals one CPU core.
+
+Untrusted byte-exact workloads require a pinned image/environment digest,
+runtime and dependency versions, fixed locale/timezone/UTF-8/newlines/dialect,
+explicit invalid-row policy and algorithm options, canonical ordering, stable
+archive metadata, golden fixtures, two independently provisioned workers, and
+local/distributed plus retry/completion-order parity tests.
+
+## Compatibility evolution
+
+The stable release retains the current one-input/one-result task contract.
+First, a composite manifest artifact may reference multiple logical inputs;
+later, ordered input and output artifact collections can become first-class.
+The transition must be versioned and retain old workload compatibility.
+
+Discovery should use an installed Python package, manifest, pinned environment
+metadata, explicit entry points, and golden fixtures. It must be allowlisted;
+never scan or execute user-provided module paths.
+
+## General workload model: a versioned artifact workflow
+
+Map/reduce is the first execution shape, not the limit of the SDK. The target
+abstraction is an acyclic **workflow graph**: typed artifact ports connect
+versioned stages, and a stage may fan out, fan in, or run once per job. This
+allows the same SDK to express scientific ETL, simulations, parameter sweeps,
+multi-step pipelines, model inference, image/video analysis, and the current
+molecular workloads without placing scientific logic in the coordinator.
+
+```text
+Job inputs -> validate -> plan -> [map/partition stages] -> [join/reduce stages]
+                                      |                         |
+                                  accepted artifacts -----------+-> verify -> final manifest
+```
+
+The coordinator persists the graph, task attempts, leases, and artifact
+ownership. The SDK declares stage behavior; it does not receive database access
+or arbitrary commands. The initial `DistributedWorkload` protocol maps to a
+single input, many map tasks, one reducer, and one final artifact. It remains a
+supported compatibility profile rather than being replaced abruptly.
+
+### Workflow and stage contracts
+
+| Concept | Target responsibility |
+| --- | --- |
+| `WorkflowSpec` | Versioned DAG, external input ports, terminal outputs, global limits, and failure policy. |
+| `StageSpec` | Stable stage ID, kind (`map`, `reduce`, `service`, `verify`), input/output port schemas, retry and resource policy. |
+| `TaskSpec` | One concrete deterministic unit: stage ID, ordered artifact bindings, parameters, execution profile, and expected output manifest. |
+| `ArtifactSchema` | Logical media type, schema version, cardinality, size bound, canonicalization rules, and privacy/retention class. |
+| `ArtifactCollection` | Ordered, named, or keyed artifact set; used for shards, paired inputs, model bundles, and multiple outputs. |
+| `OutputManifest` | Every output's artifact reference, schema/version/digest, metrics, provenance, and verifier evidence. |
+| `FailurePolicy` | Retryable versus terminal errors, timeout, cancellation, partial-output disposal, and compensating cleanup rules. |
+
+A stage is a pure artifact transformation wherever possible. Interactive,
+long-running, or external-side-effect stages must declare that fact explicitly
+and are initially trusted-only. A workflow cannot form cycles, read a
+worker-local path from another stage, mutate a sealed input artifact, or produce
+undeclared output ports. Dynamic fan-out is permitted only through a bounded,
+versioned manifest emitted by an accepted planning stage; the coordinator must
+enforce declared task, artifact, scratch, and output limits.
+
+### Artifact and data-shape generality
+
+The SDK must support more than CSV while retaining streamability and audit
+trails. An `ArtifactSchema` can describe tabular records, scientific arrays,
+images, meshes, molecular structures, model weights, archives, JSON manifests,
+binary checkpoints, or opaque domain formats. It always declares how a consumer
+validates structure and bounds bytes/records/dimensions before loading it.
+
+Collections solve multi-input/multi-output work without an immediate database
+rewrite. A task can initially receive one composite manifest artifact whose
+entries name ordered or keyed logical inputs; it can return a composite output
+manifest. Later protocol versions may persist first-class collection edges. The
+collection manifest itself is immutable, coordinator-owned, schema-versioned,
+and hash-addressed, so the old one-input/one-result API remains compatible.
+
+## Execution model: Worker Agent, slots, and isolation
+
+The Worker Agent is a resource manager, not a scientific runtime. One physical
+machine registers one Agent. The Agent advertises a finite inventory and creates
+isolated **execution slots**; each leased Task owns exactly one slot until it
+finishes, loses its lease, or is cancelled.
+
+```text
+machine -> Worker Agent -> CPU / GPU / memory / scratch slot -> task subprocess -> attempt directory
+```
+
+`ExecutionProfile` declares whether a task uses a single process, a bounded
+process pool, a distributed runtime, or an accelerator backend. It also carries
+environment image/digest, entry-point identity, timeout, network policy,
+scratch/output bounds, checkpoint policy, and determinism declaration. The
+Agent—not a workload—sets environment variables, process groups, filesystem
+roots, credentials, resource limits, and lifecycle signals.
+
+### CPU parallelism
+
+`cpu_cores` is a reservation, while `max_concurrency` is the number of slots;
+neither is inferred from the other. CPU-bound Python work normally uses a
+process pool constrained to the task's allocated cores. A workload must declare
+its own internal parallelism and thread-library limits (for example OpenMP,
+BLAS, Torch, or RDKit-related native code) so nested pools cannot oversubscribe
+the host. The Agent starts independent heartbeat supervision per task and never
+claims a task if it cannot reserve all declared resources.
+
+Graceful draining means: stop new claims, continue heartbeat for active
+attempts, request checkpoint/cancellation at deadline, then clean only that
+attempt directory. Checkpoints are immutable artifacts and may be resumed only
+when the workload's manifest explicitly supports checkpoint compatibility; they
+are never treated as a completed result.
+
+### Accelerator support
+
+GPU/accelerator capability is generic inventory, not a coordinator-specific
+CUDA feature. A future Agent reports device kind/vendor, UUID, compute
+capability, memory, driver/runtime/image digest, supported backends, and
+allocatable slot count. The coordinator only matches `ResourceRequirements` to
+this inventory. The Agent assigns exclusive or shareable devices, sets device
+visibility (for example `CUDA_VISIBLE_DEVICES`), reserves memory where the
+platform supports it, starts the subprocess, measures usage, and releases the
+slot.
+
+The workload implementation chooses CUDA, ROCm, Metal, TPU, FPGA, SIMD, or a
+CPU fallback; batches work and manages model/device memory; and declares the
+scientific equivalence policy. Reducers and verifiers compare domain outputs,
+not device-specific logs or floating-point text. GPU work cannot be enabled for
+untrusted quorum merely because it runs: it additionally needs pinned images,
+appropriate verifier/trust mode, and CPU/GPU or domain-valid parity evidence.
+
+## Determinism, verification, and scientific validity
+
+`DeterminismProfile` separates reproducibility from correctness:
+
+| Profile | Examples | Minimum acceptance route |
+| --- | --- | --- |
+| `byte_exact` | canonical descriptors, fingerprints, sorted ETL | Exact artifact SHA-256 from independent owners. |
+| `canonical_exact` | format-normalized records, deterministic structures | Versioned parser/canonicalizer then exact records. |
+| `numeric_tolerance` | numerical solvers, GPU linear algebra | Structured comparison with absolute/relative/ULP tolerances and invariants. |
+| `seeded_stochastic` | conformers, randomized search | Recorded seed, repeated-run policy, statistical/domain verifier. |
+| `search_or_optimization` | routing, docking, retrosynthesis | Objective/constraint/domain evidence; often trusted execution. |
+| `side_effecting` | instrument control, external database writes | Trusted-only, idempotency key, audit/compensation policy. |
+
+The verifier consumes `OutputManifest` values, declared schemas, and bounded
+streams; it returns accept/reject/inconclusive plus evidence. `inconclusive`
+must never become success by a reducer default. Verification may be run by a
+coordinator adapter, a pinned Python verifier subprocess, or a separate trusted
+service—selection remains an open architectural decision. The manifest versions
+the verifier configuration, tolerance values, canonicalizer, reference data,
+and environment assumptions so historical results remain interpretable.
+
+## Existing-workload migration matrix
+
+| Existing capability | SDK workflow profile | Future adapter path |
+| --- | --- | --- |
+| Local `similarity-search` | single-process map + bounded top-k reduce | Keep local algorithm; expose a manifest and use current distributed planner/reducer. |
+| Distributed `similarity-search` | deterministic shard map -> ordered reduce | Compatibility workload v1; later attach exact verifier and provenance manifest. |
+| Local `similarity-graph` | triangular pair-partition map -> edge-set reduce | Preserve pair-coverage invariant as stage verifier. |
+| Distributed `similarity-graph` | planned block-pair DAG -> duplicate-safe reduce | First major non-linear partition reference; implement before SDK generalization. |
+| `descriptor-batch` | row-partition map -> ordered concatenation | First SDK reference workload and byte-exact quorum candidate. |
+| Future ML/docking/QM/MD | parameter sweep, ensemble, or iterative workflow | Use numeric/domain/trusted verifier profile and explicit resource/environment contracts. |
+
+## Authoring and operational lifecycle
+
+An installed workload package should contain a signed or administrator-approved
+manifest, Python entry points, schema migrations where required, pinned
+environment metadata, golden fixtures, test vectors, and documentation. An
+administrator controls enablement; users choose only among enabled manifests and
+validated parameter ranges. Workload installation is separate from job
+submission, preventing a user from sending code through the normal API.
+
+The eventual author workflow remains deliberate: initialize a template, define
+schemas and bounds, implement local scientific core, add planner/runner/reducer/
+verifier adapters, generate fixtures, test local parity, test two-worker and
+retry behavior, package, review, and enable. Any future CLI names are examples,
+not implemented commands.
+
+## Delivery sequence
+
+1. Finish distributed `similarity-graph` and reliability/cross-language CI.
+2. Stabilize manifest, schema, planner/runner/reducer interfaces, exact verifier,
+   compatibility metadata, and an author guide.
+3. Deliver `descriptor-batch` as the reference workload: pinned RDKit 2D
+   descriptors; canonical one-row-per-input CSV; shard-index concatenation with
+   one header; byte-identical local/distributed output and two-worker quorum.
+4. Add standardization, SMARTS screening, fingerprint export, fixed-template
+   reaction enumeration, then reaction validation/descriptors.
+5. Generalize composite artifacts, process slots, resource requests, and richer
+   verifier policies.
+6. Only then consider pinned, trusted/domain-verified numeric, ML, docking, QM,
+   MD, and GPU workloads.
+
+Future developer tooling may include `scimesh workload init`, `validate`,
+`test-local`, `test-distributed`, `golden`, and `package`; these commands do not
+exist today. A template should generate a manifest, schemas, planner, runner,
+reducer, verifier, unit tests, golden fixture, two-worker integration test, and
+documentation.
+
+## Open decisions
+
+- Is the SDK part of `scimesh` or a separately versioned Python distribution?
+- What stable bridge connects Go orchestration to Python planners/reducers?
+- Where do future verifiers execute, and how are environments attested?
+- Which trust modes may run each verifier profile?
+- Who may install/enable workloads in multi-user deployments?
+- How are composite I/O, version negotiation, output-growth limits, and
+  numeric-tolerance access governed without breaking the existing API?

docs/sdk-handoff.md

@@ -0,0 +1,172 @@
+# Workload SDK handoff
+
+**Audience:** the engineer/AI continuing SciMesh Workload SDK implementation.
+**Date:** 2026-08-01. **Baseline:** uncommitted working tree on `main` (`11e9333`
+plus the CTX-16 SDK changes); `python -m pytest -q` reports **225 passed**.
+
+Read first, in this order: `AGENTS.md` (binding repo rules),
+`docs/scimesh-sdk-roadmap.md` (delivery order — it governs, this file does not),
+`docs/scimesh-sdk-contract.md` (normative target semantics),
+`docs/workload-sdk.md` (author guide for what exists), and the CTX-16 entry in
+`PLAN.md`.
+
+## What is already done (do not redo)
+
+**Legacy removal (2026-08-01):** the CTX-07 `DistributedWorkload` protocol
+package (`scimesh/distributed/`), the SDK compatibility adapter
+(`scimesh/sdk/compat/`), and `library.similarity_search_sdk_adapter` were
+removed. The worker now executes the SDK-built workloads directly:
+`scimesh/worker/runners.py` builds a `TaskSpec` with the workload's pins,
+negotiates, reserves resources, runs the workload's own Runner through
+`LocalTaskContext` (store-backed catalog/sink), and uploads the sealed
+partial over the unchanged v1 wire. `run_search_shard` + the full-precision
+partial writer moved to `scimesh/workloads/search/core.py`; the partial
+format is unchanged, so the Go reducer and UI keep working. The runner
+resolves `query_id` per task and rejects plan-time `max_rows`.
+
+**Default hooks + molwt-filter (2026-08-01):** `MapReduceWorkload` now provides default `partition_input` (row-bounded, header-preserving sharding for delimited inputs, `shard_rows` class attr) and default `reduce_partials` (`concatenate_partial_tables`, one header, byte-identical). A new built-in `molwt-filter@1.0.0` (`scimesh/workloads/molwt_filter/`) demonstrates the minimal authoring surface: only `compute_shard` is workload code. descriptor-batch dropped its now-redundant partition/reduce overrides.
+
+**MkDocs site (2026-08-02):** the standalone documentation site lives in `mkdocs/` (`docs_dir: mkdocs`) and does not use the project's `docs/` directory. It contains guides (`mkdocs/sdk/`: overview, authoring-workloads, cli, worker-integration), the full auto-generated API reference for all `scimesh.sdk` modules (`mkdocs/api/`, mkdocstrings `::: scimesh.sdk.<module>` — set `show_if_no_docstring: true`), and the writing rules (`mkdocs/approach.md`). `make docs` builds it; the UI serves it at `/ui/docs/`. All public SDK members now carry Google-style docstrings.
+
+**Go worker agent (2026-08-02):** the Python worker daemon was removed. `coordinator/internal/agent/` + `cmd/worker-agent` (build: `make agent`) now owns the full lifecycle: register/claim/heartbeat/download(checksum)/spawn-task/upload/submit/fail, static bearer or worker-key JWT auth with 401 refresh, attempt-dir cleanup (`CLEANUP_AFTER_SECONDS`), backoff, idle/max-tasks exit. `scimesh/worker/` keeps only the per-task Python execution: `task.py` (exit 0/3/1), `runners.py` (SDK bridge, `SCIMESH_WORKLOAD_ALLOWLIST` discovery), `models.py` (claim payload). The `scimesh-worker` console script and the daemon/auth/transport modules are gone. Demo (`make demo-ui`) and smoke (`make smoke-two-worker`) run the Go agent; smoke passes 4/4 with two agents. `coordinator/internal/agent/` (config, models, client, sanitize, taskrunner, daemon) + `coordinator/cmd/worker-agent`, built with `make agent`. It mirrors the Python worker's v1 lifecycle; per-task SDK execution happens in a Python subprocess (`scimesh/worker/task.py`: exits 0 on success, 3 permanent, 1 retryable). Default `TASK_RUNNER` is `python -m scimesh.worker.task` — set it to the venv python in source checkouts. Verified E2E against the demo coordinator. Open items: JWT refresh, resource slots/limits, attempt-dir cleanup, protocol-v2 features.
+
+**Authoring scaffold (2026-08-01):** `scimesh/sdk/batch.py` adds
+`MapReduceWorkload` — the primary authoring surface for `core-batch-v1`. A
+subclass declares identity/parameters/ports and three scientific hooks
+(`partition_input`, `compute_shard`, `reduce_partials`); the SDK assembles the
+manifest, map/reduce stages, workflow, digest-pinned handlers, and the
+exact-artifact verifier. Overridable hooks: `domain_validate`,
+`resolved_parameters`, `resolved_parameters_for_plan`, `plan_tasks`,
+`parse_partial_key`/`validate_partial_keys`, `map_stage_inputs` (multi-input
+map stages share the external input schema). All three built-in workloads are
+refactored onto it. Generic `scimesh workload list|run` CLI added (no
+workload-specific logic). The worker loads workloads generically:
+`SCIMESH_WORKLOAD_ALLOWLIST` (JSON `{distribution, name, version, digest}`,
+discovery via entry points) or built-in fallback; `SCIMESH_CAPABILITIES`
+overrides advertised capabilities; workloads with multi-input map stages are
+rejected by the v1 bridge. `query_id` resolution moved into the search
+workload's `run_search_shard`; the worker passes task parameters through and
+the workload validates them.
+
+CTX-16 "Workload SDK foundation" is complete and tested. `scimesh/sdk/`
+implements the `core-batch-v1` profile:
+
+- Immutable, JSON-strict value objects: `identity.py`, `artifacts.py`,
+  `workflow.py`, `manifest.py`, `plans.py`, `execution.py`, `resources.py`.
+- Fail-closed compatibility negotiation: `runtime.py` (`negotiate_manifest`)
+  plus request-level checks in `registry.py`.
+- Installed-package registry with administrator allowlist, exact version +
+  `sha256:` digest pinning, entry-point discovery with digest measured before
+  and after import: `registry.py`, `integrity.py`.
+- Verifier primitives `ExactArtifactVerifier`, `CanonicalRecordVerifier`,
+  `NumericToleranceVerifier` with bounded sanitized evidence: `verification.py`.
+- Local conformance harness: `LocalArtifactStore`, `LocalCoreBatchExecutor`,
+  `ResourcePool` (atomic all-or-nothing reservation): `conformance.py`.
+- SDK-built workloads living outside the SDK: `scimesh/workloads/search/`,
+  `scimesh/workloads/graph/`, `scimesh/workloads/descriptors/` (each `core.py`
+  + `definition.py`), composed by `scimesh/workloads/library.py`
+  (`default_sdk_registry`, `default_sdk_runtime`); entry points for all three
+  declared in `pyproject.toml`.
+- Tests: `tests/test_sdk_{models,resources,verification,compatibility,registry}.py`
+  including fail-closed rejection coverage for every advanced profile
+  declaration (gang, GPU modes, pools, checkpoints, retries, secrets, streams,
+  loops, side effects), plus `tests/test_sdk_{search,graph,descriptors}.py`
+  and the worker bridge tests in `tests/test_worker_daemon.py`.
+  `tests/test_distributed*.py` were removed with the protocol.
+
+## What remains, in delivery order
+
+1. ~~**`descriptor-batch` reference workload**~~ — **done** (2026-08-01):
+   `scimesh/workloads/descriptors/` (`core.py` + `definition.py`) is the first
+   SDK-built workload. Pinned 81-name RDKit 2D descriptor set (validated at
+   definition build time), canonical one-row-per-input CSV with `%.6f` floats,
+   deterministic row-bounded shards, shard-index concatenation with one header,
+   byte-identical local/distributed output, `skip_invalid` explicit policy, and
+   `untrusted_quorum` + exact-artifact@1 declared in the manifest. Tests:
+   `tests/test_sdk_descriptors.py`.
+2. ~~**SDK-built `similarity-search` and `similarity-graph`**~~ — **done**
+   (2026-08-01). Both local workloads are SDK-built packages outside the SDK:
+   `scimesh/workloads/search/` and `scimesh/workloads/graph/` (each `core.py` +
+   `definition.py`, manifest + planner/runner/reducer, byte_exact +
+   exact-artifact@1, trusted + untrusted_quorum). Search resolves the query at
+   plan time and merges partials with the reference heap (byte-identical to the
+   CLI). Graph plans one task per block pair `(i,j)` with `i <= j`, reducer
+   enforces pair-coverage and duplicate-pair rejection, output byte-identical
+   to the local brute-force reference for both directions and any block size.
+   Tests: `tests/test_sdk_search.py`, `tests/test_sdk_graph.py`.
+   **Architecture note:** `scimesh.sdk/` is the framework ONLY (no workload
+   code); workloads are user scripts/packages under `scimesh/workloads/` that
+   import the SDK. Keep new workloads out of the SDK package.
+2. **Distributed `similarity-graph`** (CTX-10, roadmap step 1). The coordinator
+   currently rejects `similarity-graph` uploads; it needs cross-shard block-pair
+   planning and duplicate-safe reduction. STATUS.md names this the next
+   recommended assignment overall.
+3. **Coordinator/Worker protocol v2** (needs CTX-10, then CTX-13 in-worker CPU
+   parallelism and CTX-14 GPU execution; Go + Python). The protocol-v1
+   coordinator persists only flat one-input/one-result tasks: no resource
+   requirements, stage edges, package versions, device allocations, or gang
+   leases. Until a versioned rollout lands, SDK declarations for those features
+   must stay fail-closed — do not silently "enable" them.
+4. **More chemistry workloads** (roadmap step 4): standardization, SMARTS
+   screening, fingerprint export, fixed-template reaction enumeration, then
+   reaction validation/descriptors.
+5. **Composite artifacts and richer verifier policies** (roadmap step 5):
+   first-class ordered/keyed `ArtifactCollection` edges instead of composite
+   manifest artifacts; decide where verifiers execute (open decision in the
+   roadmap).
+6. **Authoring CLI** (future tooling, does not exist today): `scimesh workload
+   init`, `validate`, `test-local`, `test-distributed`, `golden`, `package`.
+   Per AGENTS.md, keep CLI parsing in workload modules and register through
+   `scimesh/core/registry.py`; no workload-specific logic in the main CLI.
+7. **Open decisions** (listed at the end of the roadmap): SDK distribution
+   split, Go↔Python planner bridge, verifier execution/attestation, trust-mode
+   governance, multi-user enablement. Do not pick one unilaterally — surface it.
+
+## Known traps (cost the previous session real time)
+
+- **Architecture boundary:** `scimesh.sdk/` is the framework only and must
+  never import `scimesh.workloads` (SDK depends on nothing workload-specific).
+  Workload packages live under `scimesh/workloads/` (each `core.py` +
+  `definition.py`), and built-in wiring lives in `scimesh/workloads/library.py`.
+  The digest helpers are in `scimesh/workloads/environment.py`; the SDK keeps
+  only the generic `installed_distribution_digest` in `scimesh/sdk/integrity.py`.
+- The legacy adapter pins its own manifest (`adapter.manifest`). If a test
+  changes limits/workflow on the manifest, the adapter's copy must be replaced
+  too, or `registry.plan` fails with "planner plan does not carry the selected
+  immutable workload pin".
+- `WorkloadDefinition` validation: a PLAN stage's `entry_point` must equal
+  `planner.entry_point`; every non-REDUCE stage's `entry_point` must be a key
+  in `runners` (REDUCE → `reducers`); verifier handlers are keyed by
+  `ComponentRef.canonical` and must expose a matching `.identity`.
+- Negotiation requires each triggering property's feature to be declared
+  separately: e.g. `PROCESS_POOL` needs `process-pools` **and** `multi-process`
+  for `max_processes > 1`. Runtime must also advertise every declared required
+  feature, or negotiation fails with `feature-unavailable`.
+- `feature-fallback-disallowed` in `scimesh/sdk/registry.py` is currently
+  unreachable via `registry.plan` (the `feature-unavailable` check fires first
+  for any runtime that produced a fallback). Behavior is still fail-closed;
+  decide whether to reorder or delete the branch.
+- The local executor is deliberately trusted/in-process: it rejects anything
+  but `TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-threaded CPU
+  map/reduce without retries/gangs/accelerators/secrets/checkpoints. That is a
+  contract, not a bug — test rejections, don't "fix" them.
+- `JobRequest` parameters and failure/evidence payloads reject local paths and
+  URIs by design; keep new payloads location-free.
+- There is a stray nested clone `SciMesh/` in the repo root (same repo at an
+  older commit). Ignore it and never `git add` it; consider deleting it.
+- The full ChEMBL extract `chembl_37_chemreps.txt` (~2.9M rows) makes the
+  single-threaded local executor run for many minutes; tests must use small
+  TSV fixtures (see `_write_tiny_dataset`).
+
+## Working agreement
+
+- Verify with `source .venv/bin/activate && python -m pytest -q`; the baseline
+  is 225 passing tests and it must stay green. Add a regression test for every
+  behavioral change; similarity code needs a brute-force/sorted reference and
+  determinism across block sizes.
+- Legacy `similarity-search` wire schema, worker alias boundary, and scientific
+  output must not change. Worker code never talks to PostgreSQL directly;
+  results go through the coordinator; failures go to `/failure`, never as
+  `file://`/`worker://` result URIs.
+- Do not commit datasets, generated CSV/PNG, tokens, or local worker artifacts.
+- One CTX task per pull request; link the CTX item from `PLAN.md`.

docs/user-service-api-contract.md

@@ -0,0 +1,47 @@
+# SciMesh User Service API contract (v1)
+
+**Status:** `v1`. The User Service owns user accounts and issues access tokens.
+The coordinator never receives user passwords and never accesses the User
+Service database.
+
+## Authentication boundary
+
+- User Service signs access tokens; coordinator verifies them before accepting
+  user-scoped requests.
+- Tokens contain a UUID `sub`, `role` (`user` or `admin`), `verified`, `iat`,
+  and `exp` claims.
+- A user-authenticated caller may operate only workers whose `owner_id` equals
+  `sub`. This applies to claim, heartbeat, result, failure, and artifact upload.
+- Worker traffic authenticated with the coordinator's shared worker token has
+  no user identity and remains an operator-only compatibility path.
+- Role or verification changes take effect when the access token is renewed.
+  Deployments needing immediate revocation must use a short token lifetime or a
+  revocation mechanism before enabling volunteer-worker trust.
+
+## Endpoints
+
+All JSON request bodies reject unknown fields and are size-limited. Error
+responses are JSON with a stable `error` value and request ID.
+
+| Method | Path | Auth | Success |
+| --- | --- | --- | --- |
+| `GET` | `/health` | none | `200 {"status":"ok"}` |
+| `POST` | `/register` | none | `201` user object |
+| `POST` | `/login` | none | `200` user object and access token |
+| `GET` | `/me` | Bearer access token | `200` current user |
+| `POST` | `/users/{id}/verify` | Bearer admin token | `204` |
+| `POST` | `/users/{id}/unverify` | Bearer admin token | `204` |
+| `POST` | `/users/{id}/promote` | Bearer admin token | `204` |
+| `POST` | `/users/{id}/demote` | Bearer admin token | `204` |
+
+`POST /register` accepts `{ "email": string, "password": string }` and
+always creates role `user` with `verified: false`. `POST /login` accepts the
+same shape and returns `{ "token": string, "user": User }`. Password hashes,
+JWT signing material, and raw tokens must never be logged.
+
+## Coordinator integration tests
+
+The coordinator must test that a JWT user cannot claim or mutate another
+user's worker lease, including heartbeat, failure, result, and artifact upload.
+Job and artifact access is restricted to the job owner unless the caller has
+the admin role.

docs/workload-sdk.md

@@ -0,0 +1,468 @@
+# SciMesh Workload SDK v1
+
+SciMesh now ships a public Python SDK under `scimesh.sdk`. The implemented
+authoring profile is **`core-batch-v1`**: installed and digest-pinned workload
+definitions, strict JSON manifests, typed artifact ports and collections, a
+static map/reduce workflow, CPU/memory/scratch eligibility, atomic local
+resource reservation, exact/canonical/numeric verifier primitives, and a
+compatibility adapter for the existing `DistributedWorkload` protocol. Its
+local executor is deliberately a trusted, in-process conformance harness; the
+production subprocess/lease sandbox remains a coordinator/Worker milestone.
+
+The full target contract remains in
+[`scimesh-sdk-contract.md`](scimesh-sdk-contract.md). Dynamic expansion,
+streaming, accelerators, gang execution, and side effects have typed bounded
+declarations, but the current coordinator/Worker runtime does not advertise
+their features. Compatibility negotiation therefore rejects those workflows
+before planner code runs.
+
+## SDK versus workloads
+
+`scimesh.sdk` is the framework only: strict manifests, plans, artifacts,
+registry, verifiers, and the local conformance executor. It contains no
+scientific workload code. Workloads are user Python scripts and packages that
+import the SDK and live outside it. The built-in SciMesh workloads are under
+`scimesh/workloads/`:
+
+- `scimesh/workloads/search/` — SDK-built `similarity-search@1.0.0`;
+- `scimesh/workloads/graph/` — SDK-built `similarity-graph@1.0.0`;
+- `scimesh/workloads/descriptors/` — SDK-built `descriptor-batch@1.0.0`;
+- `scimesh/workloads/molwt_filter/` — SDK-built `molwt-filter@1.0.0`, the
+  minimal authoring example: it only declares identity, parameters, ports,
+  and the `compute_shard` hook, using the scaffold's default sharding and
+  concatenation;
+- `scimesh/workloads/library.py` — the built-in library wiring: a default
+  registry containing all three definitions and a runtime advertising their
+  capabilities;
+- the plain `scimesh/workloads/*.py` modules remain the local CLI scientific
+  cores and their `Workload` registry.
+
+Each SDK-built workload is a small package with `core.py` (scientific code)
+and `definition.py` (manifest plus planner/runner/reducer handlers). A future
+external workload library can follow the same shape: its own distribution, one
+`scimesh.workloads` entry point per workload version, and an administrator
+allowlist.
+
+## What authors import
+
+The stable authoring surface is exported from `scimesh.sdk`:
+
+- `WorkloadManifest`, `WorkloadId`, `VersionRange`, `PackageSpec`, and
+  `EnvironmentSpec` pin identity and compatibility;
+- `ArtifactSchema`, `PortSpec`, `ArtifactRef`, and `ArtifactCollection` define
+  immutable data boundaries without transport URLs or local paths;
+- `WorkflowSpec`, `StageSpec`, `ArtifactEdge`, `TaskSpec`, and `WorkflowPlan`
+  define a typed acyclic plan and pin package/manifest digests plus trust mode;
+- `ResourceRequirements` and `ExecutionProfile` separate per-task resources
+  from Agent `max_concurrency`;
+- `Planner`, `Runner`, `Reducer`, and `Verifier` are the package handler
+  protocols;
+- `OutputManifest` and `Provenance` describe sealed durable results;
+- `WorkloadRegistry` resolves an exact name, version, package digest, runtime,
+  environment, and feature set. It never selects an implicit latest version;
+- `MapReduceWorkload` is the primary authoring scaffold for `core-batch-v1`:
+  a subclass declares its identity, parameter schema, artifact ports, and
+  three scientific hooks (partition, compute, merge), and the SDK assembles
+  the manifest, map/reduce stages, workflow, digest-pinned handlers, and the
+  exact-artifact verifier. See "Authoring a workload" below.
+
+Persisted manifests, requests, plans, tasks, expansions, outputs, candidates,
+decisions, and failures are frozen, recursively immutable, JSON-safe,
+canonically serialized, and strict about unknown fields; their enclosing wire
+contracts carry schema versions.
+Artifact identities contain a coordinator-owned UUID, schema, checksum, media
+type, and bounds; a scientific handler never persists a filesystem path.
+
+## Try the built-in SDK workloads
+
+This example runs the SDK-built `similarity-search` without starting
+PostgreSQL or the coordinator:
+
+```python
+from pathlib import Path
+
+from scimesh.sdk import (
+    ArtifactCollection,
+    JobRequest,
+    LocalArtifactStore,
+    LocalCoreBatchExecutor,
+)
+from scimesh.workloads.library import (
+    default_sdk_registry,
+    default_sdk_runtime,
+    similarity_search_sdk_definition,
+)
+
+root = Path("sdk-run")
+store = LocalArtifactStore(root / "artifacts")
+workload = similarity_search_sdk_definition(shard_rows=1_000)
+
+dataset = store.import_file(
+    Path("chembl_37_chemreps.txt"),
+    declaration=workload.manifest.inputs["input"].schema,
+)
+request = JobRequest(
+    workload=workload.manifest.workload,
+    parameters={"query_smiles": "CCO", "top_k": 20},
+    inputs={"input": ArtifactCollection.single(dataset)},
+)
+
+result = LocalCoreBatchExecutor(
+    default_sdk_registry(shard_rows=1_000),
+    default_sdk_runtime(),
+    store,
+    root / "attempts",
+).execute(request, workload.manifest.package.digest)
+
+result_ref = result.outputs["result"].items[0].artifact
+print(store.materialize(result_ref))
+```
+
+`LocalCoreBatchExecutor` is a correctness/conformance runtime, not a substitute
+for coordinator leases or multi-machine scheduling. It accepts only
+`TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-process/single-threaded CPU
+map/reduce stages without secrets, checkpoints, retries, gangs, or
+accelerators. It does not claim network, timeout, process, or credential
+isolation. Unsupported declarations are rejected before a handler runs. The
+harness runs the SDK-built workload handlers themselves, and their parity
+against the single-process references is covered by automated tests.
+
+## The descriptor-batch reference workload
+
+`descriptor-batch@1.0.0` is the first SDK-native reference workload: it is
+built directly on the manifest/planner/runner/reducer contracts, and it is
+the intended first `untrusted_quorum` candidate
+(`byte_exact` plus `exact-artifact@1`). Its scientific contract is pinned:
+
+- one output CSV row per valid input molecule, in input order, with RDKit
+  canonical SMILES recomputed by RDKit;
+- an explicit 81-name pinned RDKit 2D descriptor set (see
+  `scimesh/workloads/descriptors/core.py`), validated against the installed RDKit at
+  definition build time;
+- `%.6f` float formatting, `utf-8` CSV with one header, and row-bounded
+  deterministic shards;
+- `skip_invalid` is the only parameter (default `true`): invalid SMILES rows
+  are counted and skipped, or fail the run when `false`;
+- the reducer concatenates shard partials by shard index with exactly one
+  header, so the distributed output is byte-identical to the single-process
+  reference for the same input rows.
+
+```python
+from pathlib import Path
+
+from scimesh.sdk import (
+    ArtifactCollection,
+    JobRequest,
+    LocalArtifactStore,
+    LocalCoreBatchExecutor,
+    WorkloadRegistry,
+)
+from scimesh.workloads.descriptors import descriptor_batch_sdk_definition
+from scimesh.workloads.library import default_sdk_runtime
+
+root = Path("descriptor-run")
+store = LocalArtifactStore(root / "artifacts")
+workload = descriptor_batch_sdk_definition(shard_rows=1_000)
+
+dataset = store.import_file(
+    Path("chembl_37_chemreps.txt"),
+    declaration=workload.manifest.inputs["input"].schema,
+)
+request = JobRequest(
+    workload=workload.manifest.workload,
+    parameters={"skip_invalid": True},
+    inputs={"input": ArtifactCollection.single(dataset)},
+)
+registry = WorkloadRegistry()
+registry.register(workload.definition(), enabled=True)
+
+result = LocalCoreBatchExecutor(
+    registry,
+    default_sdk_runtime(),
+    store,
+    root / "attempts",
+).execute(request, workload.manifest.package.digest)
+
+result_ref = result.outputs["result"].items[0].artifact
+print(store.materialize(result_ref))
+```
+
+The descriptor-batch entry point `descriptor-batch@1.0.0` is declared in
+`pyproject.toml`; discovery loads it only when an administrator supplies a
+matching `AllowedPackage` allowlist entry. Its manifest declares both
+`trusted` and `untrusted_quorum` trust modes and the exact-artifact verifier,
+so the same definition can later run under coordinator quorum once protocol-v2
+leases exist.
+
+## The SDK-built similarity workloads
+
+`similarity-search@1.0.0` and `similarity-graph@1.0.0` are SDK-built workloads
+under `scimesh/workloads/search/` and `scimesh/workloads/graph/`; both reuse
+the local scientific cores from `scimesh/workloads/similarity_search.py` and
+`similarity_graph.py` and declare `byte_exact` + `exact-artifact@1`:
+
+- the search workload resolves `query_id` exactly once at plan time, shards
+  the input deterministically, computes a local top-k per shard with the
+  reference heap, and merges the sorted partials with the same tie-breakers,
+  so the final CSV is byte-identical to the single-process CLI output;
+- the graph workload parses molecules once into deterministic row-ordered
+  blocks, plans one map task per block pair `(i, j)` with `i <= j`, and its
+  reducer enforces the pair-coverage invariant (every unordered molecule pair
+  compared exactly once, no duplicates) before emitting the same
+  deterministically sorted edge list as the local brute-force reference, for
+  either threshold direction and any block size;
+- the v1 worker executes SDK-built workloads directly:
+  `scimesh/worker/runners.py` is a workload-generic wire bridge that builds a
+  `TaskSpec` with the workload's own pins, negotiates against a runtime
+  derived from the loaded definitions, reserves resources, seals the partial
+  through a content-addressed store, and uploads the resulting CSV over the
+  unchanged coordinator contract. The worker loads workloads from
+  `SCIMESH_WORKLOAD_ALLOWLIST` (a JSON array of
+  `{distribution, name, version, digest}` entries matched against installed
+  `scimesh.workloads` entry points) or falls back to the built-in
+  `similarity-search`; advertised capabilities come from
+  `SCIMESH_CAPABILITIES`. Workloads whose map stage needs more than one input
+  port are rejected with a clear message until the coordinator contract
+  supports them.
+
+## Authoring a workload
+
+A workload is a user script that imports the SDK. For the standard
+`core-batch-v1` shape (one input dataset, shards, partials, one merged result)
+subclass `MapReduceWorkload` and implement the three scientific hooks; the
+framework provides everything else:
+
+```python
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+from scimesh.sdk import (
+    ArtifactSchema,
+    ComponentRef,
+    MapReduceWorkload,
+    PortSpec,
+    SchemaRef,
+    WorkloadId,
+)
+
+class CountRowsWorkload(MapReduceWorkload):
+    workload_id = WorkloadId("count-rows", "1.0.0")
+    description = "Count TSV data rows per shard and concatenate the counts."
+    parameters_schema = {
+        "type": "object",
+        "additionalProperties": False,
+        "properties": {"prefix": {"type": "string", "minLength": 1, "maxLength": 50}},
+    }
+    input_port = PortSpec(ArtifactSchema(
+        SchemaRef("molecule-table", 1), "text/tab-separated-values", "utf-8",
+        max_bytes=10**9, validator=ComponentRef("delimited-table", 1),
+        validator_configuration={"required_columns": ["canonical_smiles", "chembl_id"]},
+    ))
+    partial_port = output_port = PortSpec(ArtifactSchema(
+        SchemaRef("count-table", 1), "text/csv", "utf-8",
+        max_bytes=10**9, validator=ComponentRef("delimited-table", 1),
+        validator_configuration={"columns": ["id", "rows"]},
+    ))
+    map_parameter_names = ("prefix",)
+
+    def partition_input(self, input_path, parameters, workspace):  # -> list[Path]
+        ...  # deterministic shard files, one per map task
+
+    def compute_shard(self, inputs, parameters, output_path):  # -> Mapping[str, int|float]
+        ...  # one map task; inputs maps each map port to a materialized file
+
+    def reduce_partials(self, partial_paths, parameters, output_path):  # -> Mapping[str, int|float]
+        ...  # deterministic merge of the accepted partials
+```
+
+The base class then provides `validate`, `plan`, `run`, `reduce`, and
+`definition()`. For workloads whose map output is a simple filtered or
+transformed table, the scaffold's defaults already cover partitioning
+(row-bounded shards that keep the header) and reduction (concatenation with
+one header), so only `compute_shard` has to be written — that is exactly what
+the built-in `molwt-filter` workload does. The registry, negotiation, resource
+reservation, verification, and the local conformance executor treat the
+result like any other workload:
+
+```python
+from scimesh.sdk import (
+    ArtifactCollection,
+    JobRequest,
+    LocalArtifactStore,
+    LocalCoreBatchExecutor,
+    WorkloadRegistry,
+)
+from scimesh.workloads.library import default_sdk_runtime
+
+workload = CountRowsWorkload(package_digest=..., environment_digest=...)
+registry = WorkloadRegistry()
+registry.register(workload.definition(), enabled=True)
+
+store = LocalArtifactStore(Path("artifacts"))
+artifact = store.import_file(Path("tiny.tsv"), declaration=workload.manifest.inputs["input"].schema)
+request = JobRequest(workload=workload.manifest.workload, parameters={"prefix": "x"},
+                     inputs={"input": ArtifactCollection.single(artifact)})
+result = LocalCoreBatchExecutor(registry, default_sdk_runtime(), store, Path("work")) \
+    .execute(request, workload.manifest.package.digest)
+```
+
+Hooks you can override beyond the three scientific ones:
+
+- `domain_validate(parameters)` — extra job-parameter validation (the JSON
+  schema already ran);
+- `resolved_parameters(request)` / `resolved_parameters_for_plan(job, input_path, resolved)`
+  — values persisted into the plan (for example one-time query resolution);
+- `plan_tasks(...)` — custom task construction (the graph workload uses this
+  to plan one task per block pair with two block inputs);
+- `parse_partial_key(key)` / `validate_partial_keys(parsed)` — partial-key
+  policy (default: `map.<eight-digit-index>`, contiguous; the graph workload
+  parses `map.<i>x<j>` and enforces the pair-coverage invariant);
+- `map_stage_inputs` — a map stage with more than one input port (each extra
+  port must share the external input schema).
+
+Anything outside this model uses the lower-level SDK value objects directly.
+Authoring rules: keep the scientific core callable without a coordinator,
+inline a strict JSON parameter schema, declare artifact schemas with bounds,
+return only sink-sealed artifacts, and select a verifier compatible with
+determinism and trust.
+
+To run a workload from the command line without writing any program code:
+
+```bash
+scimesh workload list
+scimesh workload run count-rows --input tiny.tsv --params '{"prefix": "x"}' -o result.csv
+```
+
+`scimesh workload` is a generic SDK tool; it contains no workload-specific
+logic, so new workloads do not require changes to the CLI or any other part of
+the program.
+
+## Package shape and registration
+
+A workload distribution provides one explicit entry point per workload
+version. The built-in workloads are part of the `scimesh` distribution:
+
+```toml
+[project.entry-points."scimesh.workloads"]
+"similarity-search@1.0.0" = "scimesh.workloads.search:workload_definition"
+"similarity-graph@1.0.0" = "scimesh.workloads.graph:workload_definition"
+"descriptor-batch@1.0.0" = "scimesh.workloads.descriptors:workload_definition"
+```
+
+The factory returns a `WorkloadDefinition` containing its manifest and handler
+objects. An administrator supplies an `AllowedPackage` with the same
+distribution, exact `WorkloadId`, and `sha256:` package digest. Discovery
+filters installed metadata before importing an entry point and fails
+transactionally if an allowlisted definition is missing or mismatched. Job
+parameters cannot name a module, entry point, package path, or executable.
+The measured digest covers package payload files and installed entry-point
+declarations and is checked before and after loading. It is a content pin, not
+a signature or image attestation; production discovery should run in a fresh
+trusted control-plane process so a pre-populated Python module cache is not an
+integrity boundary.
+
+Direct registration is useful for tests and embedded deployments:
+
+```python
+registry = WorkloadRegistry()
+registry.register(definition, enabled=False)
+registry.enable(
+    definition.manifest.workload.name,
+    definition.manifest.workload.version,
+    definition.manifest.package.digest,
+)
+```
+
+Both version and digest are required when resolving or planning. Upgrading an
+installed definition does not change the identity of an existing Job.
+
+## Authoring rules
+
+1. Keep the scientific core callable without a coordinator.
+2. Inline a strict JSON parameter schema with `type: object` and
+   `additionalProperties: false`; the planner still performs domain validation.
+3. Give every external and stage port an `ArtifactSchema` with a media type,
+   schema version, and byte/record/dimension bounds.
+4. Connect stage ports with `ArtifactEdge` values. `WorkflowSpec` checks source
+   and target schemas, complete input bindings, declared dependencies, and
+   acyclicity.
+5. Declare one `ResourceRequirements` and `ExecutionProfile` per stage. A task
+   cannot run until its entire request is eligible and atomically reserved.
+6. Return only sink-sealed artifacts in `OutputManifest`; the local harness
+   binds task key/provenance itself and rejects fabricated references,
+   unexpected/missing ports, wrong schema/media type, and cumulative output or
+   artifact-limit violations.
+7. Select a verifier compatible with determinism and trust. SDK v1 permits
+   `untrusted_quorum` only for `byte_exact` plus `exact-artifact@1`.
+8. Add golden fixtures, local/distributed parity, retry/completion-order, and
+   verifier failure tests before enabling a package.
+
+`ArtifactSink` and `ArtifactCatalog` are bridge-owned protocols. They let
+scientific handlers materialize verified inputs and seal outputs without bearer
+tokens, database credentials, upload URLs, or durable local paths.
+
+## Verification
+
+The SDK includes:
+
+- `ExactArtifactVerifier`: compares logical port/collection/schema/content
+  digests while ignoring coordinator UUIDs, timestamps, metrics, and worker
+  identity. Quorum inputs use coordinator-created `CandidateOutput` envelopes,
+  count at most one vote per owner, and require a `VerificationBinding` for the
+  exact task, inputs, parameters, package, manifest, and environment;
+- `CanonicalRecordVerifier`: applies a package-owned bounded canonicalizer and
+  compares length-framed canonical records;
+- `NumericToleranceVerifier`: recursively checks structure plus explicit
+  absolute, relative, ULP, and NaN policy, returning bounded sanitized evidence.
+
+Canonical and numeric objects expose direct bounded comparison methods. To use
+them as manifest `Verifier` handlers, the package supplies an artifact-to-record
+or artifact-to-structured-value loader; without one, verification returns
+`inconclusive` rather than accepting bytes it did not parse.
+
+A decision is `accepted`, `rejected`, or `inconclusive`; only `accepted`
+satisfies a stage. Evidence is limited to 16 KiB and cannot contain local paths
+or transport URLs.
+
+## Resources and current runtime boundary
+
+`ResourcePool` provides a lock-protected all-or-nothing local reservation for
+CPU cores, memory, scratch, and accelerator device/partition IDs, including
+whole-device versus partition conflict fencing. It enforces aggregate capacity
+and execution-slot count. `ExecutionProfile` produces only
+allocation-derived OpenMP/BLAS and device-visibility values; credentials never
+belong to scientific parameters.
+
+The current protocol-v1 coordinator stores one input/result per flat task and
+does not persist resource requirements, device allocations, stage edges, or
+package versions. The production Worker also remains serial. Consequently:
+
+- SDK `core-batch-v1` can be authored, validated, tested, discovered, and run
+  through the trusted local conformance harness now;
+- existing production `similarity-search` remains on its compatible v1 wire
+  path and is not renamed;
+- real concurrent claims, GPU scheduling, multi-output DAG execution, dynamic
+  loops, streaming, and gang leases require the versioned coordinator/Worker
+  changes listed in [`scimesh-sdk-roadmap.md`](scimesh-sdk-roadmap.md);
+- merely declaring a GPU or gang request never enables it. Missing runtime
+  features or inventory fail before the planner executes.
+
+## Conformance commands
+
+Install development tools and run the SDK suite:
+
+```bash
+pip install -e '.[dev]'
+pytest tests/test_sdk_models.py \
+       tests/test_sdk_resources.py \
+       tests/test_sdk_verification.py \
+       tests/test_sdk_compatibility.py \
+       tests/test_sdk_registry.py \
+       tests/test_sdk_descriptors.py \
+       tests/test_sdk_search.py \
+       tests/test_sdk_graph.py
+```
+
+Run `pytest` for the full Worker, local-science, and SDK regression suite. Package authors can reuse `LocalArtifactStore`,
+`LocalCoreBatchExecutor`, and `assert_manifest_round_trip` in their own golden
+tests.

mkdocs.yml

@@ -0,0 +1,75 @@
+site_name: SciMesh
+site_description: Local-first distributed scientific computation for molecular workloads
+site_url: https://github.com/emil28092005/SciMesh
+repo_url: https://github.com/emil28092005/SciMesh
+edit_uri: blob/main/mkdocs/
+docs_dir: mkdocs
+
+theme:
+  name: material
+  palette:
+    scheme: slate
+    primary: indigo
+    accent: cyan
+  features:
+    - navigation.instant
+    - navigation.tracking
+    - navigation.top
+    - navigation.expand
+    - search.suggest
+    - content.code.copy
+    - content.code.annotate
+  icon:
+    repo: fontawesome/brands/github
+
+plugins:
+  - search
+  - mkdocstrings:
+      handlers:
+        python:
+          options:
+            show_root_heading: true
+            show_symbol_type_heading: true
+            show_source: false
+            members_order: source
+            show_if_no_docstring: true
+
+markdown_extensions:
+  - admonition
+  - toc:
+      permalink: true
+  - pymdownx.superfences
+  - pymdownx.highlight:
+      anchor_linenums: true
+  - pymdownx.details
+  - pymdownx.tabbed:
+      alternate_style: true
+
+nav:
+  - Home: index.md
+  - SDK:
+      - Overview: sdk/overview.md
+      - Authoring workloads: sdk/authoring-workloads.md
+      - Workload CLI: sdk/cli.md
+      - Worker integration: sdk/worker-integration.md
+  - API reference:
+      - Read me: api/index.md
+      - Artifacts and ports: api/sdk-artifacts.md
+      - Batch scaffold (MapReduceWorkload): api/sdk-batch.md
+      - Conformance runtime: api/sdk-conformance.md
+      - Execution profiles: api/sdk-execution.md
+      - Identities and versions: api/sdk-identity.md
+      - Package integrity: api/sdk-integrity.md
+      - Manifests: api/sdk-manifest.md
+      - Plans and tasks: api/sdk-plans.md
+      - Handler protocols: api/sdk-protocols.md
+      - Registry and discovery: api/sdk-registry.md
+      - Resources: api/sdk-resources.md
+      - Runtime negotiation: api/sdk-runtime.md
+      - Parameter schemas: api/sdk-schema.md
+      - Verification: api/sdk-verification.md
+      - Workflow DAGs: api/sdk-workflow.md
+  - Documentation approach: approach.md
+
+extra:
+  generator: false

mkdocs/api/index.md

@@ -0,0 +1,45 @@
+# API reference
+
+This section is **generated from docstrings** by
+[`mkdocstrings`](https://mkdocstrings.github.io) — it is the complete public
+API surface of `scimesh.sdk`. Markdown pages in `api/` are thin wrappers
+(`::: scimesh.sdk.<module>`) and must not be hand-edited; change the code and
+rebuild with `make docs`.
+
+All value objects are frozen, recursively immutable, JSON-safe, canonically
+serialized, and strict about unknown fields. Constructing them performs
+full validation; invalid input raises `ValueError`.
+
+## Module map
+
+| Page | Module | Contents |
+| --- | --- | --- |
+| [Artifacts and ports](sdk-artifacts.md) | `scimesh.sdk.artifacts` | `ArtifactSchema`, `PortSpec`, `ArtifactRef`, `ArtifactCollection`, `OutputManifest`, `Provenance` |
+| [Batch scaffold](sdk-batch.md) | `scimesh.sdk.batch` | `MapReduceWorkload`, `concatenate_partial_tables` |
+| [Conformance runtime](sdk-conformance.md) | `scimesh.sdk.conformance` | `LocalArtifactStore`, `LocalCoreBatchExecutor`, scoped contexts, round-trip helper |
+| [Execution profiles](sdk-execution.md) | `scimesh.sdk.execution` | `ExecutionProfile`, `RetryPolicy`, `CheckpointPolicy`, `FailureReport` |
+| [Identities](sdk-identity.md) | `scimesh.sdk.identity` | `WorkloadId`, `VersionRange`, `SchemaRef`, `ComponentRef`, `FeatureRequirement` |
+| [Package integrity](sdk-integrity.md) | `scimesh.sdk.integrity` | `installed_distribution_digest` |
+| [Manifests](sdk-manifest.md) | `scimesh.sdk.manifest` | `WorkloadManifest`, `PackageSpec`, `EnvironmentSpec`, `VerifierSpec`, `WorkloadLimits`, trust/determinism enums |
+| [Plans and tasks](sdk-plans.md) | `scimesh.sdk.plans` | `JobRequest`, `ValidatedJob`, `TaskSpec`, `WorkflowPlan`, `ExpansionManifest` |
+| [Handler protocols](sdk-protocols.md) | `scimesh.sdk.protocols` | `Planner`, `Runner`, `Reducer`, `Verifier`, contexts, catalog/sink |
+| [Registry](sdk-registry.md) | `scimesh.sdk.registry` | `WorkloadRegistry`, `WorkloadDefinition`, `AllowedPackage`, discovery |
+| [Resources](sdk-resources.md) | `scimesh.sdk.resources` | `ResourceRequirements`, `ResourceInventory`, `ResourcePool`, accelerators |
+| [Runtime negotiation](sdk-runtime.md) | `scimesh.sdk.runtime` | `RuntimeCapabilities`, `negotiate_manifest`, `CompatibilityError` |
+| [Parameter schemas](sdk-schema.md) | `scimesh.sdk.schema` | Bounded JSON Schema subset |
+| [Verification](sdk-verification.md) | `scimesh.sdk.verification` | Verifiers, decisions, bindings, candidate envelopes |
+| [Workflow DAGs](sdk-workflow.md) | `scimesh.sdk.workflow` | `WorkflowSpec`, `StageSpec`, `ArtifactEdge`, advanced declarations |
+
+## Reading the generated pages
+
+- **Classes** show their full signature, validation rules, and public
+  methods; properties are listed with their type.
+- **Module-level functions** (for example `negotiate_manifest`) document
+  their exact contract and failure modes.
+- Cross-references to other SDK symbols link automatically.
+
+To keep the reference correct:
+
+- write docstrings in **Google style** (`Args:` / `Returns:` / `Raises:`);
+- document validation failures and fail-closed behavior;
+- rebuild with `make docs` after any docstring change.