Serve documentation from the operator UI

mkdocs/api/index.md

@@ -0,0 +1,45 @@
+# API reference
+
+This section is **generated from docstrings** by
+[`mkdocstrings`](https://mkdocstrings.github.io) — it is the complete public
+API surface of `scimesh.sdk`. Markdown pages in `api/` are thin wrappers
+(`::: scimesh.sdk.<module>`) and must not be hand-edited; change the code and
+rebuild with `make docs`.
+
+All value objects are frozen, recursively immutable, JSON-safe, canonically
+serialized, and strict about unknown fields. Constructing them performs
+full validation; invalid input raises `ValueError`.
+
+## Module map
+
+| Page | Module | Contents |
+| --- | --- | --- |
+| [Artifacts and ports](sdk-artifacts.md) | `scimesh.sdk.artifacts` | `ArtifactSchema`, `PortSpec`, `ArtifactRef`, `ArtifactCollection`, `OutputManifest`, `Provenance` |
+| [Batch scaffold](sdk-batch.md) | `scimesh.sdk.batch` | `MapReduceWorkload`, `concatenate_partial_tables` |
+| [Conformance runtime](sdk-conformance.md) | `scimesh.sdk.conformance` | `LocalArtifactStore`, `LocalCoreBatchExecutor`, scoped contexts, round-trip helper |
+| [Execution profiles](sdk-execution.md) | `scimesh.sdk.execution` | `ExecutionProfile`, `RetryPolicy`, `CheckpointPolicy`, `FailureReport` |
+| [Identities](sdk-identity.md) | `scimesh.sdk.identity` | `WorkloadId`, `VersionRange`, `SchemaRef`, `ComponentRef`, `FeatureRequirement` |
+| [Package integrity](sdk-integrity.md) | `scimesh.sdk.integrity` | `installed_distribution_digest` |
+| [Manifests](sdk-manifest.md) | `scimesh.sdk.manifest` | `WorkloadManifest`, `PackageSpec`, `EnvironmentSpec`, `VerifierSpec`, `WorkloadLimits`, trust/determinism enums |
+| [Plans and tasks](sdk-plans.md) | `scimesh.sdk.plans` | `JobRequest`, `ValidatedJob`, `TaskSpec`, `WorkflowPlan`, `ExpansionManifest` |
+| [Handler protocols](sdk-protocols.md) | `scimesh.sdk.protocols` | `Planner`, `Runner`, `Reducer`, `Verifier`, contexts, catalog/sink |
+| [Registry](sdk-registry.md) | `scimesh.sdk.registry` | `WorkloadRegistry`, `WorkloadDefinition`, `AllowedPackage`, discovery |
+| [Resources](sdk-resources.md) | `scimesh.sdk.resources` | `ResourceRequirements`, `ResourceInventory`, `ResourcePool`, accelerators |
+| [Runtime negotiation](sdk-runtime.md) | `scimesh.sdk.runtime` | `RuntimeCapabilities`, `negotiate_manifest`, `CompatibilityError` |
+| [Parameter schemas](sdk-schema.md) | `scimesh.sdk.schema` | Bounded JSON Schema subset |
+| [Verification](sdk-verification.md) | `scimesh.sdk.verification` | Verifiers, decisions, bindings, candidate envelopes |
+| [Workflow DAGs](sdk-workflow.md) | `scimesh.sdk.workflow` | `WorkflowSpec`, `StageSpec`, `ArtifactEdge`, advanced declarations |
+
+## Reading the generated pages
+
+- **Classes** show their full signature, validation rules, and public
+  methods; properties are listed with their type.
+- **Module-level functions** (for example `negotiate_manifest`) document
+  their exact contract and failure modes.
+- Cross-references to other SDK symbols link automatically.
+
+To keep the reference correct:
+
+- write docstrings in **Google style** (`Args:` / `Returns:` / `Raises:`);
+- document validation failures and fail-closed behavior;
+- rebuild with `make docs` after any docstring change.

mkdocs/api/sdk-artifacts.md

@@ -0,0 +1,8 @@
+# Artifacts and ports
+
+Typed artifact schemas, port declarations, immutable artifact references and
+collections, output manifests, and provenance.
+
+::: scimesh.sdk.artifacts
+    options:
+      show_root_heading: false

mkdocs/api/sdk-batch.md

@@ -0,0 +1,9 @@
+# Batch scaffold
+
+The primary authoring surface for `core-batch-v1`: `MapReduceWorkload` and
+the default partial-concatenation helper. See the
+[authoring guide](../sdk/authoring-workloads.md) for a full walkthrough.
+
+::: scimesh.sdk.batch
+    options:
+      show_root_heading: false

mkdocs/api/sdk-conformance.md

@@ -0,0 +1,8 @@
+# Conformance runtime
+
+The trusted, in-process correctness harness: content-addressed artifact
+store, scoped catalog/sink contexts, and `LocalCoreBatchExecutor`.
+
+::: scimesh.sdk.conformance
+    options:
+      show_root_heading: false

mkdocs/api/sdk-execution.md

@@ -0,0 +1,7 @@
+# Execution profiles
+
+Execution, retry, checkpoint, and failure declarations.
+
+::: scimesh.sdk.execution
+    options:
+      show_root_heading: false

mkdocs/api/sdk-identity.md

@@ -0,0 +1,8 @@
+# Identities and versions
+
+Versioned identities used across the SDK: workload ids, version ranges,
+schema and component references, and feature requirements.
+
+::: scimesh.sdk.identity
+    options:
+      show_root_heading: false

mkdocs/api/sdk-integrity.md

@@ -0,0 +1,7 @@
+# Package integrity
+
+Content pinning of installed distributions for allowlisted discovery.
+
+::: scimesh.sdk.integrity
+    options:
+      show_root_heading: false

mkdocs/api/sdk-manifest.md

@@ -0,0 +1,7 @@
+# Manifests
+
+The installed-package manifest and cross-component compatibility contract.
+
+::: scimesh.sdk.manifest
+    options:
+      show_root_heading: false

mkdocs/api/sdk-plans.md

@@ -0,0 +1,8 @@
+# Plans and tasks
+
+Strict job, task, and workflow-plan value objects, plus dynamic expansion
+manifests.
+
+::: scimesh.sdk.plans
+    options:
+      show_root_heading: false

mkdocs/api/sdk-protocols.md

@@ -0,0 +1,8 @@
+# Handler protocols
+
+Author-facing contracts for planners, runners, reducers, verifiers, and the
+bridge-owned catalog/sink/cancellation contexts.
+
+::: scimesh.sdk.protocols
+    options:
+      show_root_heading: false

mkdocs/api/sdk-registry.md

@@ -0,0 +1,8 @@
+# Registry and discovery
+
+Explicit, digest-pinned workload package registry and safe installed
+discovery.
+
+::: scimesh.sdk.registry
+    options:
+      show_root_heading: false

mkdocs/api/sdk-resources.md

@@ -0,0 +1,8 @@
+# Resources
+
+Resource requirements, host inventory, atomic local reservation, and
+accelerator declarations.
+
+::: scimesh.sdk.resources
+    options:
+      show_root_heading: false

mkdocs/api/sdk-runtime.md

@@ -0,0 +1,7 @@
+# Runtime negotiation
+
+Fail-closed SDK/profile/feature/resource compatibility negotiation.
+
+::: scimesh.sdk.runtime
+    options:
+      show_root_heading: false

mkdocs/api/sdk-schema.md

@@ -0,0 +1,7 @@
+# Parameter schemas
+
+The bounded JSON Schema subset used for public workload parameters.
+
+::: scimesh.sdk.schema
+    options:
+      show_root_heading: false

mkdocs/api/sdk-verification.md

@@ -0,0 +1,8 @@
+# Verification
+
+Verifier primitives, decisions, coordinator bindings, and candidate
+envelopes for quorum acceptance.
+
+::: scimesh.sdk.verification
+    options:
+      show_root_heading: false

mkdocs/api/sdk-workflow.md

@@ -0,0 +1,8 @@
+# Workflow DAGs
+
+Versioned workflow graphs, stage declarations, artifact edges, and bounded
+advanced-stage declarations (loops, streams, gangs, side effects).
+
+::: scimesh.sdk.workflow
+    options:
+      show_root_heading: false

mkdocs/approach.md

@@ -0,0 +1,93 @@
+# Documentation approach
+
+These rules are the contract for this site. Every page must follow them;
+reviewers enforce them.
+
+## 1. Purpose and scope
+
+This MkDocs site documents **the Workload SDK**: how to use it, how to
+author workloads, and the complete `scimesh.sdk` API. It does **not** host
+the project's internal documentation (contracts, task briefs, planning
+documents) — those live in the repository's `docs/` directory and are not
+part of the site. Where a guide needs a contract detail, link to the
+repository file in prose; do not copy its content.
+
+## 2. Page types and status markers
+
+Every page starts with one of:
+
+- **concept** — explains a model (for example the SDK overview);
+- **guide** — how to accomplish a task (authoring workloads, CLI, worker);
+- **reference** — generated API documentation, never hand-written.
+
+Guides may be marked with an admonition when a section describes future or
+fail-closed behavior:
+
+```markdown
+!!! warning "Not yet supported"
+    The v1 coordinator contract does not persist resource requirements...
+```
+
+## 3. The reference is generated, not written
+
+- `api/` pages contain only mkdocstrings directives
+  (`::: scimesh.sdk.<module>`); editing them by hand is an error.
+- Public API must be documented in **Google-style docstrings** in the code:
+  `Args:`, `Returns:`, `Raises:`.
+- Every validation failure and fail-closed path must be documented in the
+  docstring.
+- After any docstring change, rebuild: `make docs`.
+
+## 4. Terminology
+
+Use the single glossary from `PLAN.md`:
+
+| Term | Meaning |
+| --- | --- |
+| Job / Run | A user-requested full computation |
+| Task | One independently executable unit of a job |
+| Attempt | A lease for one task execution |
+| Artifact | A durable input, shard, partial, or final result |
+| Workload | A user script (package) built on the SDK |
+
+Never introduce synonyms (`pipeline`, `run` for task, etc.). The word
+"workload" in this site means an SDK workload (a user script), not a
+"workload" in the performance sense.
+
+## 5. Code and output conventions
+
+- Use language-tagged fenced blocks: ```python, ```bash, ```text, ```json.
+- Never include local machine paths, tokens, or private data in examples.
+- Show complete runnable examples; prefer the real built-in workloads
+  (`molwt-filter`, `descriptor-batch`) over invented ones.
+- Keep command output minimal and accurate; regenerate it, don't retype it.
+
+## 6. Linking
+
+- Relative links inside `mkdocs/` (for example `../api/sdk-batch.md`).
+- Repository files outside the site (`docs/`, `PLAN.md`) are referenced in
+  prose with their path, not linked as site pages.
+- Every guide must link to the relevant API pages.
+
+## 7. When to write or update
+
+- **New workload** → update `sdk/authoring-workloads.md` examples and the
+  UI workload catalog (`make workloads-export`).
+- **SDK API change** → update docstrings; the reference rebuilds.
+- **Worker/coordinator behavior change** → update
+  `sdk/worker-integration.md` and the fail-closed warnings.
+- **New CLI surface** → update `sdk/cli.md`.
+- Behavior changes without documentation updates are incomplete changes.
+
+## 8. Build and verification
+
+```bash
+make docs          # build into site/
+make docs-serve    # http://localhost:8000
+```
+
+- `mkdocs build` must succeed with no errors.
+- New or changed pages must render (check the generated HTML, not just the
+  markdown).
+- The site is served inside the coordinator UI at `/ui/docs/`; the demo
+  mounts `site/` automatically.

mkdocs/index.md

@@ -0,0 +1,86 @@
+# SciMesh
+
+SciMesh is a local-first platform for scientific computation on molecular
+datasets. It turns a scientific run into independent tasks, dispatches them
+to Python workers, and deterministically combines the partial results into a
+checksum-protected final artifact.
+
+The two halves of the project:
+
+- **The Workload SDK (`scimesh.sdk`)** — a strict Python framework for
+  authoring scientific workloads. Workloads are ordinary user scripts built
+  on the SDK; they run locally, in the conformance harness, and on claimed
+  coordinator tasks without touching any other part of the program.
+- **The coordinator and worker** — a Go/PostgreSQL coordinator with an
+  operator UI and Python worker agents that execute SDK-built workloads over
+  an HTTP contract.
+
+## What is implemented
+
+- **SDK-built workloads**: `similarity-search` (exact top-k Tanimoto search),
+  `similarity-graph` (exact sparse similarity graph with pair-coverage),
+  `descriptor-batch` (pinned RDKit 2D descriptors), and `molwt-filter`
+  (molecular-weight filter — the minimal authoring example).
+- **`MapReduceWorkload`**: the primary authoring scaffold. A subclass
+  declares identity, parameters, ports, and scientific hooks; the SDK
+  assembles the manifest, map/reduce stages, the digest-pinned
+  planner/runner/reducer, and the exact-artifact verifier.
+- **A local conformance runtime** (`LocalCoreBatchExecutor`): a trusted,
+  in-process harness that validates scientific parity, sealed outputs,
+  provenance, and limits.
+- **A distributed worker** that executes the same SDK workload handlers on
+  tasks claimed from the coordinator, with digest-pinned `TaskSpec`s,
+  resource reservation, and allowlist-driven workload discovery.
+- **An operator UI** served by the coordinator: the control room, a workload
+  library page, and this documentation site at `/ui/docs/`.
+
+## Quick start
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e '.[dev]'
+```
+
+List the installed SDK workloads and run one locally:
+
+```bash
+scimesh workload list
+scimesh workload run molwt-filter \
+  --input molecules.tsv \
+  --params '{"min_molwt": 40.0}' \
+  -o filtered.csv
+```
+
+Run the local scientific CLI workloads:
+
+```bash
+scimesh help
+```
+
+Start the full demo (PostgreSQL, coordinator, UI, two workers):
+
+```bash
+make demo-ui
+# open http://localhost:18080/ui  (root@scimesh.local / rootpassword)
+```
+
+Build and serve this documentation site:
+
+```bash
+make docs
+make docs-serve      # http://localhost:8000
+```
+
+## Where to go next
+
+- [SDK overview](sdk/overview.md) — what the SDK is and is not.
+- [Authoring workloads](sdk/authoring-workloads.md) — write your first
+  workload with `MapReduceWorkload`.
+- [Workload CLI](sdk/cli.md) — list, run, and export workloads from the
+  command line.
+- [Worker integration](sdk/worker-integration.md) — how the distributed
+  worker executes SDK workloads.
+- [API reference](api/index.md) — the complete `scimesh.sdk` API, generated
+  from docstrings.
+- [Documentation approach](approach.md) — the rules this site is written by.

mkdocs/sdk/authoring-workloads.md

@@ -0,0 +1,262 @@
+# Authoring workloads
+
+A workload is a **user script** that imports the SDK. For the standard
+`core-batch-v1` shape — one input dataset, deterministic shards, one merged
+result — subclass [`MapReduceWorkload`](../api/sdk-batch.md) and implement
+the scientific hooks. The SDK assembles everything else: the immutable
+manifest, the map/reduce stages, the workflow DAG, the digest-pinned
+planner/runner/reducer handlers, and the exact-artifact verifier.
+
+## The minimal workload
+
+This is the complete `molwt-filter` workload (the built-in minimal example):
+it filters molecules by exact RDKit molecular weight and needs only **one**
+scientific hook, because the scaffold's default sharding and concatenation
+already cover partitioning and reduction.
+
+```python
+from pathlib import Path
+from typing import Any, Mapping
+
+from scimesh.sdk import (
+    ArtifactSchema,
+    ComponentRef,
+    MapReduceWorkload,
+    PortSpec,
+    SchemaRef,
+    WorkloadId,
+)
+from scimesh.sdk.registry import WorkloadDefinition
+
+
+class MolwtFilterWorkload(MapReduceWorkload):
+    workload_id = WorkloadId("molwt-filter", "1.0.0")
+    description = (
+        "Filter molecules by exact RDKit molecular weight, one canonical "
+        "CSV row per kept input molecule, in deterministic input order."
+    )
+    parameters_schema = {
+        "type": "object",
+        "additionalProperties": False,
+        "properties": {
+            "min_molwt": {"type": "number", "minimum": 0},
+            "max_molwt": {"type": "number", "minimum": 0},
+            "skip_invalid": {"type": "boolean", "default": True},
+        },
+    }
+    input_port = PortSpec(ArtifactSchema(
+        SchemaRef("molecule-table", 1),
+        "text/tab-separated-values",
+        "utf-8",
+        max_bytes=10 * 1024 * 1024 * 1024,
+        validator=ComponentRef("delimited-table", 1),
+        validator_configuration={"required_columns": ["canonical_smiles", "chembl_id"]},
+        max_records=100_000_000,
+        canonicalizer="scimesh-tsv-v1",
+    ))
+    partial_port = output_port = PortSpec(ArtifactSchema(
+        SchemaRef("molwt-filtered-table", 1),
+        "text/csv",
+        "utf-8",
+        max_bytes=100 * 1024 * 1024 * 1024,
+        validator=ComponentRef("delimited-table", 1),
+        validator_configuration={"columns": ["chembl_id", "canonical_smiles", "molwt"]},
+        max_records=100_000_000,
+        canonicalizer="molwt-filtered-table-v1",
+    ))
+    map_parameter_names = ("min_molwt", "max_molwt", "skip_invalid")
+
+    def domain_validate(self, parameters: Mapping[str, Any]) -> None:
+        if parameters.get("min_molwt") is None and parameters.get("max_molwt") is None:
+            raise ValueError("at least one of min_molwt or max_molwt is required")
+
+    def compute_shard(self, inputs, parameters, output_path):
+        # `inputs` maps every map-stage input port to a materialized file;
+        # here there is one port: "input".
+        return filter_molecules_by_molwt(
+            inputs["input"],
+            output_path,
+            min_molwt=parameters.get("min_molwt"),
+            max_molwt=parameters.get("max_molwt"),
+            skip_invalid=parameters.get("skip_invalid", True),
+        )
+```
+
+Because the scaffold provides default `partition_input` (row-bounded shards
+that keep the header, `shard_rows` rows each) and default `reduce_partials`
+(concatenation with one header), no further code is required.
+
+## Required class attributes
+
+| Attribute | Meaning |
+| --- | --- |
+| `workload_id` | `WorkloadId("name", "1.0.0")` — the immutable identity |
+| `description` | Shown in `scimesh workload list`, the UI library, and the catalog export |
+| `parameters_schema` | Strict JSON object schema (`additionalProperties: false`); the registry validates jobs against it before the planner runs |
+| `input_port` | External input port (`PortSpec`) |
+| `partial_port` | One map output artifact (`PortSpec`) |
+| `output_port` | Final result artifact (`PortSpec`) |
+
+## Optional class attributes
+
+| Attribute | Default | Meaning |
+| --- | --- | --- |
+| `map_stage_inputs` | `{"input": input_port}` | Map-stage input ports; extra ports must share the external input schema |
+| `map_parameter_names` | `()` | Parameter projection for map tasks |
+| `reduce_parameter_names` | `map_parameter_names` | Parameter projection for the reducer |
+| `capabilities` | `(workload_id.name,)` | Advertised capabilities |
+| `trust_modes` | `(trusted, untrusted_quorum)` | Declared trust modes |
+| `workflow_id` | `"<name>-map-reduce-v1"` | Workflow identity |
+| `limits` | derived from port bounds | `WorkloadLimits` |
+| `resources` / `execution` | CPU-1 core defaults | Per-task resource and execution profile |
+| `shard_rows` | `1000` | Rows per shard for the default `partition_input` |
+| `map_entry_point` / `reduce_entry_point` | derived from the module | Handler keys (can stay default) |
+
+## Scientific hooks
+
+Override only what your workload needs:
+
+| Hook | Default | Purpose |
+| --- | --- | --- |
+| `domain_validate(parameters)` | none | Extra job-parameter validation (the JSON schema already ran) |
+| `resolved_parameters(request)` | `dict(request.parameters)` | Values persisted into the plan |
+| `resolved_parameters_for_plan(job, input_path, resolved)` | unchanged | Plan-time enrichment (e.g. one-time query resolution) |
+| `partition_input(input_path, parameters, workspace)` | row-bounded sharding | Deterministic shard files, one per map task |
+| `plan_tasks(shard_paths, resolved, job, negotiated, map_stage, context)` | one task per shard | Custom task construction |
+| `task_parameters(resolved)` | filtered projection | Map-task parameters |
+| `compute_shard(inputs, parameters, output_path)` | **required** | One map task; returns metrics |
+| `parse_partial_key(key)` / `validate_partial_keys(parsed)` | `map.<8-digit>`, contiguous | Partial-key policy for the reducer |
+| `reduce_partials(partial_paths, parameters, output_path)` | header-preserving concatenation | Deterministic merge |
+
+Hooks must be **deterministic**: identical inputs and parameters must
+produce byte-identical partials, in any worker, in any completion order.
+Floats should be formatted with a fixed precision (for example `f"{v:.6f}"`),
+and output row order must be canonical.
+
+## Running a workload locally
+
+```python
+from scimesh.sdk import (
+    ArtifactCollection,
+    JobRequest,
+    LocalArtifactStore,
+    LocalCoreBatchExecutor,
+    WorkloadRegistry,
+)
+from scimesh.workloads.library import default_sdk_runtime
+
+workload = MolwtFilterWorkload(
+    package_digest=current_scimesh_package_digest(),
+    environment_digest=current_environment_digest(),
+)
+registry = WorkloadRegistry()
+registry.register(workload.definition(), enabled=True)
+
+store = LocalArtifactStore(Path("artifacts"))
+artifact = store.import_file(
+    Path("molecules.tsv"),
+    declaration=workload.manifest.inputs["input"].schema,
+)
+request = JobRequest(
+    workload=workload.manifest.workload,
+    parameters={"min_molwt": 40.0},
+    inputs={"input": ArtifactCollection.single(artifact)},
+)
+result = LocalCoreBatchExecutor(
+    registry, default_sdk_runtime(), store, Path("work"),
+).execute(request, workload.manifest.package.digest)
+
+print(store.materialize(result.outputs["result"].items[0].artifact).read_text())
+```
+
+`LocalCoreBatchExecutor` runs the full pipeline — negotiation, planning,
+map tasks, stage verification, reduce, final verification — in-process. It
+is a correctness harness, not an isolation boundary: it accepts only
+trusted, single-threaded, trusted-network profiles and rejects everything
+else before a handler runs.
+
+## Custom planning: block pairs
+
+Workloads that need more than one input per task override `plan_tasks` and
+`map_stage_inputs`. The built-in `similarity-graph` plans one task per block
+pair `(i, j)` with `i <= j`:
+
+```python
+map_stage_inputs = {"left": block_port, "right": block_port}
+
+def plan_tasks(self, shard_paths, resolved, job, negotiated, map_stage, context):
+    block_refs = [
+        context.sink.seal(path, declaration=self.input_port.schema)
+        for path in shard_paths
+    ]
+    tasks = []
+    for left in range(len(block_refs)):
+        for right in range(left, len(block_refs)):
+            tasks.append(self.task_spec(
+                map_stage, job, negotiated,
+                f"map/{left:04d}x{right:04d}",
+                {"left_block": left, "right_block": right,
+                 "threshold": resolved["threshold"]},
+                {"left": ArtifactCollection.single(block_refs[left]),
+                 "right": ArtifactCollection.single(block_refs[right])},
+            ))
+    return tasks
+```
+
+Its reducer overrides `parse_partial_key`/`validate_partial_keys` to parse
+`map.<i>x<j>` keys and enforce the pair-coverage invariant (every unordered
+molecule pair compared exactly once).
+
+## Packaging and discovery
+
+Workloads are installed as part of a Python distribution and declared as
+entry points:
+
+```toml
+[project.entry-points."scimesh.workloads"]
+"my-workload@1.0.0" = "my_package.workload:workload_definition"
+```
+
+The factory returns a `WorkloadDefinition` (or a `MapReduceWorkload`
+instance with a `definition()` method). An administrator then supplies an
+`AllowedPackage(distribution, WorkloadId, "sha256:...")` allowlist entry;
+discovery loads the entry point only when the installed package content
+matches the pinned digest.
+
+```python
+from scimesh.sdk import AllowedPackage, WorkloadId, WorkloadRegistry
+
+registry = WorkloadRegistry()
+registry.discover_installed((
+    AllowedPackage("my-dist", WorkloadId("my-workload", "1.0.0"), "sha256:" + "a" * 64),
+))
+```
+
+## Tests and golden parity
+
+Add a regression test for every behavioral change:
+
+- **Byte parity**: run the workload through `LocalCoreBatchExecutor` and
+  compare the final artifact bytes with a single-process reference computed
+  by the scientific core directly.
+- **Determinism**: planning twice must produce identical JSON; results must
+  be invariant to shard/block sizes.
+- **Fail-closed**: invalid parameters, missing ports, forged outputs, and
+  unsupported trust modes must be rejected.
+- **Verifier policy**: for `untrusted_quorum`, two distinct owners with
+  identical outputs must be accepted, conflicting outputs rejected.
+
+Use small TSV fixtures — never the full ChEMBL extract, which takes minutes
+even for one shard.
+
+## Rules
+
+1. Keep the scientific core callable without a coordinator.
+2. Inline a strict JSON parameter schema (`additionalProperties: false`);
+   the planner still performs domain validation.
+3. Give every external and stage port an `ArtifactSchema` with bounds.
+4. Return only sink-sealed artifacts in `OutputManifest`; the harness binds
+   task key and provenance itself.
+5. Select a verifier compatible with determinism and trust: v1 permits
+   `untrusted_quorum` only for `byte_exact` plus `exact-artifact@1`.
+6. Never put a filesystem path or transport URL into a plan or task.

mkdocs/sdk/cli.md

@@ -0,0 +1,76 @@
+# Workload CLI
+
+`scimesh workload` is a generic SDK tool: it contains no workload-specific
+logic, so new workloads never require changes to the CLI or any other part
+of the program.
+
+```text
+scimesh workload list|run|export
+```
+
+## list
+
+Show every installed and enabled SDK workload:
+
+```bash
+scimesh workload list
+```
+
+Output: `name  version  description  [enabled <digest-prefix>]`. With
+`SCIMESH_WORKLOAD_ALLOWLIST` set, allowlisted installed workloads are shown
+instead of (or in addition to) the built-ins.
+
+## run
+
+Execute one workload locally against an input file:
+
+```bash
+scimesh workload run molwt-filter \
+  --input molecules.tsv \
+  --params '{"min_molwt": 40.0}' \
+  --shard-rows 1000 \
+  -o filtered.csv
+```
+
+| Option | Meaning |
+| --- | --- |
+| `name` | Workload name, for example `descriptor-batch` |
+| `--version` | Exact workload version (default: the enabled one) |
+| `--input FILE` | Input dataset file |
+| `--params JSON` | Job parameters as a JSON object |
+| `--shard-rows N` | Rows per planned shard (default 10000) |
+| `-o, --output FILE` | Output path for the final artifact |
+| `--work-dir DIR` | Working directory (default: a fresh temporary directory) |
+
+```bash
+scimesh workload run similarity-search \
+  --input molecules.tsv \
+  --params '{"query_smiles": "CCO", "top_k": 20, "progress_every": 0}'
+```
+
+The runner prints the saved path and the final metrics.
+
+## export
+
+Write the workload library as a JSON catalog — the same catalog the
+coordinator UI embeds on its **Workloads** page:
+
+```bash
+scimesh workload export -o workloads.json
+```
+
+Regenerate the coordinator's embedded catalog with:
+
+```bash
+make workloads-export
+```
+
+## Environment
+
+| Variable | Meaning |
+| --- | --- |
+| `SCIMESH_WORKLOAD_ALLOWLIST` | JSON array of `{distribution, name, version, digest}` entries; discovery loads the matching installed `scimesh.workloads` entry points |
+| `SCIMESH_CAPABILITIES` | Comma-separated capabilities the worker advertises (default `similarity-search,similarity_search`) |
+
+Both variables are read by the worker (`scimesh-worker`) and the workload
+CLI.

mkdocs/sdk/overview.md

@@ -0,0 +1,90 @@
+# SDK overview
+
+`scimesh.sdk` is the **framework only**. It contains no scientific workload
+code. Workloads are user Python scripts and packages that import the SDK and
+live outside it — the built-in SciMesh workloads under `scimesh/workloads/`
+are exactly such scripts, and a future workload library can follow the same
+shape.
+
+```text
+scientific implementation -> SDK manifest/plan -> map tasks -> reduce -> verifier
+```
+
+## The `core-batch-v1` profile
+
+The implemented authoring profile is **`core-batch-v1`**: a static
+map/reduce workflow with one external input and one final output.
+
+- one input dataset (a delimited table, typically TSV);
+- deterministic row-bounded (or otherwise partitioned) shards;
+- one **map** task per shard, each producing one partial artifact;
+- one **reduce** task merging the accepted partials into the final artifact;
+- an acceptance **verifier** for every output-producing stage.
+
+The SDK's strict value objects make the whole contract explicit:
+
+- `WorkloadManifest` pins identity, compatibility ranges, package and
+  environment digests, parameter schema, workflow, ports, determinism,
+  trust modes, verifier, and limits;
+- `WorkflowSpec`/`StageSpec` describe a typed acyclic DAG;
+- `TaskSpec`/`WorkflowPlan` carry the exact workload pin (package, manifest,
+  environment digests, trust mode, negotiated features);
+- `OutputManifest`/`Provenance` describe sealed durable results;
+- artifacts are content-addressed (`sha256`), immutable, and free of
+  transport URLs and local paths.
+
+## What the SDK provides
+
+| Area | Modules | Purpose |
+| --- | --- | --- |
+| Identity | `identity`, `schema` | `WorkloadId`, versions, schema refs, bounded JSON parameter schemas |
+| Declarations | `manifest`, `workflow`, `artifacts`, `execution`, `resources` | Manifest, DAG stages, typed ports, execution/resource profiles |
+| Planning | `plans` | `JobRequest`, `TaskSpec`, `WorkflowPlan` |
+| Registry | `registry`, `integrity` | Allowlisted discovery, digest pinning |
+| Negotiation | `runtime` | Fail-closed compatibility negotiation |
+| Execution | `conformance` | `LocalCoreBatchExecutor`, `LocalArtifactStore` |
+| Verification | `verification` | Exact, canonical, and numeric verifier primitives |
+| Authoring | `batch` | `MapReduceWorkload` scaffold |
+
+## Security model
+
+- Workload discovery requires an **administrator allowlist**: exact
+  distribution, workload name/version, and a `sha256:` package digest.
+  Discovery measures the installed package before and after importing the
+  entry point and fails transactionally on any mismatch. Job parameters can
+  never name a module, entry point, or executable.
+- Compatibility negotiation is **fail-closed**: if the runtime does not
+  advertise a declared feature (gangs, GPU, streams, checkpoints, secrets,
+  retries, process pools, dynamic expansion), the job is rejected before any
+  workload code runs. Declaring an advanced profile never silently enables
+  it.
+- Handlers receive **bridge-owned contexts**: `ArtifactCatalog` for verified
+  input materialization and `ArtifactSink` for sealing outputs. They never
+  see database credentials, upload URLs, or coordinator tokens.
+- The **local conformance executor** is deliberately trusted and in-process.
+  It rejects anything but `TrustMode.TRUSTED`, a single non-nested host
+  thread, and the trusted network policy — a contract, not a limitation.
+
+## What is not supported yet
+
+The coordinator contract (v1) persists flat one-input/one-result tasks.
+Until a versioned protocol rollout lands, the following remain **fail-closed
+by design**:
+
+- distributed execution of multi-input map stages (for example the
+  block-pair `similarity-graph` tasks) and of workloads beyond the v1
+  contract;
+- coordinator-backed GPU scheduling, streams, gang leases, checkpoints,
+  retries, and secret injection;
+- dynamic (plan-stage) expansion.
+
+The verifier primitives `CanonicalRecordVerifier` and
+`NumericToleranceVerifier` exist and are tested, but only the
+`ExactArtifactVerifier` (whole-file SHA-256) is eligible for
+`untrusted_quorum` in v1.
+
+## Next
+
+- [Authoring workloads](authoring-workloads.md) — build a workload with
+  `MapReduceWorkload`.
+- [API reference](../api/index.md) — the complete SDK surface.

mkdocs/sdk/worker-integration.md

@@ -0,0 +1,82 @@
+# Worker integration
+
+The Worker Agent (`scimesh-worker`) is a coordinator client, never a
+database client. It polls the coordinator over HTTP, executes SDK-built
+workloads, and uploads partial results through the coordinator — results
+never carry `file://` or `worker://` URIs, and failures go to `/failure`.
+
+The same scientific handlers run in three places: the local CLI cores, the
+`LocalCoreBatchExecutor` conformance harness, and the worker — because the
+worker executes the workload's own SDK runner.
+
+## Claim lifecycle
+
+```text
+register -> claim (one task) -> download input + verify sha256
+         -> run via SDK bridge -> upload partial -> submit result
+```
+
+- **Register**: the worker advertises its capabilities (`similarity-search`
+  by default; extend with `SCIMESH_CAPABILITIES`).
+- **Claim**: atomic lease of one task; `204` means idle.
+- **Download**: the input is streamed and its SHA-256 verified; the bearer
+  token is stripped on cross-origin redirects.
+- **Heartbeat**: a background thread renews the lease from the returned
+  deadline at less than half the remaining TTL.
+- **Upload**: the partial CSV is streamed to the coordinator with
+  `X-Worker-ID` / `X-Task-Attempt` headers, then the completion is submitted
+  referencing the coordinator-owned artifact id.
+- **Failure**: sanitized `error_code` + message (≤300 chars, no local
+  paths, no tracebacks); transient transport errors are retried.
+
+## The SDK execution bridge
+
+`scimesh/worker/runners.py` is workload-generic. For a claimed task it:
+
+1. normalizes the workload name (underscores → hyphens) and looks up the
+   loaded definition;
+2. runs compatibility negotiation against a runtime derived from the loaded
+   definitions (capabilities + pinned environment digests) and the worker
+   inventory (CPU/memory from configuration);
+3. verifies the workload's map stage fits the v1 contract — a single
+   `input` port and a single `partial` output — otherwise it fails closed
+   with a clear message;
+4. imports the downloaded input into a content-addressed local store;
+5. builds a digest-pinned `TaskSpec` (package/manifest/environment digests,
+   trust mode, negotiated features, stage resources and execution profile);
+6. reserves resources through `ResourcePool` and runs the workload's own
+   `Runner` with a `LocalTaskContext` (scoped catalog/sink, provenance,
+   cancellation flag);
+7. validates the returned `OutputManifest` (task key, provenance, sealed
+   vs. declared artifacts, byte budget) and returns the sealed partial for
+   upload.
+
+Scientific policy lives in the workload: `query_id` resolution, parameter
+validation, and `max_rows` rejection are all handled by the workload's own
+hooks — the bridge passes task parameters through unchanged.
+
+## Loading workloads
+
+The worker loads workloads from `SCIMESH_WORKLOAD_ALLOWLIST` (a JSON array
+of `{distribution, name, version, digest}` entries matched against installed
+`scimesh.workloads` entry points). Discovery measures the installed package
+before and after importing and fails transactionally on any mismatch. When
+no allowlist is configured, the worker falls back to the built-in
+`similarity-search`.
+
+```bash
+SCIMESH_WORKLOAD_ALLOWLIST='[{"distribution": "scimesh",
+  "name": "descriptor-batch", "version": "1.0.0",
+  "digest": "sha256:..."}]' scimesh-worker --coordinator-url https://...
+```
+
+## v1 contract limits
+
+The coordinator protocol v1 persists flat one-input/one-result tasks. Until
+a versioned protocol rollout:
+
+- map stages with more than one input port (for example
+  `similarity-graph`'s block pairs) are **rejected by the bridge** — the
+  coordinator does not create such tasks anyway;
+- `max_rows` is a plan-time option and is rejected per task;
+- workloads beyond the allowlisted set are rejected as unsupported.