Update workload SDK roadmap

PLAN.md

@@ -5,11 +5,13 @@
 > platform. It is intentionally detailed enough to split into independent task
 > briefs for developers or coding agents.
 >
-> **Planning baseline.** This branch starts from `Workers`: the Python package
+> **Current planning baseline (2026-08-01).** The Go/PostgreSQL coordinator,
-> has local `similarity-search` and `similarity-graph` workloads plus a Worker
+> Python Worker Agent, versioned distributed-workload protocol, artifact-backed
-> Daemon client. The coordinator and PostgreSQL implementation do not yet
+> task lifecycle, reducer orchestration, operator UI, User Service, and
-> exist. The Worker contract and the Go/PostgreSQL design briefs in `docs/` are
+> distributed `similarity-search` are implemented on `main`. The evidence-based
-> part of this plan.
+> completion tracker is [`STATUS.md`](STATUS.md); this document defines the
+> remaining direction and dependencies. Earlier descriptions of a missing
+> coordinator are historical context, not current work.
 
 ---
 
@@ -51,7 +53,7 @@ Coordinator reducer -> final artifact -> download/status API
 
 ### 2.1 In scope
 
-- Go 1.22+ coordinator service with PostgreSQL 15+;
+- Go 1.25+ coordinator service with PostgreSQL 15+;
 - Python Worker Daemon running existing SciMesh workloads locally;
 - durable job, task, worker, and artifact metadata;
 - local coordinator-managed artifact storage for the first deployment;
@@ -66,8 +68,8 @@ Coordinator reducer -> final artifact -> download/status API
 
 - cloud object storage, Kubernetes, autoscaling, and multi-region operation;
 - arbitrary shell commands sent by coordinator to workers;
-- user accounts, multi-tenancy, billing, or sophisticated authorization
+- billing and sophisticated multi-tenant administration beyond the implemented
-  (planned after the first release in CTX-15);
+  User Service and owner scoping;
 - GPU scheduling and multiprocessing inside a worker;
 - Docker as a required runtime dependency;
 - video/CV processing implementation;
@@ -491,14 +493,82 @@ brute-force graph for both `greater` and `less` threshold directions.
 
 ### 7.3 Future workload policy
 
-A new workload is accepted only when it supplies:
+A workload is more than a runner. It must define validation, planner and
+reducer behavior, worker allowlist/capabilities, input and output artifacts,
+UI/API parameters, reproducible execution environment, result verification,
+golden cross-worker fixtures, and applicable resource limits. The future public
+contract is described in [`docs/scimesh-sdk-roadmap.md`](docs/scimesh-sdk-roadmap.md).
 
-- an input/parameter validator;
+Every workload declaration must classify its task decomposition and input/output
-- an explicit sharding strategy;
+artifact shapes, determinism, reduction semantics, verifier mode, supported
-- bounded-memory task execution;
+trust modes, CPU/memory/accelerator needs, and maximum output growth. The
-- deterministic reduction semantics;
+initial profiles are:
-- fixture-based local and distributed correctness tests;
+
-- a `describe()` payload for UI/API discovery.
+| Profile | Current acceptance policy |
+| --- | --- |
+| Byte-exact deterministic | Supported for untrusted quorum when whole artifacts have identical SHA-256. |
+| Canonical-exact deterministic | Deferred until the parser, schema, ordering, encoding, and serializer are versioned. |
+| Numeric deterministic with tolerance | Deferred until structured numeric comparison exists. |
+| Stochastic/search-based | Requires domain-specific evidence, repeated runs, or trusted execution. |
+| Trusted-only or domain-verified | May be planned only with an explicit trust policy and verifier. |
+
+The current untrusted quorum records one vote per owner and accepts a task only
+when distinct owners upload artifacts with the same complete-file SHA-256. It
+therefore supports only the byte-exact profile (or a workload that first makes
+its output byte-identical through a specified canonicalization step). Reducers
+must fail safely rather than silently merge inconsistent partial results.
+
+Before a workload is admitted to untrusted execution it needs a reproducibility
+gate: pinned environment/container digest and dependency versions; fixed locale,
+timezone, UTF-8/newline/CSV settings; explicit invalid-row and algorithm
+options; canonical representation and ordering; deterministic filenames/archive
+metadata; golden fixtures from two independently provisioned workers; local vs
+distributed parity; and retry/out-of-order completion tests. A loose dependency
+constraint is insufficient for byte-exact quorum.
+
+Near-term critical path:
+
+```text
+distributed similarity-graph
+  -> reliability, security, and cross-language CI
+    -> stable first release
+      -> SDK foundation and descriptor-batch
+        -> additional deterministic workloads
+```
+
+Initial deterministic-workload backlog: `descriptor-batch` (the first SDK
+reference workload), molecule standardization, SMARTS screening, fingerprint
+export, fixed-template SMIRKS enumeration with strict caps, and reaction
+validation/descriptors. `similarity-graph` remains ahead of this backlog.
+Bounded combinatorial libraries and seeded conformers need specialized controls.
+ML, retrosynthesis, docking, QM, molecular dynamics, and GPU workloads are
+deferred until verifier/trust and reproducibility requirements are met.
+
+### 7.4 Future verification, concurrency, and accelerators
+
+Verification is a future versioned workload capability, not permanent
+whole-file-SHA logic. Planned modes are `ExactArtifactVerifier`,
+`CanonicalRecordVerifier`, `NumericToleranceVerifier`, `DomainSpecificVerifier`,
+and `TrustedWorkerPolicy`. Exact SHA-256 remains the first and safest mode;
+canonical and numeric modes must compare bounded structured data and publish
+sanitized evidence and failure reasons.
+
+Worker concurrency remains **1** until implemented and tested. Its target model
+is one physical machine running one Worker Agent with `N` execution slots and
+one isolated subprocess per active Task, rather than one registered worker per
+CPU core. `max_concurrency` must be separate from `cpu_count`; each task keeps
+its own heartbeat, attempt directory, lease lifecycle, resource request, and
+graceful-drain behavior. CPU-bound scientific code should use processes and
+avoid nested oversubscription.
+
+Accelerator support is also deferred. The coordinator matches generic resource
+requirements; the Worker Agent discovers and isolates devices (including
+`CUDA_VISIBLE_DEVICES`) and owns process/accounting lifecycle; the Python
+workload owns batching, memory strategy, deterministic output, and scientific
+validation; reducers/verifiers define CPU/GPU-independent semantics. CUDA and
+scientific kernels do not belong in the Go coordinator. GPU work follows stable
+CPU slices, generic resource requirements, pinned worker images, and tested
+CPU/GPU or domain-valid equivalence.
 
 ---
 

STATUS.md

@@ -1,7 +1,7 @@
 # SciMesh Status
 
-**Updated:** 2026-07-27
+**Updated:** 2026-08-01
-**Branch baseline:** `main` at `f5ead0a` (team and scaling-roadmap documentation)
+**Branch baseline:** `main` at `b9a975b` (self-service worker enrollment)
 
 ## Current state
 
@@ -27,6 +27,9 @@ The User Service is merged into `main`. It owns user accounts, authentication,
 roles, and verified-contributor status; the coordinator scopes user jobs and
 worker operations to the authenticated owner. Its documented v1 contract is in
 [`docs/user-service-api-contract.md`](docs/user-service-api-contract.md).
+Users can create and revoke worker keys for self-service Worker Agent
+enrollment. Untrusted workers require quorum agreement from distinct owners on
+the complete result-artifact SHA-256 before a task is accepted.
 
 ## Milestone tracker
 
@@ -45,7 +48,7 @@ worker operations to the authenticated owner. Its documented v1 contract is in
 | CTX-10 Distributed similarity-graph | Not started | Local reference exists. |
 | CTX-11 Dashboard/operator view | Implemented | Protected live control room: recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, and bounded polling. |
 | CTX-12 Reliability, security, CI | In progress | Unit, race, PostgreSQL integration, and smoke checks exist; CI hardening remains. |
-| CTX-15 User Service and access control | Implemented | Go unit/race checks, Docker/PostgreSQL integration, and coordinator ownership regression checks pass locally. |
+| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, self-service enrollment, and quorum-backed untrusted workers are merged; local Go/Python and Docker/PostgreSQL checks passed. |
 
 ## Next recommended assignment
 

docs/scimesh-sdk-roadmap.md

@@ -0,0 +1,123 @@
+# SciMesh Workload SDK roadmap
+
+**Status:** future design and sequencing document. No SDK package, commands, or
+general verifier abstraction described here is implemented yet.
+
+## Purpose and boundaries
+
+The SDK should let a scientific developer add an allowlisted workload without
+learning coordinator internals or writing SQL, while keeping one scientific
+implementation usable locally and in distributed execution:
+
+```text
+scientific implementation -> local adapter -> planner/tasks -> reducer -> verifier
+```
+
+It must not execute arbitrary code or shell commands supplied by a coordinator.
+SDK v1 is not a public marketplace, generic container/job runner, cross-language
+SDK, automatic correctness-proof system, or immediate route to stochastic ML or
+GPU workloads.
+
+The current foundation is the Python `DistributedWorkload` protocol and registry
+under `scimesh/distributed/`, the Worker Agent under `scimesh/worker/`, and the
+coordinator API contract. See [CTX-07](ctx-07-distributed-workload-protocol.md),
+[worker-building guide](building-workers.md), and [API contract](api-contract.md).
+
+## Proposed public concepts
+
+| Concept | Responsibility |
+| --- | --- |
+| `WorkloadDefinition` / `WorkloadManifest` | Name, versions, schemas, execution and verification metadata. |
+| `ParameterSchema`, `InputSpec`, `OutputSpec`, `ArtifactRef` | Typed public inputs and durable artifact shapes. |
+| `TaskPlan`, `Planner`, `Runner`, `Reducer` | Validate, split, execute, and deterministically combine work. |
+| `Verifier` | Accept or reject result evidence; never silently downgrade checks. |
+| `ResourceRequirements`, `ExecutionProfile`, `ReproducibilityProfile` | Bounded resource needs and pinned execution assumptions. |
+
+A manifest should include workload/version and SDK compatibility versions,
+description, parameter/input/output schemas, planner/runner/reducer/verifier
+types, determinism and trust profiles, resource/output limits, worker
+capabilities, and environment or image digest. Compatibility must be explicit
+among SDK, coordinator protocol, worker runtime, workload, output schema, and
+verifier versions.
+
+## Contracts
+
+**Planner:** validates before durable Job/Task creation; produces versioned,
+JSON-serializable plans that refer only to durable artifacts; gives stable task
+order and expected resources/outputs; fails transactionally without a partial
+task graph.
+
+**Runner:** receives typed parameters and owned artifact references; runs only
+allowlisted SDK code; writes to its attempt directory; produces output manifest,
+metrics, and sanitized failures; respects cancellation/lease loss when platform
+support exists; never uses `shell=True` or unnecessarily exposes credentials.
+
+**Reducer:** consumes only accepted partial artifacts in stable order; is
+idempotent or coordinator-state protected; creates a versioned final manifest;
+defines missing, duplicate, and malformed-shard failures.
+
+**Verifier:** is versioned with the workload and states one of byte-exact,
+canonical, numeric, domain-specific, or trust-policy comparison. It processes
+structured manifests and bounded streams where practical, records sanitized
+evidence, and rejects inconsistent results.
+
+Current untrusted quorum is only `ExactArtifactVerifier`: distinct owners must
+produce whole files with identical SHA-256. Future modes are
+`CanonicalRecordVerifier`, `NumericToleranceVerifier`,
+`DomainSpecificVerifier`, and `TrustedWorkerPolicy`. Canonical mode requires a
+specified parser/schema/order/encoding/serialization; numeric mode compares
+structured values, not CSV text.
+
+## Resources and reproducibility
+
+The extensible requirement model is `cpu_cores`, `memory_mb`, `scratch_mb`,
+`gpu_count`, `gpu_memory_mb`, `accelerator_kind`, `exclusive_device`, and
+`estimated_output_bytes`. It must not imply one Task equals one CPU core.
+
+Untrusted byte-exact workloads require a pinned image/environment digest,
+runtime and dependency versions, fixed locale/timezone/UTF-8/newlines/dialect,
+explicit invalid-row policy and algorithm options, canonical ordering, stable
+archive metadata, golden fixtures, two independently provisioned workers, and
+local/distributed plus retry/completion-order parity tests.
+
+## Compatibility evolution
+
+The stable release retains the current one-input/one-result task contract.
+First, a composite manifest artifact may reference multiple logical inputs;
+later, ordered input and output artifact collections can become first-class.
+The transition must be versioned and retain old workload compatibility.
+
+Discovery should use an installed Python package, manifest, pinned environment
+metadata, explicit entry points, and golden fixtures. It must be allowlisted;
+never scan or execute user-provided module paths.
+
+## Delivery sequence
+
+1. Finish distributed `similarity-graph` and reliability/cross-language CI.
+2. Stabilize manifest, schema, planner/runner/reducer interfaces, exact verifier,
+   compatibility metadata, and an author guide.
+3. Deliver `descriptor-batch` as the reference workload: pinned RDKit 2D
+   descriptors; canonical one-row-per-input CSV; shard-index concatenation with
+   one header; byte-identical local/distributed output and two-worker quorum.
+4. Add standardization, SMARTS screening, fingerprint export, fixed-template
+   reaction enumeration, then reaction validation/descriptors.
+5. Generalize composite artifacts, process slots, resource requests, and richer
+   verifier policies.
+6. Only then consider pinned, trusted/domain-verified numeric, ML, docking, QM,
+   MD, and GPU workloads.
+
+Future developer tooling may include `scimesh workload init`, `validate`,
+`test-local`, `test-distributed`, `golden`, and `package`; these commands do not
+exist today. A template should generate a manifest, schemas, planner, runner,
+reducer, verifier, unit tests, golden fixture, two-worker integration test, and
+documentation.
+
+## Open decisions
+
+- Is the SDK part of `scimesh` or a separately versioned Python distribution?
+- What stable bridge connects Go orchestration to Python planners/reducers?
+- Where do future verifiers execute, and how are environments attested?
+- Which trust modes may run each verifier profile?
+- Who may install/enable workloads in multi-user deployments?
+- How are composite I/O, version negotiation, output-growth limits, and
+  numeric-tolerance access governed without breaking the existing API?