Add workload SDK foundation

docs/scimesh-sdk-contract.md

@@ -1,13 +1,16 @@
 # SciMesh Workload SDK contract
 
-**Status:** design draft `0.1`; not implemented. Normative words **MUST**,
-**MUST NOT**, **SHOULD**, and **MAY** describe the intended future contract,
-not capabilities of the current release.
+**Status:** contract `0.1`. The Python `core-batch-v1` foundation is implemented
+in `scimesh.sdk`; dynamic, streaming, accelerator, gang, side-effect, and
+coordinator protocol-v2 behavior remains a normative target. Normative words
+**MUST**, **MUST NOT**, **SHOULD**, and **MAY** apply to an implementation only
+when it advertises the affected profile or feature.
 
 This document defines the compatibility boundary for approved SciMesh workload
 packages. The sequencing and unresolved product decisions remain in
-[`scimesh-sdk-roadmap.md`](scimesh-sdk-roadmap.md). The currently implemented
-protocol is still [`ctx-07-distributed-workload-protocol.md`](ctx-07-distributed-workload-protocol.md).
+[`scimesh-sdk-roadmap.md`](scimesh-sdk-roadmap.md). The production coordinator
+wire compatibility profile remains
+[`ctx-07-distributed-workload-protocol.md`](ctx-07-distributed-workload-protocol.md).
 
 ## 1. Scope and invariants
 
@@ -240,7 +243,11 @@ indefinitely while holding resources.
 `TaskSpec` is the concrete unit leased to a Worker Agent:
 
 ```yaml
-task_schema_version: 1
+schema_version: 1
+workload: descriptor-batch@1.2.0
+package_digest: sha256:...
+manifest_digest: ...
+trust_mode: trusted
 task_key: calculate/shard-000042
 stage_id: calculate
 parameters: {...validated JSON...}
@@ -279,9 +286,10 @@ represent a collection. A later protocol may persist collection edges directly.
 
 Before completion, a runner uploads an `OutputManifest` listing every declared
 output artifact, checksum, schema, size, record/dimension summary, metrics, and
-provenance. Provenance includes resolved versions, package/environment digest,
-Worker runtime, allocated resource IDs, parameters digest, input collection
-digest, timestamps, random seed where applicable, and checkpoint lineage.
+provenance. Provenance includes resolved versions, package/environment and
+manifest digests, Worker runtime, allocated resource IDs, parameters digest,
+input collection digest, timestamps, random seed where applicable, and
+checkpoint lineage.
 
 Unexpected ports, missing required outputs, extra artifacts, schema failures,
 or limit violations reject the Attempt. Logs and checkpoints are separate
@@ -419,6 +427,13 @@ declare an appropriate verifier/trust combination. Reducers consume only
 accepted partial outputs and MUST detect missing, duplicate, conflicting, or
 inconclusive inputs.
 
+Quorum candidates MUST be coordinator-authenticated envelopes with unique
+Attempt/candidate identity and an owner identity. A verifier counts at most one
+vote per owner. It also receives a coordinator-owned binding for workload,
+task, package/manifest/environment, parameters, and input-collection digests;
+outputs from another job or code pin are invalid even when their result bytes
+match.
+
 ## 8. Failure, retry, cancellation, and checkpoint semantics
 
 Every failure has a stable sanitized code, category (`input`, `scientific`,
@@ -472,26 +487,34 @@ isolation is ineligible rather than silently unsandboxed.
 
 ## 10. SDK interfaces and conformance
 
-The future Python API SHOULD expose protocols equivalent to:
+The Python API exposes protocols equivalent to:
 
 ```python
 class Planner(Protocol):
     def validate(self, request: JobRequest) -> ValidatedJob: ...
-    def plan(self, job: ValidatedJob, artifacts: ArtifactCatalog) -> WorkflowPlan: ...
+    def plan(self, job: ValidatedJob, context: PlanningContext) -> WorkflowPlan: ...
 
 class Runner(Protocol):
     def run(self, context: TaskContext) -> OutputManifest: ...
 
 class Reducer(Protocol):
-    def reduce(self, context: ReduceContext, inputs: AcceptedOutputs) -> OutputManifest: ...
+    def reduce(self, context: ReduceContext) -> OutputManifest: ...
 
 class Verifier(Protocol):
     def verify(self, context: VerifyContext, candidates: CandidateOutputs) -> VerificationDecision: ...
 ```
 
-Concrete public value objects are immutable, typed, JSON-safe, schema-versioned,
-and reject unknown fields. Scientific cores SHOULD remain callable without a
-coordinator so the same implementation powers local and distributed adapters.
+Concrete public value objects are immutable, typed, JSON-safe, strict about
+unknown fields, and canonically serialized inside versioned wire contracts.
+Scientific cores SHOULD remain callable without a coordinator so the same
+implementation powers local and distributed adapters.
+
+The shipped `LocalCoreBatchExecutor` is a trusted in-process conformance
+harness, not the `core-batch-v1` production isolation boundary. It rejects
+restricted-network, parallel-process/thread, accelerator, secret, checkpoint,
+retry, gang, and advanced-stage declarations. Subprocess isolation, hard
+timeouts, leases, and credential enforcement remain requirements for an Agent
+runtime that advertises those guarantees.
 
 An SDK conformance suite MUST test manifest/schema validation, deterministic
 planning, no local-path/URI leakage, output bounds, local/distributed parity,

docs/scimesh-sdk-roadmap.md

@@ -1,7 +1,11 @@
 # SciMesh Workload SDK roadmap
 
-**Status:** future design and sequencing document. No SDK package, commands, or
-general verifier abstraction described here is implemented yet.
+**Status:** active delivery roadmap. The Python `scimesh.sdk` package now
+implements the `core-batch-v1` foundation, verifier primitives, resource
+eligibility/local allocation, installed-package registry, local conformance
+runtime, and legacy similarity-search adapter. Coordinator-backed generalized
+DAG execution, Worker concurrency, accelerators, streaming, gang execution,
+and authoring CLI commands remain future phases.
 
 The normative future API, workflow, execution, resource, security, and failure
 semantics are specified in the design-draft
@@ -69,8 +73,10 @@ canonical, numeric, domain-specific, or trust-policy comparison. It processes
 structured manifests and bounded streams where practical, records sanitized
 evidence, and rejects inconsistent results.
 
-Current untrusted quorum is only `ExactArtifactVerifier`: distinct owners must
-produce whole files with identical SHA-256. Future modes are
+Current untrusted quorum is only `ExactArtifactVerifier`: coordinator-created
+candidate envelopes from distinct owners must share the exact workload, task,
+package/manifest/environment, parameters, and input binding and produce whole
+files with identical SHA-256. Future modes are
 `CanonicalRecordVerifier`, `NumericToleranceVerifier`,
 `DomainSpecificVerifier`, and `TrustedWorkerPolicy`. Canonical mode requires a
 specified parser/schema/order/encoding/serialization; numeric mode compares

docs/sdk-handoff.md

@@ -0,0 +1,115 @@
+# Workload SDK handoff
+
+**Audience:** the engineer/AI continuing SciMesh Workload SDK implementation.
+**Date:** 2026-08-01. **Baseline:** uncommitted working tree on `main` (`11e9333`
+plus the CTX-16 SDK changes); `python -m pytest -q` reports **225 passed**.
+
+Read first, in this order: `AGENTS.md` (binding repo rules),
+`docs/scimesh-sdk-roadmap.md` (delivery order — it governs, this file does not),
+`docs/scimesh-sdk-contract.md` (normative target semantics),
+`docs/workload-sdk.md` (author guide for what exists), and the CTX-16 entry in
+`PLAN.md`.
+
+## What is already done (do not redo)
+
+CTX-16 "Workload SDK foundation" is complete and tested. `scimesh/sdk/`
+implements the `core-batch-v1` profile:
+
+- Immutable, JSON-strict value objects: `identity.py`, `artifacts.py`,
+  `workflow.py`, `manifest.py`, `plans.py`, `execution.py`, `resources.py`.
+- Fail-closed compatibility negotiation: `runtime.py` (`negotiate_manifest`)
+  plus request-level checks in `registry.py`.
+- Installed-package registry with administrator allowlist, exact version +
+  `sha256:` digest pinning, entry-point discovery with digest measured before
+  and after import: `registry.py`, `integrity.py`.
+- Verifier primitives `ExactArtifactVerifier`, `CanonicalRecordVerifier`,
+  `NumericToleranceVerifier` with bounded sanitized evidence: `verification.py`.
+- Local conformance harness: `LocalArtifactStore`, `LocalCoreBatchExecutor`,
+  `ResourcePool` (atomic all-or-nothing reservation): `conformance.py`.
+- Legacy adapter exposing distributed `similarity-search` through the SDK
+  without changing its wire schema: `compat/distributed_v1.py`, `builtins.py`;
+  entry point `similarity-search@1.0.0` is declared in `pyproject.toml`.
+- Tests: `tests/test_sdk_{models,resources,verification,compatibility,registry}.py`
+  including fail-closed rejection coverage for every advanced profile
+  declaration (gang, GPU modes, pools, checkpoints, retries, secrets, streams,
+  loops, side effects).
+
+## What remains, in delivery order
+
+1. **`descriptor-batch` reference workload** (roadmap step 3 — the recommended
+   next task; it is pure Python and needs no coordinator changes). Pinned RDKit
+   2D descriptors, canonical one-row-per-input CSV, shard-index concatenation
+   with one header, byte-identical local/distributed output, two-worker quorum.
+   Build it as an SDK-native package (manifest + planner/runner/reducer/
+   verifier handlers), not through the legacy adapter; reuse the
+   `similarity-search` adapter (`scimesh/sdk/compat/distributed_v1.py`) and
+   `builtins.py` as the structural template, and the
+   `tests/test_sdk_compatibility.py` fixtures as the test template. This is the
+   intended first `untrusted_quorum` candidate (byte_exact + exact-artifact@1).
+2. **Distributed `similarity-graph`** (CTX-10, roadmap step 1). The coordinator
+   currently rejects `similarity-graph` uploads; it needs cross-shard block-pair
+   planning and duplicate-safe reduction. STATUS.md names this the next
+   recommended assignment overall.
+3. **Coordinator/Worker protocol v2** (needs CTX-10, then CTX-13 in-worker CPU
+   parallelism and CTX-14 GPU execution; Go + Python). The protocol-v1
+   coordinator persists only flat one-input/one-result tasks: no resource
+   requirements, stage edges, package versions, device allocations, or gang
+   leases. Until a versioned rollout lands, SDK declarations for those features
+   must stay fail-closed — do not silently "enable" them.
+4. **More chemistry workloads** (roadmap step 4): standardization, SMARTS
+   screening, fingerprint export, fixed-template reaction enumeration, then
+   reaction validation/descriptors.
+5. **Composite artifacts and richer verifier policies** (roadmap step 5):
+   first-class ordered/keyed `ArtifactCollection` edges instead of composite
+   manifest artifacts; decide where verifiers execute (open decision in the
+   roadmap).
+6. **Authoring CLI** (future tooling, does not exist today): `scimesh workload
+   init`, `validate`, `test-local`, `test-distributed`, `golden`, `package`.
+   Per AGENTS.md, keep CLI parsing in workload modules and register through
+   `scimesh/core/registry.py`; no workload-specific logic in the main CLI.
+7. **Open decisions** (listed at the end of the roadmap): SDK distribution
+   split, Go↔Python planner bridge, verifier execution/attestation, trust-mode
+   governance, multi-user enablement. Do not pick one unilaterally — surface it.
+
+## Known traps (cost the previous session real time)
+
+- The legacy adapter pins its own manifest (`adapter.manifest`). If a test
+  changes limits/workflow on the manifest, the adapter's copy must be replaced
+  too, or `registry.plan` fails with "planner plan does not carry the selected
+  immutable workload pin".
+- `WorkloadDefinition` validation: a PLAN stage's `entry_point` must equal
+  `planner.entry_point`; every non-REDUCE stage's `entry_point` must be a key
+  in `runners` (REDUCE → `reducers`); verifier handlers are keyed by
+  `ComponentRef.canonical` and must expose a matching `.identity`.
+- Negotiation requires each triggering property's feature to be declared
+  separately: e.g. `PROCESS_POOL` needs `process-pools` **and** `multi-process`
+  for `max_processes > 1`. Runtime must also advertise every declared required
+  feature, or negotiation fails with `feature-unavailable`.
+- `feature-fallback-disallowed` in `scimesh/sdk/registry.py` is currently
+  unreachable via `registry.plan` (the `feature-unavailable` check fires first
+  for any runtime that produced a fallback). Behavior is still fail-closed;
+  decide whether to reorder or delete the branch.
+- The local executor is deliberately trusted/in-process: it rejects anything
+  but `TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-threaded CPU
+  map/reduce without retries/gangs/accelerators/secrets/checkpoints. That is a
+  contract, not a bug — test rejections, don't "fix" them.
+- `JobRequest` parameters and failure/evidence payloads reject local paths and
+  URIs by design; keep new payloads location-free.
+- There is a stray nested clone `SciMesh/` in the repo root (same repo at an
+  older commit). Ignore it and never `git add` it; consider deleting it.
+- The full ChEMBL extract `chembl_37_chemreps.txt` (~2.9M rows) makes the
+  single-threaded local executor run for many minutes; tests must use small
+  TSV fixtures (see `_write_tiny_dataset`).
+
+## Working agreement
+
+- Verify with `source .venv/bin/activate && python -m pytest -q`; the baseline
+  is 225 passing tests and it must stay green. Add a regression test for every
+  behavioral change; similarity code needs a brute-force/sorted reference and
+  determinism across block sizes.
+- Legacy `similarity-search` wire schema, worker alias boundary, and scientific
+  output must not change. Worker code never talks to PostgreSQL directly;
+  results go through the coordinator; failures go to `/failure`, never as
+  `file://`/`worker://` result URIs.
+- Do not commit datasets, generated CSV/PNG, tokens, or local worker artifacts.
+- One CTX task per pull request; link the CTX item from `PLAN.md`.

docs/workload-sdk.md

@@ -0,0 +1,220 @@
+# SciMesh Workload SDK v1
+
+SciMesh now ships a public Python SDK under `scimesh.sdk`. The implemented
+authoring profile is **`core-batch-v1`**: installed and digest-pinned workload
+definitions, strict JSON manifests, typed artifact ports and collections, a
+static map/reduce workflow, CPU/memory/scratch eligibility, atomic local
+resource reservation, exact/canonical/numeric verifier primitives, and a
+compatibility adapter for the existing `DistributedWorkload` protocol. Its
+local executor is deliberately a trusted, in-process conformance harness; the
+production subprocess/lease sandbox remains a coordinator/Worker milestone.
+
+The full target contract remains in
+[`scimesh-sdk-contract.md`](scimesh-sdk-contract.md). Dynamic expansion,
+streaming, accelerators, gang execution, and side effects have typed bounded
+declarations, but the current coordinator/Worker runtime does not advertise
+their features. Compatibility negotiation therefore rejects those workflows
+before planner code runs.
+
+## What authors import
+
+The stable authoring surface is exported from `scimesh.sdk`:
+
+- `WorkloadManifest`, `WorkloadId`, `VersionRange`, `PackageSpec`, and
+  `EnvironmentSpec` pin identity and compatibility;
+- `ArtifactSchema`, `PortSpec`, `ArtifactRef`, and `ArtifactCollection` define
+  immutable data boundaries without transport URLs or local paths;
+- `WorkflowSpec`, `StageSpec`, `ArtifactEdge`, `TaskSpec`, and `WorkflowPlan`
+  define a typed acyclic plan and pin package/manifest digests plus trust mode;
+- `ResourceRequirements` and `ExecutionProfile` separate per-task resources
+  from Agent `max_concurrency`;
+- `Planner`, `Runner`, `Reducer`, and `Verifier` are the package handler
+  protocols;
+- `OutputManifest` and `Provenance` describe sealed durable results;
+- `WorkloadRegistry` resolves an exact name, version, package digest, runtime,
+  environment, and feature set. It never selects an implicit latest version.
+
+Persisted manifests, requests, plans, tasks, expansions, outputs, candidates,
+decisions, and failures are frozen, recursively immutable, JSON-safe,
+canonically serialized, and strict about unknown fields; their enclosing wire
+contracts carry schema versions.
+Artifact identities contain a coordinator-owned UUID, schema, checksum, media
+type, and bounds; a scientific handler never persists a filesystem path.
+
+## Try the built-in SDK workload
+
+This example executes the current distributed `similarity-search` through the
+SDK without starting PostgreSQL or the coordinator:
+
+```python
+from pathlib import Path
+
+from scimesh.sdk import (
+    ArtifactCollection,
+    JobRequest,
+    LocalArtifactStore,
+    LocalCoreBatchExecutor,
+    default_sdk_registry,
+    default_sdk_runtime,
+    similarity_search_sdk_adapter,
+)
+
+root = Path("sdk-run")
+store = LocalArtifactStore(root / "artifacts")
+adapter = similarity_search_sdk_adapter(shard_rows=1_000)
+
+dataset = store.import_file(
+    Path("chembl_37_chemreps.txt"),
+    declaration=adapter.input_port.schema,
+)
+request = JobRequest(
+    workload=adapter.manifest.workload,
+    parameters={"query_smiles": "CCO", "top_k": 20},
+    inputs={"input": ArtifactCollection.single(dataset)},
+)
+
+result = LocalCoreBatchExecutor(
+    default_sdk_registry(shard_rows=1_000),
+    default_sdk_runtime(),
+    store,
+    root / "attempts",
+).execute(request, adapter.manifest.package.digest)
+
+result_ref = result.outputs["result"].items[0].artifact
+print(store.materialize(result_ref))
+```
+
+`LocalCoreBatchExecutor` is a correctness/conformance runtime, not a substitute
+for coordinator leases or multi-machine scheduling. It accepts only
+`TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-process/single-threaded CPU
+map/reduce stages without secrets, checkpoints, retries, gangs, or
+accelerators. It does not claim network, timeout, process, or credential
+isolation. Unsupported declarations are rejected before a handler runs. The
+harness uses the same legacy scientific planner, shard runner, and reducer as
+the distributed `similarity-search`, and its parity is covered by automated
+tests.
+
+## Package shape and registration
+
+An SDK distribution provides one explicit entry point per workload version:
+
+```toml
+[project.entry-points."scimesh.workloads"]
+"descriptor-batch@1.0.0" = "scimesh_descriptors.sdk:workload_definition"
+```
+
+The factory returns a `WorkloadDefinition` containing its manifest and handler
+objects. An administrator supplies an `AllowedPackage` with the same
+distribution, exact `WorkloadId`, and `sha256:` package digest. Discovery
+filters installed metadata before importing an entry point and fails
+transactionally if an allowlisted definition is missing or mismatched. Job
+parameters cannot name a module, entry point, package path, or executable.
+The measured digest covers package payload files and installed entry-point
+declarations and is checked before and after loading. It is a content pin, not
+a signature or image attestation; production discovery should run in a fresh
+trusted control-plane process so a pre-populated Python module cache is not an
+integrity boundary.
+
+Direct registration is useful for tests and embedded deployments:
+
+```python
+registry = WorkloadRegistry()
+registry.register(definition, enabled=False)
+registry.enable(
+    definition.manifest.workload.name,
+    definition.manifest.workload.version,
+    definition.manifest.package.digest,
+)
+```
+
+Both version and digest are required when resolving or planning. Upgrading an
+installed definition does not change the identity of an existing Job.
+
+## Authoring rules
+
+1. Keep the scientific core callable without a coordinator.
+2. Inline a strict JSON parameter schema with `type: object` and
+   `additionalProperties: false`; the planner still performs domain validation.
+3. Give every external and stage port an `ArtifactSchema` with a media type,
+   schema version, and byte/record/dimension bounds.
+4. Connect stage ports with `ArtifactEdge` values. `WorkflowSpec` checks source
+   and target schemas, complete input bindings, declared dependencies, and
+   acyclicity.
+5. Declare one `ResourceRequirements` and `ExecutionProfile` per stage. A task
+   cannot run until its entire request is eligible and atomically reserved.
+6. Return only sink-sealed artifacts in `OutputManifest`; the local harness
+   binds task key/provenance itself and rejects fabricated references,
+   unexpected/missing ports, wrong schema/media type, and cumulative output or
+   artifact-limit violations.
+7. Select a verifier compatible with determinism and trust. SDK v1 permits
+   `untrusted_quorum` only for `byte_exact` plus `exact-artifact@1`.
+8. Add golden fixtures, local/distributed parity, retry/completion-order, and
+   verifier failure tests before enabling a package.
+
+`ArtifactSink` and `ArtifactCatalog` are bridge-owned protocols. They let
+scientific handlers materialize verified inputs and seal outputs without bearer
+tokens, database credentials, upload URLs, or durable local paths.
+
+## Verification
+
+The SDK includes:
+
+- `ExactArtifactVerifier`: compares logical port/collection/schema/content
+  digests while ignoring coordinator UUIDs, timestamps, metrics, and worker
+  identity. Quorum inputs use coordinator-created `CandidateOutput` envelopes,
+  count at most one vote per owner, and require a `VerificationBinding` for the
+  exact task, inputs, parameters, package, manifest, and environment;
+- `CanonicalRecordVerifier`: applies a package-owned bounded canonicalizer and
+  compares length-framed canonical records;
+- `NumericToleranceVerifier`: recursively checks structure plus explicit
+  absolute, relative, ULP, and NaN policy, returning bounded sanitized evidence.
+
+Canonical and numeric objects expose direct bounded comparison methods. To use
+them as manifest `Verifier` handlers, the package supplies an artifact-to-record
+or artifact-to-structured-value loader; without one, verification returns
+`inconclusive` rather than accepting bytes it did not parse.
+
+A decision is `accepted`, `rejected`, or `inconclusive`; only `accepted`
+satisfies a stage. Evidence is limited to 16 KiB and cannot contain local paths
+or transport URLs.
+
+## Resources and current runtime boundary
+
+`ResourcePool` provides a lock-protected all-or-nothing local reservation for
+CPU cores, memory, scratch, and accelerator device/partition IDs, including
+whole-device versus partition conflict fencing. It enforces aggregate capacity
+and execution-slot count. `ExecutionProfile` produces only
+allocation-derived OpenMP/BLAS and device-visibility values; credentials never
+belong to scientific parameters.
+
+The current protocol-v1 coordinator stores one input/result per flat task and
+does not persist resource requirements, device allocations, stage edges, or
+package versions. The production Worker also remains serial. Consequently:
+
+- SDK `core-batch-v1` can be authored, validated, tested, discovered, and run
+  through the trusted local conformance harness now;
+- existing production `similarity-search` remains on its compatible v1 wire
+  path and is not renamed;
+- real concurrent claims, GPU scheduling, multi-output DAG execution, dynamic
+  loops, streaming, and gang leases require the versioned coordinator/Worker
+  changes listed in [`scimesh-sdk-roadmap.md`](scimesh-sdk-roadmap.md);
+- merely declaring a GPU or gang request never enables it. Missing runtime
+  features or inventory fail before the planner executes.
+
+## Conformance commands
+
+Install development tools and run the SDK suite:
+
+```bash
+pip install -e '.[dev]'
+pytest tests/test_sdk_models.py \
+       tests/test_sdk_resources.py \
+       tests/test_sdk_verification.py \
+       tests/test_sdk_compatibility.py \
+       tests/test_sdk_registry.py
+```
+
+Run `pytest` for the full legacy, Worker, local-science, and SDK regression
+suite. Package authors can reuse `LocalArtifactStore`,
+`LocalCoreBatchExecutor`, and `assert_manifest_round_trip` in their own golden
+tests.