SciMesh/docs/workload-sdk.md

SciMesh Workload SDK v1

SciMesh now ships a public Python SDK under scimesh.sdk. The implemented authoring profile is core-batch-v1: installed and digest-pinned workload definitions, strict JSON manifests, typed artifact ports and collections, a static map/reduce workflow, CPU/memory/scratch eligibility, atomic local resource reservation, exact/canonical/numeric verifier primitives, and a compatibility adapter for the existing DistributedWorkload protocol. Its local executor is deliberately a trusted, in-process conformance harness; the production subprocess/lease sandbox remains a coordinator/Worker milestone.

The full target contract remains in scimesh-sdk-contract.md. Dynamic expansion, streaming, accelerators, gang execution, and side effects have typed bounded declarations, but the current coordinator/Worker runtime does not advertise their features. Compatibility negotiation therefore rejects those workflows before planner code runs.

What authors import

The stable authoring surface is exported from scimesh.sdk:

• WorkloadManifest, WorkloadId, VersionRange, PackageSpec, and EnvironmentSpec pin identity and compatibility;
• ArtifactSchema, PortSpec, ArtifactRef, and ArtifactCollection define immutable data boundaries without transport URLs or local paths;
• WorkflowSpec, StageSpec, ArtifactEdge, TaskSpec, and WorkflowPlan define a typed acyclic plan and pin package/manifest digests plus trust mode;
• ResourceRequirements and ExecutionProfile separate per-task resources from Agent max_concurrency;
• Planner, Runner, Reducer, and Verifier are the package handler protocols;
• OutputManifest and Provenance describe sealed durable results;
• WorkloadRegistry resolves an exact name, version, package digest, runtime, environment, and feature set. It never selects an implicit latest version.

Persisted manifests, requests, plans, tasks, expansions, outputs, candidates, decisions, and failures are frozen, recursively immutable, JSON-safe, canonically serialized, and strict about unknown fields; their enclosing wire contracts carry schema versions. Artifact identities contain a coordinator-owned UUID, schema, checksum, media type, and bounds; a scientific handler never persists a filesystem path.

Try the built-in SDK workload

This example executes the current distributed similarity-search through the SDK without starting PostgreSQL or the coordinator:

from pathlib import Path

from scimesh.sdk import (
    ArtifactCollection,
    JobRequest,
    LocalArtifactStore,
    LocalCoreBatchExecutor,
    default_sdk_registry,
    default_sdk_runtime,
    similarity_search_sdk_adapter,
)

root = Path("sdk-run")
store = LocalArtifactStore(root / "artifacts")
adapter = similarity_search_sdk_adapter(shard_rows=1_000)

dataset = store.import_file(
    Path("chembl_37_chemreps.txt"),
    declaration=adapter.input_port.schema,
)
request = JobRequest(
    workload=adapter.manifest.workload,
    parameters={"query_smiles": "CCO", "top_k": 20},
    inputs={"input": ArtifactCollection.single(dataset)},
)

result = LocalCoreBatchExecutor(
    default_sdk_registry(shard_rows=1_000),
    default_sdk_runtime(),
    store,
    root / "attempts",
).execute(request, adapter.manifest.package.digest)

result_ref = result.outputs["result"].items[0].artifact
print(store.materialize(result_ref))

LocalCoreBatchExecutor is a correctness/conformance runtime, not a substitute for coordinator leases or multi-machine scheduling. It accepts only TrustMode.TRUSTED, NetworkPolicy.TRUSTED, single-process/single-threaded CPU map/reduce stages without secrets, checkpoints, retries, gangs, or accelerators. It does not claim network, timeout, process, or credential isolation. Unsupported declarations are rejected before a handler runs. The harness uses the same legacy scientific planner, shard runner, and reducer as the distributed similarity-search, and its parity is covered by automated tests.

Package shape and registration

An SDK distribution provides one explicit entry point per workload version:

[project.entry-points."scimesh.workloads"]
"descriptor-batch@1.0.0" = "scimesh_descriptors.sdk:workload_definition"

The factory returns a WorkloadDefinition containing its manifest and handler objects. An administrator supplies an AllowedPackage with the same distribution, exact WorkloadId, and sha256: package digest. Discovery filters installed metadata before importing an entry point and fails transactionally if an allowlisted definition is missing or mismatched. Job parameters cannot name a module, entry point, package path, or executable. The measured digest covers package payload files and installed entry-point declarations and is checked before and after loading. It is a content pin, not a signature or image attestation; production discovery should run in a fresh trusted control-plane process so a pre-populated Python module cache is not an integrity boundary.

Direct registration is useful for tests and embedded deployments:

registry = WorkloadRegistry()
registry.register(definition, enabled=False)
registry.enable(
    definition.manifest.workload.name,
    definition.manifest.workload.version,
    definition.manifest.package.digest,
)

Both version and digest are required when resolving or planning. Upgrading an installed definition does not change the identity of an existing Job.

Authoring rules

1. Keep the scientific core callable without a coordinator.
2. Inline a strict JSON parameter schema with type: object and additionalProperties: false; the planner still performs domain validation.
3. Give every external and stage port an ArtifactSchema with a media type, schema version, and byte/record/dimension bounds.
4. Connect stage ports with ArtifactEdge values. WorkflowSpec checks source and target schemas, complete input bindings, declared dependencies, and acyclicity.
5. Declare one ResourceRequirements and ExecutionProfile per stage. A task cannot run until its entire request is eligible and atomically reserved.
6. Return only sink-sealed artifacts in OutputManifest; the local harness binds task key/provenance itself and rejects fabricated references, unexpected/missing ports, wrong schema/media type, and cumulative output or artifact-limit violations.
7. Select a verifier compatible with determinism and trust. SDK v1 permits untrusted_quorum only for byte_exact plus exact-artifact@1.
8. Add golden fixtures, local/distributed parity, retry/completion-order, and verifier failure tests before enabling a package.

ArtifactSink and ArtifactCatalog are bridge-owned protocols. They let scientific handlers materialize verified inputs and seal outputs without bearer tokens, database credentials, upload URLs, or durable local paths.

Verification

The SDK includes:

• ExactArtifactVerifier: compares logical port/collection/schema/content digests while ignoring coordinator UUIDs, timestamps, metrics, and worker identity. Quorum inputs use coordinator-created CandidateOutput envelopes, count at most one vote per owner, and require a VerificationBinding for the exact task, inputs, parameters, package, manifest, and environment;
• CanonicalRecordVerifier: applies a package-owned bounded canonicalizer and compares length-framed canonical records;
• NumericToleranceVerifier: recursively checks structure plus explicit absolute, relative, ULP, and NaN policy, returning bounded sanitized evidence.

Canonical and numeric objects expose direct bounded comparison methods. To use them as manifest Verifier handlers, the package supplies an artifact-to-record or artifact-to-structured-value loader; without one, verification returns inconclusive rather than accepting bytes it did not parse.

A decision is accepted, rejected, or inconclusive; only accepted satisfies a stage. Evidence is limited to 16 KiB and cannot contain local paths or transport URLs.

Resources and current runtime boundary

ResourcePool provides a lock-protected all-or-nothing local reservation for CPU cores, memory, scratch, and accelerator device/partition IDs, including whole-device versus partition conflict fencing. It enforces aggregate capacity and execution-slot count. ExecutionProfile produces only allocation-derived OpenMP/BLAS and device-visibility values; credentials never belong to scientific parameters.

The current protocol-v1 coordinator stores one input/result per flat task and does not persist resource requirements, device allocations, stage edges, or package versions. The production Worker also remains serial. Consequently:

• SDK core-batch-v1 can be authored, validated, tested, discovered, and run through the trusted local conformance harness now;
• existing production similarity-search remains on its compatible v1 wire path and is not renamed;
• real concurrent claims, GPU scheduling, multi-output DAG execution, dynamic loops, streaming, and gang leases require the versioned coordinator/Worker changes listed in scimesh-sdk-roadmap.md;
• merely declaring a GPU or gang request never enables it. Missing runtime features or inventory fail before the planner executes.

Conformance commands

Install development tools and run the SDK suite:

pip install -e '.[dev]'
pytest tests/test_sdk_models.py \
       tests/test_sdk_resources.py \
       tests/test_sdk_verification.py \
       tests/test_sdk_compatibility.py \
       tests/test_sdk_registry.py

Run pytest for the full legacy, Worker, local-science, and SDK regression suite. Package authors can reuse LocalArtifactStore, LocalCoreBatchExecutor, and assert_manifest_round_trip in their own golden tests.