SciMesh/mkdocs/sdk/overview.md

SDK overview

scimesh.sdk is the framework only. It contains no scientific workload code. Workloads are user Python scripts and packages that import the SDK and live outside it — the built-in SciMesh workloads under scimesh/workloads/ are exactly such scripts, and a future workload library can follow the same shape.

scientific implementation -> SDK manifest/plan -> map tasks -> reduce -> verifier

The core-batch-v1 profile

The implemented authoring profile is core-batch-v1: a static map/reduce workflow with one external input and one final output.

• one input dataset (a delimited table, typically TSV);
• deterministic row-bounded (or otherwise partitioned) shards;
• one map task per shard, each producing one partial artifact;
• one reduce task merging the accepted partials into the final artifact;
• an acceptance verifier for every output-producing stage.

The SDK's strict value objects make the whole contract explicit:

• WorkloadManifest pins identity, compatibility ranges, package and environment digests, parameter schema, workflow, ports, determinism, trust modes, verifier, and limits;
• WorkflowSpec/StageSpec describe a typed acyclic DAG;
• TaskSpec/WorkflowPlan carry the exact workload pin (package, manifest, environment digests, trust mode, negotiated features);
• OutputManifest/Provenance describe sealed durable results;
• artifacts are content-addressed (sha256), immutable, and free of transport URLs and local paths.

What the SDK provides

Area	Modules	Purpose
Identity	identity, schema	WorkloadId, versions, schema refs, bounded JSON parameter schemas
Declarations	manifest, workflow, artifacts, execution, resources	Manifest, DAG stages, typed ports, execution/resource profiles
Planning	plans	JobRequest, TaskSpec, WorkflowPlan
Registry	registry, integrity	Allowlisted discovery, digest pinning
Negotiation	runtime	Fail-closed compatibility negotiation
Execution	conformance	LocalCoreBatchExecutor, LocalArtifactStore
Verification	verification	Exact, canonical, and numeric verifier primitives
Authoring	batch	MapReduceWorkload scaffold

Security model

• Workload discovery requires an administrator allowlist: exact distribution, workload name/version, and a sha256: package digest. Discovery measures the installed package before and after importing the entry point and fails transactionally on any mismatch. Job parameters can never name a module, entry point, or executable.
• Compatibility negotiation is fail-closed: if the runtime does not advertise a declared feature (gangs, GPU, streams, checkpoints, secrets, retries, process pools, dynamic expansion), the job is rejected before any workload code runs. Declaring an advanced profile never silently enables it.
• Handlers receive bridge-owned contexts: ArtifactCatalog for verified input materialization and ArtifactSink for sealing outputs. They never see database credentials, upload URLs, or coordinator tokens.
• The local conformance executor is deliberately trusted and in-process. It rejects anything but TrustMode.TRUSTED, a single non-nested host thread, and the trusted network policy — a contract, not a limitation.

What is not supported yet

The coordinator contract (v1) persists flat one-input/one-result tasks. Until a versioned protocol rollout lands, the following remain fail-closed by design:

• distributed execution of multi-input map stages (for example the block-pair similarity-graph tasks) and of workloads beyond the v1 contract;
• coordinator-backed GPU scheduling, streams, gang leases, checkpoints, retries, and secret injection;
• dynamic (plan-stage) expansion.

The verifier primitives CanonicalRecordVerifier and NumericToleranceVerifier exist and are tested, but only the ExactArtifactVerifier (whole-file SHA-256) is eligible for untrusted_quorum in v1.

Next

• Authoring workloads — build a workload with MapReduceWorkload.
• API reference — the complete SDK surface.