SciMesh/docs/scimesh-sdk-roadmap.md

SciMesh Workload SDK roadmap

Status: future design and sequencing document. No SDK package, commands, or general verifier abstraction described here is implemented yet.

Purpose and boundaries

The SDK should let a scientific developer add an allowlisted workload without learning coordinator internals or writing SQL, while keeping one scientific implementation usable locally and in distributed execution:

scientific implementation -> local adapter -> planner/tasks -> reducer -> verifier

It must not execute arbitrary code or shell commands supplied by a coordinator. SDK v1 is not a public marketplace, generic container/job runner, cross-language SDK, automatic correctness-proof system, or immediate route to stochastic ML or GPU workloads.

The current foundation is the Python DistributedWorkload protocol and registry under scimesh/distributed/, the Worker Agent under scimesh/worker/, and the coordinator API contract. See CTX-07, worker-building guide, and API contract.

Proposed public concepts

Concept	Responsibility
WorkloadDefinition / WorkloadManifest	Name, versions, schemas, execution and verification metadata.
ParameterSchema, InputSpec, OutputSpec, ArtifactRef	Typed public inputs and durable artifact shapes.
TaskPlan, Planner, Runner, Reducer	Validate, split, execute, and deterministically combine work.
Verifier	Accept or reject result evidence; never silently downgrade checks.
ResourceRequirements, ExecutionProfile, ReproducibilityProfile	Bounded resource needs and pinned execution assumptions.

A manifest should include workload/version and SDK compatibility versions, description, parameter/input/output schemas, planner/runner/reducer/verifier types, determinism and trust profiles, resource/output limits, worker capabilities, and environment or image digest. Compatibility must be explicit among SDK, coordinator protocol, worker runtime, workload, output schema, and verifier versions.

Contracts

Planner: validates before durable Job/Task creation; produces versioned, JSON-serializable plans that refer only to durable artifacts; gives stable task order and expected resources/outputs; fails transactionally without a partial task graph.

Runner: receives typed parameters and owned artifact references; runs only allowlisted SDK code; writes to its attempt directory; produces output manifest, metrics, and sanitized failures; respects cancellation/lease loss when platform support exists; never uses shell=True or unnecessarily exposes credentials.

Reducer: consumes only accepted partial artifacts in stable order; is idempotent or coordinator-state protected; creates a versioned final manifest; defines missing, duplicate, and malformed-shard failures.

Verifier: is versioned with the workload and states one of byte-exact, canonical, numeric, domain-specific, or trust-policy comparison. It processes structured manifests and bounded streams where practical, records sanitized evidence, and rejects inconsistent results.

Current untrusted quorum is only ExactArtifactVerifier: distinct owners must produce whole files with identical SHA-256. Future modes are CanonicalRecordVerifier, NumericToleranceVerifier, DomainSpecificVerifier, and TrustedWorkerPolicy. Canonical mode requires a specified parser/schema/order/encoding/serialization; numeric mode compares structured values, not CSV text.

Resources and reproducibility

The extensible requirement model is cpu_cores, memory_mb, scratch_mb, gpu_count, gpu_memory_mb, accelerator_kind, exclusive_device, and estimated_output_bytes. It must not imply one Task equals one CPU core.

Untrusted byte-exact workloads require a pinned image/environment digest, runtime and dependency versions, fixed locale/timezone/UTF-8/newlines/dialect, explicit invalid-row policy and algorithm options, canonical ordering, stable archive metadata, golden fixtures, two independently provisioned workers, and local/distributed plus retry/completion-order parity tests.

Compatibility evolution

The stable release retains the current one-input/one-result task contract. First, a composite manifest artifact may reference multiple logical inputs; later, ordered input and output artifact collections can become first-class. The transition must be versioned and retain old workload compatibility.

Discovery should use an installed Python package, manifest, pinned environment metadata, explicit entry points, and golden fixtures. It must be allowlisted; never scan or execute user-provided module paths.

Delivery sequence

1. Finish distributed similarity-graph and reliability/cross-language CI.
2. Stabilize manifest, schema, planner/runner/reducer interfaces, exact verifier, compatibility metadata, and an author guide.
3. Deliver descriptor-batch as the reference workload: pinned RDKit 2D descriptors; canonical one-row-per-input CSV; shard-index concatenation with one header; byte-identical local/distributed output and two-worker quorum.
4. Add standardization, SMARTS screening, fingerprint export, fixed-template reaction enumeration, then reaction validation/descriptors.
5. Generalize composite artifacts, process slots, resource requests, and richer verifier policies.
6. Only then consider pinned, trusted/domain-verified numeric, ML, docking, QM, MD, and GPU workloads.

Future developer tooling may include scimesh workload init, validate, test-local, test-distributed, golden, and package; these commands do not exist today. A template should generate a manifest, schemas, planner, runner, reducer, verifier, unit tests, golden fixture, two-worker integration test, and documentation.

Open decisions

• Is the SDK part of scimesh or a separately versioned Python distribution?
• What stable bridge connects Go orchestration to Python planners/reducers?
• Where do future verifiers execute, and how are environments attested?
• Which trust modes may run each verifier profile?
• Who may install/enable workloads in multi-user deployments?
• How are composite I/O, version negotiation, output-growth limits, and numeric-tolerance access governed without breaking the existing API?