Add workload SDK foundation
This commit is contained in:
Emil
@@ -0,0 +1,220 @@
|
||||
# SciMesh Workload SDK v1
|
||||
|
||||
SciMesh now ships a public Python SDK under `scimesh.sdk`. The implemented
|
||||
authoring profile is **`core-batch-v1`**: installed and digest-pinned workload
|
||||
definitions, strict JSON manifests, typed artifact ports and collections, a
|
||||
static map/reduce workflow, CPU/memory/scratch eligibility, atomic local
|
||||
resource reservation, exact/canonical/numeric verifier primitives, and a
|
||||
compatibility adapter for the existing `DistributedWorkload` protocol. Its
|
||||
local executor is deliberately a trusted, in-process conformance harness; the
|
||||
production subprocess/lease sandbox remains a coordinator/Worker milestone.
|
||||
|
||||
The full target contract remains in
|
||||
[`scimesh-sdk-contract.md`](scimesh-sdk-contract.md). Dynamic expansion,
|
||||
streaming, accelerators, gang execution, and side effects have typed bounded
|
||||
declarations, but the current coordinator/Worker runtime does not advertise
|
||||
their features. Compatibility negotiation therefore rejects those workflows
|
||||
before planner code runs.
|
||||
|
||||
## What authors import
|
||||
|
||||
The stable authoring surface is exported from `scimesh.sdk`:
|
||||
|
||||
- `WorkloadManifest`, `WorkloadId`, `VersionRange`, `PackageSpec`, and
|
||||
`EnvironmentSpec` pin identity and compatibility;
|
||||
- `ArtifactSchema`, `PortSpec`, `ArtifactRef`, and `ArtifactCollection` define
|
||||
immutable data boundaries without transport URLs or local paths;
|
||||
- `WorkflowSpec`, `StageSpec`, `ArtifactEdge`, `TaskSpec`, and `WorkflowPlan`
|
||||
define a typed acyclic plan and pin package/manifest digests plus trust mode;
|
||||
- `ResourceRequirements` and `ExecutionProfile` separate per-task resources
|
||||
from Agent `max_concurrency`;
|
||||
- `Planner`, `Runner`, `Reducer`, and `Verifier` are the package handler
|
||||
protocols;
|
||||
- `OutputManifest` and `Provenance` describe sealed durable results;
|
||||
- `WorkloadRegistry` resolves an exact name, version, package digest, runtime,
|
||||
environment, and feature set. It never selects an implicit latest version.
|
||||
|
||||
Persisted manifests, requests, plans, tasks, expansions, outputs, candidates,
|
||||
decisions, and failures are frozen, recursively immutable, JSON-safe,
|
||||
canonically serialized, and strict about unknown fields; their enclosing wire
|
||||
contracts carry schema versions.
|
||||
Artifact identities contain a coordinator-owned UUID, schema, checksum, media
|
||||
type, and bounds; a scientific handler never persists a filesystem path.
|
||||
|
||||
## Try the built-in SDK workload
|
||||
|
||||
This example executes the current distributed `similarity-search` through the
|
||||
SDK without starting PostgreSQL or the coordinator:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from scimesh.sdk import (
|
||||
ArtifactCollection,
|
||||
JobRequest,
|
||||
LocalArtifactStore,
|
||||
LocalCoreBatchExecutor,
|
||||
default_sdk_registry,
|
||||
default_sdk_runtime,
|
||||
similarity_search_sdk_adapter,
|
||||
)
|
||||
|
||||
root = Path("sdk-run")
|
||||
store = LocalArtifactStore(root / "artifacts")
|
||||
adapter = similarity_search_sdk_adapter(shard_rows=1_000)
|
||||
|
||||
dataset = store.import_file(
|
||||
Path("chembl_37_chemreps.txt"),
|
||||
declaration=adapter.input_port.schema,
|
||||
)
|
||||
request = JobRequest(
|
||||
workload=adapter.manifest.workload,
|
||||
parameters={"query_smiles": "CCO", "top_k": 20},
|
||||
inputs={"input": ArtifactCollection.single(dataset)},
|
||||
)
|
||||
|
||||
result = LocalCoreBatchExecutor(
|
||||
default_sdk_registry(shard_rows=1_000),
|
||||
default_sdk_runtime(),
|
||||
store,
|
||||
root / "attempts",
|
||||
).execute(request, adapter.manifest.package.digest)
|
||||
|
||||
result_ref = result.outputs["result"].items[0].artifact
|
||||
print(store.materialize(result_ref))
|
||||
```
|
||||
|
||||
`LocalCoreBatchExecutor` is a correctness/conformance runtime, not a substitute
|
||||
for coordinator leases or multi-machine scheduling. It accepts only
|
||||
`TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-process/single-threaded CPU
|
||||
map/reduce stages without secrets, checkpoints, retries, gangs, or
|
||||
accelerators. It does not claim network, timeout, process, or credential
|
||||
isolation. Unsupported declarations are rejected before a handler runs. The
|
||||
harness uses the same legacy scientific planner, shard runner, and reducer as
|
||||
the distributed `similarity-search`, and its parity is covered by automated
|
||||
tests.
|
||||
|
||||
## Package shape and registration
|
||||
|
||||
An SDK distribution provides one explicit entry point per workload version:
|
||||
|
||||
```toml
|
||||
[project.entry-points."scimesh.workloads"]
|
||||
"descriptor-batch@1.0.0" = "scimesh_descriptors.sdk:workload_definition"
|
||||
```
|
||||
|
||||
The factory returns a `WorkloadDefinition` containing its manifest and handler
|
||||
objects. An administrator supplies an `AllowedPackage` with the same
|
||||
distribution, exact `WorkloadId`, and `sha256:` package digest. Discovery
|
||||
filters installed metadata before importing an entry point and fails
|
||||
transactionally if an allowlisted definition is missing or mismatched. Job
|
||||
parameters cannot name a module, entry point, package path, or executable.
|
||||
The measured digest covers package payload files and installed entry-point
|
||||
declarations and is checked before and after loading. It is a content pin, not
|
||||
a signature or image attestation; production discovery should run in a fresh
|
||||
trusted control-plane process so a pre-populated Python module cache is not an
|
||||
integrity boundary.
|
||||
|
||||
Direct registration is useful for tests and embedded deployments:
|
||||
|
||||
```python
|
||||
registry = WorkloadRegistry()
|
||||
registry.register(definition, enabled=False)
|
||||
registry.enable(
|
||||
definition.manifest.workload.name,
|
||||
definition.manifest.workload.version,
|
||||
definition.manifest.package.digest,
|
||||
)
|
||||
```
|
||||
|
||||
Both version and digest are required when resolving or planning. Upgrading an
|
||||
installed definition does not change the identity of an existing Job.
|
||||
|
||||
## Authoring rules
|
||||
|
||||
1. Keep the scientific core callable without a coordinator.
|
||||
2. Inline a strict JSON parameter schema with `type: object` and
|
||||
`additionalProperties: false`; the planner still performs domain validation.
|
||||
3. Give every external and stage port an `ArtifactSchema` with a media type,
|
||||
schema version, and byte/record/dimension bounds.
|
||||
4. Connect stage ports with `ArtifactEdge` values. `WorkflowSpec` checks source
|
||||
and target schemas, complete input bindings, declared dependencies, and
|
||||
acyclicity.
|
||||
5. Declare one `ResourceRequirements` and `ExecutionProfile` per stage. A task
|
||||
cannot run until its entire request is eligible and atomically reserved.
|
||||
6. Return only sink-sealed artifacts in `OutputManifest`; the local harness
|
||||
binds task key/provenance itself and rejects fabricated references,
|
||||
unexpected/missing ports, wrong schema/media type, and cumulative output or
|
||||
artifact-limit violations.
|
||||
7. Select a verifier compatible with determinism and trust. SDK v1 permits
|
||||
`untrusted_quorum` only for `byte_exact` plus `exact-artifact@1`.
|
||||
8. Add golden fixtures, local/distributed parity, retry/completion-order, and
|
||||
verifier failure tests before enabling a package.
|
||||
|
||||
`ArtifactSink` and `ArtifactCatalog` are bridge-owned protocols. They let
|
||||
scientific handlers materialize verified inputs and seal outputs without bearer
|
||||
tokens, database credentials, upload URLs, or durable local paths.
|
||||
|
||||
## Verification
|
||||
|
||||
The SDK includes:
|
||||
|
||||
- `ExactArtifactVerifier`: compares logical port/collection/schema/content
|
||||
digests while ignoring coordinator UUIDs, timestamps, metrics, and worker
|
||||
identity. Quorum inputs use coordinator-created `CandidateOutput` envelopes,
|
||||
count at most one vote per owner, and require a `VerificationBinding` for the
|
||||
exact task, inputs, parameters, package, manifest, and environment;
|
||||
- `CanonicalRecordVerifier`: applies a package-owned bounded canonicalizer and
|
||||
compares length-framed canonical records;
|
||||
- `NumericToleranceVerifier`: recursively checks structure plus explicit
|
||||
absolute, relative, ULP, and NaN policy, returning bounded sanitized evidence.
|
||||
|
||||
Canonical and numeric objects expose direct bounded comparison methods. To use
|
||||
them as manifest `Verifier` handlers, the package supplies an artifact-to-record
|
||||
or artifact-to-structured-value loader; without one, verification returns
|
||||
`inconclusive` rather than accepting bytes it did not parse.
|
||||
|
||||
A decision is `accepted`, `rejected`, or `inconclusive`; only `accepted`
|
||||
satisfies a stage. Evidence is limited to 16 KiB and cannot contain local paths
|
||||
or transport URLs.
|
||||
|
||||
## Resources and current runtime boundary
|
||||
|
||||
`ResourcePool` provides a lock-protected all-or-nothing local reservation for
|
||||
CPU cores, memory, scratch, and accelerator device/partition IDs, including
|
||||
whole-device versus partition conflict fencing. It enforces aggregate capacity
|
||||
and execution-slot count. `ExecutionProfile` produces only
|
||||
allocation-derived OpenMP/BLAS and device-visibility values; credentials never
|
||||
belong to scientific parameters.
|
||||
|
||||
The current protocol-v1 coordinator stores one input/result per flat task and
|
||||
does not persist resource requirements, device allocations, stage edges, or
|
||||
package versions. The production Worker also remains serial. Consequently:
|
||||
|
||||
- SDK `core-batch-v1` can be authored, validated, tested, discovered, and run
|
||||
through the trusted local conformance harness now;
|
||||
- existing production `similarity-search` remains on its compatible v1 wire
|
||||
path and is not renamed;
|
||||
- real concurrent claims, GPU scheduling, multi-output DAG execution, dynamic
|
||||
loops, streaming, and gang leases require the versioned coordinator/Worker
|
||||
changes listed in [`scimesh-sdk-roadmap.md`](scimesh-sdk-roadmap.md);
|
||||
- merely declaring a GPU or gang request never enables it. Missing runtime
|
||||
features or inventory fail before the planner executes.
|
||||
|
||||
## Conformance commands
|
||||
|
||||
Install development tools and run the SDK suite:
|
||||
|
||||
```bash
|
||||
pip install -e '.[dev]'
|
||||
pytest tests/test_sdk_models.py \
|
||||
tests/test_sdk_resources.py \
|
||||
tests/test_sdk_verification.py \
|
||||
tests/test_sdk_compatibility.py \
|
||||
tests/test_sdk_registry.py
|
||||
```
|
||||
|
||||
Run `pytest` for the full legacy, Worker, local-science, and SDK regression
|
||||
suite. Package authors can reuse `LocalArtifactStore`,
|
||||
`LocalCoreBatchExecutor`, and `assert_manifest_round_trip` in their own golden
|
||||
tests.
|
||||
Reference in New Issue
Block a user