Files

T

Emil bc76f386e5 Add MapReduceWorkload scaffold and generic workload execution

2026-08-02 01:02:24 +03:00

20 KiB

Raw Blame History

SciMesh Workload SDK v1

SciMesh now ships a public Python SDK under scimesh.sdk. The implemented authoring profile is core-batch-v1: installed and digest-pinned workload definitions, strict JSON manifests, typed artifact ports and collections, a static map/reduce workflow, CPU/memory/scratch eligibility, atomic local resource reservation, exact/canonical/numeric verifier primitives, and a compatibility adapter for the existing DistributedWorkload protocol. Its local executor is deliberately a trusted, in-process conformance harness; the production subprocess/lease sandbox remains a coordinator/Worker milestone.

The full target contract remains in scimesh-sdk-contract.md. Dynamic expansion, streaming, accelerators, gang execution, and side effects have typed bounded declarations, but the current coordinator/Worker runtime does not advertise their features. Compatibility negotiation therefore rejects those workflows before planner code runs.

SDK versus workloads

scimesh.sdk is the framework only: strict manifests, plans, artifacts, registry, verifiers, and the local conformance executor. It contains no scientific workload code. Workloads are user Python scripts and packages that import the SDK and live outside it. The built-in SciMesh workloads are under scimesh/workloads/:

scimesh/workloads/search/ — SDK-built similarity-search@1.0.0;
scimesh/workloads/graph/ — SDK-built similarity-graph@1.0.0;
scimesh/workloads/descriptors/ — SDK-built descriptor-batch@1.0.0;
scimesh/workloads/library.py — the built-in library wiring: a default registry containing all three definitions and a runtime advertising their capabilities;
the plain scimesh/workloads/*.py modules remain the local CLI scientific cores and their Workload registry.

Each SDK-built workload is a small package with core.py (scientific code) and definition.py (manifest plus planner/runner/reducer handlers). A future external workload library can follow the same shape: its own distribution, one scimesh.workloads entry point per workload version, and an administrator allowlist.

What authors import

The stable authoring surface is exported from scimesh.sdk:

WorkloadManifest, WorkloadId, VersionRange, PackageSpec, and EnvironmentSpec pin identity and compatibility;
ArtifactSchema, PortSpec, ArtifactRef, and ArtifactCollection define immutable data boundaries without transport URLs or local paths;
WorkflowSpec, StageSpec, ArtifactEdge, TaskSpec, and WorkflowPlan define a typed acyclic plan and pin package/manifest digests plus trust mode;
ResourceRequirements and ExecutionProfile separate per-task resources from Agent max_concurrency;
Planner, Runner, Reducer, and Verifier are the package handler protocols;
OutputManifest and Provenance describe sealed durable results;
WorkloadRegistry resolves an exact name, version, package digest, runtime, environment, and feature set. It never selects an implicit latest version;
MapReduceWorkload is the primary authoring scaffold for core-batch-v1: a subclass declares its identity, parameter schema, artifact ports, and three scientific hooks (partition, compute, merge), and the SDK assembles the manifest, map/reduce stages, workflow, digest-pinned handlers, and the exact-artifact verifier. See "Authoring a workload" below.

Persisted manifests, requests, plans, tasks, expansions, outputs, candidates, decisions, and failures are frozen, recursively immutable, JSON-safe, canonically serialized, and strict about unknown fields; their enclosing wire contracts carry schema versions. Artifact identities contain a coordinator-owned UUID, schema, checksum, media type, and bounds; a scientific handler never persists a filesystem path.

Try the built-in SDK workloads

This example runs the SDK-built similarity-search without starting PostgreSQL or the coordinator:

from pathlib import Path

from scimesh.sdk import (
    ArtifactCollection,
    JobRequest,
    LocalArtifactStore,
    LocalCoreBatchExecutor,
)
from scimesh.workloads.library import (
    default_sdk_registry,
    default_sdk_runtime,
    similarity_search_sdk_definition,
)

root = Path("sdk-run")
store = LocalArtifactStore(root / "artifacts")
workload = similarity_search_sdk_definition(shard_rows=1_000)

dataset = store.import_file(
    Path("chembl_37_chemreps.txt"),
    declaration=workload.manifest.inputs["input"].schema,
)
request = JobRequest(
    workload=workload.manifest.workload,
    parameters={"query_smiles": "CCO", "top_k": 20},
    inputs={"input": ArtifactCollection.single(dataset)},
)

result = LocalCoreBatchExecutor(
    default_sdk_registry(shard_rows=1_000),
    default_sdk_runtime(),
    store,
    root / "attempts",
).execute(request, workload.manifest.package.digest)

result_ref = result.outputs["result"].items[0].artifact
print(store.materialize(result_ref))

LocalCoreBatchExecutor is a correctness/conformance runtime, not a substitute for coordinator leases or multi-machine scheduling. It accepts only TrustMode.TRUSTED, NetworkPolicy.TRUSTED, single-process/single-threaded CPU map/reduce stages without secrets, checkpoints, retries, gangs, or accelerators. It does not claim network, timeout, process, or credential isolation. Unsupported declarations are rejected before a handler runs. The harness runs the SDK-built workload handlers themselves, and their parity against the single-process references is covered by automated tests.

The descriptor-batch reference workload

descriptor-batch@1.0.0 is the first SDK-native reference workload: it is built directly on the manifest/planner/runner/reducer contracts, and it is the intended first untrusted_quorum candidate (byte_exact plus exact-artifact@1). Its scientific contract is pinned:

one output CSV row per valid input molecule, in input order, with RDKit canonical SMILES recomputed by RDKit;
an explicit 81-name pinned RDKit 2D descriptor set (see scimesh/workloads/descriptors/core.py), validated against the installed RDKit at definition build time;
%.6f float formatting, utf-8 CSV with one header, and row-bounded deterministic shards;
skip_invalid is the only parameter (default true): invalid SMILES rows are counted and skipped, or fail the run when false;
the reducer concatenates shard partials by shard index with exactly one header, so the distributed output is byte-identical to the single-process reference for the same input rows.

from pathlib import Path

from scimesh.sdk import (
    ArtifactCollection,
    JobRequest,
    LocalArtifactStore,
    LocalCoreBatchExecutor,
    WorkloadRegistry,
)
from scimesh.workloads.descriptors import descriptor_batch_sdk_definition
from scimesh.workloads.library import default_sdk_runtime

root = Path("descriptor-run")
store = LocalArtifactStore(root / "artifacts")
workload = descriptor_batch_sdk_definition(shard_rows=1_000)

dataset = store.import_file(
    Path("chembl_37_chemreps.txt"),
    declaration=workload.manifest.inputs["input"].schema,
)
request = JobRequest(
    workload=workload.manifest.workload,
    parameters={"skip_invalid": True},
    inputs={"input": ArtifactCollection.single(dataset)},
)
registry = WorkloadRegistry()
registry.register(workload.definition(), enabled=True)

result = LocalCoreBatchExecutor(
    registry,
    default_sdk_runtime(),
    store,
    root / "attempts",
).execute(request, workload.manifest.package.digest)

result_ref = result.outputs["result"].items[0].artifact
print(store.materialize(result_ref))

The descriptor-batch entry point descriptor-batch@1.0.0 is declared in pyproject.toml; discovery loads it only when an administrator supplies a matching AllowedPackage allowlist entry. Its manifest declares both trusted and untrusted_quorum trust modes and the exact-artifact verifier, so the same definition can later run under coordinator quorum once protocol-v2 leases exist.

The SDK-built similarity workloads

similarity-search@1.0.0 and similarity-graph@1.0.0 are SDK-built workloads under scimesh/workloads/search/ and scimesh/workloads/graph/; both reuse the local scientific cores from scimesh/workloads/similarity_search.py and similarity_graph.py and declare byte_exact + exact-artifact@1:

the search workload resolves query_id exactly once at plan time, shards the input deterministically, computes a local top-k per shard with the reference heap, and merges the sorted partials with the same tie-breakers, so the final CSV is byte-identical to the single-process CLI output;
the graph workload parses molecules once into deterministic row-ordered blocks, plans one map task per block pair (i, j) with i <= j, and its reducer enforces the pair-coverage invariant (every unordered molecule pair compared exactly once, no duplicates) before emitting the same deterministically sorted edge list as the local brute-force reference, for either threshold direction and any block size;
the v1 worker executes SDK-built workloads directly: scimesh/worker/runners.py is a workload-generic wire bridge that builds a TaskSpec with the workload's own pins, negotiates against a runtime derived from the loaded definitions, reserves resources, seals the partial through a content-addressed store, and uploads the resulting CSV over the unchanged coordinator contract. The worker loads workloads from SCIMESH_WORKLOAD_ALLOWLIST (a JSON array of {distribution, name, version, digest} entries matched against installed scimesh.workloads entry points) or falls back to the built-in similarity-search; advertised capabilities come from SCIMESH_CAPABILITIES. Workloads whose map stage needs more than one input port are rejected with a clear message until the coordinator contract supports them.

Authoring a workload

A workload is a user script that imports the SDK. For the standard core-batch-v1 shape (one input dataset, shards, partials, one merged result) subclass MapReduceWorkload and implement the three scientific hooks; the framework provides everything else:

from pathlib import Path
from typing import Any, Mapping, Sequence

from scimesh.sdk import (
    ArtifactSchema,
    ComponentRef,
    MapReduceWorkload,
    PortSpec,
    SchemaRef,
    WorkloadId,
)

class CountRowsWorkload(MapReduceWorkload):
    workload_id = WorkloadId("count-rows", "1.0.0")
    description = "Count TSV data rows per shard and concatenate the counts."
    parameters_schema = {
        "type": "object",
        "additionalProperties": False,
        "properties": {"prefix": {"type": "string", "minLength": 1, "maxLength": 50}},
    }
    input_port = PortSpec(ArtifactSchema(
        SchemaRef("molecule-table", 1), "text/tab-separated-values", "utf-8",
        max_bytes=10**9, validator=ComponentRef("delimited-table", 1),
        validator_configuration={"required_columns": ["canonical_smiles", "chembl_id"]},
    ))
    partial_port = output_port = PortSpec(ArtifactSchema(
        SchemaRef("count-table", 1), "text/csv", "utf-8",
        max_bytes=10**9, validator=ComponentRef("delimited-table", 1),
        validator_configuration={"columns": ["id", "rows"]},
    ))
    map_parameter_names = ("prefix",)

    def partition_input(self, input_path, parameters, workspace):  # -> list[Path]
        ...  # deterministic shard files, one per map task

    def compute_shard(self, inputs, parameters, output_path):  # -> Mapping[str, int|float]
        ...  # one map task; inputs maps each map port to a materialized file

    def reduce_partials(self, partial_paths, parameters, output_path):  # -> Mapping[str, int|float]
        ...  # deterministic merge of the accepted partials

The base class then provides validate, plan, run, reduce, and definition(); the registry, negotiation, resource reservation, verification, and the local conformance executor treat the result like any other workload:

from scimesh.sdk import (
    ArtifactCollection,
    JobRequest,
    LocalArtifactStore,
    LocalCoreBatchExecutor,
    WorkloadRegistry,
)
from scimesh.workloads.library import default_sdk_runtime

workload = CountRowsWorkload(package_digest=..., environment_digest=...)
registry = WorkloadRegistry()
registry.register(workload.definition(), enabled=True)

store = LocalArtifactStore(Path("artifacts"))
artifact = store.import_file(Path("tiny.tsv"), declaration=workload.manifest.inputs["input"].schema)
request = JobRequest(workload=workload.manifest.workload, parameters={"prefix": "x"},
                     inputs={"input": ArtifactCollection.single(artifact)})
result = LocalCoreBatchExecutor(registry, default_sdk_runtime(), store, Path("work")) \
    .execute(request, workload.manifest.package.digest)

Hooks you can override beyond the three scientific ones:

domain_validate(parameters) — extra job-parameter validation (the JSON schema already ran);
resolved_parameters(request) / resolved_parameters_for_plan(job, input_path, resolved) — values persisted into the plan (for example one-time query resolution);
plan_tasks(...) — custom task construction (the graph workload uses this to plan one task per block pair with two block inputs);
parse_partial_key(key) / validate_partial_keys(parsed) — partial-key policy (default: map.<eight-digit-index>, contiguous; the graph workload parses map.<i>x<j> and enforces the pair-coverage invariant);
map_stage_inputs — a map stage with more than one input port (each extra port must share the external input schema).

Anything outside this model uses the lower-level SDK value objects directly. Authoring rules: keep the scientific core callable without a coordinator, inline a strict JSON parameter schema, declare artifact schemas with bounds, return only sink-sealed artifacts, and select a verifier compatible with determinism and trust.

To run a workload from the command line without writing any program code:

scimesh workload list
scimesh workload run count-rows --input tiny.tsv --params '{"prefix": "x"}' -o result.csv

scimesh workload is a generic SDK tool; it contains no workload-specific logic, so new workloads do not require changes to the CLI or any other part of the program.

Package shape and registration

A workload distribution provides one explicit entry point per workload version. The built-in workloads are part of the scimesh distribution:

[project.entry-points."scimesh.workloads"]
"similarity-search@1.0.0" = "scimesh.workloads.search:workload_definition"
"similarity-graph@1.0.0" = "scimesh.workloads.graph:workload_definition"
"descriptor-batch@1.0.0" = "scimesh.workloads.descriptors:workload_definition"

The factory returns a WorkloadDefinition containing its manifest and handler objects. An administrator supplies an AllowedPackage with the same distribution, exact WorkloadId, and sha256: package digest. Discovery filters installed metadata before importing an entry point and fails transactionally if an allowlisted definition is missing or mismatched. Job parameters cannot name a module, entry point, package path, or executable. The measured digest covers package payload files and installed entry-point declarations and is checked before and after loading. It is a content pin, not a signature or image attestation; production discovery should run in a fresh trusted control-plane process so a pre-populated Python module cache is not an integrity boundary.

Direct registration is useful for tests and embedded deployments:

registry = WorkloadRegistry()
registry.register(definition, enabled=False)
registry.enable(
    definition.manifest.workload.name,
    definition.manifest.workload.version,
    definition.manifest.package.digest,
)

Both version and digest are required when resolving or planning. Upgrading an installed definition does not change the identity of an existing Job.

Authoring rules

Keep the scientific core callable without a coordinator.
Inline a strict JSON parameter schema with type: object and additionalProperties: false; the planner still performs domain validation.
Give every external and stage port an ArtifactSchema with a media type, schema version, and byte/record/dimension bounds.
Connect stage ports with ArtifactEdge values. WorkflowSpec checks source and target schemas, complete input bindings, declared dependencies, and acyclicity.
Declare one ResourceRequirements and ExecutionProfile per stage. A task cannot run until its entire request is eligible and atomically reserved.
Return only sink-sealed artifacts in OutputManifest; the local harness binds task key/provenance itself and rejects fabricated references, unexpected/missing ports, wrong schema/media type, and cumulative output or artifact-limit violations.
Select a verifier compatible with determinism and trust. SDK v1 permits untrusted_quorum only for byte_exact plus exact-artifact@1.
Add golden fixtures, local/distributed parity, retry/completion-order, and verifier failure tests before enabling a package.

ArtifactSink and ArtifactCatalog are bridge-owned protocols. They let scientific handlers materialize verified inputs and seal outputs without bearer tokens, database credentials, upload URLs, or durable local paths.

Verification

The SDK includes:

ExactArtifactVerifier: compares logical port/collection/schema/content digests while ignoring coordinator UUIDs, timestamps, metrics, and worker identity. Quorum inputs use coordinator-created CandidateOutput envelopes, count at most one vote per owner, and require a VerificationBinding for the exact task, inputs, parameters, package, manifest, and environment;
CanonicalRecordVerifier: applies a package-owned bounded canonicalizer and compares length-framed canonical records;
NumericToleranceVerifier: recursively checks structure plus explicit absolute, relative, ULP, and NaN policy, returning bounded sanitized evidence.

Canonical and numeric objects expose direct bounded comparison methods. To use them as manifest Verifier handlers, the package supplies an artifact-to-record or artifact-to-structured-value loader; without one, verification returns inconclusive rather than accepting bytes it did not parse.

A decision is accepted, rejected, or inconclusive; only accepted satisfies a stage. Evidence is limited to 16 KiB and cannot contain local paths or transport URLs.

Resources and current runtime boundary

ResourcePool provides a lock-protected all-or-nothing local reservation for CPU cores, memory, scratch, and accelerator device/partition IDs, including whole-device versus partition conflict fencing. It enforces aggregate capacity and execution-slot count. ExecutionProfile produces only allocation-derived OpenMP/BLAS and device-visibility values; credentials never belong to scientific parameters.

The current protocol-v1 coordinator stores one input/result per flat task and does not persist resource requirements, device allocations, stage edges, or package versions. The production Worker also remains serial. Consequently:

SDK core-batch-v1 can be authored, validated, tested, discovered, and run through the trusted local conformance harness now;
existing production similarity-search remains on its compatible v1 wire path and is not renamed;
real concurrent claims, GPU scheduling, multi-output DAG execution, dynamic loops, streaming, and gang leases require the versioned coordinator/Worker changes listed in scimesh-sdk-roadmap.md;
merely declaring a GPU or gang request never enables it. Missing runtime features or inventory fail before the planner executes.

Conformance commands

Install development tools and run the SDK suite:

pip install -e '.[dev]'
pytest tests/test_sdk_models.py \
       tests/test_sdk_resources.py \
       tests/test_sdk_verification.py \
       tests/test_sdk_compatibility.py \
       tests/test_sdk_registry.py \
       tests/test_sdk_descriptors.py \
       tests/test_sdk_search.py \
       tests/test_sdk_graph.py

Run pytest for the full Worker, local-science, and SDK regression suite. Package authors can reuse LocalArtifactStore, LocalCoreBatchExecutor, and assert_manifest_round_trip in their own golden tests.

20 KiB Raw Blame History