Emil_Shanaty/SciMesh
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace legacy distributed protocol with SDK-built workloads

Browse Source
This commit is contained in:
Emil
2026-08-01 23:57:41 +03:00
parent 96169086f0
commit 19fbb8e926
34 changed files with 3009 additions and 1986 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sdk-handoff.md
+43 -13
View File
@@ -12,6 +12,18 @@ Read first, in this order: `AGENTS.md` (binding repo rules),
## What is already done (do not redo)
**Legacy removal (2026-08-01):** the CTX-07 `DistributedWorkload` protocol
package (`scimesh/distributed/`), the SDK compatibility adapter
(`scimesh/sdk/compat/`), and `library.similarity_search_sdk_adapter` were
removed. The worker now executes the SDK-built workloads directly:
`scimesh/worker/runners.py` builds a `TaskSpec` with the workload's pins,
negotiates, reserves resources, runs the workload's own Runner through
`LocalTaskContext` (store-backed catalog/sink), and uploads the sealed
partial over the unchanged v1 wire. `run_search_shard` + the full-precision
partial writer moved to `scimesh/workloads/search/core.py`; the partial
format is unchanged, so the Go reducer and UI keep working. The runner
resolves `query_id` per task and rejects plan-time `max_rows`.
CTX-16 "Workload SDK foundation" is complete and tested. `scimesh/sdk/`
implements the `core-batch-v1` profile:
@@ -26,29 +38,41 @@ implements the `core-batch-v1` profile:
`NumericToleranceVerifier` with bounded sanitized evidence: `verification.py`.
- Local conformance harness: `LocalArtifactStore`, `LocalCoreBatchExecutor`,
`ResourcePool` (atomic all-or-nothing reservation): `conformance.py`.
- Legacy adapter exposing distributed `similarity-search` through the SDK
without changing its wire schema: `compat/distributed_v1.py`, `builtins.py`;
entry point `similarity-search@1.0.0` is declared in `pyproject.toml`.
- SDK-built workloads living outside the SDK: `scimesh/workloads/search/`,
`scimesh/workloads/graph/`, `scimesh/workloads/descriptors/` (each `core.py`
+ `definition.py`), composed by `scimesh/workloads/library.py`
(`default_sdk_registry`, `default_sdk_runtime`); entry points for all three
declared in `pyproject.toml`.
- Tests: `tests/test_sdk_{models,resources,verification,compatibility,registry}.py`
including fail-closed rejection coverage for every advanced profile
declaration (gang, GPU modes, pools, checkpoints, retries, secrets, streams,
loops, side effects).
loops, side effects), plus `tests/test_sdk_{search,graph,descriptors}.py`
and the worker bridge tests in `tests/test_worker_daemon.py`.
`tests/test_distributed*.py` were removed with the protocol.
## What remains, in delivery order
1. ~~**`descriptor-batch` reference workload**~~ — **done** (2026-08-01):
`scimesh/sdk/descriptors/` (`core.py` + `definition.py`) is the first
SDK-native workload. Pinned 81-name RDKit 2D descriptor set (validated at
`scimesh/workloads/descriptors/` (`core.py` + `definition.py`) is the first
SDK-built workload. Pinned 81-name RDKit 2D descriptor set (validated at
definition build time), canonical one-row-per-input CSV with `%.6f` floats,
deterministic row-bounded shards, shard-index concatenation with one header,
byte-identical local/distributed output, `skip_invalid` explicit policy, and
`untrusted_quorum` + exact-artifact@1 declared in the manifest. Entry point
`descriptor-batch@1.0.0` is in `pyproject.toml`; `default_sdk_runtime` now
advertises the `descriptor-batch` capability. Tests:
`tests/test_sdk_descriptors.py` (8 tests: manifest/negotiation, local-vs-
reference byte parity, deterministic path-free planning, explicit invalid-
row policy, strict parameter schema, two-owner quorum accept, conflicting-
quorum reject, allowlist discovery). Total suite: 233 passing.
`untrusted_quorum` + exact-artifact@1 declared in the manifest. Tests:
`tests/test_sdk_descriptors.py`.
2. ~~**SDK-built `similarity-search` and `similarity-graph`**~~ — **done**
(2026-08-01). Both local workloads are SDK-built packages outside the SDK:
`scimesh/workloads/search/` and `scimesh/workloads/graph/` (each `core.py` +
`definition.py`, manifest + planner/runner/reducer, byte_exact +
exact-artifact@1, trusted + untrusted_quorum). Search resolves the query at
plan time and merges partials with the reference heap (byte-identical to the
CLI). Graph plans one task per block pair `(i,j)` with `i <= j`, reducer
enforces pair-coverage and duplicate-pair rejection, output byte-identical
to the local brute-force reference for both directions and any block size.
Tests: `tests/test_sdk_search.py`, `tests/test_sdk_graph.py`.
**Architecture note:** `scimesh.sdk/` is the framework ONLY (no workload
code); workloads are user scripts/packages under `scimesh/workloads/` that
import the SDK. Keep new workloads out of the SDK package.
2. **Distributed `similarity-graph`** (CTX-10, roadmap step 1). The coordinator
currently rejects `similarity-graph` uploads; it needs cross-shard block-pair
planning and duplicate-safe reduction. STATUS.md names this the next
@@ -76,6 +100,12 @@ implements the `core-batch-v1` profile:
## Known traps (cost the previous session real time)
- **Architecture boundary:** `scimesh.sdk/` is the framework only and must
never import `scimesh.workloads` (SDK depends on nothing workload-specific).
Workload packages live under `scimesh/workloads/` (each `core.py` +
`definition.py`), and built-in wiring lives in `scimesh/workloads/library.py`.
The digest helpers are in `scimesh/workloads/environment.py`; the SDK keeps
only the generic `installed_distribution_digest` in `scimesh/sdk/integrity.py`.
- The legacy adapter pins its own manifest (`adapter.manifest`). If a test
changes limits/workflow on the manifest, the adapter's copy must be replaced
too, or `registry.plan` fails with "planner plan does not carry the selected
docs/workload-sdk.md
+72 -21
View File
@@ -16,6 +16,29 @@ declarations, but the current coordinator/Worker runtime does not advertise
their features. Compatibility negotiation therefore rejects those workflows
before planner code runs.
## SDK versus workloads
`scimesh.sdk` is the framework only: strict manifests, plans, artifacts,
registry, verifiers, and the local conformance executor. It contains no
scientific workload code. Workloads are user Python scripts and packages that
import the SDK and live outside it. The built-in SciMesh workloads are under
`scimesh/workloads/`:
- `scimesh/workloads/search/` — SDK-built `similarity-search@1.0.0`;
- `scimesh/workloads/graph/` — SDK-built `similarity-graph@1.0.0`;
- `scimesh/workloads/descriptors/` — SDK-built `descriptor-batch@1.0.0`;
- `scimesh/workloads/library.py` — the built-in library wiring: a default
registry containing all three definitions and a runtime advertising their
capabilities;
- the plain `scimesh/workloads/*.py` modules remain the local CLI scientific
cores and their `Workload` registry.
Each SDK-built workload is a small package with `core.py` (scientific code)
and `definition.py` (manifest plus planner/runner/reducer handlers). A future
external workload library can follow the same shape: its own distribution, one
`scimesh.workloads` entry point per workload version, and an administrator
allowlist.
## What authors import
The stable authoring surface is exported from `scimesh.sdk`:
@@ -41,10 +64,10 @@ contracts carry schema versions.
Artifact identities contain a coordinator-owned UUID, schema, checksum, media
type, and bounds; a scientific handler never persists a filesystem path.
## Try the built-in SDK workload
## Try the built-in SDK workloads
This example executes the current distributed `similarity-search` through the
SDK without starting PostgreSQL or the coordinator:
This example runs the SDK-built `similarity-search` without starting
PostgreSQL or the coordinator:
```python
from pathlib import Path
@@ -54,21 +77,23 @@ from scimesh.sdk import (
JobRequest,
LocalArtifactStore,
LocalCoreBatchExecutor,
)
from scimesh.workloads.library import (
default_sdk_registry,
default_sdk_runtime,
similarity_search_sdk_adapter,
similarity_search_sdk_definition,
)
root = Path("sdk-run")
store = LocalArtifactStore(root / "artifacts")
adapter = similarity_search_sdk_adapter(shard_rows=1_000)
workload = similarity_search_sdk_definition(shard_rows=1_000)
dataset = store.import_file(
Path("chembl_37_chemreps.txt"),
declaration=adapter.input_port.schema,
declaration=workload.manifest.inputs["input"].schema,
)
request = JobRequest(
workload=adapter.manifest.workload,
workload=workload.manifest.workload,
parameters={"query_smiles": "CCO", "top_k": 20},
inputs={"input": ArtifactCollection.single(dataset)},
)
@@ -78,7 +103,7 @@ result = LocalCoreBatchExecutor(
default_sdk_runtime(),
store,
root / "attempts",
).execute(request, adapter.manifest.package.digest)
).execute(request, workload.manifest.package.digest)
result_ref = result.outputs["result"].items[0].artifact
print(store.materialize(result_ref))
@@ -90,21 +115,20 @@ for coordinator leases or multi-machine scheduling. It accepts only
map/reduce stages without secrets, checkpoints, retries, gangs, or
accelerators. It does not claim network, timeout, process, or credential
isolation. Unsupported declarations are rejected before a handler runs. The
harness uses the same legacy scientific planner, shard runner, and reducer as
the distributed `similarity-search`, and its parity is covered by automated
tests.
harness runs the SDK-built workload handlers themselves, and their parity
against the single-process references is covered by automated tests.
## The descriptor-batch reference workload
`descriptor-batch@1.0.0` is the first SDK-native reference workload: it is
built directly on the manifest/planner/runner/reducer contracts instead of the
legacy adapter, and it is the intended first `untrusted_quorum` candidate
built directly on the manifest/planner/runner/reducer contracts, and it is
the intended first `untrusted_quorum` candidate
(`byte_exact` plus `exact-artifact@1`). Its scientific contract is pinned:
- one output CSV row per valid input molecule, in input order, with RDKit
canonical SMILES recomputed by RDKit;
- an explicit 81-name pinned RDKit 2D descriptor set (see
`scimesh/sdk/descriptors/core.py`), validated against the installed RDKit at
`scimesh/workloads/descriptors/core.py`), validated against the installed RDKit at
definition build time;
- `%.6f` float formatting, `utf-8` CSV with one header, and row-bounded
deterministic shards;
@@ -123,9 +147,9 @@ from scimesh.sdk import (
LocalArtifactStore,
LocalCoreBatchExecutor,
WorkloadRegistry,
default_sdk_runtime,
)
from scimesh.sdk.descriptors import descriptor_batch_sdk_definition
from scimesh.workloads.descriptors import descriptor_batch_sdk_definition
from scimesh.workloads.library import default_sdk_runtime
root = Path("descriptor-run")
store = LocalArtifactStore(root / "artifacts")
@@ -161,13 +185,39 @@ matching `AllowedPackage` allowlist entry. Its manifest declares both
so the same definition can later run under coordinator quorum once protocol-v2
leases exist.
## The SDK-built similarity workloads
`similarity-search@1.0.0` and `similarity-graph@1.0.0` are SDK-built workloads
under `scimesh/workloads/search/` and `scimesh/workloads/graph/`; both reuse
the local scientific cores from `scimesh/workloads/similarity_search.py` and
`similarity_graph.py` and declare `byte_exact` + `exact-artifact@1`:
- the search workload resolves `query_id` exactly once at plan time, shards
the input deterministically, computes a local top-k per shard with the
reference heap, and merges the sorted partials with the same tie-breakers,
so the final CSV is byte-identical to the single-process CLI output;
- the graph workload parses molecules once into deterministic row-ordered
blocks, plans one map task per block pair `(i, j)` with `i <= j`, and its
reducer enforces the pair-coverage invariant (every unordered molecule pair
compared exactly once, no duplicates) before emitting the same
deterministically sorted edge list as the local brute-force reference, for
either threshold direction and any block size;
- the v1 worker executes the SDK-built `similarity-search` runner directly:
`scimesh/worker/runners.py` is a small wire bridge that builds a `TaskSpec`
with the workload's own pins, reserves resources, seals the partial through
a content-addressed store, and uploads the resulting CSV over the unchanged
coordinator contract.
## Package shape and registration
An SDK distribution provides one explicit entry point per workload version:
A workload distribution provides one explicit entry point per workload
version. The built-in workloads are part of the `scimesh` distribution:
```toml
[project.entry-points."scimesh.workloads"]
"descriptor-batch@1.0.0" = "scimesh_descriptors.sdk:workload_definition"
"similarity-search@1.0.0" = "scimesh.workloads.search:workload_definition"
"similarity-graph@1.0.0" = "scimesh.workloads.graph:workload_definition"
"descriptor-batch@1.0.0" = "scimesh.workloads.descriptors:workload_definition"
```
The factory returns a `WorkloadDefinition` containing its manifest and handler
@@ -279,10 +329,11 @@ pytest tests/test_sdk_models.py \
tests/test_sdk_verification.py \
tests/test_sdk_compatibility.py \
tests/test_sdk_registry.py \
tests/test_sdk_descriptors.py
tests/test_sdk_descriptors.py \
tests/test_sdk_search.py \
tests/test_sdk_graph.py
```
Run `pytest` for the full legacy, Worker, local-science, and SDK regression
suite. Package authors can reuse `LocalArtifactStore`,
Run `pytest` for the full Worker, local-science, and SDK regression suite. Package authors can reuse `LocalArtifactStore`,
`LocalCoreBatchExecutor`, and `assert_manifest_round_trip` in their own golden
tests.