Add descriptor-batch SDK reference workload

This commit is contained in:
Emil
2026-08-01 23:27:53 +03:00
parent c43af32495
commit 96169086f0
9 changed files with 1363 additions and 14 deletions
+13 -10
View File
@@ -36,16 +36,19 @@ implements the `core-batch-v1` profile:
## What remains, in delivery order
1. **`descriptor-batch` reference workload** (roadmap step 3 — the recommended
next task; it is pure Python and needs no coordinator changes). Pinned RDKit
2D descriptors, canonical one-row-per-input CSV, shard-index concatenation
with one header, byte-identical local/distributed output, two-worker quorum.
Build it as an SDK-native package (manifest + planner/runner/reducer/
verifier handlers), not through the legacy adapter; reuse the
`similarity-search` adapter (`scimesh/sdk/compat/distributed_v1.py`) and
`builtins.py` as the structural template, and the
`tests/test_sdk_compatibility.py` fixtures as the test template. This is the
intended first `untrusted_quorum` candidate (byte_exact + exact-artifact@1).
1. ~~**`descriptor-batch` reference workload**~~ — **done** (2026-08-01):
`scimesh/sdk/descriptors/` (`core.py` + `definition.py`) is the first
SDK-native workload. Pinned 81-name RDKit 2D descriptor set (validated at
definition build time), canonical one-row-per-input CSV with `%.6f` floats,
deterministic row-bounded shards, shard-index concatenation with one header,
byte-identical local/distributed output, `skip_invalid` explicit policy, and
`untrusted_quorum` + exact-artifact@1 declared in the manifest. Entry point
`descriptor-batch@1.0.0` is in `pyproject.toml`; `default_sdk_runtime` now
advertises the `descriptor-batch` capability. Tests:
`tests/test_sdk_descriptors.py` (8 tests: manifest/negotiation, local-vs-
reference byte parity, deterministic path-free planning, explicit invalid-
row policy, strict parameter schema, two-owner quorum accept, conflicting-
quorum reject, allowlist discovery). Total suite: 233 passing.
2. **Distributed `similarity-graph`** (CTX-10, roadmap step 1). The coordinator
currently rejects `similarity-graph` uploads; it needs cross-shard block-pair
planning and duplicate-safe reduction. STATUS.md names this the next
+69 -1
View File
@@ -94,6 +94,73 @@ harness uses the same legacy scientific planner, shard runner, and reducer as
the distributed `similarity-search`, and its parity is covered by automated
tests.
## The descriptor-batch reference workload
`descriptor-batch@1.0.0` is the first SDK-native reference workload: it is
built directly on the manifest/planner/runner/reducer contracts instead of the
legacy adapter, and it is the intended first `untrusted_quorum` candidate
(`byte_exact` plus `exact-artifact@1`). Its scientific contract is pinned:
- one output CSV row per valid input molecule, in input order, with RDKit
canonical SMILES recomputed by RDKit;
- an explicit 81-name pinned RDKit 2D descriptor set (see
`scimesh/sdk/descriptors/core.py`), validated against the installed RDKit at
definition build time;
- `%.6f` float formatting, `utf-8` CSV with one header, and row-bounded
deterministic shards;
- `skip_invalid` is the only parameter (default `true`): invalid SMILES rows
are counted and skipped, or fail the run when `false`;
- the reducer concatenates shard partials by shard index with exactly one
header, so the distributed output is byte-identical to the single-process
reference for the same input rows.
```python
from pathlib import Path
from scimesh.sdk import (
ArtifactCollection,
JobRequest,
LocalArtifactStore,
LocalCoreBatchExecutor,
WorkloadRegistry,
default_sdk_runtime,
)
from scimesh.sdk.descriptors import descriptor_batch_sdk_definition
root = Path("descriptor-run")
store = LocalArtifactStore(root / "artifacts")
workload = descriptor_batch_sdk_definition(shard_rows=1_000)
dataset = store.import_file(
Path("chembl_37_chemreps.txt"),
declaration=workload.manifest.inputs["input"].schema,
)
request = JobRequest(
workload=workload.manifest.workload,
parameters={"skip_invalid": True},
inputs={"input": ArtifactCollection.single(dataset)},
)
registry = WorkloadRegistry()
registry.register(workload.definition(), enabled=True)
result = LocalCoreBatchExecutor(
registry,
default_sdk_runtime(),
store,
root / "attempts",
).execute(request, workload.manifest.package.digest)
result_ref = result.outputs["result"].items[0].artifact
print(store.materialize(result_ref))
```
The descriptor-batch entry point `descriptor-batch@1.0.0` is declared in
`pyproject.toml`; discovery loads it only when an administrator supplies a
matching `AllowedPackage` allowlist entry. Its manifest declares both
`trusted` and `untrusted_quorum` trust modes and the exact-artifact verifier,
so the same definition can later run under coordinator quorum once protocol-v2
leases exist.
## Package shape and registration
An SDK distribution provides one explicit entry point per workload version:
@@ -211,7 +278,8 @@ pytest tests/test_sdk_models.py \
tests/test_sdk_resources.py \
tests/test_sdk_verification.py \
tests/test_sdk_compatibility.py \
tests/test_sdk_registry.py
tests/test_sdk_registry.py \
tests/test_sdk_descriptors.py
```
Run `pytest` for the full legacy, Worker, local-science, and SDK regression