Emil_Shanaty/SciMesh
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add distributed similarity search
python / test (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
Emil
2026-07-24 14:38:11 +03:00
parent 0bef7604fd
commit 0f3a2d92d8
10 changed files with 749 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATUS.md
+12 -10
View File
@@ -32,8 +32,8 @@ Docker PostgreSQL stack on 2026-07-23.
| CTX-04 Worker registry and HTTP API | Implemented | Registration, claim, heartbeat, result, failure, and status endpoints. |
| CTX-05 Artifact storage | Implemented | Coordinator-owned inputs/results, checksum verification, and upload flow. |
| CTX-06 Python Worker live-contract alignment | Implemented | Worker completed a real uploaded shard via HTTP on 2026-07-23. |
| CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering are in `scimesh/distributed/`. The concrete molecular planner/reducer remains CTX-08/09. |
| CTX-08 Distributed similarity-search | Not started | Local reference exists. |
| CTX-07 Distributed workload protocol | Implemented | Versioned Python contract models, registry, strict plan validation, and deterministic reduction ordering are in `scimesh/distributed/`. |
| CTX-08 Distributed similarity-search | Implemented (scientific layer) | Python planner resolves `query_id` once, creates deterministic shard plans, worker adapter emits exact partial top-k CSVs/metrics, and reducer matches the local reference. Coordinator persistence/orchestration remains CTX-09. |
| CTX-09 Reducer and final-result API | Not started | Depends on CTX-07 and CTX-08. |
| CTX-10 Distributed similarity-graph | Not started | Local reference exists. |
| CTX-11 Dashboard/operator view | Implemented (diagnostic scope) | Protected local view: job/task/worker status, validated similarity-search upload, diagnostic partial-artifact download, and bounded polling. Final-result reduction remains CTX-09. |
@@ -41,20 +41,22 @@ Docker PostgreSQL stack on 2026-07-23.
## Next recommended assignment
Assign **CTX-08** to the workload role: implement the molecular
`similarity-search` planner and worker adapter on top of the accepted CTX-07
contract.
Assign **CTX-09** to the coordinator role: materialize planned shards,
persist them transactionally, invoke the registered reducer once, and expose a
durable final artifact.
## Known constraints
- The CTX-07 protocol is implemented, but no concrete molecular planner or
reducer is registered yet; the operator UI labels `partial_result` files as
diagnostic and cannot present them as final output.
- The Python `similarity-search` planner/reducer is implemented, but the Go
coordinator does not yet invoke it or persist its final artifact. The
operator UI labels `partial_result` files as diagnostic and cannot present
them as final output.
Use the local `scimesh` CLI for complete workload results.
- The worker/coordinator flow currently accepts both underscore API workload
names and hyphenated CLI names while the contract is consolidated.
- A real-stack worker test uses a small `query_smiles` shard. Resolving a
`query_id` once and sharing it across shards belongs to CTX-07.
- A real-stack worker test uses a small `query_smiles` shard. The Python
planner resolves `query_id` once and shares `query_smiles`; connecting that
planner to uploaded coordinator jobs belongs to CTX-09.
- The coordinator accepts uploaded distributed jobs only for
`similarity-search` with `query_smiles`. It rejects `similarity-graph` until
CTX-10 supplies cross-shard pair planning.