Add distributed similarity search
python / test (push) Canceled after 0s

This commit is contained in:
Emil
2026-07-24 14:38:11 +03:00
parent 0bef7604fd
commit 0f3a2d92d8
10 changed files with 749 additions and 47 deletions
+13 -8
View File
@@ -4,9 +4,11 @@
This document is the implementation contract for CTX-07. Its generic protocol,
registry, strict JSON models, and deterministic reduction ordering are
implemented in `scimesh/distributed/`. It does not implement a molecular
planner, reducer, API endpoint, database migration, or final artifact. Until
CTX-08 and CTX-09 are complete, shard CSVs remain diagnostic partial results.
implemented in `scimesh/distributed/`. CTX-08 implements the molecular
similarity-search planner, worker adapter, and pure reducer on top of it. This
document does not implement a coordinator API endpoint, database migration, or
durable final artifact. Until CTX-09 is complete, shard CSVs remain diagnostic
partial results.
The protocol gives local scientific workloads a coordinator-independent way to
validate a job, plan artifact-backed tasks, and later reduce completed outputs.
@@ -154,7 +156,10 @@ rank,chembl_id,canonical_smiles,similarity
```
- `rank` is one-based local rank.
- `similarity` uses the local CLI's six-decimal formatting.
- `similarity` uses a round-trip decimal representation of the computed float
(for Python, `repr(similarity)`). This preserves exact cross-shard ranking;
the reducer writes the user-facing final CSV with the local CLI's six-decimal
display formatting.
- Rows are sorted by `(-similarity, chembl_id, canonical_smiles)` for
`threshold_direction=greater`, or `(similarity, chembl_id,
canonical_smiles)` for `less`.
@@ -203,7 +208,7 @@ multiplicity. Reduction is independent of worker completion order and uses
## Deferred work
CTX-08 implements the similarity-search planner, runner adapter, reducer, and
comparison against the local CLI. CTX-09 persists the final artifact and job
state. CTX-10 defines graph-specific triangular block plans; it must not reuse
the search shard scheme without its pair-coverage invariants.
CTX-09 materializes the planned shard files as coordinator artifacts, invokes
the registered reducer once, and persists its final artifact/job state. CTX-10
defines graph-specific triangular block plans; it must not reuse the search
shard scheme without its pair-coverage invariants.
+9 -3
View File
@@ -122,7 +122,10 @@ Content-Type: application/json
},
"metrics": {
"elapsed_seconds": 12.4,
"processed_rows": 10000
"scanned_rows": 10000,
"valid_molecules": 9876,
"invalid_smiles": 124,
"matches_emitted": 20
}
}
```
@@ -189,8 +192,11 @@ class Runner(Protocol):
"""Run one task and return output artifacts plus safe metrics."""
```
`SciMeshRunner` should map `workload` and validated parameters to the existing
SciMesh CLI. For example, a `similarity-search` task invokes:
`SciMeshRunner` maps an allowlisted workload and validated parameters to the
local SciMesh reference functions. A planned `similarity-search` task contains
a resolved `query_smiles` (never `query_id`) and writes one exact local top-k
partial CSV plus the metrics above. Legacy single-shard tasks may still use the
CLI compatibility path:
```text
scimesh similarity-search <local-input> --query-id ... --output <task-dir>/result.csv