SciMesh
SciMesh is a scientific-workload framework for molecular datasets. Its public CLI
runs exact similarity search and sparse similarity-graph construction locally in
one Python process; it creates no dense similarity matrix. The Go/PostgreSQL
coordinator and Python worker can run a shard-based similarity-search
pipeline locally. After every shard succeeds, the coordinator deterministically
merges its candidates into one final global top-k CSV. See
STATUS.md.
The ChEMBL TSV database is intentionally not included in this repository. Download it separately and pass its path to the commands below. The expected columns are chembl_id and canonical_smiles.
Installation
SciMesh requires Python 3.10+ and RDKit.
python -m venv .venv
source .venv/bin/activate
pip install -e .
RDKit can alternatively be installed from conda-forge:
conda install -c conda-forge rdkit
pip install -e .
Quick start
Run the built-in help command for copy-paste examples of both workloads:
scimesh help
It includes environment setup, output-directory creation, similarity search by ChEMBL ID or SMILES, and similarity-graph construction. Use the standard help for the complete option reference:
scimesh similarity-search --help
scimesh similarity-graph --help
Manual pipeline demo
To inspect the coordinator, Web UI, and distributed similarity-search
pipeline by hand, install development dependencies once and start the isolated
demo from the repository root:
python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
make demo-ui
Open http://localhost:18080/ui and sign in with username operator and
password demo-ui-secret. The command starts PostgreSQL, the coordinator, and
two local reference workers. Upload a small ChEMBL TSV, then use the job page
to follow shard progress, inspect bounded Preview CSV results, and see a
live processing-speed chart in shards per minute. The Workloads page shows
the installed SDK workload library (descriptions, parameters, and artifact
schemas) from the embedded catalog; regenerate it with
make workloads-export (or scimesh workload export) whenever workloads
change. To change the worker count, run make demo-ui WORKERS=3; stop
everything with make demo-down.
Run make help to display these commands in the terminal.
Similarity search
similarity-search finds the top-k molecules most similar to a query. The query is supplied either by ChEMBL ID or by SMILES. It uses Morgan fingerprints with radius=2 and fpSize=2048, Tanimoto similarity, streaming TSV reads, and a bounded heap. Invalid SMILES and the query molecule are skipped.
scimesh similarity-search chembl_37_chemreps.txt \
--query-id CHEMBL939 \
--top-k 20 \
--output results.csv
Use a SMILES query when it is not identified by ChEMBL ID:
scimesh similarity-search chembl_37_chemreps.txt \
--query-smiles 'COc1cc2ncnc(Nc3ccc(F)c(Cl)c3)c2cc1OCCCN1CCOCC1' \
--top-k 20 \
--output results.csv
The output CSV contains rank,chembl_id,canonical_smiles,similarity. Search progress and valid/invalid-SMILES statistics are written to the terminal. --max-rows limits the candidate scan for small tests, while --progress-every 0 disables progress reports.
To find the least similar molecules, use --threshold-direction less. This ranks
results from the lowest similarity upward; --threshold optionally limits them
to values less than or equal to a cutoff:
scimesh similarity-search chembl_37_chemreps.txt \
--query-id CHEMBL939 \
--threshold-direction less \
--threshold 0.1 \
--top-k 20 \
--output least_similar.csv
To render the query and retained candidates:
scimesh similarity-search chembl_37_chemreps.txt \
--query-id CHEMBL939 \
--images-dir structures
This writes query.png and top_candidates.png into structures.
Similarity graph
similarity-graph constructs an exact sparse undirected graph. Every valid molecule is a vertex; an edge is emitted when Tanimoto similarity satisfies the selected threshold direction (>= by default, or <= with --threshold-direction less). Each fingerprint is calculated once. Comparisons are processed block by block, each pair is tested once (i < j), and no dense N×N matrix is created or stored.
scimesh similarity-graph chembl_37_chemreps.txt \
--max-rows 10000 \
--threshold 0.7 \
--block-size 1000 \
--output similarity_graph.csv
The deterministic edge-list CSV has source_id,target_id,similarity columns. The command reports valid molecules, checked pairs, emitted edges, rate, and elapsed time. --block-size changes only how comparisons are grouped, not the result.
Development
pip install -e '.[dev]'
pytest
The package separates common dataset parsing and fingerprints from independent workloads. Add future workloads through the workload registry without changing the main CLI.
Workload SDK
scimesh.sdk is the framework only: strict and immutable workload manifests,
typed artifact ports, static map/reduce plans, resource eligibility and local
reservations, exact/canonical/numeric verifier primitives, installed-package
allowlisting, and a local conformance executor. It contains no scientific
workload code. Workloads are user scripts built on the SDK: the built-in
similarity-search, similarity-graph, and descriptor-batch live in
scimesh/workloads/ (each a small package with core.py + definition.py),
composed by scimesh/workloads/library.py and registered through
scimesh.workloads entry points. The Worker Agent executes those SDK-built
workloads directly (see scimesh/worker/runners.py), so the same scientific
handlers run locally, in conformance, and on claimed coordinator tasks.
scimesh workload list and scimesh workload run run any SDK workload from
the command line. See the
SDK author guide, contract,
and delivery roadmap.
Dynamic workflows, real Worker concurrency, coordinator-backed GPU allocation, streaming, and gang execution remain fail-closed until their versioned runtime features are implemented; declaring those profiles does not silently enable them.
The included LocalCoreBatchExecutor is a trusted, single-threaded in-process
conformance harness. It validates scientific parity, sealed outputs, provenance,
and limits, but intentionally refuses profiles that claim network/process
isolation, secrets, accelerators, gangs, checkpoints, or retries; those require
the future enforcing Agent runtime.