Efremenko Arhip
f1c3163be4
Replaces the repository stubs with real pgx queries, so the queue now works end
to end: a job is split into tasks, leased to workers one at a time, heartbeated,
completed, and reflected in job progress.
Task claiming is a single statement — SELECT ... FOR UPDATE SKIP LOCKED feeding
an UPDATE — so concurrent coordinators lease different rows instead of blocking
on the same one. Writes use optimistic concurrency: the entity increments its
version in memory, and the UPDATE guards on the previous value.
Retries moved to the transaction level. Once Postgres aborts a transaction with
a serialization failure, replaying one statement inside it cannot help; the unit
of retry is Begin -> fn -> Commit, which is safe because each attempt re-reads
its rows through GetForUpdate.
Adds integration tests behind the `integration` build tag, run against a real
PostgreSQL through TEST_DATABASE_URL: concurrent claiming hands each task to
exactly one worker, job creation rolls back whole, stale writes are refused,
completed results keep chunk order, and expired leases return to the queue.
Two bugs the tests caught:
- a nil parameters map reached a NOT NULL jsonb column as SQL NULL, since pgx
sends NULL rather than omitting the column and letting DEFAULT '{}' apply;
- replaying an already-recorded result returned 409. The idempotent path leaves
the entity untouched, so the version guard matched nothing and a successful
no-op looked like a conflict. CompleteTask now skips the write when the
entity did not change.
SciMesh
SciMesh is a small local framework for scientific workloads on molecular datasets. It currently provides exact molecular similarity search and exact sparse similarity-graph construction. It runs in one local Python process: there is no network service, multiprocessing, coordinator, database, or dense similarity matrix.
The ChEMBL TSV database is intentionally not included in this repository. Download it separately and pass its path to the commands below. The expected columns are chembl_id and canonical_smiles.
Installation
SciMesh requires Python 3.10+ and RDKit.
python -m venv .venv
source .venv/bin/activate
pip install -e .
RDKit can alternatively be installed from conda-forge:
conda install -c conda-forge rdkit
pip install -e .
Quick start
Run the built-in help command for copy-paste examples of both workloads:
scimesh help
It includes environment setup, output-directory creation, similarity search by ChEMBL ID or SMILES, and similarity-graph construction. Use the standard help for the complete option reference:
scimesh similarity-search --help
scimesh similarity-graph --help
Similarity search
similarity-search finds the top-k molecules most similar to a query. The query is supplied either by ChEMBL ID or by SMILES. It uses Morgan fingerprints with radius=2 and fpSize=2048, Tanimoto similarity, streaming TSV reads, and a bounded heap. Invalid SMILES and the query molecule are skipped.
scimesh similarity-search chembl_37_chemreps.txt \
--query-id CHEMBL939 \
--top-k 20 \
--output results.csv
Use a SMILES query when it is not identified by ChEMBL ID:
scimesh similarity-search chembl_37_chemreps.txt \
--query-smiles 'COc1cc2ncnc(Nc3ccc(F)c(Cl)c3)c2cc1OCCCN1CCOCC1' \
--top-k 20 \
--output results.csv
The output CSV contains rank,chembl_id,canonical_smiles,similarity. Search progress and valid/invalid-SMILES statistics are written to the terminal. --max-rows limits the candidate scan for small tests, while --progress-every 0 disables progress reports.
To find the least similar molecules, use --threshold-direction less. This ranks
results from the lowest similarity upward; --threshold optionally limits them
to values less than or equal to a cutoff:
scimesh similarity-search chembl_37_chemreps.txt \
--query-id CHEMBL939 \
--threshold-direction less \
--threshold 0.1 \
--top-k 20 \
--output least_similar.csv
To render the query and retained candidates:
scimesh similarity-search chembl_37_chemreps.txt \
--query-id CHEMBL939 \
--images-dir structures
This writes query.png and top_candidates.png into structures.
Similarity graph
similarity-graph constructs an exact sparse undirected graph. Every valid molecule is a vertex; an edge is emitted when Tanimoto similarity satisfies the selected threshold direction (>= by default, or <= with --threshold-direction less). Each fingerprint is calculated once. Comparisons are processed block by block, each pair is tested once (i < j), and no dense N×N matrix is created or stored.
scimesh similarity-graph chembl_37_chemreps.txt \
--max-rows 10000 \
--threshold 0.7 \
--block-size 1000 \
--output similarity_graph.csv
The deterministic edge-list CSV has source_id,target_id,similarity columns. The command reports valid molecules, checked pairs, emitted edges, rate, and elapsed time. --block-size changes only how comparisons are grouped, not the result.
Development
pip install -e '.[dev]'
pytest
The package separates common dataset parsing and fingerprints from independent workloads. Add future workloads through the workload registry without changing the main CLI.