From 746958884ee0809236f28f1dc8750e613cb7fbbe Mon Sep 17 00:00:00 2001 From: Emil Date: Fri, 24 Jul 2026 17:37:46 +0300 Subject: [PATCH] Document local pipeline demo --- Makefile | 14 +++++++++++++- README.md | 21 +++++++++++++++++++++ coordinator/Makefile | 15 ++++++++++++++- coordinator/README.md | 7 +++++++ scimesh/workloads/help.py | 16 ++++++++++++++++ tests/test_cli_help.py | 1 + 6 files changed, 72 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 982f53f..8a6fc43 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,16 @@ -.PHONY: demo-ui demo-down demo-logs +.DEFAULT_GOAL := help + +.PHONY: help demo-ui demo-down demo-logs + +help: + @printf '%s\n' \ + 'SciMesh developer commands:' \ + ' make demo-ui Start the local UI pipeline demo with 2 workers.' \ + ' make demo-ui WORKERS=3 Start the demo with 3 local workers.' \ + ' make demo-logs Follow coordinator logs for the demo.' \ + ' make demo-down Stop demo containers and workers.' \ + '' \ + 'After make demo-ui: open http://localhost:18080/ui (operator / demo-ui-secret).' # Convenient entry points from the repository root. Extra settings are passed # through, for example: make demo-ui WORKERS=3 diff --git a/README.md b/README.md index 947e27e..0342055 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,27 @@ scimesh similarity-search --help scimesh similarity-graph --help ``` +## Manual pipeline demo + +To inspect the coordinator, Web UI, and distributed `similarity-search` +pipeline by hand, install development dependencies once and start the isolated +demo from the repository root: + +```bash +python3 -m venv .venv +.venv/bin/pip install -e '.[dev]' +make demo-ui +``` + +Open `http://localhost:18080/ui` and sign in with username `operator` and +password `demo-ui-secret`. The command starts PostgreSQL, the coordinator, and +two local reference workers. Upload a small ChEMBL TSV, then use the job page +to follow shard progress, inspect bounded **Preview CSV** results, and see a +live processing-speed chart in shards per minute. To change the worker count, +run `make demo-ui WORKERS=3`; stop everything with `make demo-down`. + +Run `make help` to display these commands in the terminal. + ## Similarity search `similarity-search` finds the top-k molecules most similar to a query. The query is supplied either by ChEMBL ID or by SMILES. It uses Morgan fingerprints with `radius=2` and `fpSize=2048`, Tanimoto similarity, streaming TSV reads, and a bounded heap. Invalid SMILES and the query molecule are skipped. diff --git a/coordinator/Makefile b/coordinator/Makefile index 4e339f4..6f17ed6 100644 --- a/coordinator/Makefile +++ b/coordinator/Makefile @@ -1,4 +1,6 @@ -.PHONY: build run test test-integration vet lint tidy check migrate-up migrate-down up down down-clean logs ps rebuild psql smoke demo-ui demo-down demo-logs +.DEFAULT_GOAL := help + +.PHONY: help build run test test-integration vet lint tidy check migrate-up migrate-down up down down-clean logs ps rebuild psql smoke demo-ui demo-down demo-logs # `check` deliberately uses its own Compose project and host ports. This keeps # it from connecting to or replacing a developer's local PostgreSQL instance. @@ -24,6 +26,17 @@ DEMO_WORKERS ?= 2 WORKERS ?= $(DEMO_WORKERS) DEMO_DIR ?= .demo +help: + @printf '%s\n' \ + 'SciMesh coordinator commands:' \ + ' make up / make down Start or stop the normal coordinator stack.' \ + ' make demo-ui [WORKERS=3] Start isolated UI demo services and local workers.' \ + ' make demo-logs Follow coordinator logs for the UI demo.' \ + ' make demo-down Stop the demo services and workers.' \ + ' make test / make vet Run Go verification.' \ + '' \ + 'Demo UI: http://localhost:18080/ui (operator / demo-ui-secret).' + demo-ui: @DEMO_PROJECT="$(DEMO_PROJECT)" \ DEMO_POSTGRES_PORT="$(DEMO_POSTGRES_PORT)" \ diff --git a/coordinator/README.md b/coordinator/README.md index 83f9d95..7240b99 100644 --- a/coordinator/README.md +++ b/coordinator/README.md @@ -109,6 +109,13 @@ This uses a separate Docker project and ports `18080` (coordinator) and it. Change the worker count with `make demo-ui WORKERS=3`; stop all demo services and workers with `make demo-down`. +The job page shows a live **Processing speed** graph in completed shards per +minute. It uses the coordinator snapshots observed by the open browser tab, so +it is a transparent local-session measurement rather than a persisted metric. +Use **Preview CSV** before downloading a partial diagnostic or completed final +result. Run `make help` from either the repository root or this directory for +the full list of demo commands. + `up` starts three services in order: Postgres waits until `pg_isready` passes, a one-shot `migrate` container applies the schema and exits, and only then does the coordinator start — so it never queries a database that has no tables. diff --git a/scimesh/workloads/help.py b/scimesh/workloads/help.py index d361fad..71e1872 100644 --- a/scimesh/workloads/help.py +++ b/scimesh/workloads/help.py @@ -52,6 +52,22 @@ HELP_TEXT = dedent( --block-size 250 \\ --output results/similarity_graph.csv + 6. Start a local distributed pipeline demo with a coordinator, PostgreSQL, + and two reference workers. Run this from the repository root after + installing the project with its development dependencies: + + make demo-ui + + # Use more workers when you want to compare throughput: + make demo-ui WORKERS=3 + + # Open http://localhost:18080/ui + # Username: operator Password: demo-ui-secret + + The job page includes a live completed-shards-per-minute chart. It is + measured from coordinator snapshots observed by the open browser tab. + Stop the local demo with: make demo-down + Use --max-rows for quick local tests; omit it to process the full dataset. For all options, run: diff --git a/tests/test_cli_help.py b/tests/test_cli_help.py index ee193d2..9a78385 100644 --- a/tests/test_cli_help.py +++ b/tests/test_cli_help.py @@ -9,3 +9,4 @@ def test_help_command_prints_runnable_examples(capsys: object) -> None: assert "scimesh similarity-search" in output assert "scimesh similarity-graph" in output assert "mkdir -p results" in output + assert "make demo-ui" in output