SciMesh/docs/web-interface-plan.md

SciMesh: plan for the initial web interface

1. Purpose and outcome

Build a small, local-first web interface for manually checking the complete SciMesh pipeline. A person should be able to open one address, upload a small ChEMBL-style TSV, configure a supported run, observe workers and task progress, inspect failures, and download artifacts without composing raw HTTP requests.

This is not a public multi-tenant product. It is an operator and demo interface for a trusted local team. The coordinator remains the only process with direct database and artifact-storage access; the browser never calls PostgreSQL and never receives a worker bearer token.

Current delivered scope

The initial operator UI and CTX-09 final reduction are now implemented. The control room polls a bounded, coordinator-owned read model every two seconds while a tab is visible. It shows the worker fleet, recent jobs, safe shard diagnostics, the actual reducing phase, and final-result availability. A job detail page renders the concrete pipeline stages—input accepted, shards, worker CSVs, reduction, final CSV—from coordinator state and replaces task and artifact views as work changes. All browser mutations remain limited to validated dataset upload and operator cancellation.

The interface must distinguish an in-progress distributed search from a run whose reducer has produced a durable final result:

Mode	What it proves	What it must not claim
In-progress run	Upload, task creation, claim, heartbeat, artifact upload, task completion, retries, and shard diagnostics work end-to-end.	That the partial CSVs are a global scientific answer.
Final run	A reducer has produced a durable final CSV for the full job.	Available for similarity-search after CTX-09; graph remains unavailable until CTX-10.

Never label a partial artifact as a final molecular result. The UI must show a clear waiting or reducing stage until a final artifact exists and the job is completed.

2. Constraints and decisions

• Serve the UI from the Go coordinator at the same origin. No React, Vue, Node build, CDN, or separate frontend service.
• Use Go html/template, embed, ordinary CSS, and small vanilla JavaScript modules. The page works locally with docker compose up.
• Keep worker APIs and UI APIs separate. The UI handlers call Go use cases; they do not make HTTP calls to worker endpoints.
• Add a distinct UI_AUTH_TOKEN for browser/operator access. It must never reuse WORKER_AUTH_TOKEN, appear in page HTML, localStorage, logs, URLs, or error messages. For the initial local UI use HTTP Basic Auth over a trusted local/reverse-proxied connection. If UI_AUTH_TOKEN is unset, /ui and /ui/api/* return 404 and the coordinator stays API-only.
• Keep all user-controllable text escaped by html/template; JavaScript renders API fields through textContent, never innerHTML.
• All downloads go through coordinator-owned UI endpoints with authorization. Do not expose filesystem paths, storage_key, database errors, worker tokens, or raw worker tracebacks.
• The API vocabulary should be canonicalised before UI forms are implemented. Choose one external workload spelling (similarity-search and similarity-graph recommended, matching the CLI) and keep legacy underscore aliases only at the worker boundary. Update docs/api-contract.md and docs/openapi.yaml in the same change.

3. Target user journey

1. Start coordinator/PostgreSQL and one or more scimesh-worker processes.
2. Open http://localhost:8080/ui, authenticate with the UI token, and see readiness plus the registered-worker table.
3. Choose New run, select a workload, fill validated parameters, choose a TSV, and submit it.
4. The UI redirects to /ui/jobs/{job_id} and polls every two seconds.
5. The operator sees counters, per-task attempt/lease/error state, and worker activity. They may copy the worker launch command but cannot start arbitrary processes from the browser.
6. During a pipeline check, download an input shard or partial CSV to validate manually. For a final run, download the final CSV only when the job status is completed and a final artifact exists.
7. A failed run exposes only its sanitized failure reason and retry state. The browser offers a safe retry action only after an explicit future API supports it; v1 never fabricates a retry by changing task rows directly.

4. What exists today and required gaps

Capability	Current state	UI plan
Upload/chunk dataset	POST /jobs/upload exists	Reuse through a UI handler with server-side multipart validation.
Aggregate status	GET /jobs/{id} exists	Add list/detail read models for the UI.
Worker registration/lease flow	Implemented	Add a read-only worker list; no browser worker controls.
Task diagnostics	No public list/detail response	Add sanitized job task list with attempt, status, lease owner, expiry and error.
Artifact download	Worker endpoint exists	Add UI-authorized, job-scoped download proxy.
Final result	CTX-09 final artifact and download route exist	Show the reducing stage, then make the final CSV prominent only for completed.
Distributed graph correctness	Planner/reducer unavailable	Do not advertise a multi-shard graph as final until CTX-10.

5. Proposed structure

coordinator/
  web/
    templates/
      layout.html
      dashboard.html
      job_new.html
      job_detail.html
      error.html
    static/
      app.css
      dashboard.js
      job-detail.js
  internal/
    transport/http/
      ui_handlers.go
      ui_dto.go
      ui_auth.go
      ui_handlers_test.go
    usecase/
      ui.go                 # read-only DTO orchestration, no HTML
    domain/
      ui.go                 # only if a shared value object is genuinely needed
    storage/postgres/
      ui_read_repo.go       # parameterized listing/detail queries

Embed web/templates and web/static into the coordinator binary with go:embed. No assets are generated at runtime; go test ./... must work without Node/npm.

6. UI surface and routes

HTML routes

Route	Purpose	Availability
GET /ui	Dashboard: readiness, recent jobs, workers, quick actions.	WUI-03
GET /ui/jobs/new	New-run form and parameter help.	WUI-04
GET /ui/jobs/{id}	Job detail and polling shell.	WUI-03
GET /ui/jobs/{id}/artifacts/{artifact_id}	Authorized download proxy with attachment headers.	WUI-05

JSON routes used only by the pages

Route	Response / action	Notes
GET /ui/api/overview	readiness, recent jobs, workers	No secrets, no storage paths.
GET /ui/api/jobs	cursor/page of compact job cards	Default 20, deterministic created_at DESC, id DESC.
POST /ui/api/jobs/upload	validates form, streams dataset, returns 201 {job_id}	Same input limits as /jobs/upload; form fields first, file last.
GET /ui/api/jobs/{id}	job detail, counters, tasks, allowed artifacts	Polling endpoint, no raw DB model.
GET /ui/api/jobs/{id}/events	deferred	Start with polling; no SSE/WebSocket in v1.

All /ui routes use UI authentication. Existing worker API routes keep worker authentication and are not relaxed for the browser.

7. Read models and data minimisation

Create UI-specific DTOs; do not return domain/database entities directly.

JobCard:
  id, workload, created_at, status,
  total, pending, leased, running, completed, failed

JobDetail:
  JobCard fields,
  parameters (allowlisted/redacted),
  tasks: [{id, chunk_index, status, attempt, max_attempts,
           lease_owner_display, lease_expires_at, error_code, error_message}],
  artifacts: [{id, kind, filename, size_bytes, sha256, downloadable}]

WorkerCard:
  id, name, capabilities, status, last_heartbeat_at, created_at

Rules:

• Display a shortened UUID by default but provide a copy button with the full value; never interpolate it into HTML.
• Do not expose storage_key, absolute artifact paths, raw metrics containing unexpected values, auth configuration, or worker-local directories.
• An artifact is downloadable only when it belongs to the requested job. A final_result is downloadable only after the job is completed; partial artifacts are marked diagnostic.
• SQL uses explicit columns, pagination/cursors, deterministic ordering, and joins constrained by job_id.

8. Workload forms and validation

8.1 Common fields

• TSV file, required, streamed; show expected columns chembl_id and canonical_smiles.
• chunk_rows: integer 1--100000, default 1000.
• max_rows: optional positive integer. The coordinator creates shards only from the first N data rows, so a user can test a large upload without creating thousands of tasks. It does not truncate the stored source blob.
• optional human-readable run name is a later schema/API addition; v1 does not silently store it.
• display file name and client-side size only as convenience; server limits and validation remain authoritative.

8.2 Similarity search

Inputs: exactly one query_smiles or query_id, top_k, optional threshold, threshold direction, max_rows, and progress_every.

The current upload form accepts query_smiles, because resolving a cross-shard query_id has not yet been connected to coordinator uploads. The detail page calls an artifact a partial top-k CSV until all shards are complete and CTX-09 reduction stores the final global result.

8.3 Similarity graph

Inputs: threshold, threshold direction, block size, max_rows, and progress interval. The form may show a disabled Experimental — not globally reduced card, but it must not submit multi-shard graph jobs until CTX-10 implements block-pair planning and deterministic reduction. A one-shard pipeline check is allowed only behind an explicit acknowledgement and produces a diagnostic edge list, not a final graph.

Validation exists in three places: HTML constraints for feedback, a small JavaScript schema for form behaviour, and authoritative Go validation mapped to typed workload parameters. Never pass arbitrary parameter maps straight from the browser to workers.

9. Delivery packages

Each package is a separate PR/task context. Do not start a later package until its listed dependency and tests are green.

WUI-00 — Freeze UI contract and demo scope

Depends on: current main.

Deliver: this plan, canonical workload naming decision, and updates to docs/api-contract.md/docs/openapi.yaml if names or statuses change.

Acceptance: API has a precise distinction between diagnostic artifacts and final results; UI security model has a distinct token; CTX-07/09/10 limitations are visible.

WUI-01 — UI read models and PostgreSQL queries

Depends on: WUI-00.

Deliver: repository/use-case methods for deterministic job lists, sanitized job details, task summaries, artifact metadata, and worker lists. Add indexes only if EXPLAIN ANALYZE on a realistic list query shows need.

Acceptance: no N+1 query path; no storage key/secret leaks; unknown job is 404; artifact lookup is constrained to its job; Go unit plus real-PostgreSQL integration tests cover ordering, empty lists and ownership boundaries.

WUI-02 — UI auth and embedded asset foundation

Depends on: WUI-01.

Deliver: UI_AUTH_TOKEN config validation, Basic Auth middleware, embedded template/static serving, security headers, and an API-only fallback when UI is disabled.

Acceptance: worker token never authorizes /ui; UI token never authorizes worker routes; /ui is 404 when disabled; HTML/content-security headers are tested; no token appears in logs or errors.

WUI-03 — Read-only dashboard and job detail

Depends on: WUI-01, WUI-02.

Deliver: dashboard, worker table, job list, detail page, two-second polling with pause when the tab is hidden, error/retry display, and accessible empty/ loading/error states.

Acceptance: a manually created job changes from pending to running on the page without reload; all text is escaped; polling stops on terminal states; task attempts and sanitized errors are visible; handler/template tests cover XSS-shaped names and errors.

WUI-04 — New-run upload form

Depends on: WUI-02, WUI-03.

Deliver: workload-specific form, typed Go validation, streamed upload, progress/submit state, redirect to job detail, and copyable worker launch instructions.

Acceptance: invalid query combinations fail with 400 and clear UI feedback; valid small similarity-search upload creates deterministic shard count; upload limits are enforced; browser never sees worker auth; test covers malformed TSV, large/invalid fields, duplicate form fields, and failed coordinator storage.

WUI-05 — Safe artifact downloads and diagnostic preview

Depends on: WUI-01--04.

Deliver: job-scoped download proxy, CSV preview limited by byte/row count, checksum/size metadata display, and prominent partial/final labels.

Acceptance: artifact from another job is 404; path traversal cannot select a file; Content-Disposition is safe; preview never loads an unbounded CSV; no final-result button exists before CTX-09.

WUI-06 — Final-result UX after CTX-09 — implemented

Depends on: CTX-09 and WUI-05.

Deliver: reducing state, final artifact card, final CSV preview/download, and deterministic result metadata.

Acceptance: final link is shown only for completed; reducer failure is sanitized; page refresh/restart preserves final artifact; end-to-end test compares downloaded final search CSV with local reference output.

WUI-07 — Full similarity-graph UX after CTX-10

Depends on: CTX-10 and WUI-06.

Deliver: enabled graph form, block-pair planning summary, graph-specific progress, final edge-list preview/download, and warnings for low thresholds.

Acceptance: graph result equals local brute force on a small fixture; no dense matrix is introduced; threshold direction is displayed and preserved; result order is deterministic.

WUI-08 — Manual-demo script and CI browser checks

Depends on: WUI-05; extend after WUI-06/07.

Deliver: make demo-ui (or documented compose profile), a tiny tracked TSV fixture, start/stop instructions, and headless browser smoke tests.

Acceptance: clean checkout can start coordinator, a worker, open UI, submit fixture, observe task completion, and download a diagnostic artifact. CI covers auth, upload, polling JSON, job isolation, and download permission.

10. Testing strategy

Layer	Required checks
Go domain/use case	status projection, artifact/job ownership, pagination ordering, redaction.
Go HTTP	UI auth separation, malformed multipart, CSRF-safe same-origin policy, 404/401, headers, escaping.
PostgreSQL	fresh migrations, list/detail query ordering, cross-job artifact denial, completed/final state.
Browser	form validation, upload success/failure, polling transition, terminal state, safe text rendering.
End-to-end	real PostgreSQL + coordinator + Python worker + small fixture; verify actual bytes/checksum.

Use Playwright only if it can run in CI without adding a production runtime dependency. Otherwise start with Go handler tests and a small curl/HTML smoke script, then add browser automation in WUI-08.

11. Manual verification script after WUI-05

# terminal 1
cd coordinator
UI_AUTH_TOKEN='local-ui-secret' make up

# terminal 2: use a separate worker token; do not paste it into the browser
SCIMESH_COORDINATOR_URL=http://localhost:8080 \
SCIMESH_BEARER_TOKEN='worker-secret' \
scimesh-worker --work-dir ./worker-data

# browser
# http://localhost:8080/ui
# authenticate with the UI token, upload a tiny TSV, then watch /ui/jobs/{id}

The actual configuration variable names, compose wiring, and launch command are implemented in WUI-02/WUI-08; this block is the acceptance target, not a claim that the interface exists today.

12. Explicit non-goals for the initial interface

• no database browser or SQL console;
• no browser-side RDKit or scientific calculation;
• no worker start/stop shell execution from UI;
• no multi-user accounts, RBAC, password reset, public internet exposure, or user-provided storage credentials;
• no WebSocket/SSE, React/Vue, Docker requirement for the Python package, or deployment platform;
• no claim that a distributed graph or search result is final before its planner/reducer acceptance criteria are met.

13. Definition of done for the first hand-testable release

The hand-testable release is complete when a clean local checkout can run a trusted, authenticated local UI; display workers, jobs, pipeline stages, tasks and safe errors; submit a valid small search; poll it through reducing; and download the coordinator-owned final CSV only after completion. The page must make the distinction between partial diagnostics and the final result impossible to miss.