SciMesh/docs/web-interface-plan.md

# SciMesh: plan for the initial web interface

## 1. Purpose and outcome

Build a small, local-first web interface for manually checking the complete
SciMesh pipeline. A person should be able to open one address, upload a small
ChEMBL-style TSV, configure a supported run, observe workers and task progress,
inspect failures, and download artifacts without composing raw HTTP requests.

This is not a public multi-tenant product. It is an operator and demo interface
for a trusted local team. The coordinator remains the only process with direct
database and artifact-storage access; the browser never calls PostgreSQL and
never receives a worker bearer token.

The first release must be useful before CTX-07--CTX-10 are complete. Therefore
it has two visibly different modes:

| Mode | What it proves | What it must not claim |
| --- | --- | --- |
| **Pipeline check** | Upload, task creation, claim, heartbeat, artifact upload, task completion, retries, and downloads work end-to-end. | That multiple shard results have been scientifically reduced into one answer. |
| **Final run** | A reducer has produced a durable final CSV for the full job. | Available only after CTX-09, and for graph only after CTX-10. |

Never label a partial artifact as a final molecular result. The UI must show a
clear `Pipeline check — partial results` badge while a reducer is unavailable.

## 2. Constraints and decisions

- Serve the UI from the Go coordinator at the same origin. No React, Vue, Node
  build, CDN, or separate frontend service.
- Use Go `html/template`, `embed`, ordinary CSS, and small vanilla JavaScript
  modules. The page works locally with `docker compose up`.
- Keep worker APIs and UI APIs separate. The UI handlers call Go use cases;
  they do not make HTTP calls to worker endpoints.
- Add a distinct `UI_AUTH_TOKEN` for browser/operator access. It must never
  reuse `WORKER_AUTH_TOKEN`, appear in page HTML, localStorage, logs, URLs, or
  error messages. For the initial local UI use HTTP Basic Auth over a trusted
  local/reverse-proxied connection. If `UI_AUTH_TOKEN` is unset, `/ui` and
  `/ui/api/*` return `404` and the coordinator stays API-only.
- Keep all user-controllable text escaped by `html/template`; JavaScript renders
  API fields through `textContent`, never `innerHTML`.
- All downloads go through coordinator-owned UI endpoints with authorization.
  Do not expose filesystem paths, `storage_key`, database errors, worker tokens,
  or raw worker tracebacks.
- The API vocabulary should be canonicalised before UI forms are implemented.
  Choose one external workload spelling (`similarity-search` and
  `similarity-graph` recommended, matching the CLI) and keep legacy underscore
  aliases only at the worker boundary. Update `docs/api-contract.md` and
  `docs/openapi.yaml` in the same change.

## 3. Target user journey

1. Start coordinator/PostgreSQL and one or more `scimesh-worker` processes.
2. Open `http://localhost:8080/ui`, authenticate with the UI token, and see
   readiness plus the registered-worker table.
3. Choose **New run**, select a workload, fill validated parameters, choose a
   TSV, and submit it.
4. The UI redirects to `/ui/jobs/{job_id}` and polls every two seconds.
5. The operator sees counters, per-task attempt/lease/error state, and worker
   activity. They may copy the worker launch command but cannot start arbitrary
   processes from the browser.
6. During a pipeline check, download an input shard or partial CSV to validate
   manually. For a final run, download the final CSV only when the job status is
   `completed` and a final artifact exists.
7. A failed run exposes only its sanitized failure reason and retry state. The
   browser offers a safe retry action only after an explicit future API supports
   it; v1 never fabricates a retry by changing task rows directly.

## 4. What exists today and required gaps

| Capability | Current state | UI plan |
| --- | --- | --- |
| Upload/chunk dataset | `POST /jobs/upload` exists | Reuse through a UI handler with server-side multipart validation. |
| Aggregate status | `GET /jobs/{id}` exists | Add list/detail read models for the UI. |
| Worker registration/lease flow | Implemented | Add a read-only worker list; no browser worker controls. |
| Task diagnostics | No public list/detail response | Add sanitized job task list with attempt, status, lease owner, expiry and error. |
| Artifact download | Worker endpoint exists | Add UI-authorized, job-scoped download proxy. |
| Final result | Reducer is not implemented | Gate behind CTX-09; show partial diagnostic artifacts meanwhile. |
| Distributed graph correctness | Planner/reducer unavailable | Do not advertise a multi-shard graph as final until CTX-10. |

## 5. Proposed structure

```text
coordinator/
  web/
    templates/
      layout.html
      dashboard.html
      job_new.html
      job_detail.html
      error.html
    static/
      app.css
      dashboard.js
      job-detail.js
  internal/
    transport/http/
      ui_handlers.go
      ui_dto.go
      ui_auth.go
      ui_handlers_test.go
    usecase/
      ui.go                 # read-only DTO orchestration, no HTML
    domain/
      ui.go                 # only if a shared value object is genuinely needed
    storage/postgres/
      ui_read_repo.go       # parameterized listing/detail queries
```

Embed `web/templates` and `web/static` into the coordinator binary with
`go:embed`. No assets are generated at runtime; `go test ./...` must work
without Node/npm.

## 6. UI surface and routes

### HTML routes

| Route | Purpose | Availability |
| --- | --- | --- |
| `GET /ui` | Dashboard: readiness, recent jobs, workers, quick actions. | WUI-03 |
| `GET /ui/jobs/new` | New-run form and parameter help. | WUI-04 |
| `GET /ui/jobs/{id}` | Job detail and polling shell. | WUI-03 |
| `GET /ui/jobs/{id}/artifacts/{artifact_id}` | Authorized download proxy with attachment headers. | WUI-05 |

### JSON routes used only by the pages

| Route | Response / action | Notes |
| --- | --- | --- |
| `GET /ui/api/overview` | readiness, recent jobs, workers | No secrets, no storage paths. |
| `GET /ui/api/jobs` | cursor/page of compact job cards | Default 20, deterministic `created_at DESC, id DESC`. |
| `POST /ui/api/jobs/upload` | validates form, streams dataset, returns `201 {job_id}` | Same input limits as `/jobs/upload`; form fields first, file last. |
| `GET /ui/api/jobs/{id}` | job detail, counters, tasks, allowed artifacts | Polling endpoint, no raw DB model. |
| `GET /ui/api/jobs/{id}/events` | **deferred** | Start with polling; no SSE/WebSocket in v1. |

All `/ui` routes use UI authentication. Existing worker API routes keep worker
authentication and are not relaxed for the browser.

## 7. Read models and data minimisation

Create UI-specific DTOs; do not return domain/database entities directly.

```text
JobCard:
  id, workload, created_at, status,
  total, pending, leased, running, completed, failed

JobDetail:
  JobCard fields,
  parameters (allowlisted/redacted),
  tasks: [{id, chunk_index, status, attempt, max_attempts,
           lease_owner_display, lease_expires_at, error_code, error_message}],
  artifacts: [{id, kind, filename, size_bytes, sha256, downloadable}]

WorkerCard:
  id, name, capabilities, status, last_heartbeat_at, created_at
```

Rules:

- Display a shortened UUID by default but provide a copy button with the full
  value; never interpolate it into HTML.
- Do not expose `storage_key`, absolute artifact paths, raw metrics containing
  unexpected values, auth configuration, or worker-local directories.
- An artifact is downloadable only when it belongs to the requested job. A
  `final_result` is downloadable only after the job is `completed`; partial
  artifacts are marked diagnostic.
- SQL uses explicit columns, pagination/cursors, deterministic ordering, and
  joins constrained by `job_id`.

## 8. Workload forms and validation

### 8.1 Common fields

- TSV file, required, streamed; show expected columns
  `chembl_id` and `canonical_smiles`.
- `chunk_rows`: integer 1--100000, default 1000.
- `max_rows`: optional positive integer. The coordinator creates shards only
  from the first N data rows, so a user can test a large upload without
  creating thousands of tasks. It does not truncate the stored source blob.
- optional human-readable run name is a later schema/API addition; v1 does not
  silently store it.
- display file name and client-side size only as convenience; server limits and
  validation remain authoritative.

### 8.2 Similarity search

Inputs: exactly one `query_smiles` or `query_id`, `top_k`, optional threshold,
threshold direction, `max_rows`, and `progress_every`.

For a runnable manual pipeline check before CTX-08, offer `query_smiles` and
default `chunk_rows` large enough to create one shard. A `query_id` across
multiple shards is disabled with an explanation until CTX-07 resolves it once
before fan-out. The detail page calls an artifact a **partial top-k CSV**, not
a global top-k, until CTX-09 reduction exists.

### 8.3 Similarity graph

Inputs: threshold, threshold direction, block size, `max_rows`, and progress
interval. The form may show a disabled **Experimental — not globally reduced**
card, but it must not submit multi-shard graph jobs until CTX-10 implements
block-pair planning and deterministic reduction. A one-shard pipeline check is
allowed only behind an explicit acknowledgement and produces a diagnostic edge
list, not a final graph.

Validation exists in three places: HTML constraints for feedback, a small
JavaScript schema for form behaviour, and authoritative Go validation mapped to
typed workload parameters. Never pass arbitrary parameter maps straight from
the browser to workers.

## 9. Delivery packages

Each package is a separate PR/task context. Do not start a later package until
its listed dependency and tests are green.

### WUI-00 — Freeze UI contract and demo scope

**Depends on:** current `main`.

**Deliver:** this plan, canonical workload naming decision, and updates to
`docs/api-contract.md`/`docs/openapi.yaml` if names or statuses change.

**Acceptance:** API has a precise distinction between diagnostic artifacts and
final results; UI security model has a distinct token; CTX-07/09/10 limitations
are visible.

### WUI-01 — UI read models and PostgreSQL queries

**Depends on:** WUI-00.

**Deliver:** repository/use-case methods for deterministic job lists, sanitized
job details, task summaries, artifact metadata, and worker lists. Add indexes
only if `EXPLAIN ANALYZE` on a realistic list query shows need.

**Acceptance:** no N+1 query path; no storage key/secret leaks; unknown job is
404; artifact lookup is constrained to its job; Go unit plus real-PostgreSQL
integration tests cover ordering, empty lists and ownership boundaries.

### WUI-02 — UI auth and embedded asset foundation

**Depends on:** WUI-01.

**Deliver:** `UI_AUTH_TOKEN` config validation, Basic Auth middleware, embedded
template/static serving, security headers, and an API-only fallback when UI is
disabled.

**Acceptance:** worker token never authorizes `/ui`; UI token never authorizes
worker routes; `/ui` is 404 when disabled; HTML/content-security headers are
tested; no token appears in logs or errors.

### WUI-03 — Read-only dashboard and job detail

**Depends on:** WUI-01, WUI-02.

**Deliver:** dashboard, worker table, job list, detail page, two-second polling
with pause when the tab is hidden, error/retry display, and accessible empty/
loading/error states.

**Acceptance:** a manually created job changes from pending to running on the
page without reload; all text is escaped; polling stops on terminal states;
task attempts and sanitized errors are visible; handler/template tests cover
XSS-shaped names and errors.

### WUI-04 — New-run upload form

**Depends on:** WUI-02, WUI-03.

**Deliver:** workload-specific form, typed Go validation, streamed upload,
progress/submit state, redirect to job detail, and copyable worker launch
instructions.

**Acceptance:** invalid query combinations fail with 400 and clear UI feedback;
valid small similarity-search upload creates deterministic shard count; upload
limits are enforced; browser never sees worker auth; test covers malformed TSV,
large/invalid fields, duplicate form fields, and failed coordinator storage.

### WUI-05 — Safe artifact downloads and diagnostic preview

**Depends on:** WUI-01--04.

**Deliver:** job-scoped download proxy, CSV preview limited by byte/row count,
checksum/size metadata display, and prominent partial/final labels.

**Acceptance:** artifact from another job is 404; path traversal cannot select a
file; `Content-Disposition` is safe; preview never loads an unbounded CSV; no
final-result button exists before CTX-09.

### WUI-06 — Final-result UX after CTX-09

**Depends on:** CTX-09 and WUI-05.

**Deliver:** `reducing` state, final artifact card, final CSV preview/download,
and deterministic result metadata.

**Acceptance:** final link is shown only for `completed`; reducer failure is
sanitized; page refresh/restart preserves final artifact; end-to-end test
compares downloaded final search CSV with local reference output.

### WUI-07 — Full similarity-graph UX after CTX-10

**Depends on:** CTX-10 and WUI-06.

**Deliver:** enabled graph form, block-pair planning summary, graph-specific
progress, final edge-list preview/download, and warnings for low thresholds.

**Acceptance:** graph result equals local brute force on a small fixture; no
dense matrix is introduced; threshold direction is displayed and preserved;
result order is deterministic.

### WUI-08 — Manual-demo script and CI browser checks

**Depends on:** WUI-05; extend after WUI-06/07.

**Deliver:** `make demo-ui` (or documented compose profile), a tiny tracked TSV
fixture, start/stop instructions, and headless browser smoke tests.

**Acceptance:** clean checkout can start coordinator, a worker, open UI,
submit fixture, observe task completion, and download a diagnostic artifact.
CI covers auth, upload, polling JSON, job isolation, and download permission.

## 10. Testing strategy

| Layer | Required checks |
| --- | --- |
| Go domain/use case | status projection, artifact/job ownership, pagination ordering, redaction. |
| Go HTTP | UI auth separation, malformed multipart, CSRF-safe same-origin policy, 404/401, headers, escaping. |
| PostgreSQL | fresh migrations, list/detail query ordering, cross-job artifact denial, completed/final state. |
| Browser | form validation, upload success/failure, polling transition, terminal state, safe text rendering. |
| End-to-end | real PostgreSQL + coordinator + Python worker + small fixture; verify actual bytes/checksum. |

Use Playwright only if it can run in CI without adding a production runtime
dependency. Otherwise start with Go handler tests and a small `curl`/HTML
smoke script, then add browser automation in WUI-08.

## 11. Manual verification script after WUI-05

```sh
# terminal 1
cd coordinator
UI_AUTH_TOKEN='local-ui-secret' make up

# terminal 2: use a separate worker token; do not paste it into the browser
SCIMESH_COORDINATOR_URL=http://localhost:8080 \
SCIMESH_BEARER_TOKEN='worker-secret' \
scimesh-worker --work-dir ./worker-data

# browser
# http://localhost:8080/ui
# authenticate with the UI token, upload a tiny TSV, then watch /ui/jobs/{id}
```

The actual configuration variable names, compose wiring, and launch command are
implemented in WUI-02/WUI-08; this block is the acceptance target, not a claim
that the interface exists today.

## 12. Explicit non-goals for the initial interface

- no database browser or SQL console;
- no browser-side RDKit or scientific calculation;
- no worker start/stop shell execution from UI;
- no multi-user accounts, RBAC, password reset, public internet exposure, or
  user-provided storage credentials;
- no WebSocket/SSE, React/Vue, Docker requirement for the Python package, or
  deployment platform;
- no claim that a distributed graph or search result is final before its
  planner/reducer acceptance criteria are met.

## 13. Definition of done for the first hand-testable release

WUI-00 through WUI-05 are complete when a clean local checkout can run a
trusted, authenticated local UI; display coordinator readiness, workers, jobs,
tasks and safe errors; submit a valid small search pipeline check; poll it to a
terminal task state; and download/preview the coordinator-owned partial CSV.
The page must make the absence of final reduction impossible to miss.