Commit Graph
78 Commits
Author SHA1 Message Date
Efremenko Arhip 163cbe14bf fix(coordinator): bind JWT caller to worker at claim (close quarantine bypass)
The trust tier was read off the caller-supplied worker_id, so a JWT user who
knew any trusted worker's id could claim as it — draining and poisoning the
trusted queue and bypassing the untrusted-worker quarantine entirely.

Claim now requires a JWT caller to own the worker it acts as; a shared-token
caller (lab operator) may still act as any worker. Claim is the sole grantor of
a lease, so this also protects the downstream heartbeat/result/failure paths.

Tests: reject claim as another user's worker; allow claim as own worker.
2026-07-26 19:26:30 +03:00
Efremenko Arhip 80ff72a0fe feat(coordinator): worker trust tiers (C1) — enroll volunteers, quarantine untrusted
- migration 0012: workers.owner_id + trust_level (trusted/untrusted)
- verifier/authctx read the JWT verified claim; IsTrusted() = admin||verified
- /workers/register resolves trust from auth: service token or verified/admin
  JWT -> trusted; plain user JWT -> untrusted, tagged with owner_id
- claim quarantines untrusted workers (no tasks) until quorum (C2) lands
- unit tests for trust resolution, quarantine, and the verified claim

Additive and backward compatible: shared-token workers stay trusted, so the
existing worker flow and team tests are unchanged. Quorum verification (C2)
is deferred.
2026-07-26 19:18:50 +03:00
Efremenko Arhip c6a66747eb feat(users): add admin-granted verified badge for trusted contributors
- migration 0002: users.verified boolean, default false
- verified rides in the JWT (role + verified claims)
- POST /users/{id}/verify + /unverify, admin-only (403 otherwise)
- Issue now takes the whole user so trust claims travel in the token
- unit + integration + admin-flow tests
2026-07-26 19:10:01 +03:00
Efremenko Arhip 0c1f5f06d4 chore(users): remove coordinator leftovers, tidy for userservice
- delete api/requests.http and ARCHITECTURE.md (coordinator content)
- rewrite README.md and .env.example for the userservice
- drop dead RunPeriodic (reaper machinery userservice has no use for)
- fix .gitignore/.dockerignore/.golangci.yml module + artifact names
- degeneralize stale copied comments that said "coordinator"
2026-07-26 16:36:17 +03:00
Efremenko Arhip 67407220c3 ci(users): add pipeline (vet, gofmt, race, lint, migrate, integration)
Mirrors the coordinator workflow against a scimesh_users Postgres service.
Switches the test request helper to http.NewRequestWithContext so the noctx
linter passes on the go1.22 module.
2026-07-26 16:27:24 +03:00
Efremenko Arhip 1b1b971378 fix(coordinator): scan owner_id in UIReadRepo.ListJobs
jobColumns gained owner_id but ListJobs' Scan still read 12 targets,
so the query returned 13 columns and pgx failed at runtime. Only the
integration tests (real DB) caught it; memstore-backed unit tests did not.
2026-07-26 16:24:34 +03:00
Efremenko Arhip 73196579e8 test(users): cover config, usecase error paths, and /me 500 path 2026-07-26 16:15:36 +03:00
Efremenko Arhip a3db1a1e67 add users logic 2026-07-26 14:38:24 +03:00
Emil 16db1e41f7 Link team profiles in README 2026-07-25 15:02:28 +03:00
Emil ad9cc8f95c Add team section to README 2026-07-25 14:54:50 +03:00
Emil 746958884e Document local pipeline demo 2026-07-24 17:37:46 +03:00
Emil 7ad28b939d Show live processing speed chart 2026-07-24 17:28:16 +03:00
Emil 1012f5d95a Allow demo worker count override 2026-07-24 17:21:45 +03:00
Emil f5baec507c Add final result preview button 2026-07-24 17:19:31 +03:00
Emil 5a1414bee9 Add local pipeline demo launcher 2026-07-24 17:13:26 +03:00
Emil 8b738efd5d Add safe artifact previews 2026-07-24 17:04:07 +03:00
Emil d0aeb7fc95 Upgrade pipeline observability UI
coordinator / test (push) Canceled after 0s
2026-07-24 15:13:59 +03:00
Emil a055473706 Add final result reduction
coordinator / test (push) Canceled after 0s
2026-07-24 14:58:18 +03:00
Emil 6e67daa9eb Merge distributed similarity search 2026-07-24 14:43:57 +03:00
Emil 0f3a2d92d8 Add distributed similarity search
python / test (push) Canceled after 0s
2026-07-24 14:38:11 +03:00
Emil 0bef7604fd Merge distributed workload protocol 2026-07-24 14:26:58 +03:00
Emil 6ef92908a1 Add distributed workload protocol
python / test (push) Canceled after 0s
2026-07-24 14:25:38 +03:00
Emil f953112cfd Merge distributed pipeline hardening 2026-07-24 14:16:42 +03:00
Emil 19cbf7f113 Harden distributed pipeline 2026-07-24 14:16:42 +03:00
Emil 9ec8f50313 Fix worker interruption handling 2026-07-24 13:19:40 +03:00
Emil 08f5478a66 Define distributed workload protocol 2026-07-24 13:09:25 +03:00
Emil bde6cdb4ba Improve worker lifecycle controls 2026-07-24 13:08:10 +03:00
Emil 43ceec1f77 Add two-worker smoke test 2026-07-24 13:05:34 +03:00
Emil f5b16b057f Add UI development task briefs 2026-07-24 00:13:55 +03:00
Emil f8de0b2b9d Merge feat/web-interface 2026-07-23 23:43:57 +03:00
Emil 7547a30bde Add job cancellation and dataset row limit
coordinator / test (push) Canceled after 0s
2026-07-23 23:14:25 +03:00
Emil 6bac7dad3c Clarify worker failures in operator UI 2026-07-23 22:59:51 +03:00
Emil c7956c4683 Fix relative worker work directory 2026-07-23 22:48:27 +03:00
Emil d648beede2 Use English operator UI copy 2026-07-23 22:45:36 +03:00
Emil ac9b921401 Clarify operator UI workflow 2026-07-23 22:41:14 +03:00
Emil 5be87ad762 Fix UI dataset upload field order 2026-07-23 22:20:18 +03:00
Emil e83e0b5e1f Add local operator web interface 2026-07-23 22:08:58 +03:00
Emil ec861edce5 Plan initial web interface 2026-07-23 21:53:00 +03:00
Emil 2ce9687e52 Merge branch 'fix/worker-contract-hardening'
# Conflicts:
#	docs/api-contract.md
#	scimesh/worker/artifacts.py
#	scimesh/worker/coordinator.py
#	scimesh/worker/daemon.py
#	scimesh/worker/models.py
#	tests/test_worker_daemon.py
2026-07-23 21:47:26 +03:00
Emil 66836b962d Merge branch 'fix/coordinator-worker-integration' 2026-07-23 21:47:13 +03:00
Emil 484ecd0dfa Bind result artifacts to lease attempts 2026-07-23 21:44:44 +03:00
Emil 983c5843ec Fix coordinator worker integration 2026-07-23 21:33:37 +03:00
Emil b4a89dd7c2 Merge branch 'feat/coordinator'
# Conflicts:
#	docs/api-contract.md
2026-07-23 21:17:53 +03:00
Efremenko Arhip 8af8ddcf48 build: add make check — run the whole suite with one command
vet + lint + race unit tests, then bring up the stack and run the integration
suite and the end-to-end smoke test. One command for a reviewer to verify
everything.
2026-07-23 21:12:32 +03:00
Efremenko Arhip d271170dd2 docs: task brief for the worker-side agent
A self-contained handoff for whoever (human or agent) implements the Python
worker: what the coordinator already does, where the contract lives
(openapi.yaml, building-workers.md), how to run it locally, the deliverable
(CTX-06), acceptance criteria, and the do-not-break rules.
2026-07-23 21:06:16 +03:00
Emil e0ee95cbab Harden worker contract and transport 2026-07-23 20:49:50 +03:00
Emil abfda35170 Add team agent context and project contracts 2026-07-23 20:31:20 +03:00
Efremenko Arhip 6829632651 ci+docs: GitHub Actions pipeline and a worker-author guide
- .github/workflows/coordinator.yml: vet, gofmt, race unit tests, golangci-lint,
  and the integration suite against a Postgres service — on every push/PR that
  touches coordinator/.
- docs/building-workers.md: a from-scratch guide for implementing a worker (the
  claim/heartbeat/upload/complete loop, auth, lease semantics, status codes,
  and the do-not-break rules), pointing at openapi.yaml for client generation.
- README: refresh the endpoint table, status, and test sections to match reality.
2026-07-23 17:36:26 +03:00
Efremenko Arhip 3b41455b20 feat(coordinator): running state, worker liveness, request-size limits
Polish pass hardening the queue and closing plan gaps.

- Task state machine gains `running`: the first heartbeat moves a task from
  leased to running (migrations 0006/0007 add the enum value and extend the
  lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job
  progress all treat leased and running alike.
- Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps
  its last_heartbeat_at online; a second background reaper marks workers offline
  after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic).
- Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads
  capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader.
- Tests cover the running transition, liveness + offline reaper (unit over
  memstore and integration over Postgres).
2026-07-23 17:36:26 +03:00
Efremenko Arhip 4fc3c69fdf docs: add OpenAPI 3.0 spec for the Python worker side
docs/openapi.yaml is a machine-readable mirror of the api-contract, covering
every coordinator endpoint with request/response schemas, auth, and error
codes. The Python author can generate a typed client or Pydantic models from it.
Its 12 paths match the router exactly; all $refs resolve.
2026-07-23 16:58:49 +03:00