diff --git a/PLAN.md b/PLAN.md index 81540d2..91f5922 100644 --- a/PLAN.md +++ b/PLAN.md @@ -66,7 +66,8 @@ Coordinator reducer -> final artifact -> download/status API - cloud object storage, Kubernetes, autoscaling, and multi-region operation; - arbitrary shell commands sent by coordinator to workers; -- user accounts, multi-tenancy, billing, or sophisticated authorization; +- user accounts, multi-tenancy, billing, or sophisticated authorization + (planned after the first release in CTX-15); - GPU scheduling and multiprocessing inside a worker; - Docker as a required runtime dependency; - video/CV processing implementation; @@ -820,6 +821,69 @@ CTX-09 enables final result downloads. - failure/retry scenarios have automated coverage; - README contains architecture diagram, security caveat, and troubleshooting. +### CTX-13 — In-worker CPU parallelism + +**Goal:** Allow a worker to use a bounded, configured number of CPU threads or +processes while preserving the existing one-task-per-lease coordinator model. + +**Depends on:** CTX-12. + +**Acceptance criteria:** + +- worker concurrency is an explicit configuration value with a safe default of + one; +- a task's internal parallel execution has bounded memory and does not build a + dense N×N similarity matrix; +- CPU-parallel `similarity-search` and `similarity-graph` outputs match the + single-threaded local reference byte-for-byte where ordering is observable; +- result ordering is deterministic across worker counts and block sizes; +- cancellation, lease loss, and worker failure stop child work safely and do + not report a successful result; +- benchmarks and tests cover one-worker and multi-worker configurations. + +### CTX-14 — GPU-accelerated workload execution + +**Goal:** Add an optional GPU execution backend for supported molecular +workloads, while retaining the validated CPU implementation as the reference +and fallback. + +**Depends on:** CTX-13. + +**Acceptance criteria:** + +- GPU capability and backend version are advertised explicitly by a worker; +- the coordinator schedules GPU work only to compatible workers and CPU-only + workers continue to claim CPU tasks; +- unsupported hardware, unavailable drivers, and GPU execution errors produce + sanitized failures or a documented CPU fallback; +- GPU results match the CPU reference within a documented, tested numerical + tolerance and preserve deterministic output ordering; +- GPU memory use is bounded and no dense N×N similarity matrix is created; +- CPU-only CI verifies backend selection and contract behavior, with GPU + integration tests documented for compatible runners. + +### CTX-15 — User Service and access control + +**Goal:** Introduce a dedicated User Service for user identity and access +control, without coupling workers to user credentials or moving scientific +workload logic into the service. + +**Depends on:** CTX-12. + +**Acceptance criteria:** + +- the service has a versioned, documented API and owns user identity data; +- credentials and authentication tokens are stored and handled securely; they + are never logged or exposed to workers; +- authenticated identity is propagated to coordinator requests through an + explicit, validated boundary; +- authorization restricts access to jobs and artifacts to the intended user or + project; +- unauthenticated, expired-token, and cross-user access attempts have + automated failure tests; +- the existing single-operator demo remains usable through a documented local + development configuration. + --- ## 10. Suggested assignment bundles @@ -948,12 +1012,15 @@ Before merging a task, reviewer checks: Do not start these before CTX-12 is accepted. - Replace local artifact storage with S3/MinIO behind an `ArtifactStore` API. -- Add worker labels/capacity-aware scheduling and concurrency > 1. +- Add worker labels and capacity-aware scheduling. +- Implement CTX-13 for bounded in-worker CPU parallelism. +- Implement CTX-14 for optional GPU-accelerated workload execution. +- Implement CTX-15 for the User Service and authenticated user/project access. - Add cancellation propagation to workers. - Add image outputs and final PDF reporting to job artifacts. - Add CV/video workloads using the same planner/runner/reducer contract. - Add observability export (Prometheus/OpenTelemetry). -- Add per-user/project authorization and signed artifact URLs. +- Add signed artifact URLs. - Add shard caching and content-addressed input deduplication. - Add job priority and fair scheduling. - Add a CLI for submitting and monitoring remote jobs. diff --git a/README.md b/README.md index 0e79c8f..51f00f2 100644 --- a/README.md +++ b/README.md @@ -137,5 +137,6 @@ The package separates common dataset parsing and fingerprints from independent w - [Emil](https://github.com/emil28092005) — Project Lead - [Kristina](https://github.com/kristtma) — Tech Lead +- [Veniamin](https://t.me/Veniamin_Kt) — Scientific Lead - [Arkhip](https://github.com/hIpa-ussr) — Programmer -- [Makar](https://github.com/RERAN4K) — Programmer +- [Reranchik](https://github.com/RERAN4K) — Programmer diff --git a/STATUS.md b/STATUS.md index cd13285..b07dab7 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,7 +1,7 @@ # SciMesh Status -**Updated:** 2026-07-24 -**Branch baseline:** `main` at `6e67daa` (distributed similarity-search) +**Updated:** 2026-07-27 +**Branch baseline:** `main` at `f5ead0a` (team and scaling-roadmap documentation) ## Current state @@ -23,6 +23,9 @@ are reduced once into a checksum-protected final CSV, which is downloadable through the coordinator. The full Go checks (including a fresh migration and real PostgreSQL smoke test) passed on 2026-07-24. +A User Service is being developed on a separate programmer branch. It is not +yet merged, reviewed, or integrated with the coordinator/worker contract. + ## Milestone tracker | CTX | Status | Notes | @@ -40,6 +43,7 @@ real PostgreSQL smoke test) passed on 2026-07-24. | CTX-10 Distributed similarity-graph | Not started | Local reference exists. | | CTX-11 Dashboard/operator view | Implemented | Protected live control room: recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, and bounded polling. | | CTX-12 Reliability, security, CI | In progress | Unit, race, PostgreSQL integration, and smoke checks exist; CI hardening remains. | +| CTX-15 User Service and access control | In progress (separate branch) | Proposed implementation is under development; API, security review, tests, and integration are pending. | ## Next recommended assignment