diff --git a/PLAN.md b/PLAN.md index 2e43ebc..1b8ecd8 100644 --- a/PLAN.md +++ b/PLAN.md @@ -5,11 +5,13 @@ > platform. It is intentionally detailed enough to split into independent task > briefs for developers or coding agents. > -> **Planning baseline.** This branch starts from `Workers`: the Python package -> has local `similarity-search` and `similarity-graph` workloads plus a Worker -> Daemon client. The coordinator and PostgreSQL implementation do not yet -> exist. The Worker contract and the Go/PostgreSQL design briefs in `docs/` are -> part of this plan. +> **Current planning baseline (2026-08-01).** The Go/PostgreSQL coordinator, +> Python Worker Agent, versioned distributed-workload protocol, artifact-backed +> task lifecycle, reducer orchestration, operator UI, User Service, and +> distributed `similarity-search` are implemented on `main`. The evidence-based +> completion tracker is [`STATUS.md`](STATUS.md); this document defines the +> remaining direction and dependencies. Earlier descriptions of a missing +> coordinator are historical context, not current work. --- @@ -51,7 +53,7 @@ Coordinator reducer -> final artifact -> download/status API ### 2.1 In scope -- Go 1.22+ coordinator service with PostgreSQL 15+; +- Go 1.25+ coordinator service with PostgreSQL 15+; - Python Worker Daemon running existing SciMesh workloads locally; - durable job, task, worker, and artifact metadata; - local coordinator-managed artifact storage for the first deployment; @@ -66,8 +68,8 @@ Coordinator reducer -> final artifact -> download/status API - cloud object storage, Kubernetes, autoscaling, and multi-region operation; - arbitrary shell commands sent by coordinator to workers; -- user accounts, multi-tenancy, billing, or sophisticated authorization - (planned after the first release in CTX-15); +- billing and sophisticated multi-tenant administration beyond the implemented + User Service and owner scoping; - GPU scheduling and multiprocessing inside a worker; - Docker as a required runtime dependency; - video/CV processing implementation; @@ -491,14 +493,82 @@ brute-force graph for both `greater` and `less` threshold directions. ### 7.3 Future workload policy -A new workload is accepted only when it supplies: +A workload is more than a runner. It must define validation, planner and +reducer behavior, worker allowlist/capabilities, input and output artifacts, +UI/API parameters, reproducible execution environment, result verification, +golden cross-worker fixtures, and applicable resource limits. The future public +contract is described in [`docs/scimesh-sdk-roadmap.md`](docs/scimesh-sdk-roadmap.md). -- an input/parameter validator; -- an explicit sharding strategy; -- bounded-memory task execution; -- deterministic reduction semantics; -- fixture-based local and distributed correctness tests; -- a `describe()` payload for UI/API discovery. +Every workload declaration must classify its task decomposition and input/output +artifact shapes, determinism, reduction semantics, verifier mode, supported +trust modes, CPU/memory/accelerator needs, and maximum output growth. The +initial profiles are: + +| Profile | Current acceptance policy | +| --- | --- | +| Byte-exact deterministic | Supported for untrusted quorum when whole artifacts have identical SHA-256. | +| Canonical-exact deterministic | Deferred until the parser, schema, ordering, encoding, and serializer are versioned. | +| Numeric deterministic with tolerance | Deferred until structured numeric comparison exists. | +| Stochastic/search-based | Requires domain-specific evidence, repeated runs, or trusted execution. | +| Trusted-only or domain-verified | May be planned only with an explicit trust policy and verifier. | + +The current untrusted quorum records one vote per owner and accepts a task only +when distinct owners upload artifacts with the same complete-file SHA-256. It +therefore supports only the byte-exact profile (or a workload that first makes +its output byte-identical through a specified canonicalization step). Reducers +must fail safely rather than silently merge inconsistent partial results. + +Before a workload is admitted to untrusted execution it needs a reproducibility +gate: pinned environment/container digest and dependency versions; fixed locale, +timezone, UTF-8/newline/CSV settings; explicit invalid-row and algorithm +options; canonical representation and ordering; deterministic filenames/archive +metadata; golden fixtures from two independently provisioned workers; local vs +distributed parity; and retry/out-of-order completion tests. A loose dependency +constraint is insufficient for byte-exact quorum. + +Near-term critical path: + +```text +distributed similarity-graph + -> reliability, security, and cross-language CI + -> stable first release + -> SDK foundation and descriptor-batch + -> additional deterministic workloads +``` + +Initial deterministic-workload backlog: `descriptor-batch` (the first SDK +reference workload), molecule standardization, SMARTS screening, fingerprint +export, fixed-template SMIRKS enumeration with strict caps, and reaction +validation/descriptors. `similarity-graph` remains ahead of this backlog. +Bounded combinatorial libraries and seeded conformers need specialized controls. +ML, retrosynthesis, docking, QM, molecular dynamics, and GPU workloads are +deferred until verifier/trust and reproducibility requirements are met. + +### 7.4 Future verification, concurrency, and accelerators + +Verification is a future versioned workload capability, not permanent +whole-file-SHA logic. Planned modes are `ExactArtifactVerifier`, +`CanonicalRecordVerifier`, `NumericToleranceVerifier`, `DomainSpecificVerifier`, +and `TrustedWorkerPolicy`. Exact SHA-256 remains the first and safest mode; +canonical and numeric modes must compare bounded structured data and publish +sanitized evidence and failure reasons. + +Worker concurrency remains **1** until implemented and tested. Its target model +is one physical machine running one Worker Agent with `N` execution slots and +one isolated subprocess per active Task, rather than one registered worker per +CPU core. `max_concurrency` must be separate from `cpu_count`; each task keeps +its own heartbeat, attempt directory, lease lifecycle, resource request, and +graceful-drain behavior. CPU-bound scientific code should use processes and +avoid nested oversubscription. + +Accelerator support is also deferred. The coordinator matches generic resource +requirements; the Worker Agent discovers and isolates devices (including +`CUDA_VISIBLE_DEVICES`) and owns process/accounting lifecycle; the Python +workload owns batching, memory strategy, deterministic output, and scientific +validation; reducers/verifiers define CPU/GPU-independent semantics. CUDA and +scientific kernels do not belong in the Go coordinator. GPU work follows stable +CPU slices, generic resource requirements, pinned worker images, and tested +CPU/GPU or domain-valid equivalence. --- diff --git a/STATUS.md b/STATUS.md index da7d10e..a84c396 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,7 +1,7 @@ # SciMesh Status -**Updated:** 2026-07-27 -**Branch baseline:** `main` at `f5ead0a` (team and scaling-roadmap documentation) +**Updated:** 2026-08-01 +**Branch baseline:** `main` at `b9a975b` (self-service worker enrollment) ## Current state @@ -27,6 +27,9 @@ The User Service is merged into `main`. It owns user accounts, authentication, roles, and verified-contributor status; the coordinator scopes user jobs and worker operations to the authenticated owner. Its documented v1 contract is in [`docs/user-service-api-contract.md`](docs/user-service-api-contract.md). +Users can create and revoke worker keys for self-service Worker Agent +enrollment. Untrusted workers require quorum agreement from distinct owners on +the complete result-artifact SHA-256 before a task is accepted. ## Milestone tracker @@ -45,7 +48,7 @@ worker operations to the authenticated owner. Its documented v1 contract is in | CTX-10 Distributed similarity-graph | Not started | Local reference exists. | | CTX-11 Dashboard/operator view | Implemented | Protected live control room: recent-run/worker overview, real pipeline-stage visualization, shard attempts and safe failures, validated similarity-search upload, coordinator artifacts, final-result download, and bounded polling. | | CTX-12 Reliability, security, CI | In progress | Unit, race, PostgreSQL integration, and smoke checks exist; CI hardening remains. | -| CTX-15 User Service and access control | Implemented | Go unit/race checks, Docker/PostgreSQL integration, and coordinator ownership regression checks pass locally. | +| CTX-15 User Service and access control | Implemented | User/owner scoping, verified contributors, worker keys, self-service enrollment, and quorum-backed untrusted workers are merged; local Go/Python and Docker/PostgreSQL checks passed. | ## Next recommended assignment diff --git a/docs/scimesh-sdk-roadmap.md b/docs/scimesh-sdk-roadmap.md new file mode 100644 index 0000000..83e3f66 --- /dev/null +++ b/docs/scimesh-sdk-roadmap.md @@ -0,0 +1,123 @@ +# SciMesh Workload SDK roadmap + +**Status:** future design and sequencing document. No SDK package, commands, or +general verifier abstraction described here is implemented yet. + +## Purpose and boundaries + +The SDK should let a scientific developer add an allowlisted workload without +learning coordinator internals or writing SQL, while keeping one scientific +implementation usable locally and in distributed execution: + +```text +scientific implementation -> local adapter -> planner/tasks -> reducer -> verifier +``` + +It must not execute arbitrary code or shell commands supplied by a coordinator. +SDK v1 is not a public marketplace, generic container/job runner, cross-language +SDK, automatic correctness-proof system, or immediate route to stochastic ML or +GPU workloads. + +The current foundation is the Python `DistributedWorkload` protocol and registry +under `scimesh/distributed/`, the Worker Agent under `scimesh/worker/`, and the +coordinator API contract. See [CTX-07](ctx-07-distributed-workload-protocol.md), +[worker-building guide](building-workers.md), and [API contract](api-contract.md). + +## Proposed public concepts + +| Concept | Responsibility | +| --- | --- | +| `WorkloadDefinition` / `WorkloadManifest` | Name, versions, schemas, execution and verification metadata. | +| `ParameterSchema`, `InputSpec`, `OutputSpec`, `ArtifactRef` | Typed public inputs and durable artifact shapes. | +| `TaskPlan`, `Planner`, `Runner`, `Reducer` | Validate, split, execute, and deterministically combine work. | +| `Verifier` | Accept or reject result evidence; never silently downgrade checks. | +| `ResourceRequirements`, `ExecutionProfile`, `ReproducibilityProfile` | Bounded resource needs and pinned execution assumptions. | + +A manifest should include workload/version and SDK compatibility versions, +description, parameter/input/output schemas, planner/runner/reducer/verifier +types, determinism and trust profiles, resource/output limits, worker +capabilities, and environment or image digest. Compatibility must be explicit +among SDK, coordinator protocol, worker runtime, workload, output schema, and +verifier versions. + +## Contracts + +**Planner:** validates before durable Job/Task creation; produces versioned, +JSON-serializable plans that refer only to durable artifacts; gives stable task +order and expected resources/outputs; fails transactionally without a partial +task graph. + +**Runner:** receives typed parameters and owned artifact references; runs only +allowlisted SDK code; writes to its attempt directory; produces output manifest, +metrics, and sanitized failures; respects cancellation/lease loss when platform +support exists; never uses `shell=True` or unnecessarily exposes credentials. + +**Reducer:** consumes only accepted partial artifacts in stable order; is +idempotent or coordinator-state protected; creates a versioned final manifest; +defines missing, duplicate, and malformed-shard failures. + +**Verifier:** is versioned with the workload and states one of byte-exact, +canonical, numeric, domain-specific, or trust-policy comparison. It processes +structured manifests and bounded streams where practical, records sanitized +evidence, and rejects inconsistent results. + +Current untrusted quorum is only `ExactArtifactVerifier`: distinct owners must +produce whole files with identical SHA-256. Future modes are +`CanonicalRecordVerifier`, `NumericToleranceVerifier`, +`DomainSpecificVerifier`, and `TrustedWorkerPolicy`. Canonical mode requires a +specified parser/schema/order/encoding/serialization; numeric mode compares +structured values, not CSV text. + +## Resources and reproducibility + +The extensible requirement model is `cpu_cores`, `memory_mb`, `scratch_mb`, +`gpu_count`, `gpu_memory_mb`, `accelerator_kind`, `exclusive_device`, and +`estimated_output_bytes`. It must not imply one Task equals one CPU core. + +Untrusted byte-exact workloads require a pinned image/environment digest, +runtime and dependency versions, fixed locale/timezone/UTF-8/newlines/dialect, +explicit invalid-row policy and algorithm options, canonical ordering, stable +archive metadata, golden fixtures, two independently provisioned workers, and +local/distributed plus retry/completion-order parity tests. + +## Compatibility evolution + +The stable release retains the current one-input/one-result task contract. +First, a composite manifest artifact may reference multiple logical inputs; +later, ordered input and output artifact collections can become first-class. +The transition must be versioned and retain old workload compatibility. + +Discovery should use an installed Python package, manifest, pinned environment +metadata, explicit entry points, and golden fixtures. It must be allowlisted; +never scan or execute user-provided module paths. + +## Delivery sequence + +1. Finish distributed `similarity-graph` and reliability/cross-language CI. +2. Stabilize manifest, schema, planner/runner/reducer interfaces, exact verifier, + compatibility metadata, and an author guide. +3. Deliver `descriptor-batch` as the reference workload: pinned RDKit 2D + descriptors; canonical one-row-per-input CSV; shard-index concatenation with + one header; byte-identical local/distributed output and two-worker quorum. +4. Add standardization, SMARTS screening, fingerprint export, fixed-template + reaction enumeration, then reaction validation/descriptors. +5. Generalize composite artifacts, process slots, resource requests, and richer + verifier policies. +6. Only then consider pinned, trusted/domain-verified numeric, ML, docking, QM, + MD, and GPU workloads. + +Future developer tooling may include `scimesh workload init`, `validate`, +`test-local`, `test-distributed`, `golden`, and `package`; these commands do not +exist today. A template should generate a manifest, schemas, planner, runner, +reducer, verifier, unit tests, golden fixture, two-worker integration test, and +documentation. + +## Open decisions + +- Is the SDK part of `scimesh` or a separately versioned Python distribution? +- What stable bridge connects Go orchestration to Python planners/reducers? +- Where do future verifiers execute, and how are environments attested? +- Which trust modes may run each verifier profile? +- Who may install/enable workloads in multi-user deployments? +- How are composite I/O, version negotiation, output-growth limits, and + numeric-tolerance access governed without breaking the existing API?