Add workload SDK foundation
This commit is contained in:
Emil
@@ -0,0 +1,115 @@
|
||||
# Workload SDK handoff
|
||||
|
||||
**Audience:** the engineer/AI continuing SciMesh Workload SDK implementation.
|
||||
**Date:** 2026-08-01. **Baseline:** uncommitted working tree on `main` (`11e9333`
|
||||
plus the CTX-16 SDK changes); `python -m pytest -q` reports **225 passed**.
|
||||
|
||||
Read first, in this order: `AGENTS.md` (binding repo rules),
|
||||
`docs/scimesh-sdk-roadmap.md` (delivery order — it governs, this file does not),
|
||||
`docs/scimesh-sdk-contract.md` (normative target semantics),
|
||||
`docs/workload-sdk.md` (author guide for what exists), and the CTX-16 entry in
|
||||
`PLAN.md`.
|
||||
|
||||
## What is already done (do not redo)
|
||||
|
||||
CTX-16 "Workload SDK foundation" is complete and tested. `scimesh/sdk/`
|
||||
implements the `core-batch-v1` profile:
|
||||
|
||||
- Immutable, JSON-strict value objects: `identity.py`, `artifacts.py`,
|
||||
`workflow.py`, `manifest.py`, `plans.py`, `execution.py`, `resources.py`.
|
||||
- Fail-closed compatibility negotiation: `runtime.py` (`negotiate_manifest`)
|
||||
plus request-level checks in `registry.py`.
|
||||
- Installed-package registry with administrator allowlist, exact version +
|
||||
`sha256:` digest pinning, entry-point discovery with digest measured before
|
||||
and after import: `registry.py`, `integrity.py`.
|
||||
- Verifier primitives `ExactArtifactVerifier`, `CanonicalRecordVerifier`,
|
||||
`NumericToleranceVerifier` with bounded sanitized evidence: `verification.py`.
|
||||
- Local conformance harness: `LocalArtifactStore`, `LocalCoreBatchExecutor`,
|
||||
`ResourcePool` (atomic all-or-nothing reservation): `conformance.py`.
|
||||
- Legacy adapter exposing distributed `similarity-search` through the SDK
|
||||
without changing its wire schema: `compat/distributed_v1.py`, `builtins.py`;
|
||||
entry point `similarity-search@1.0.0` is declared in `pyproject.toml`.
|
||||
- Tests: `tests/test_sdk_{models,resources,verification,compatibility,registry}.py`
|
||||
including fail-closed rejection coverage for every advanced profile
|
||||
declaration (gang, GPU modes, pools, checkpoints, retries, secrets, streams,
|
||||
loops, side effects).
|
||||
|
||||
## What remains, in delivery order
|
||||
|
||||
1. **`descriptor-batch` reference workload** (roadmap step 3 — the recommended
|
||||
next task; it is pure Python and needs no coordinator changes). Pinned RDKit
|
||||
2D descriptors, canonical one-row-per-input CSV, shard-index concatenation
|
||||
with one header, byte-identical local/distributed output, two-worker quorum.
|
||||
Build it as an SDK-native package (manifest + planner/runner/reducer/
|
||||
verifier handlers), not through the legacy adapter; reuse the
|
||||
`similarity-search` adapter (`scimesh/sdk/compat/distributed_v1.py`) and
|
||||
`builtins.py` as the structural template, and the
|
||||
`tests/test_sdk_compatibility.py` fixtures as the test template. This is the
|
||||
intended first `untrusted_quorum` candidate (byte_exact + exact-artifact@1).
|
||||
2. **Distributed `similarity-graph`** (CTX-10, roadmap step 1). The coordinator
|
||||
currently rejects `similarity-graph` uploads; it needs cross-shard block-pair
|
||||
planning and duplicate-safe reduction. STATUS.md names this the next
|
||||
recommended assignment overall.
|
||||
3. **Coordinator/Worker protocol v2** (needs CTX-10, then CTX-13 in-worker CPU
|
||||
parallelism and CTX-14 GPU execution; Go + Python). The protocol-v1
|
||||
coordinator persists only flat one-input/one-result tasks: no resource
|
||||
requirements, stage edges, package versions, device allocations, or gang
|
||||
leases. Until a versioned rollout lands, SDK declarations for those features
|
||||
must stay fail-closed — do not silently "enable" them.
|
||||
4. **More chemistry workloads** (roadmap step 4): standardization, SMARTS
|
||||
screening, fingerprint export, fixed-template reaction enumeration, then
|
||||
reaction validation/descriptors.
|
||||
5. **Composite artifacts and richer verifier policies** (roadmap step 5):
|
||||
first-class ordered/keyed `ArtifactCollection` edges instead of composite
|
||||
manifest artifacts; decide where verifiers execute (open decision in the
|
||||
roadmap).
|
||||
6. **Authoring CLI** (future tooling, does not exist today): `scimesh workload
|
||||
init`, `validate`, `test-local`, `test-distributed`, `golden`, `package`.
|
||||
Per AGENTS.md, keep CLI parsing in workload modules and register through
|
||||
`scimesh/core/registry.py`; no workload-specific logic in the main CLI.
|
||||
7. **Open decisions** (listed at the end of the roadmap): SDK distribution
|
||||
split, Go↔Python planner bridge, verifier execution/attestation, trust-mode
|
||||
governance, multi-user enablement. Do not pick one unilaterally — surface it.
|
||||
|
||||
## Known traps (cost the previous session real time)
|
||||
|
||||
- The legacy adapter pins its own manifest (`adapter.manifest`). If a test
|
||||
changes limits/workflow on the manifest, the adapter's copy must be replaced
|
||||
too, or `registry.plan` fails with "planner plan does not carry the selected
|
||||
immutable workload pin".
|
||||
- `WorkloadDefinition` validation: a PLAN stage's `entry_point` must equal
|
||||
`planner.entry_point`; every non-REDUCE stage's `entry_point` must be a key
|
||||
in `runners` (REDUCE → `reducers`); verifier handlers are keyed by
|
||||
`ComponentRef.canonical` and must expose a matching `.identity`.
|
||||
- Negotiation requires each triggering property's feature to be declared
|
||||
separately: e.g. `PROCESS_POOL` needs `process-pools` **and** `multi-process`
|
||||
for `max_processes > 1`. Runtime must also advertise every declared required
|
||||
feature, or negotiation fails with `feature-unavailable`.
|
||||
- `feature-fallback-disallowed` in `scimesh/sdk/registry.py` is currently
|
||||
unreachable via `registry.plan` (the `feature-unavailable` check fires first
|
||||
for any runtime that produced a fallback). Behavior is still fail-closed;
|
||||
decide whether to reorder or delete the branch.
|
||||
- The local executor is deliberately trusted/in-process: it rejects anything
|
||||
but `TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-threaded CPU
|
||||
map/reduce without retries/gangs/accelerators/secrets/checkpoints. That is a
|
||||
contract, not a bug — test rejections, don't "fix" them.
|
||||
- `JobRequest` parameters and failure/evidence payloads reject local paths and
|
||||
URIs by design; keep new payloads location-free.
|
||||
- There is a stray nested clone `SciMesh/` in the repo root (same repo at an
|
||||
older commit). Ignore it and never `git add` it; consider deleting it.
|
||||
- The full ChEMBL extract `chembl_37_chemreps.txt` (~2.9M rows) makes the
|
||||
single-threaded local executor run for many minutes; tests must use small
|
||||
TSV fixtures (see `_write_tiny_dataset`).
|
||||
|
||||
## Working agreement
|
||||
|
||||
- Verify with `source .venv/bin/activate && python -m pytest -q`; the baseline
|
||||
is 225 passing tests and it must stay green. Add a regression test for every
|
||||
behavioral change; similarity code needs a brute-force/sorted reference and
|
||||
determinism across block sizes.
|
||||
- Legacy `similarity-search` wire schema, worker alias boundary, and scientific
|
||||
output must not change. Worker code never talks to PostgreSQL directly;
|
||||
results go through the coordinator; failures go to `/failure`, never as
|
||||
`file://`/`worker://` result URIs.
|
||||
- Do not commit datasets, generated CSV/PNG, tokens, or local worker artifacts.
|
||||
- One CTX task per pull request; link the CTX item from `PLAN.md`.
|
||||
Reference in New Issue
Block a user