116 lines
6.9 KiB
Markdown
116 lines
6.9 KiB
Markdown
# Workload SDK handoff
|
|
|
|
**Audience:** the engineer/AI continuing SciMesh Workload SDK implementation.
|
|
**Date:** 2026-08-01. **Baseline:** uncommitted working tree on `main` (`11e9333`
|
|
plus the CTX-16 SDK changes); `python -m pytest -q` reports **225 passed**.
|
|
|
|
Read first, in this order: `AGENTS.md` (binding repo rules),
|
|
`docs/scimesh-sdk-roadmap.md` (delivery order — it governs, this file does not),
|
|
`docs/scimesh-sdk-contract.md` (normative target semantics),
|
|
`docs/workload-sdk.md` (author guide for what exists), and the CTX-16 entry in
|
|
`PLAN.md`.
|
|
|
|
## What is already done (do not redo)
|
|
|
|
CTX-16 "Workload SDK foundation" is complete and tested. `scimesh/sdk/`
|
|
implements the `core-batch-v1` profile:
|
|
|
|
- Immutable, JSON-strict value objects: `identity.py`, `artifacts.py`,
|
|
`workflow.py`, `manifest.py`, `plans.py`, `execution.py`, `resources.py`.
|
|
- Fail-closed compatibility negotiation: `runtime.py` (`negotiate_manifest`)
|
|
plus request-level checks in `registry.py`.
|
|
- Installed-package registry with administrator allowlist, exact version +
|
|
`sha256:` digest pinning, entry-point discovery with digest measured before
|
|
and after import: `registry.py`, `integrity.py`.
|
|
- Verifier primitives `ExactArtifactVerifier`, `CanonicalRecordVerifier`,
|
|
`NumericToleranceVerifier` with bounded sanitized evidence: `verification.py`.
|
|
- Local conformance harness: `LocalArtifactStore`, `LocalCoreBatchExecutor`,
|
|
`ResourcePool` (atomic all-or-nothing reservation): `conformance.py`.
|
|
- Legacy adapter exposing distributed `similarity-search` through the SDK
|
|
without changing its wire schema: `compat/distributed_v1.py`, `builtins.py`;
|
|
entry point `similarity-search@1.0.0` is declared in `pyproject.toml`.
|
|
- Tests: `tests/test_sdk_{models,resources,verification,compatibility,registry}.py`
|
|
including fail-closed rejection coverage for every advanced profile
|
|
declaration (gang, GPU modes, pools, checkpoints, retries, secrets, streams,
|
|
loops, side effects).
|
|
|
|
## What remains, in delivery order
|
|
|
|
1. **`descriptor-batch` reference workload** (roadmap step 3 — the recommended
|
|
next task; it is pure Python and needs no coordinator changes). Pinned RDKit
|
|
2D descriptors, canonical one-row-per-input CSV, shard-index concatenation
|
|
with one header, byte-identical local/distributed output, two-worker quorum.
|
|
Build it as an SDK-native package (manifest + planner/runner/reducer/
|
|
verifier handlers), not through the legacy adapter; reuse the
|
|
`similarity-search` adapter (`scimesh/sdk/compat/distributed_v1.py`) and
|
|
`builtins.py` as the structural template, and the
|
|
`tests/test_sdk_compatibility.py` fixtures as the test template. This is the
|
|
intended first `untrusted_quorum` candidate (byte_exact + exact-artifact@1).
|
|
2. **Distributed `similarity-graph`** (CTX-10, roadmap step 1). The coordinator
|
|
currently rejects `similarity-graph` uploads; it needs cross-shard block-pair
|
|
planning and duplicate-safe reduction. STATUS.md names this the next
|
|
recommended assignment overall.
|
|
3. **Coordinator/Worker protocol v2** (needs CTX-10, then CTX-13 in-worker CPU
|
|
parallelism and CTX-14 GPU execution; Go + Python). The protocol-v1
|
|
coordinator persists only flat one-input/one-result tasks: no resource
|
|
requirements, stage edges, package versions, device allocations, or gang
|
|
leases. Until a versioned rollout lands, SDK declarations for those features
|
|
must stay fail-closed — do not silently "enable" them.
|
|
4. **More chemistry workloads** (roadmap step 4): standardization, SMARTS
|
|
screening, fingerprint export, fixed-template reaction enumeration, then
|
|
reaction validation/descriptors.
|
|
5. **Composite artifacts and richer verifier policies** (roadmap step 5):
|
|
first-class ordered/keyed `ArtifactCollection` edges instead of composite
|
|
manifest artifacts; decide where verifiers execute (open decision in the
|
|
roadmap).
|
|
6. **Authoring CLI** (future tooling, does not exist today): `scimesh workload
|
|
init`, `validate`, `test-local`, `test-distributed`, `golden`, `package`.
|
|
Per AGENTS.md, keep CLI parsing in workload modules and register through
|
|
`scimesh/core/registry.py`; no workload-specific logic in the main CLI.
|
|
7. **Open decisions** (listed at the end of the roadmap): SDK distribution
|
|
split, Go↔Python planner bridge, verifier execution/attestation, trust-mode
|
|
governance, multi-user enablement. Do not pick one unilaterally — surface it.
|
|
|
|
## Known traps (cost the previous session real time)
|
|
|
|
- The legacy adapter pins its own manifest (`adapter.manifest`). If a test
|
|
changes limits/workflow on the manifest, the adapter's copy must be replaced
|
|
too, or `registry.plan` fails with "planner plan does not carry the selected
|
|
immutable workload pin".
|
|
- `WorkloadDefinition` validation: a PLAN stage's `entry_point` must equal
|
|
`planner.entry_point`; every non-REDUCE stage's `entry_point` must be a key
|
|
in `runners` (REDUCE → `reducers`); verifier handlers are keyed by
|
|
`ComponentRef.canonical` and must expose a matching `.identity`.
|
|
- Negotiation requires each triggering property's feature to be declared
|
|
separately: e.g. `PROCESS_POOL` needs `process-pools` **and** `multi-process`
|
|
for `max_processes > 1`. Runtime must also advertise every declared required
|
|
feature, or negotiation fails with `feature-unavailable`.
|
|
- `feature-fallback-disallowed` in `scimesh/sdk/registry.py` is currently
|
|
unreachable via `registry.plan` (the `feature-unavailable` check fires first
|
|
for any runtime that produced a fallback). Behavior is still fail-closed;
|
|
decide whether to reorder or delete the branch.
|
|
- The local executor is deliberately trusted/in-process: it rejects anything
|
|
but `TrustMode.TRUSTED`, `NetworkPolicy.TRUSTED`, single-threaded CPU
|
|
map/reduce without retries/gangs/accelerators/secrets/checkpoints. That is a
|
|
contract, not a bug — test rejections, don't "fix" them.
|
|
- `JobRequest` parameters and failure/evidence payloads reject local paths and
|
|
URIs by design; keep new payloads location-free.
|
|
- There is a stray nested clone `SciMesh/` in the repo root (same repo at an
|
|
older commit). Ignore it and never `git add` it; consider deleting it.
|
|
- The full ChEMBL extract `chembl_37_chemreps.txt` (~2.9M rows) makes the
|
|
single-threaded local executor run for many minutes; tests must use small
|
|
TSV fixtures (see `_write_tiny_dataset`).
|
|
|
|
## Working agreement
|
|
|
|
- Verify with `source .venv/bin/activate && python -m pytest -q`; the baseline
|
|
is 225 passing tests and it must stay green. Add a regression test for every
|
|
behavioral change; similarity code needs a brute-force/sorted reference and
|
|
determinism across block sizes.
|
|
- Legacy `similarity-search` wire schema, worker alias boundary, and scientific
|
|
output must not change. Worker code never talks to PostgreSQL directly;
|
|
results go through the coordinator; failures go to `/failure`, never as
|
|
`file://`/`worker://` result URIs.
|
|
- Do not commit datasets, generated CSV/PNG, tokens, or local worker artifacts.
|
|
- One CTX task per pull request; link the CTX item from `PLAN.md`.
|