Emil_Shanaty/SciMesh
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace Python worker daemon with the Go worker agent

Browse Source
This commit is contained in:
Emil
2026-08-02 16:53:04 +03:00
parent 9a8221163a
commit 706bc85e17
31 changed files with 516 additions and 2399 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sdk-handoff.md
+1 -1
View File
@@ -28,7 +28,7 @@ resolves `query_id` per task and rejects plan-time `max_rows`.
**MkDocs site (2026-08-02):** the standalone documentation site lives in `mkdocs/` (`docs_dir: mkdocs`) and does not use the project's `docs/` directory. It contains guides (`mkdocs/sdk/`: overview, authoring-workloads, cli, worker-integration), the full auto-generated API reference for all `scimesh.sdk` modules (`mkdocs/api/`, mkdocstrings `::: scimesh.sdk.<module>` — set `show_if_no_docstring: true`), and the writing rules (`mkdocs/approach.md`). `make docs` builds it; the UI serves it at `/ui/docs/`. All public SDK members now carry Google-style docstrings.
**Go worker agent prototype (2026-08-02):** `coordinator/internal/agent/` (config, models, client, sanitize, taskrunner, daemon) + `coordinator/cmd/worker-agent`, built with `make agent`. It mirrors the Python worker's v1 lifecycle; per-task SDK execution happens in a Python subprocess (`scimesh/worker/task.py`: exits 0 on success, 3 permanent, 1 retryable). Default `TASK_RUNNER` is `python -m scimesh.worker.task` — set it to the venv python in source checkouts. Verified E2E against the demo coordinator. Open items: JWT refresh, resource slots/limits, attempt-dir cleanup, protocol-v2 features.
**Go worker agent (2026-08-02):** the Python worker daemon was removed. `coordinator/internal/agent/` + `cmd/worker-agent` (build: `make agent`) now owns the full lifecycle: register/claim/heartbeat/download(checksum)/spawn-task/upload/submit/fail, static bearer or worker-key JWT auth with 401 refresh, attempt-dir cleanup (`CLEANUP_AFTER_SECONDS`), backoff, idle/max-tasks exit. `scimesh/worker/` keeps only the per-task Python execution: `task.py` (exit 0/3/1), `runners.py` (SDK bridge, `SCIMESH_WORKLOAD_ALLOWLIST` discovery), `models.py` (claim payload). The `scimesh-worker` console script and the daemon/auth/transport modules are gone. Demo (`make demo-ui`) and smoke (`make smoke-two-worker`) run the Go agent; smoke passes 4/4 with two agents. `coordinator/internal/agent/` (config, models, client, sanitize, taskrunner, daemon) + `coordinator/cmd/worker-agent`, built with `make agent`. It mirrors the Python worker's v1 lifecycle; per-task SDK execution happens in a Python subprocess (`scimesh/worker/task.py`: exits 0 on success, 3 permanent, 1 retryable). Default `TASK_RUNNER` is `python -m scimesh.worker.task` — set it to the venv python in source checkouts. Verified E2E against the demo coordinator. Open items: JWT refresh, resource slots/limits, attempt-dir cleanup, protocol-v2 features.
**Authoring scaffold (2026-08-01):** `scimesh/sdk/batch.py` adds
`MapReduceWorkload` — the primary authoring surface for `core-batch-v1`. A