Merge branch 'fix/worker-contract-hardening'
# Conflicts: # docs/api-contract.md # scimesh/worker/artifacts.py # scimesh/worker/coordinator.py # scimesh/worker/daemon.py # scimesh/worker/models.py # tests/test_worker_daemon.py
This commit is contained in:
@@ -1,6 +1,11 @@
|
|||||||
# SciMesh
|
# SciMesh
|
||||||
|
|
||||||
SciMesh is a small local framework for scientific workloads on molecular datasets. It currently provides exact molecular similarity search and exact sparse similarity-graph construction. It runs in one local Python process: there is no network service, multiprocessing, coordinator, database, or dense similarity matrix.
|
SciMesh is a scientific-workload framework for molecular datasets. Its public CLI
|
||||||
|
currently runs exact similarity search and sparse similarity-graph construction
|
||||||
|
locally in one Python process; it creates no dense similarity matrix. A Python
|
||||||
|
Worker client and the planned Go/PostgreSQL coordinator contract are tracked in
|
||||||
|
the repository, but distributed execution is not available yet; see
|
||||||
|
[`STATUS.md`](STATUS.md).
|
||||||
|
|
||||||
The ChEMBL TSV database is intentionally not included in this repository. Download it separately and pass its path to the commands below. The expected columns are `chembl_id` and `canonical_smiles`.
|
The ChEMBL TSV database is intentionally not included in this repository. Download it separately and pass its path to the commands below. The expected columns are `chembl_id` and `canonical_smiles`.
|
||||||
|
|
||||||
|
|||||||
@@ -27,7 +27,9 @@ inside the daemon.
|
|||||||
`scimesh-worker`.
|
`scimesh-worker`.
|
||||||
2. Configuration via environment variables and CLI overrides:
|
2. Configuration via environment variables and CLI overrides:
|
||||||
- `SCIMESH_COORDINATOR_URL` (required);
|
- `SCIMESH_COORDINATOR_URL` (required);
|
||||||
- `SCIMESH_WORKER_ID` (required, stable UUID or hostname-derived value);
|
- `SCIMESH_WORKER_NAME` (optional; defaults to the hostname);
|
||||||
|
- `SCIMESH_WORKER_ID` (optional legacy/test override; production identity is
|
||||||
|
returned by registration);
|
||||||
- working directory for downloaded inputs and generated outputs;
|
- working directory for downloaded inputs and generated outputs;
|
||||||
- poll interval and request timeout;
|
- poll interval and request timeout;
|
||||||
- optional bearer token.
|
- optional bearer token.
|
||||||
@@ -43,6 +45,28 @@ inside the daemon.
|
|||||||
Use JSON over HTTPS. Claiming a task changes its state, so use `POST`, even if
|
Use JSON over HTTPS. Claiming a task changes its state, so use `POST`, even if
|
||||||
the initial diagram labels the endpoint as `GET /get_task`.
|
the initial diagram labels the endpoint as `GET /get_task`.
|
||||||
|
|
||||||
|
`docs/api-contract.md` is the authoritative API schema. This document explains
|
||||||
|
the daemon workflow and must not introduce a different request or response
|
||||||
|
shape.
|
||||||
|
|
||||||
|
### Register worker
|
||||||
|
|
||||||
|
At daemon startup, register the worker capabilities before claiming tasks:
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /workers/register
|
||||||
|
Content-Type: application/json
|
||||||
|
|
||||||
|
{
|
||||||
|
"name": "lab-worker-01",
|
||||||
|
"capabilities": ["similarity-search", "similarity-graph"],
|
||||||
|
"cpu_count": 8,
|
||||||
|
"memory_mb": 16384
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `worker_id` returned by this endpoint is used for the daemon lifetime.
|
||||||
|
|
||||||
### Claim a task
|
### Claim a task
|
||||||
|
|
||||||
```http
|
```http
|
||||||
@@ -90,8 +114,8 @@ Content-Type: application/json
|
|||||||
{
|
{
|
||||||
"worker_id": "worker-01",
|
"worker_id": "worker-01",
|
||||||
"attempt": 1,
|
"attempt": 1,
|
||||||
"status": "completed",
|
|
||||||
"result": {
|
"result": {
|
||||||
|
"artifact_id": "0d2d5a53-4c7e-467e-93d2-45ed2dc18e46",
|
||||||
"uri": "https://coordinator.example/tasks/0d2d/result.csv",
|
"uri": "https://coordinator.example/tasks/0d2d/result.csv",
|
||||||
"sha256": "...",
|
"sha256": "...",
|
||||||
"content_type": "text/csv"
|
"content_type": "text/csv"
|
||||||
@@ -121,7 +145,10 @@ The coordinator streams the artifact to its configured storage and responds:
|
|||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"uri": "https://coordinator.example/tasks/0d2d/artifacts/result.csv"
|
"artifact_id": "0d2d5a53-4c7e-467e-93d2-45ed2dc18e46",
|
||||||
|
"uri": "https://coordinator.example/tasks/0d2d/artifacts/result.csv",
|
||||||
|
"sha256": "...",
|
||||||
|
"size_bytes": 1234
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -142,7 +169,9 @@ idle -> claiming -> downloading -> running -> uploading -> submitting -> idle
|
|||||||
- Verify the input checksum before running.
|
- Verify the input checksum before running.
|
||||||
- Create one isolated task directory: `<work-dir>/<task-id>/<attempt>/`.
|
- Create one isolated task directory: `<work-dir>/<task-id>/<attempt>/`.
|
||||||
- Invoke the runner with an explicit argument list, never `shell=True`.
|
- Invoke the runner with an explicit argument list, never `shell=True`.
|
||||||
- Upload/submit exactly the produced result files listed by the runner.
|
- Upload the produced result artifact before submitting its manifest.
|
||||||
|
- Version 1 produces exactly one CSV partial result. Multi-artifact manifests
|
||||||
|
require an explicit future API-contract change.
|
||||||
- Do not mark a task completed until every submitted artifact has a durable
|
- Do not mark a task completed until every submitted artifact has a durable
|
||||||
coordinator-provided URI.
|
coordinator-provided URI.
|
||||||
- A timeout, network error, or rejected submission must leave the local task
|
- A timeout, network error, or rejected submission must leave the local task
|
||||||
|
|||||||
Reference in New Issue
Block a user