Fix worker result and lease contracts
This commit is contained in:
@@ -17,6 +17,8 @@ sketch. It does not implement the Worker Daemon or scientific/CV computation.
|
||||
- HTTP server: standard `net/http` is sufficient; do not introduce a framework
|
||||
unless it solves a concrete requirement.
|
||||
- Schema migrations: versioned SQL migrations using `golang-migrate`.
|
||||
- Result storage: a coordinator-managed local artifact directory in the first
|
||||
version; keep its interface replaceable by object storage later.
|
||||
- Identifiers: UUID.
|
||||
- Times: timezone-aware UTC timestamps.
|
||||
- API boundary: the Go coordinator owns all database access. Python workers
|
||||
@@ -155,7 +157,8 @@ must be `POST` rather than `GET`.
|
||||
| --- | --- |
|
||||
| `POST /jobs` | Validate submission, create job and pending tasks transactionally |
|
||||
| `POST /tasks/claim` | Atomically lease one compatible task; `204` when none exists |
|
||||
| `POST /tasks/{task_id}/heartbeat` | Renew the current worker's lease |
|
||||
| `POST /tasks/{task_id}/heartbeat` | Renew the current worker's lease; return the new `lease_expires_at` |
|
||||
| `PUT /tasks/{task_id}/artifacts/{filename}` | Stream one result artifact into coordinator storage; verify the current worker lease |
|
||||
| `POST /tasks/{task_id}/result` | Idempotently persist a completed result manifest |
|
||||
| `POST /tasks/{task_id}/failure` | Record a safe failure or retryable state |
|
||||
| `GET /jobs/{job_id}` | Return aggregate job/task progress |
|
||||
@@ -170,6 +173,13 @@ inputs in the handler. Do not return raw database errors to clients. Return a
|
||||
request ID in error responses and emit structured logs with the request ID,
|
||||
task ID, worker ID, and operation.
|
||||
|
||||
The artifact endpoint receives binary content with `Content-Type`,
|
||||
`X-Worker-ID`, and `X-Task-Attempt` headers. It must stream the request body to
|
||||
the coordinator artifact directory instead of buffering it in memory, calculate
|
||||
or verify its SHA-256, and return `201` with the durable artifact `uri`. Only
|
||||
the worker holding the current lease may upload. `POST /result` accepts only
|
||||
URIs returned by this endpoint for the same task and attempt.
|
||||
|
||||
## Tests and acceptance criteria
|
||||
|
||||
- `golang-migrate` upgrades an empty PostgreSQL database to the current schema
|
||||
@@ -190,7 +200,7 @@ task ID, worker ID, and operation.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Python Worker Daemon execution, input download, and result upload;
|
||||
- Python Worker Daemon execution and input download;
|
||||
- video chunk generation and trajectory stitching;
|
||||
- authentication provider, UI, object-storage implementation, and PDF report;
|
||||
- network/distributed coordinator deployment beyond the local coordinator
|
||||
|
||||
@@ -12,7 +12,7 @@ The target flow is:
|
||||
Worker Daemon -> coordinator: claim task
|
||||
coordinator -> Worker Daemon: task metadata + input location
|
||||
Worker Daemon -> local SciMesh Core / CV runner: execute
|
||||
Worker Daemon -> coordinator: submit result
|
||||
Worker Daemon -> coordinator: upload result artifact, then submit result manifest
|
||||
```
|
||||
|
||||
The architecture sketch uses video chunks and CV, while the current SciMesh
|
||||
@@ -103,9 +103,31 @@ Content-Type: application/json
|
||||
}
|
||||
```
|
||||
|
||||
For a failed execution, send `status: "failed"` with a short, sanitized
|
||||
`error_code` and `error_message`. Never send a Python traceback, access token,
|
||||
or local path outside the worker directory.
|
||||
The `result.uri` must be the durable URI returned by the artifact upload
|
||||
endpoint below; a worker-local `file://` or `worker://` path is invalid.
|
||||
|
||||
### Upload a result artifact
|
||||
|
||||
```http
|
||||
PUT /tasks/{task_id}/artifacts/{filename}
|
||||
Content-Type: text/csv
|
||||
X-Worker-ID: worker-01
|
||||
X-Task-Attempt: 1
|
||||
|
||||
<CSV bytes>
|
||||
```
|
||||
|
||||
The coordinator streams the artifact to its configured storage and responds:
|
||||
|
||||
```json
|
||||
{
|
||||
"uri": "https://coordinator.example/tasks/0d2d/artifacts/result.csv"
|
||||
}
|
||||
```
|
||||
|
||||
For a failed execution, send a short, sanitized `error_code` and
|
||||
`error_message` to `POST /tasks/{task_id}/failure`. Never send a Python
|
||||
traceback, access token, or local path outside the worker directory.
|
||||
|
||||
## Required state machine
|
||||
|
||||
@@ -121,6 +143,8 @@ idle -> claiming -> downloading -> running -> uploading -> submitting -> idle
|
||||
- Create one isolated task directory: `<work-dir>/<task-id>/<attempt>/`.
|
||||
- Invoke the runner with an explicit argument list, never `shell=True`.
|
||||
- Upload/submit exactly the produced result files listed by the runner.
|
||||
- Do not mark a task completed until every submitted artifact has a durable
|
||||
coordinator-provided URI.
|
||||
- A timeout, network error, or rejected submission must leave the local task
|
||||
directory available for diagnostics until a configurable cleanup period.
|
||||
- Treat a duplicate successful submission as success when the coordinator
|
||||
|
||||
Reference in New Issue
Block a user