Add final result reduction
coordinator / test (push) Canceled after 0s

This commit is contained in:
Emil
2026-07-24 14:58:18 +03:00
parent 6e67daa9eb
commit a055473706
28 changed files with 849 additions and 54 deletions
+31 -1
View File
@@ -26,7 +26,8 @@ must be updated in the same change as any behaviour it describes.
| `POST /tasks/{id}/heartbeat` | renew lease | ✅ done |
| `POST /tasks/{id}/result` | complete | ✅ done, references `artifact_id` |
| `POST /tasks/{id}/failure` | fail | ✅ done |
| `GET /jobs/{id}` | progress | ✅ done |
| `GET /jobs/{id}` | progress and final-result URI | ✅ done |
| `GET /jobs/{id}/result` | download final CSV | ✅ done |
| `PUT /tasks/{id}/artifacts/{name}` | upload partial | ✅ done |
| `GET /artifacts/{id}/download` | download by id | ✅ done |
| `POST /jobs/upload` | upload dataset, coordinator chunks it | ✅ done |
@@ -67,6 +68,35 @@ artifact. The coordinator splits the selected TSV rows into shard artifacts
Each resulting task's claim response carries `input.uri = /tasks/{id}/input`,
served by §5.4.
## Job progress and final result
```http
GET /jobs/{job_id}
Authorization: Bearer <token>
```
The response contains task counters and a derived status. A successful
`similarity-search` enters `reducing` after the last shard completes, then
becomes `completed` only after the coordinator stores its deterministic final
CSV. At that point `result_uri` is present:
```json
{
"id": "uuid",
"status": "completed",
"total": 3,
"pending": 0,
"leased": 0,
"completed": 3,
"failed": 0,
"cancelled": 0,
"result_uri": "/jobs/uuid/result"
}
```
`GET /jobs/{job_id}/result` downloads that final coordinator-owned CSV. Before
the job is completed (or for a failed/cancelled job), it returns `404`.
## Stop a job
```http
+4 -3
View File
@@ -56,9 +56,10 @@ Response: `{ "worker_id": "<uuid>", "heartbeat_interval_seconds": 15 }`.
- **Keep `worker_id`**. Use it as your identity in every later call. Using the
registered UUID is what lets the coordinator track your liveness (it marks
workers offline after they go silent).
- Current diagnostic uploads use `similarity-search` with `query_smiles`. The
reference worker accepts the legacy `similarity_search` spelling too. Do not
advertise `similarity-graph` until CTX-10 implements cross-shard pair planning.
- Current distributed uploads use `similarity-search` with `query_smiles`; the
coordinator merges completed shard candidates into a final CSV. The reference
worker accepts the legacy `similarity_search` spelling too. Do not advertise
`similarity-graph` until CTX-10 implements cross-shard pair planning.
## 2. Claim a task
+29 -4
View File
@@ -97,8 +97,9 @@ paths:
multipart/form-data. The text fields (`workload`, `parameters`,
`chunk_rows`, `max_rows`) MUST precede the `file` part: the file is streamed, not
buffered, so the fields have to be parsed before it arrives. Currently
only diagnostic `similarity-search` with `parameters.query_smiles` is
accepted; distributed graph planning is not implemented.
only `similarity-search` with `parameters.query_smiles` is accepted.
When every shard succeeds, the coordinator merges their candidates into
one final CSV; distributed graph planning is not implemented.
requestBody:
required: true
content:
@@ -124,7 +125,9 @@ paths:
- $ref: "#/components/parameters/JobID"
responses:
"200":
description: Progress counts and derived status.
description: >
Progress counts and derived status. A completed similarity-search
response includes `result_uri` for its final CSV.
content:
application/json:
schema: { $ref: "#/components/schemas/JobProgress" }
@@ -132,6 +135,21 @@ paths:
"401": { $ref: "#/components/responses/Unauthorized" }
"404": { $ref: "#/components/responses/NotFound" }
/jobs/{job_id}/result:
get:
tags: [jobs]
summary: Download a completed job's final result
parameters:
- $ref: "#/components/parameters/JobID"
responses:
"200":
description: Final coordinator-owned CSV.
content:
text/csv:
schema: { type: string, format: binary }
"401": { $ref: "#/components/responses/Unauthorized" }
"404": { $ref: "#/components/responses/NotFound" }
/jobs/{job_id}/cancel:
post:
tags: [jobs]
@@ -483,6 +501,13 @@ components:
completed: { type: integer }
failed: { type: integer }
cancelled: { type: integer }
result_uri:
type: string
description: Present only when the final result is available.
example: /jobs/5a4c3a7f-ccfc-47d6-b78d-2d1fa565bafd/result
error_code:
type: string
description: Sanitized terminal reducer failure code, when applicable.
ClaimRequest:
type: object
@@ -577,7 +602,7 @@ components:
JobStatus:
type: string
enum: [pending, running, completed, failed, cancelled]
enum: [pending, running, reducing, completed, failed, cancelled]
TaskStatus:
type: string