feat(coordinator): upload a dataset and chunk it into shard tasks (CTX-05, part 4)
The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.
- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
splits it into shard artifacts, and creates one shard task per shard, all in
one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
This commit is contained in:
+24
-1
@@ -24,7 +24,8 @@ must be updated in the same change as any behaviour it describes.
|
||||
| `GET /jobs/{id}` | progress | ✅ done |
|
||||
| `PUT /tasks/{id}/artifacts/{name}` | upload partial | ✅ done |
|
||||
| `GET /artifacts/{id}/download` | download by id | ✅ done |
|
||||
| `GET /tasks/{id}/input` | download shard | ❌ needs input artifacts (upload+chunking) |
|
||||
| `POST /jobs/upload` | upload dataset, coordinator chunks it | ✅ done |
|
||||
| `GET /tasks/{id}/input` | download shard | ✅ done |
|
||||
|
||||
---
|
||||
|
||||
@@ -37,6 +38,28 @@ GET /health
|
||||
`200 {"status":"ok"}` when the database is reachable; `503 {"status":"unavailable"}`
|
||||
otherwise. Unauthenticated.
|
||||
|
||||
## Submit a dataset (submitter-side)
|
||||
|
||||
```http
|
||||
POST /jobs/upload
|
||||
Authorization: Bearer <token>
|
||||
Content-Type: multipart/form-data
|
||||
```
|
||||
|
||||
Fields, in order (text fields first, file last — the file is streamed):
|
||||
`workload`, `parameters` (JSON), `chunk_rows` (int, default 1000), and the file
|
||||
part `file`. The coordinator stores the input, splits the TSV into shard
|
||||
artifacts (header repeated per shard), and creates one task per shard.
|
||||
|
||||
`201`:
|
||||
|
||||
```json
|
||||
{ "job_id": "uuid", "task_count": 3, "input_artifact_id": "uuid" }
|
||||
```
|
||||
|
||||
Each resulting task's claim response carries `input.uri = /tasks/{id}/input`,
|
||||
served by §5.4.
|
||||
|
||||
## Register worker
|
||||
|
||||
```http
|
||||
|
||||
Reference in New Issue
Block a user