feat(coordinator): upload a dataset and chunk it into shard tasks (CTX-05, part 4)
The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.
- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
splits it into shard artifacts, and creates one shard task per shard, all in
one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
This commit is contained in:
@@ -34,6 +34,41 @@ Content-Type: application/json
|
||||
|
||||
@workerId = {{register.response.body.worker_id}}
|
||||
|
||||
### 0b. Upload a dataset — the coordinator splits it into shard tasks (201)
|
||||
# Text fields first, the file part last (it is streamed, not buffered).
|
||||
# @name uploadJob
|
||||
POST {{host}}/jobs/upload
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: multipart/form-data; boundary=----scimesh
|
||||
|
||||
------scimesh
|
||||
Content-Disposition: form-data; name="workload"
|
||||
|
||||
similarity_search
|
||||
------scimesh
|
||||
Content-Disposition: form-data; name="parameters"
|
||||
|
||||
{"top_k":10}
|
||||
------scimesh
|
||||
Content-Disposition: form-data; name="chunk_rows"
|
||||
|
||||
2
|
||||
------scimesh
|
||||
Content-Disposition: form-data; name="file"; filename="chembl.tsv"
|
||||
Content-Type: text/tab-separated-values
|
||||
|
||||
id smiles
|
||||
A CC
|
||||
B CCC
|
||||
C CCCC
|
||||
D CCCCC
|
||||
------scimesh--
|
||||
|
||||
### Download a task's input shard (200) — taskId must be a shard task from an
|
||||
### uploaded job (claim one first; its input.uri is /tasks/{id}/input).
|
||||
GET {{host}}/tasks/{{taskId}}/input
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
### 1. Create a job and its chunks (201)
|
||||
# The coordinator splits the submission into one task per chunk, transactionally.
|
||||
# @name createJob
|
||||
|
||||
Reference in New Issue
Block a user