feat(coordinator): upload a dataset and chunk it into shard tasks (CTX-05, part 4)

The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.

- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
  per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
  splits it into shard artifacts, and creates one shard task per shard, all in
  one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
  input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
  the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
This commit is contained in:
Efremenko Arhip
2026-07-23 16:34:04 +03:00
parent 4a092d2e4e
commit c3243a6b7e
19 changed files with 741 additions and 42 deletions
+35
View File
@@ -34,6 +34,41 @@ Content-Type: application/json
@workerId = {{register.response.body.worker_id}}
### 0b. Upload a dataset — the coordinator splits it into shard tasks (201)
# Text fields first, the file part last (it is streamed, not buffered).
# @name uploadJob
POST {{host}}/jobs/upload
Authorization: Bearer {{token}}
Content-Type: multipart/form-data; boundary=----scimesh
------scimesh
Content-Disposition: form-data; name="workload"
similarity_search
------scimesh
Content-Disposition: form-data; name="parameters"
{"top_k":10}
------scimesh
Content-Disposition: form-data; name="chunk_rows"
2
------scimesh
Content-Disposition: form-data; name="file"; filename="chembl.tsv"
Content-Type: text/tab-separated-values
id smiles
A CC
B CCC
C CCCC
D CCCCC
------scimesh--
### Download a task's input shard (200) — taskId must be a shard task from an
### uploaded job (claim one first; its input.uri is /tasks/{id}/input).
GET {{host}}/tasks/{{taskId}}/input
Authorization: Bearer {{token}}
### 1. Create a job and its chunks (201)
# The coordinator splits the submission into one task per chunk, transactionally.
# @name createJob