feat(coordinator): upload a dataset and chunk it into shard tasks (CTX-05, part 4)

The coordinator can now ingest a dataset itself, not only accept client-supplied
chunk URIs.

- internal/chunk: a deterministic, generic TSV row splitter — repeats the header
  per shard, buffers one shard at a time, rejects header-only input. Unit-tested.
- POST /jobs/upload (multipart): streams the dataset into an input artifact,
  splits it into shard artifacts, and creates one shard task per shard, all in
  one transaction; blobs are cleaned up if the transaction fails.
- GET /tasks/{id}/input streams a task's input shard back to the worker.
- domain: NewUploadedJob, NewShardTask, Task/Job.InputArtifactID; a shard task's
  input is an artifact, not a URI. Claim response nests input:{uri,sha256} per
  the contract, with uri = /tasks/{id}/input for shards.
- migration 0005 makes input_uri nullable and adds a has-input check.
- The existing URI-based POST /jobs path is untouched; both coexist.
This commit is contained in:
Efremenko Arhip
2026-07-23 16:34:04 +03:00
parent 4a092d2e4e
commit c3243a6b7e
19 changed files with 741 additions and 42 deletions
+25 -7
View File
@@ -18,13 +18,31 @@ const (
// Job is one user submission that fans out into one or more tasks.
type Job struct {
ID uuid.UUID
Workload string
InputURI string
Parameters map[string]any
Status JobStatus
CreatedAt time.Time
CompletedAt *time.Time
ID uuid.UUID
Workload string
InputURI string // external input URI; empty for uploaded datasets
InputArtifactID *uuid.UUID // uploaded input artifact; nil for URI submissions
Parameters map[string]any
Status JobStatus
CreatedAt time.Time
CompletedAt *time.Time
}
// NewUploadedJob builds a job whose input was uploaded to the coordinator. The
// job's id is generated here so the input artifact can reference it; the reverse
// link (jobs.input_artifact_id) is left unset — the input is found via the
// artifact's job_id — which also sidesteps the circular job↔artifact FK.
func NewUploadedJob(workload string, params map[string]any, now time.Time) (*Job, error) {
if workload == "" {
return nil, ErrInvalidInput
}
return &Job{
ID: uuid.New(),
Workload: workload,
Parameters: params,
Status: JobPending,
CreatedAt: now,
}, nil
}
// ChunkSpec describes one piece a job is split into. Callers build these from