Add job cancellation and dataset row limit
coordinator / test (push) Canceled after 0s

This commit is contained in:
Emil
2026-07-23 23:14:25 +03:00
parent 6bac7dad3c
commit 7547a30bde
30 changed files with 438 additions and 54 deletions
+23 -3
View File
@@ -52,9 +52,11 @@ Content-Type: multipart/form-data
```
Fields, in order (text fields first, file last — the file is streamed):
`workload`, `parameters` (JSON), `chunk_rows` (int, default 1000), and the file
part `file`. The coordinator stores the input, splits the TSV into shard
artifacts (header repeated per shard), and creates one task per shard.
`workload`, `parameters` (JSON), `chunk_rows` (int, default 1000), optional
`max_rows` (positive int), and the file part `file`. `max_rows` limits the
leading data rows that become shards; it does not change the stored source
artifact. The coordinator splits the selected TSV rows into shard artifacts
(header repeated per shard) and creates one task per shard.
`201`:
@@ -65,6 +67,24 @@ artifacts (header repeated per shard), and creates one task per shard.
Each resulting task's claim response carries `input.uri = /tasks/{id}/input`,
served by §5.4.
## Stop a job
```http
POST /jobs/{job_id}/cancel
Authorization: Bearer <token>
```
The coordinator transactionally marks every pending, leased, or running shard
as `cancelled`, invalidates its lease, and marks the job `cancelled`. Completed
and terminally failed shards remain as history. Repeating a cancellation of an
already cancelled job is safe.
`200`:
```json
{ "job_id": "uuid", "status": "cancelled", "cancelled_tasks": 12 }
```
## Register worker
```http
+32 -2
View File
@@ -95,7 +95,7 @@ paths:
summary: Upload a dataset; the coordinator chunks it into shard tasks
description: >
multipart/form-data. The text fields (`workload`, `parameters`,
`chunk_rows`) MUST precede the `file` part: the file is streamed, not
`chunk_rows`, `max_rows`) MUST precede the `file` part: the file is streamed, not
buffered, so the fields have to be parsed before it arrives.
requestBody:
required: true
@@ -130,6 +130,23 @@ paths:
"401": { $ref: "#/components/responses/Unauthorized" }
"404": { $ref: "#/components/responses/NotFound" }
/jobs/{job_id}/cancel:
post:
tags: [jobs]
summary: Cancel a job and invalidate all unfinished task leases
parameters:
- $ref: "#/components/parameters/JobID"
responses:
"200":
description: The job is cancelled. Completed and terminally failed tasks remain unchanged.
content:
application/json:
schema: { $ref: "#/components/schemas/CancelJobResponse" }
"400": { $ref: "#/components/responses/BadRequest" }
"401": { $ref: "#/components/responses/Unauthorized" }
"404": { $ref: "#/components/responses/NotFound" }
"409": { $ref: "#/components/responses/Conflict" }
/tasks/claim:
post:
tags: [tasks]
@@ -429,6 +446,11 @@ components:
type: integer
description: Data rows per shard. Default 1000.
example: 1000
max_rows:
type: integer
minimum: 1
description: Optional leading data-row limit for a small pipeline check.
example: 500
file:
type: string
format: binary
@@ -441,6 +463,13 @@ components:
task_count: { type: integer, example: 3 }
input_artifact_id: { type: string, format: uuid }
CancelJobResponse:
type: object
properties:
job_id: { type: string, format: uuid }
status: { type: string, enum: [cancelled] }
cancelled_tasks: { type: integer }
JobProgress:
type: object
properties:
@@ -451,6 +480,7 @@ components:
leased: { type: integer }
completed: { type: integer }
failed: { type: integer }
cancelled: { type: integer }
ClaimRequest:
type: object
@@ -549,4 +579,4 @@ components:
TaskStatus:
type: string
enum: [pending, leased, completed, failed, cancelled]
enum: [pending, leased, running, completed, failed, cancelled]
+3
View File
@@ -173,6 +173,9 @@ Rules:
- TSV file, required, streamed; show expected columns
`chembl_id` and `canonical_smiles`.
- `chunk_rows`: integer 1--100000, default 1000.
- `max_rows`: optional positive integer. The coordinator creates shards only
from the first N data rows, so a user can test a large upload without
creating thousands of tasks. It does not truncate the stored source blob.
- optional human-readable run name is a later schema/API addition; v1 does not
silently store it.
- display file name and client-side size only as convenience; server limits and