feat(coordinator): running state, worker liveness, request-size limits

Polish pass hardening the queue and closing plan gaps.

- Task state machine gains `running`: the first heartbeat moves a task from
  leased to running (migrations 0006/0007 add the enum value and extend the
  lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job
  progress all treat leased and running alike.
- Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps
  its last_heartbeat_at online; a second background reaper marks workers offline
  after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic).
- Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads
  capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader.
- Tests cover the running transition, liveness + offline reaper (unit over
  memstore and integration over Postgres).
This commit is contained in:
Efremenko Arhip
2026-07-23 17:36:26 +03:00
parent 4fc3c69fdf
commit 3b41455b20
23 changed files with 332 additions and 32 deletions
@@ -0,0 +1,4 @@
-- PostgreSQL cannot drop a single enum value without recreating the type and
-- rewriting every dependent column. Leaving 'running' in place is harmless: no
-- code writes it after the down of 0007 restores the leased-only transitions.
SELECT 1;
@@ -0,0 +1,5 @@
-- 'running' means the worker has acknowledged start via its first heartbeat.
-- Kept in its own migration, without an explicit transaction: an enum value
-- added in a transaction cannot be USED in that same transaction, and the next
-- migration references it.
ALTER TYPE task_status ADD VALUE IF NOT EXISTS 'running';
@@ -0,0 +1,8 @@
BEGIN;
ALTER TABLE tasks DROP CONSTRAINT IF EXISTS ck_tasks_leased_owner;
ALTER TABLE tasks ADD CONSTRAINT ck_tasks_leased_owner CHECK (
status <> 'leased' OR (lease_owner IS NOT NULL AND lease_expires_at IS NOT NULL)
);
COMMIT;
@@ -0,0 +1,10 @@
BEGIN;
-- A running task holds a lease just like a leased one, so the lease-integrity
-- check must cover both states.
ALTER TABLE tasks DROP CONSTRAINT IF EXISTS ck_tasks_leased_owner;
ALTER TABLE tasks ADD CONSTRAINT ck_tasks_leased_owner CHECK (
status NOT IN ('leased','running') OR (lease_owner IS NOT NULL AND lease_expires_at IS NOT NULL)
);
COMMIT;