feat(coordinator): running state, worker liveness, request-size limits
Polish pass hardening the queue and closing plan gaps. - Task state machine gains `running`: the first heartbeat moves a task from leased to running (migrations 0006/0007 add the enum value and extend the lease-integrity check). verifyLease, ExpireLease, the reaper SQL, and job progress all treat leased and running alike. - Worker liveness: a heartbeat from a registered worker (UUID worker_id) bumps its last_heartbeat_at online; a second background reaper marks workers offline after WORKER_OFFLINE_AFTER of silence (RunReaper generalized to RunPeriodic). - Request-size limits: JSON bodies capped at 1 MiB; dataset/artifact uploads capped at MAX_UPLOAD_BYTES (default 1 GiB) via http.MaxBytesReader. - Tests cover the running transition, liveness + offline reaper (unit over memstore and integration over Postgres).
This commit is contained in:
@@ -0,0 +1,4 @@
|
||||
-- PostgreSQL cannot drop a single enum value without recreating the type and
|
||||
-- rewriting every dependent column. Leaving 'running' in place is harmless: no
|
||||
-- code writes it after the down of 0007 restores the leased-only transitions.
|
||||
SELECT 1;
|
||||
@@ -0,0 +1,5 @@
|
||||
-- 'running' means the worker has acknowledged start via its first heartbeat.
|
||||
-- Kept in its own migration, without an explicit transaction: an enum value
|
||||
-- added in a transaction cannot be USED in that same transaction, and the next
|
||||
-- migration references it.
|
||||
ALTER TYPE task_status ADD VALUE IF NOT EXISTS 'running';
|
||||
@@ -0,0 +1,8 @@
|
||||
BEGIN;
|
||||
|
||||
ALTER TABLE tasks DROP CONSTRAINT IF EXISTS ck_tasks_leased_owner;
|
||||
ALTER TABLE tasks ADD CONSTRAINT ck_tasks_leased_owner CHECK (
|
||||
status <> 'leased' OR (lease_owner IS NOT NULL AND lease_expires_at IS NOT NULL)
|
||||
);
|
||||
|
||||
COMMIT;
|
||||
@@ -0,0 +1,10 @@
|
||||
BEGIN;
|
||||
|
||||
-- A running task holds a lease just like a leased one, so the lease-integrity
|
||||
-- check must cover both states.
|
||||
ALTER TABLE tasks DROP CONSTRAINT IF EXISTS ck_tasks_leased_owner;
|
||||
ALTER TABLE tasks ADD CONSTRAINT ck_tasks_leased_owner CHECK (
|
||||
status NOT IN ('leased','running') OR (lease_owner IS NOT NULL AND lease_expires_at IS NOT NULL)
|
||||
);
|
||||
|
||||
COMMIT;
|
||||
Reference in New Issue
Block a user