Add worker and database implementation briefs
This commit is contained in:
Emil
@@ -0,0 +1,197 @@
|
||||
# Task: Go/PostgreSQL task queue service
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a durable task-queue service in Go, backed by PostgreSQL. It must
|
||||
store task state, atomically lease one pending task to a worker, persist result
|
||||
metadata, and expose the data required by a future stitcher/report stage.
|
||||
|
||||
This task implements the **queue and persistence layer** from the architecture
|
||||
sketch. It does not implement the Worker Daemon or scientific/CV computation.
|
||||
|
||||
## Technical baseline
|
||||
|
||||
- Language: Go 1.22+.
|
||||
- Database: PostgreSQL 15+.
|
||||
- PostgreSQL driver and pool: `github.com/jackc/pgx/v5/pgxpool`.
|
||||
- HTTP server: standard `net/http` is sufficient; do not introduce a framework
|
||||
unless it solves a concrete requirement.
|
||||
- Schema migrations: versioned SQL migrations using `golang-migrate`.
|
||||
- Identifiers: UUID.
|
||||
- Times: timezone-aware UTC timestamps.
|
||||
- API boundary: the Go coordinator owns all database access. Python workers
|
||||
communicate only over HTTP and never receive database credentials.
|
||||
|
||||
If the coordinator application does not yet exist, create it with this minimal
|
||||
layout. Do not add a distributed scheduler.
|
||||
|
||||
```text
|
||||
coordinator/
|
||||
cmd/coordinator/main.go
|
||||
internal/config/
|
||||
internal/httpapi/
|
||||
internal/queue/
|
||||
internal/store/postgres/
|
||||
migrations/
|
||||
```
|
||||
|
||||
`main.go` must create one `pgxpool.Pool`, apply no migrations automatically in
|
||||
production, wire dependencies, and perform graceful shutdown. Migrations are a
|
||||
separate explicit command in CI/deployment.
|
||||
|
||||
## Data model
|
||||
|
||||
### `jobs`
|
||||
|
||||
One user submission that may be split into several tasks.
|
||||
|
||||
| Column | Type | Notes |
|
||||
| --- | --- | --- |
|
||||
| `id` | UUID PK | Generated by the service |
|
||||
| `workload` | text | Allowed workload name |
|
||||
| `input_uri` | text | Original dataset/video location |
|
||||
| `parameters` | JSONB | Validated job-level parameters |
|
||||
| `status` | enum | `pending`, `running`, `completed`, `failed`, `cancelled` |
|
||||
| `created_at`, `completed_at` | timestamptz | UTC |
|
||||
|
||||
### `tasks`
|
||||
|
||||
One independently executable chunk. For an unsplit SciMesh workload, create one
|
||||
task. For a video job, create one task per chunk.
|
||||
|
||||
| Column | Type | Notes |
|
||||
| --- | --- | --- |
|
||||
| `id` | UUID PK | Task identifier returned to workers |
|
||||
| `job_id` | UUID FK | References `jobs.id` |
|
||||
| `chunk_index` | integer | Unique within a job |
|
||||
| `workload` | text | Copied from job or an explicit task override |
|
||||
| `input_uri` | text | Chunk or dataset artifact |
|
||||
| `input_sha256` | text | Required checksum |
|
||||
| `parameters` | JSONB | Task-specific validated parameters |
|
||||
| `status` | enum | `pending`, `leased`, `completed`, `failed`, `cancelled` |
|
||||
| `attempt` | integer | Starts at 0; incremented atomically on claim |
|
||||
| `max_attempts` | integer | Default configurable, e.g. 3 |
|
||||
| `lease_owner` | text nullable | Worker ID |
|
||||
| `lease_expires_at` | timestamptz nullable | UTC |
|
||||
| `result_uri` | text nullable | Result CSV/artifact location |
|
||||
| `result_sha256` | text nullable | Checksum of result artifact |
|
||||
| `metrics` | JSONB nullable | Safe execution metrics |
|
||||
| `error_code`, `error_message` | text nullable | Sanitized failure information |
|
||||
| `created_at`, `started_at`, `completed_at` | timestamptz | UTC |
|
||||
| `version` | integer | Optimistic-concurrency/version marker |
|
||||
|
||||
Required constraints and indexes:
|
||||
|
||||
- unique `(job_id, chunk_index)`;
|
||||
- index for claims: `(status, lease_expires_at, created_at)`;
|
||||
- index on `job_id`;
|
||||
- check `attempt >= 0` and `max_attempts > 0`;
|
||||
- completed tasks must have `result_uri` and `result_sha256`;
|
||||
- a leased task must have `lease_owner` and `lease_expires_at`.
|
||||
|
||||
## Required Go service operations
|
||||
|
||||
Keep HTTP handlers thin. Define a small queue service interface and a PostgreSQL
|
||||
implementation; do not expose `pgx` rows or SQL details to handlers.
|
||||
|
||||
```go
|
||||
type QueueService interface {
|
||||
CreateJobWithTasks(ctx context.Context, input CreateJobInput) (Job, error)
|
||||
ClaimNextTask(ctx context.Context, input ClaimInput) (*ClaimedTask, error)
|
||||
RenewLease(ctx context.Context, input RenewLeaseInput) (ClaimedTask, error)
|
||||
CompleteTask(ctx context.Context, input CompleteTaskInput) (Task, error)
|
||||
FailTask(ctx context.Context, input FailTaskInput) (Task, error)
|
||||
ExpireLeases(ctx context.Context, now time.Time) (int64, error)
|
||||
GetJobStatus(ctx context.Context, jobID uuid.UUID) (JobStatus, error)
|
||||
ListCompletedResults(ctx context.Context, jobID uuid.UUID) ([]ResultManifest, error)
|
||||
}
|
||||
```
|
||||
|
||||
Every database call must receive a request-scoped `context.Context`. Configure
|
||||
connection-pool size, database URL, and query/request timeouts via environment
|
||||
variables. Use parameterized pgx queries only; do not build SQL with string
|
||||
interpolation.
|
||||
|
||||
`claim_next_task` must be atomic and safe with multiple coordinator processes.
|
||||
Use one transaction with `SELECT ... FOR UPDATE SKIP LOCKED`, then update the
|
||||
selected row to `leased`, increment `attempt`, and set the lease fields. Do not
|
||||
implement claiming as `SELECT` followed by a separate unguarded `UPDATE`.
|
||||
|
||||
Pseudo-SQL:
|
||||
|
||||
```sql
|
||||
WITH candidate AS (
|
||||
SELECT id
|
||||
FROM tasks
|
||||
WHERE status = 'pending'
|
||||
AND attempt < max_attempts
|
||||
AND workload = ANY($1)
|
||||
ORDER BY created_at, chunk_index
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
UPDATE tasks
|
||||
SET status = 'leased',
|
||||
attempt = attempt + 1,
|
||||
lease_owner = $2,
|
||||
lease_expires_at = $3,
|
||||
started_at = COALESCE(started_at, $4),
|
||||
version = version + 1
|
||||
FROM candidate
|
||||
WHERE tasks.id = candidate.id
|
||||
RETURNING tasks.*;
|
||||
```
|
||||
|
||||
Expired leases must be returned to `pending` only when attempts remain; otherwise
|
||||
mark the task `failed` with a lease-expired error. This should run before or as
|
||||
part of each claim and as a periodic coordinator maintenance operation.
|
||||
|
||||
## Go coordinator HTTP endpoints
|
||||
|
||||
The Worker Daemon task depends on these endpoints. Claiming mutates state, so it
|
||||
must be `POST` rather than `GET`.
|
||||
|
||||
| Endpoint | Behavior |
|
||||
| --- | --- |
|
||||
| `POST /jobs` | Validate submission, create job and pending tasks transactionally |
|
||||
| `POST /tasks/claim` | Atomically lease one compatible task; `204` when none exists |
|
||||
| `POST /tasks/{task_id}/heartbeat` | Renew the current worker's lease |
|
||||
| `POST /tasks/{task_id}/result` | Idempotently persist a completed result manifest |
|
||||
| `POST /tasks/{task_id}/failure` | Record a safe failure or retryable state |
|
||||
| `GET /jobs/{job_id}` | Return aggregate job/task progress |
|
||||
|
||||
Every task-mutating handler must validate JSON, verify `worker_id` and
|
||||
`attempt`, and map typed service errors to HTTP responses. Reject a stale or
|
||||
foreign lease with `409 Conflict`. Completing the same task/attempt with the
|
||||
same result manifest must be idempotent; a different manifest is a conflict.
|
||||
|
||||
Use JSON request/response DTOs in `internal/httpapi`; map them to typed queue
|
||||
inputs in the handler. Do not return raw database errors to clients. Return a
|
||||
request ID in error responses and emit structured logs with the request ID,
|
||||
task ID, worker ID, and operation.
|
||||
|
||||
## Tests and acceptance criteria
|
||||
|
||||
- `golang-migrate` upgrades an empty PostgreSQL database to the current schema
|
||||
and can roll back the latest migration in a test database.
|
||||
- Creating a job either creates all of its tasks or creates none.
|
||||
- Concurrent claim tests prove that each pending task is leased to one worker
|
||||
only; use a real PostgreSQL instance supplied through `TEST_DATABASE_URL`,
|
||||
not SQLite or an in-memory mock.
|
||||
- Lease expiry returns an unfinished task to the queue or fails it after the
|
||||
final permitted attempt.
|
||||
- A worker cannot renew, fail, or complete a task leased to another worker.
|
||||
- Result submission is idempotent for the same attempt and manifest.
|
||||
- `list_completed_results(job_id)` returns a deterministic order by
|
||||
`chunk_index`, suitable for the future stitcher.
|
||||
- API tests cover `204` for an empty queue, `409` for stale attempts, and
|
||||
aggregate job status.
|
||||
- `go test ./...` passes; run `go vet ./...` in CI.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Python Worker Daemon execution, input download, and result upload;
|
||||
- video chunk generation and trajectory stitching;
|
||||
- authentication provider, UI, object-storage implementation, and PDF report;
|
||||
- network/distributed coordinator deployment beyond the local coordinator
|
||||
process.
|
||||
Reference in New Issue
Block a user