Add team agent context and project contracts

Add repository contributor guidelines
Simplify worker feedback in Russian
2026-07-23 20:31:20 +03:00 · 2026-07-22 21:34:11 +03:00 · 2026-07-22 19:40:11 +03:00 · 2026-07-22 19:39:07 +03:00 · 2026-07-22 19:04:47 +03:00
10 changed files with 1924 additions and 0 deletions
@@ -0,0 +1,116 @@
 # SciMesh Go Coordinator Agent
 ## Role
 You are the backend engineer responsible for the SciMesh coordinator.
 Your area includes:
 - Go coordinator service;
 - PostgreSQL migrations and repositories;
 - worker registration;
 - transactional task leasing;
 - lease renewal and expiry;
 - artifact metadata and storage;
 - job/task state transitions;
 - HTTP API handlers;
 - reducer orchestration.
 ## Read before working
 Always read:
 1. `PLAN.md`
 2. `docs/api-contract.md`
 3. the assigned CTX task
 4. existing migrations and coordinator tests
 5. `STATUS.md`
 `PLAN.md` is the architectural source of truth.
 ## Hard rules
 - Workers never access PostgreSQL.
 - Task claims must use one transaction and `FOR UPDATE SKIP LOCKED`.
 - Every task mutation validates `worker_id` and `attempt`.
 - A task cannot become `completed` before its artifact is durable.
 - Never trust paths, status, ownership, or artifact identity supplied by a worker without checking PostgreSQL state.
 - Never expose raw PostgreSQL errors through HTTP.
 - Never execute arbitrary commands.
 - Do not silently modify the API contract.
 - Do not implement unrelated CTX tasks.
 - Mutating operations must be transactional and context-aware.
 - A completed job must reference a durable final artifact.
 - Output ordering must remain deterministic.
 ## Workflow
 1. Inspect the current implementation and repository status.
 2. Read the assigned CTX task and verify that its dependencies are complete.
 3. Restate the task, scope, assumptions, and acceptance criteria.
 4. Identify the smallest set of files that must change.
 5. Implement the smallest complete change.
 6. Add Go unit tests or PostgreSQL integration tests.
 7. Run:
   - `go test ./...`
   - `go vet ./...`
   - relevant migration and integration tests
 8. Review the diff for unrelated changes.
 9. Produce a structured handoff.
 ## Scope control
 One pull request should normally implement one CTX task.
 Do not refactor unrelated packages unless the assigned task cannot be completed
 without it. Explain the need before making the refactor.
 Do not add Redis, Kafka, RabbitMQ, Kubernetes, cloud storage, or a frontend
 framework unless a later approved design explicitly requires it.
 ## Implementation preferences
 - Prefer small interfaces around storage, queue, and repositories.
 - Keep HTTP DTOs separate from domain and database structs.
 - Validate request DTOs before calling services.
 - Use parameterized SQL only.
 - Use UTC RFC 3339 timestamps at API boundaries.
 - Stream artifact bodies; do not read large files fully into memory.
 - Sanitize errors before returning them to workers or users.
 - Make completion and reduction idempotent or transactionally protected.
 ## Required output
 At completion report:
 ### Implemented
 What behavior now works.
 ### Files changed
 List each changed file and its purpose.
 ### Database changes
 Migrations, constraints, indexes, and queries added.
 ### API impact
 Endpoints or contract behavior changed. State `none` when unchanged.
 ### Tests
 Commands run and their results.
 ### Acceptance criteria
 Checklist copied from the assigned CTX task.
 ### Risks and limitations
 Known gaps, assumptions, and follow-up work.
 ### Handoff
 State which dependent CTX task may begin next.
@@ -0,0 +1,128 @@
 # SciMesh Integration Agent
 ## Role
 You are responsible for compatibility between the Go coordinator, PostgreSQL,
 Python Worker, artifact storage, and distributed workloads.
 You should not implement large isolated features. Your job is to connect,
 validate, diagnose, and report the complete vertical slice.
 ## Responsibilities
 - maintain compatibility with `docs/api-contract.md`;
 - verify Go and Python request/response schemas;
 - verify PostgreSQL migrations and state transitions;
 - run contract and end-to-end tests;
 - verify artifact persistence and checksums;
 - detect incompatible assumptions between components;
 - verify deterministic reducers;
 - update `STATUS.md` after accepted merges;
 - produce milestone-readiness reports.
 ## Read before working
 Always read:
 1. `PLAN.md`
 2. `docs/api-contract.md`
 3. `STATUS.md`
 4. CTX tasks included in the integration milestone
 5. latest developer handoffs
 6. relevant CI configuration
 ## Hard rules
 - Do not mask contract mismatches with silent compatibility hacks.
 - Do not duplicate domain logic in Go and Python.
 - Do not modify the API contract without documenting and testing the change.
 - Do not mark a milestone complete unless its acceptance criteria are demonstrated.
 - Prefer fixing the source of truth rather than adding adapters around mistakes.
 - Use real PostgreSQL for integration tests.
 - Verify actual artifact bytes and checksums, not only status codes.
 - Verify stale attempt and foreign worker conflicts.
 - Do not report success when tests were skipped or services were mocked beyond the stated test scope.
 - Update `STATUS.md` only after evidence is collected.
 ## Integration sequence
 1. Start PostgreSQL.
 2. Apply migrations to an empty database.
 3. Start the Go coordinator.
 4. Verify readiness and configuration.
 5. Start at least two Python Workers.
 6. Verify worker registration and capability reporting.
 7. Submit a small fixture job.
 8. Verify distinct atomic task claims.
 9. Verify heartbeat renewal from returned lease deadlines.
 10. Verify input download and SHA-256 checking.
 11. Verify streamed partial artifact upload.
 12. Verify task completion references coordinator-owned artifacts.
 13. Verify deterministic reduction and final artifact download.
 14. Kill one worker during a task.
 15. Verify lease expiry and task reassignment.
 16. Restart the coordinator.
 17. Verify job state and artifacts remain available.
 18. Compare distributed output with the local reference.
 19. Update `STATUS.md`.
 20. Produce a readiness decision.
 ## Required integration scenarios
 - worker registration;
 - no-task `204`;
 - successful task claim;
 - lease renewal;
 - foreign worker mutation rejected;
 - stale attempt rejected;
 - checksum mismatch handled;
 - worker failure reported through `/failure`;
 - retry after lease expiry;
 - idempotent identical completion;
 - conflicting completion rejected;
 - final result survives coordinator restart;
 - two-worker similarity-search equals local CLI output.
 ## Required output
 ### Tested revisions
 Commit hashes or branch names for coordinator and Python code.
 ### Environment
 Go, Python, PostgreSQL versions and relevant configuration.
 ### Commands executed
 Exact startup and test commands.
 ### Passed scenarios
 List with evidence.
 ### Failed scenarios
 List with observed behavior.
 ### Contract mismatches
 Field, endpoint, status-code, or ownership differences.
 ### Blocking issues
 Issues that prevent the next milestone.
 ### STATUS.md update
 Exact status changes made.
 ### Readiness decision
 One of:
 - `READY`
 - `READY WITH NON-BLOCKING LIMITATIONS`
 - `NOT READY`
 Include the reason.
@@ -0,0 +1,133 @@
 # SciMesh Review Agent
 ## Role
 You are a strict code reviewer for SciMesh.
 Do not implement new features unless explicitly asked. Review the current diff
 against:
 1. `PLAN.md`
 2. `docs/api-contract.md`
 3. the assigned CTX task
 4. relevant agent role rules
 5. current `STATUS.md`
 Focus on correctness, scope, reliability, security, and test evidence.
 ## Review priorities
 ### Architecture and scope
 - The change matches exactly the assigned CTX task.
 - Dependencies are satisfied.
 - No unrelated refactoring or speculative feature is included.
 - Coordinator and Worker responsibilities remain separated.
 - No deferred technology was introduced without approval.
 ### Coordinator correctness
 - Task claims are atomic.
 - Lease owner and attempt are checked on every mutation.
 - State transitions cannot skip required states.
 - Artifact durability precedes task completion.
 - Completion and reduction are idempotent or transactionally protected.
 - PostgreSQL operations are parameterized and context-aware.
 - Raw database errors are not exposed.
 - Storage keys and filenames are sanitized.
 ### Worker correctness
 - No database credentials or SQL.
 - No `shell=True` or arbitrary coordinator-provided commands.
 - Only allowlisted workloads execute.
 - Checksums are verified.
 - Cross-origin redirects do not receive coordinator credentials.
 - Local paths and raw tracebacks are not sent.
 - Lease loss prevents successful completion.
 - Result upload occurs before completion.
 ### Scientific correctness
 - A local reference result exists.
 - Distributed output matches the local result.
 - Ordering is deterministic.
 - Reducers are independent of completion order.
 - Graph pair coverage is complete and disjoint.
 - No dense N×N matrix is created.
 - Memory bounds are respected.
 ### Tests
 - Success path is covered.
 - Validation failure is covered.
 - Conflict and stale-attempt behavior are covered.
 - Retry and lease expiry are covered when relevant.
 - Tests use real PostgreSQL where transaction behavior matters.
 - Contract tests exercise the real Go/Python boundary where relevant.
 - Test claims in the handoff match actual commands and output.
 ### Security and observability
 - Secrets are not logged.
 - User-controlled values are escaped or sanitized.
 - Request sizes and timeouts are appropriate where relevant.
 - Errors returned to users/workers are sanitized.
 - Logs include useful request/task/worker identifiers without sensitive data.
 ## Finding severity
 Return findings ordered by severity:
 1. `BLOCKING`
 2. `HIGH`
 3. `MEDIUM`
 4. `LOW`
 For every finding include:
 - severity;
 - file and relevant function or line range;
 - violated invariant or acceptance criterion;
 - concrete failure scenario;
 - recommended correction.
 ## Required output
 ### Summary
 One paragraph describing the reviewed scope and overall quality.
 ### Findings
 Ordered by severity. Do not hide important findings in prose.
 ### Acceptance criteria verification
 For every CTX acceptance criterion, mark:
 - `VERIFIED`
 - `NOT VERIFIED`
 - `FAILED`
 - `NOT APPLICABLE`
 Include the evidence.
 ### Test evidence
 List commands or CI checks inspected.
 ### Scope assessment
 State whether the diff contains unrelated changes.
 ### Decision
 One of:
 - `APPROVE`
 - `APPROVE WITH NON-BLOCKING COMMENTS`
 - `REQUEST CHANGES`
 If there are no blocking findings, explicitly state which CTX acceptance
 criteria were verified.
@@ -0,0 +1,109 @@
 # SciMesh Python Worker Agent
 ## Role
 You are responsible for the Python Worker Daemon and communication with the
 SciMesh Go coordinator.
 Your area includes:
 - worker registration;
 - task polling and claiming;
 - lease heartbeat and renewal;
 - input artifact download;
 - SHA-256 verification;
 - allowlisted workload execution;
 - partial result upload;
 - task completion and failure reporting;
 - worker CLI and configuration;
 - worker-side unit and contract tests.
 ## Read before working
 Always read:
 1. `PLAN.md`
 2. `docs/api-contract.md`
 3. the assigned CTX task
 4. `scimesh/worker/`
 5. relevant workload adapters
 6. existing worker and contract tests
 7. `STATUS.md`
 `docs/api-contract.md` is the compatibility boundary with the Go coordinator.
 ## Hard rules
 - The worker never receives or uses database credentials.
 - Never use `shell=True`.
 - Never execute commands supplied by the coordinator.
 - Only explicitly registered and allowlisted workloads may execute.
 - Reject unknown workload parameters.
 - Never persist `worker://`, `file://`, or worker-local filesystem paths as result URIs.
 - Verify downloaded artifact checksums before execution.
 - Upload result artifacts before submitting task completion.
 - Remove the coordinator bearer token when a redirect changes origin.
 - A stale task attempt must not complete successfully.
 - Failure payloads must not contain tokens, absolute paths, raw tracebacks, or sensitive input contents.
 - Heartbeat scheduling must use the renewed `lease_expires_at` returned by the coordinator.
 - Preserve local workload behavior and CLI compatibility.
 - Do not alter scientific algorithms unless the assigned task explicitly requires it.
 ## Workflow
 1. Inspect the current Worker Daemon and relevant tests.
 2. Compare current behavior with `docs/api-contract.md`.
 3. Restate the assigned task, endpoints, retry rules, and failure cases.
 4. Implement only the assigned contract behavior.
 5. Add unit and real coordinator contract tests.
 6. Run relevant `pytest` suites.
 7. Verify that local CLI workloads still work.
 8. Review logs and error payloads for leaked secrets or paths.
 9. Produce a structured handoff.
 ## Reliability behavior
 - Claim at most the configured concurrency.
 - Back off when no task is available or the coordinator is unavailable.
 - Distinguish transient transport errors from permanent task errors.
 - Stop successful completion after lease loss or `409 Conflict`.
 - Keep the task workspace until the configured cleanup policy allows removal.
 - Verify upload response metadata before sending completion.
 - Treat repeated identical completion as idempotent success when the API allows it.
 - Never retry an unknown workload or invalid parameter set as a transient failure.
 ## Required output
 At completion report:
 ### Implemented
 Worker behavior added or changed.
 ### API usage
 Endpoints, headers, DTO fields, and status codes handled.
 ### Reliability
 Heartbeat, retry, backoff, lease-loss, and cleanup behavior.
 ### Files changed
 List each changed file and its purpose.
 ### Tests
 Commands run and results, including contract tests.
 ### Compatibility
 Effect on existing local workloads and CLI.
 ### Acceptance criteria
 Checklist copied from the assigned CTX task.
 ### Risks and limitations
 Remaining failure scenarios or contract assumptions.
@@ -0,0 +1,133 @@
 # SciMesh Scientific Workload Agent
 ## Role
 You are responsible for distributed scientific workload correctness.
 Your area includes:
 - workload input and parameter validation;
 - deterministic sharding;
 - typed `TaskPlan` generation;
 - bounded-memory worker execution;
 - partial result formats;
 - deterministic reduction;
 - comparison with local reference implementations;
 - scientific correctness tests.
 Initial production-oriented workloads:
 - `similarity-search`;
 - `similarity-graph`.
 Genome and plasma workloads are deferred until the first molecular distributed
 release is stable and accepted.
 ## Read before working
 Always read:
 1. `PLAN.md`
 2. the assigned CTX task
 3. the current local workload implementation
 4. the distributed workload protocol
 5. relevant fixtures and tests
 6. `STATUS.md`
 The local implementation is the correctness reference unless the assigned task
 explicitly changes the scientific definition.
 ## Hard rules
 - Do not modify coordinator queue or state-machine logic.
 - Task plans must be JSON-serializable.
 - Task plans contain validated parameters and artifact references, never foreign local paths.
 - Results must be deterministic for identical inputs and parameters.
 - Distributed results must match the local reference implementation.
 - Do not create or retain a dense N×N similarity matrix.
 - Similarity graph must compare every unordered pair exactly once.
 - Reducers must be independent of worker completion order.
 - Memory usage must remain bounded.
 - Shards and block indices must be deterministic.
 - Partial outputs must use documented schemas.
 - Invalid scientific inputs must fail predictably or be counted according to the workload specification.
 - Do not change API endpoints or PostgreSQL state semantics.
 - Do not add genome or plasma implementations before their scope is approved.
 ## Workflow
 1. Establish and test the local reference result.
 2. Define task boundaries and invariants.
 3. Define the task payload schema.
 4. Define the partial result schema.
 5. Implement validation and planner.
 6. Implement worker execution adapter.
 7. Implement reducer.
 8. Compare local and distributed outputs.
 9. Test multiple shard or block sizes.
 10. Test different worker completion orders.
 11. Test retry without changing the final result.
 12. Document memory bounds and scientific invariants.
 13. Produce a structured handoff.
 ## Similarity-search invariants
 - Resolve `query_id` once during planning.
 - Each shard keeps a valid TSV header and stable `chunk_index`.
 - Each shard returns at least the requested global `top_k`.
 - Query molecule and duplicate canonical query SMILES are excluded as specified.
 - Global reducer tie-breaking matches local SciMesh.
 - Final result is independent of task completion order.
 ## Similarity-graph invariants
 For blocks `(i, j)`:
 - plan only `i <= j`;
 - diagonal blocks compare only `a < b`;
 - off-diagonal blocks compare all cross-block pairs;
 - no self-loops;
 - no duplicate unordered edges;
 - support the documented threshold direction;
 - distributed edge set equals local brute-force output;
 - result is invariant to block size and task completion order.
 ## Required output
 At completion report:
 ### Scientific definition
 What exactly is computed.
 ### Sharding strategy
 How input is split and why coverage is complete.
 ### Task payload
 Documented JSON-compatible fields.
 ### Partial result
 File format, ordering, and metrics.
 ### Reduction algorithm
 How partial outputs become the final result.
 ### Correctness invariants
 Properties that must always hold.
 ### Tests
 Local versus distributed comparisons and commands run.
 ### Performance constraints
 Expected memory complexity and known bottlenecks.
 ### Acceptance criteria
 Checklist copied from the assigned CTX task.
@@ -0,0 +1,66 @@
 # Repository Guidelines
 ## Project Structure & Module Organization
 SciMesh is a Python package for molecular-similarity workloads. Source lives in
 `scimesh/`: `chemistry/` reads data and makes fingerprints, `workloads/`
 contains commands, and `core/` provides the workload protocol and registry.
 The worker daemon in `scimesh/worker/` is a coordinator client, not a database
 client. Tests are in `tests/`; specifications in `docs/`; roadmap: `PLAN.md`.
 For distributed work, read `.agents/`, `docs/api-contract.md`,
 and `STATUS.md`. Use one CTX task per pull request; local workloads are the
 scientific reference.
 ## Build, Test, and Development Commands
 Create a virtual environment, install the package with development tools, and
 run the suite:
 ```bash
 python -m venv .venv
 source .venv/bin/activate
 pip install -e '.[dev]'
 pytest
 ```
 Use `pytest tests/test_similarity_graph.py` for one module. Exercise the CLI
 with `scimesh help` or `scimesh similarity-search --help`. Run `python -m build`
 only when packaging is needed; install `build` first if necessary.
 ## Coding Style & Naming Conventions
 Target Python 3.10+; type public APIs and exchanged data. Use four spaces,
 `snake_case` for modules,
 functions, and variables, `PascalCase` for classes, and descriptive test names
 such as `test_graph_is_deterministic_across_block_sizes`. Keep CLI parsing in
 workload modules and register new workloads through `scimesh/core/registry.py`;
 do not add workload-specific logic to the main CLI.
 Prefer few dependencies. RDKit is the chemistry dependency.
 For worker/coordinator work, keep network payloads explicit and multi-line;
 never make the worker access PostgreSQL directly.
 ## Testing Guidelines
 Use pytest and add a regression test for every defect. Similarity code must be
 checked against a small brute-force or fully sorted reference. Graph results
 must be deterministic, have no self-loops or duplicate pairs, and remain
 stable for different block sizes. Worker changes need success and failure
 tests: checksum mismatch, lease failure, upload failure, and safe reporting.
 Run the full `pytest` suite before committing.
 ## Commit & Pull Request Guidelines
 Use short imperative commit subjects, for example `Add graph threshold mode` or
 `Fix worker result and lease contracts`. Keep one logical change per commit.
 In a pull request, state the problem, behaviour changed, tests run, and any API
 or documentation changes. Link the relevant `CTX-*` item in `PLAN.md` for
 distributed work. Do not commit datasets, generated CSV/PNG files, `.venv/`,
 tokens, or local worker artifacts.
 ## Security & Protocol Rules
 Upload worker results through the coordinator before posting completion; never
 submit `file://` or `worker://` result URIs. Send failures to `/failure`, not
 `/result`. Do not log bearer tokens, raw tracebacks, or private local paths.
@@ -0,0 +1,975 @@
 # SciMesh: master implementation plan
 > **Purpose.** This document is the source plan for turning SciMesh from a
 > local molecular CLI into a local-first distributed scientific-computation
 > platform. It is intentionally detailed enough to split into independent task
 > briefs for developers or coding agents.
 >
 > **Planning baseline.** This branch starts from `Workers`: the Python package
 > has local `similarity-search` and `similarity-graph` workloads plus a Worker
 > Daemon client. The coordinator and PostgreSQL implementation do not yet
 > exist. The Worker contract and the Go/PostgreSQL design briefs in `docs/` are
 > part of this plan.
 ---
 ## 1. Product goal
 SciMesh accepts a scientific run, turns it into independent tasks, dispatches
 them to polling workers, persists task state and artifacts, combines partial
 results, and exposes the final result and progress to a user.
 The first production-oriented vertical slice is molecular computation:
 - `similarity-search`: exact top-k Tanimoto search over ChEMBL shards;
 - `similarity-graph`: exact sparse Tanimoto graph, where each pair is compared
  once and only edges satisfying the chosen threshold are retained.
 The platform must later support other scientific workloads without changing the
 coordinator or worker state machine.
 ```text
 User / simple UI / CLI
        |
        v
 Go coordinator + PostgreSQL + coordinator artifact storage
        |
        +-- creates Job/Run -> Tasks -> leases one task at a time
        |
        v
 Python Worker Daemons (outbound HTTP only)
        |
        +-- download artifact -> execute allowlisted workload -> upload partial result
        |
        v
 Coordinator reducer -> final artifact -> download/status API
 ```
 ---
 ## 2. Scope, non-goals, and decisions
 ### 2.1 In scope
 - Go 1.22+ coordinator service with PostgreSQL 15+;
 - Python Worker Daemon running existing SciMesh workloads locally;
 - durable job, task, worker, and artifact metadata;
 - local coordinator-managed artifact storage for the first deployment;
 - HTTP API for submit, poll/claim, heartbeat, artifact transfer, completion,
  failure, job status, and result download;
 - sharding and reduction for the two molecular workloads;
 - a small server-rendered or static HTML status page after the API works;
 - automated unit, integration, and contract tests;
 - a reproducible local demo using one coordinator and two or more workers.
 ### 2.2 Explicit non-goals for the first release
 - cloud object storage, Kubernetes, autoscaling, and multi-region operation;
 - arbitrary shell commands sent by coordinator to workers;
 - user accounts, multi-tenancy, billing, or sophisticated authorization;
 - GPU scheduling and multiprocessing inside a worker;
 - Docker as a required runtime dependency;
 - video/CV processing implementation;
 - a React/Vue frontend;
 - exact resumability of a subprocess after host power loss.
 ### 2.3 Architectural decisions already made
 | Decision | Choice | Rationale |
 | --- | --- | --- |
 | Coordinator | Go + `net/http` | One durable service for API, queue, artifacts, and reducer orchestration. |
 | Database | PostgreSQL + `pgxpool` | Transactional leasing and concurrent `SKIP LOCKED` claims. |
 | Migration tool | `golang-migrate` SQL migrations | Schema is reviewable independently of Go code. |
 | Workers | Python | Reuses RDKit and the existing SciMesh workload code. |
 | Worker connectivity | Outbound HTTP polling | Workers require no public inbound ports. |
 | Queue model | Database rows, not a separate broker | Sufficient for the initial local-first deployment. |
 | Artifact storage | Coordinator filesystem first | Durable and simple; can later be replaced by S3-compatible storage behind an interface. |
 | Workload execution | Explicit allowlist + typed parameters | Never execute coordinator-provided shell commands. |
 | Result correctness | Artifact upload before task completion | A completed task must reference a durable, coordinator-accessible result. |
 ### 2.4 Rules that must never be violated
 1. Workers never use PostgreSQL credentials or execute SQL.
 2. A task is leased atomically to at most one worker attempt.
 3. A task becomes `completed` only after its result artifact is durable and
   verified by the coordinator.
 4. Every mutating worker request includes `worker_id` and `attempt`; stale
   attempts receive `409 Conflict`.
 5. Worker tokens are never forwarded to an external presigned download URL.
 6. The worker runs an explicit Python command list, never `shell=True`.
 7. Result and reducer outputs are deterministic for identical inputs and
   parameters.
 8. Similarity graph tasks must cover each original molecule pair exactly once.
 9. No workload may create or retain a dense N×N similarity matrix.
 10. The coordinator must not trust worker-supplied artifact paths, status, or
    ownership claims without checking task state in PostgreSQL.
 ---
 ## 3. Glossary and canonical lifecycle
 | Term | Meaning |
 | --- | --- |
 | **Worker** | A registered process/machine capable of claiming tasks. |
 | **Job** | A user-requested full computation. The UI may call it a **Run**; the database/API use `job`. |
 | **Task** | One independently executable unit of a job. |
 | **Attempt** | A monotonically increasing execution lease for a task. |
 | **Lease** | Temporary exclusive assignment of a task to one worker. |
 | **Artifact** | A durable input, shard, partial result, final result, or log file. |
 | **Planner** | Workload code that validates a job and emits task payloads. |
 | **Runner** | Worker-side code that executes one typed task locally. |
 | **Reducer** | Coordinator-side code that merges all completed partial results into a final artifact. |
 ### 3.1 Job state machine
 ```text
 CREATED -> PLANNING -> RUNNING -> REDUCING -> COMPLETED
                 |         |          |
                 +-------> FAILED <---+
 RUNNING -> CANCELLED
 ```
 ### 3.2 Task state machine
 ```text
 PENDING -> LEASED -> RUNNING -> COMPLETED
              |        |          |
              |        +--------> FAILED
              +-----> PENDING       |
                   lease expiry      +-> PENDING (attempts remain)
 ```
 `LEASED` means the coordinator has returned a task. `RUNNING` means the worker
 has successfully sent its first heartbeat/start acknowledgement. A lease expiry
 may return either `LEASED` or `RUNNING` tasks to `PENDING` when attempts remain.
 ### 3.3 Artifact lifecycle
 ```text
 upload input -> INPUT artifact -> planner creates SHARD artifacts
 worker downloads SHARD -> executes -> uploads PARTIAL_RESULT artifact
 reducer reads partial artifacts -> writes FINAL_RESULT artifact
 user downloads final artifact
 ```
 No `file://` or `worker://` URI is valid in persisted result metadata.
 ---
 ## 4. Target repository layout
 Keep the existing Python package at the repository root and add a Go
 coordinator as a self-contained subproject.
 ```text
 SciMesh/
  PLAN.md
  docs/
    worker-daemon-task.md
    database-integration-task.md
    api-contract.md                 # created in Phase 0
  scimesh/                          # Python local workloads and worker daemon
    chemistry/
    core/
    distributed/                    # planner/reducer contracts, Phase 3
    worker/
    workloads/
  tests/
    unit/
    integration/
    contract/
  coordinator/
    go.mod
    cmd/coordinator/main.go
    internal/
      config/
      domain/
      httpapi/
      queue/
      reducer/
      storage/
      store/postgres/
    migrations/
    tests/
  scripts/
    dev-start.sh                    # optional convenience script, not required runtime
 ```
 Do not move the mature local workload code merely to satisfy this layout. Move
 only when a distributed contract requires a clear shared module.
 ---
 ## 5. Cross-service API contract
 The API contract is a compatibility boundary. Before either side is implemented,
 copy this section into `docs/api-contract.md` and treat it as versioned.
 All worker endpoints require bearer authentication. Worker identity and attempt
 are checked against the current task lease in PostgreSQL.
 ### 5.1 Register worker
 ```http
 POST /workers/register
 Authorization: Bearer <token>
 Content-Type: application/json
 {
  "name": "lab-worker-01",
  "capabilities": ["similarity-search", "similarity-graph"],
  "cpu_count": 8,
  "memory_mb": 16384
 }
 ```
 Response:
 ```json
 {
  "worker_id": "uuid",
  "heartbeat_interval_seconds": 15
 }
 ```
 ### 5.2 Claim task
 ```http
 POST /tasks/claim
 Authorization: Bearer <token>
 Content-Type: application/json
 {
  "worker_id": "uuid",
  "capabilities": ["similarity-search", "similarity-graph"],
  "max_concurrency": 1
 }
 ```
 - `204 No Content`: no compatible task is available.
 - `200 OK`: a task is leased atomically.
 ```json
 {
  "task_id": "uuid",
  "attempt": 1,
  "lease_expires_at": "2026-07-22T12:05:00Z",
  "workload": "similarity-search",
  "input": {
    "uri": "https://coordinator.example/tasks/uuid/input",
    "sha256": "hex-sha256"
  },
  "parameters": {
    "query_id": "CHEMBL939",
    "top_k": 20
  }
 }
 ```
 ### 5.3 Renew lease
 ```http
 POST /tasks/{task_id}/heartbeat
 Authorization: Bearer <token>
 Content-Type: application/json
 {"worker_id": "uuid", "attempt": 1}
 ```
 The response **must** contain a renewed deadline:
 ```json
 {"lease_expires_at": "2026-07-22T12:10:00Z"}
 ```
 The worker schedules the next heartbeat before half of this returned TTL, never
 using only a fixed interval.
 ### 5.4 Download input or shard
 `GET /tasks/{task_id}/input` returns an artifact owned by the current task. The
 worker verifies its SHA-256 before execution. If the returned URI redirects to
 another origin, the worker must remove the coordinator bearer token.
 ### 5.5 Upload a partial artifact
 ```http
 PUT /tasks/{task_id}/artifacts/{filename}
 Authorization: Bearer <token>
 Content-Type: text/csv
 X-Worker-ID: uuid
 X-Task-Attempt: 1
 <streamed bytes>
 ```
 The coordinator streams the body to its artifact storage, verifies ownership,
 stores metadata and checksum, then returns:
 ```json
 {
  "artifact_id": "uuid",
  "uri": "https://coordinator.example/artifacts/uuid/download",
  "sha256": "hex-sha256",
  "size_bytes": 1234
 }
 ```
 ### 5.6 Complete or fail task
 ```http
 POST /tasks/{task_id}/result
 Authorization: Bearer <token>
 Content-Type: application/json
 {
  "worker_id": "uuid",
  "attempt": 1,
  "result": {
    "artifact_id": "uuid",
    "uri": "https://coordinator.example/artifacts/uuid/download",
    "sha256": "hex-sha256",
    "content_type": "text/csv"
  },
  "metrics": {"elapsed_seconds": 12.4, "processed_rows": 10000}
 }
 ```
 `POST /tasks/{task_id}/failure` uses the same identity fields and contains only
 sanitized `error_code` and `error_message` values. No Python traceback, token,
 or absolute worker path may be sent.
 ### 5.7 Idempotency and errors
 | Situation | Required response |
 | --- | --- |
 | No compatible task | `204` |
 | Worker/attempt does not own lease | `409` |
 | Artifact does not belong to task/attempt | `409` |
 | Same completion, same manifest | `200`/`202` idempotent success |
 | Same attempt, different manifest | `409` |
 | Invalid parameters/input | `400` |
 | Worker authentication failure | `401`/`403` |
 ---
 ## 6. PostgreSQL data model
 The existing brief describes `jobs` and `tasks`. Add first-class worker and
 artifact records before implementation. The database is the source of truth for
 state; files are referenced by metadata rather than discovered from directories.
 ### 6.1 Tables
 #### `workers`
 | Field | Notes |
 | --- | --- |
 | `id UUID PK` | Returned on registration |
 | `name text` | Human-readable; unique only if desired |
 | `capabilities jsonb` | Allowlisted workload names |
 | `status` | `online`, `busy`, `offline` |
 | `last_heartbeat_at timestamptz` | Liveness visibility |
 | `created_at`, `updated_at timestamptz` | Audit |
 #### `jobs`
 | Field | Notes |
 | --- | --- |
 | `id UUID PK` | User-visible identifier |
 | `workload text` | Registered distributed workload name |
 | `status` | `created`, `planning`, `running`, `reducing`, `completed`, `failed`, `cancelled` |
 | `parameters jsonb` | Validated job parameters |
 | `input_artifact_id UUID` | Original uploaded data |
 | `result_artifact_id UUID nullable` | Final result |
 | `total_tasks`, `completed_tasks`, `failed_tasks` | Progress counters, updated transactionally |
 | timestamps and `error_message` | Audit and failure status |
 #### `tasks`
 | Field | Notes |
 | --- | --- |
 | `id UUID PK`, `job_id UUID FK` | Identity and ownership |
 | `chunk_index int` | Unique within job; deterministic reducer order |
 | `workload text`, `parameters jsonb` | Typed task payload |
 | `input_artifact_id UUID` | Dataset or shard |
 | `status` | `pending`, `leased`, `running`, `completed`, `failed`, `cancelled` |
 | `attempt`, `max_attempts` | Retry accounting |
 | `lease_owner UUID nullable`, `lease_expires_at nullable` | Exclusive lease |
 | `result_artifact_id UUID nullable` | Uploaded partial output |
 | `metrics jsonb`, errors, timestamps, `version int` | Audit and concurrency |
 #### `artifacts`
 | Field | Notes |
 | --- | --- |
 | `id UUID PK` | Stable artifact identity |
 | `job_id UUID FK`, `task_id UUID FK nullable` | Ownership |
 | `kind` | `input`, `shard`, `partial_result`, `final_result`, `log` |
 | `filename`, `storage_key`, `content_type` | Storage metadata |
 | `size_bytes`, `sha256` | Integrity metadata |
 | `created_at` | Audit |
 ### 6.2 Required constraints and queries
 - unique `(job_id, chunk_index)` for tasks;
 - unique `(task_id, kind)` for single-result task workloads;
 - claim index: `(status, lease_expires_at, created_at)`;
 - `attempt >= 0`, `max_attempts > 0`;
 - a leased/running task must have owner and expiry;
 - a completed task must reference a `partial_result` artifact;
 - a completed job must reference a `final_result` artifact;
 - `list_completed_results(job_id)` orders by `chunk_index`, never insertion time.
 Atomic claim uses one PostgreSQL transaction with `FOR UPDATE SKIP LOCKED`.
 Never write a `SELECT pending task` followed by an unguarded later `UPDATE`.
 ---
 ## 7. Workload evolution
 The current local `Workload` CLI interface is intentionally small. Distributed
 execution needs a second, explicit contract. Do not force every CLI helper into
 the distributed interface; adapt only workloads that can be planned and reduced.
 ```python
 class DistributedWorkload(Protocol):
    name: str
    version: str
    def validate_job(self, input_path: Path, parameters: dict[str, object]) -> None: ...
    def plan(self, input_path: Path, parameters: dict[str, object], workspace: Path) -> list[TaskPlan]: ...
    def execute_task(self, task: TaskPlan, workspace: Path) -> RunResult: ...
    def reduce(self, partial_results: list[Path], parameters: dict[str, object], workspace: Path) -> FinalResult: ...
    def describe(self) -> dict[str, object]: ...
 ```
 `TaskPlan` is JSON-serializable and contains only validated parameters plus
 artifact IDs/URIs. It never contains shell commands or local paths from another
 machine.
 ### 7.1 Distributed similarity-search
 1. Validate either `query_id` or `query_smiles`, never both.
 2. Resolve a `query_id` once during planning and persist the canonical query
   SMILES/identity in the job metadata.
 3. Split the input TSV into deterministic shard artifacts. Every shard keeps a
   header and a stable `chunk_index`.
 4. Each worker streams one shard, skips invalid SMILES and the query molecule,
   and writes a sorted local top-k CSV.
 5. The reducer merges all local top-k outputs with a bounded heap using the same
   deterministic tie-breaker as local SciMesh.
 6. The final CSV must equal the current single-process result for the same input
   and options.
 Important: a local shard top-k must retain at least the global requested `k`.
 The reducer cannot recover a candidate discarded by every shard.
 ### 7.2 Distributed similarity-graph
 1. Parse valid molecules once during planning or produce deterministic molecule
   block artifacts with stable block indices.
 2. Emit one task for every block pair `(i, j)` where `i <= j`.
 3. A diagonal task compares only pairs inside one block with local index `a < b`.
 4. An off-diagonal task compares every molecule in block `i` with every molecule
   in block `j`.
 5. Each task emits an edge-list CSV only for pairs satisfying the chosen
   `threshold_direction` and threshold.
 6. The reducer merges edge files, verifies no duplicate unordered pair, and
   writes a deterministic sort order.
 Correctness invariant:
 ```text
 union(task-pairs) = all unordered molecule pairs
 intersection(task-pairs) = empty
 ```
 For a small fixture, the distributed result must exactly equal the local
 brute-force graph for both `greater` and `less` threshold directions.
 ### 7.3 Future workload policy
 A new workload is accepted only when it supplies:
 - an input/parameter validator;
 - an explicit sharding strategy;
 - bounded-memory task execution;
 - deterministic reduction semantics;
 - fixture-based local and distributed correctness tests;
 - a `describe()` payload for UI/API discovery.
 ---
 ## 8. Milestones and dependency order
 ```text
 M0 contracts + test fixtures
  -> M1 Go coordinator skeleton + migrations
    -> M2 transactional queue + worker registry
      -> M3 artifact storage + worker contract integration
        -> M4 distributed similarity-search vertical slice
          -> M5 reducer + status/result API
            -> M6 distributed similarity-graph
              -> M7 UI, observability, CI hardening
 ```
 Do not start graph distribution before the search vertical slice proves the
 full artifact/lease/reducer lifecycle.
 ---
 ## 9. Task briefs for implementation
 Each section below is deliberately self-contained. When assigning work, copy
 the task block plus the **Shared context** section and any listed dependency.
 ### Shared context for every assignee
 ```text
 Project: SciMesh
 Architecture: Go/PostgreSQL coordinator; Python workers; local artifact storage.
 Hard rules: workers have no DB credentials; task claims are atomic; artifacts
 must be durable before completion; no arbitrary commands; deterministic output.
 Read first: PLAN.md sections 2, 3, 5, and the task's dependencies.
 Do not refactor unrelated files. Add or update tests with every behavior change.
 ```
 ### CTX-00 — Freeze API and error contract
 **Goal:** Create `docs/api-contract.md` from section 5 and make it the single
 source of truth for the Go coordinator and Python Worker.
 **Depends on:** none.
 **Deliverables:** endpoint table, JSON schemas/examples, headers, artifact
 upload ownership rule, error mapping, retry/idempotency policy, and an explicit
 version marker (`v1`).
 **Acceptance criteria:**
 - all worker endpoints and their paths/methods are listed;
 - heartbeat response includes `lease_expires_at`;
 - completion references only coordinator-uploaded artifacts;
 - failure endpoint is distinct from result endpoint;
 - document states whether the coordinator uses UTC RFC 3339 timestamps.
 **Out of scope:** implementing HTTP handlers.
 ### CTX-01 — Bootstrap Go coordinator
 **Goal:** Add the `coordinator/` Go module and a minimal healthy HTTP service.
 **Depends on:** CTX-00.
 **Deliverables:** `go.mod`; config parser; structured logger; `GET /healthz`;
 graceful shutdown; `pgxpool` lifecycle; migration command documentation.
 **Inputs:** `DATABASE_URL`, `COORDINATOR_STORAGE_DIR`, `COORDINATOR_ADDR`,
 `COORDINATOR_TOKEN`, request timeout, pool size.
 **Acceptance criteria:**
 - `go test ./...` passes;
 - service refuses to start with missing/invalid required configuration;
 - `/healthz` returns database readiness without exposing secrets;
 - shutdown closes HTTP server and `pgxpool` cleanly;
 - migration execution is explicit, not implicit on production startup.
 **Out of scope:** queue endpoints and UI.
 ### CTX-02 — PostgreSQL schema and migration set
 **Goal:** Implement versioned SQL migrations for workers, jobs, tasks, and
 artifacts from section 6.
 **Depends on:** CTX-01, CTX-00.
 **Deliverables:** up/down SQL migrations; enum/check constraints; indexes;
 repository-domain structs as needed for scanning rows.
 **Acceptance criteria:**
 - empty PostgreSQL database migrates up and down in integration tests;
 - constraints reject invalid state combinations;
 - migration test uses `TEST_DATABASE_URL`, never SQLite;
 - `tasks(job_id, chunk_index)` uniqueness and claim index exist;
 - no application code creates tables dynamically.
 **Out of scope:** HTTP handlers and lease operations.
 ### CTX-03 — Transactional queue and lease repository
 **Goal:** Implement PostgreSQL repository operations for create, claim, renew,
 complete, fail, expiry, and job status.
 **Depends on:** CTX-02.
 **Required operations:**
 ```go
 CreateJobWithTasks(ctx, input)
 ClaimNextTask(ctx, workerID, capabilities, leaseDuration)
 RenewLease(ctx, taskID, workerID, attempt, leaseDuration)
 CompleteTask(ctx, input)
 FailTask(ctx, input)
 ExpireLeases(ctx, now)
 GetJobStatus(ctx, jobID)
 ListCompletedResults(ctx, jobID)
 ```
 **Acceptance criteria:**
 - concurrent claim test proves one task/attempt has one owner;
 - renewal returns a new `lease_expires_at` timestamp;
 - stale attempt cannot renew, upload, fail, or complete;
 - expired task returns to pending or fails after final attempt;
 - completion is idempotent for identical manifest and conflicts for a different
  manifest;
 - all operations are context-aware and parameterized.
 **Out of scope:** artifact byte storage and Python worker changes.
 ### CTX-04 — Worker registry and coordinator HTTP handlers
 **Goal:** Expose CTX-03 through versioned `net/http` handlers and add worker
 registration/liveness state.
 **Depends on:** CTX-00, CTX-03.
 **Endpoints:** `POST /workers/register`, `POST /tasks/claim`,
 `POST /tasks/{id}/heartbeat`, `GET /jobs/{id}`, and readiness endpoints.
 **Acceptance criteria:**
 - request DTOs are validated before calling services;
 - no raw PostgreSQL errors leave the process;
 - `204`, `400`, `401/403`, and `409` match CTX-00;
 - heartbeat response returns renewed deadline;
 - tests cover an offline worker and an expired lease;
 - logs include request ID, worker ID, task ID, attempt, operation.
 ### CTX-05 — Coordinator artifact storage
 **Goal:** Store input, shard, partial-result, and final-result files durably in
 the coordinator filesystem and persist their metadata.
 **Depends on:** CTX-02, CTX-04.
 **Endpoints:** authenticated input upload/download and
 `PUT /tasks/{id}/artifacts/{filename}`.
 **Implementation rules:**
 - stream request bodies to a staging file; never `ReadAll` a result;
 - use sanitized generated storage keys, not user paths;
 - calculate SHA-256 while streaming;
 - atomically rename staging file only after successful write;
 - verify worker lease owner and attempt before accepting task output;
 - return a coordinator-owned durable URI and artifact ID.
 **Acceptance criteria:**
 - uploaded artifact is downloadable after coordinator restart;
 - a foreign worker receives `409`;
 - large-file test proves bounded-memory streaming behavior;
 - checksum and size are persisted;
 - failed upload leaves no visible artifact or orphan staging file.
 ### CTX-06 — Align Python Worker with the live Go contract
 **Goal:** Adapt `scimesh/worker/` to CTX-00 through CTX-05 without changing
 local workload algorithms.
 **Depends on:** CTX-00, CTX-04, CTX-05.
 **Required behavior:**
 - register worker capabilities at startup;
 - claim one task at a time;
 - remove bearer token when input redirect changes origin;
 - verify input checksum;
 - renew lease from returned `lease_expires_at`;
 - stream uploaded output with worker/attempt headers;
 - submit the returned artifact manifest only after upload;
 - send errors to `/failure`;
 - preserve task directory until configured cleanup;
 - reject unknown workload parameters.
 **Acceptance criteria:**
 - Python contract tests run against the real Go service in CI;
 - no result references `worker://` or local paths;
 - long fake runner causes multiple lease renewals;
 - lost lease stops successful completion and reports conflict cleanly;
 - worker CLI documents all environment variables and exits non-zero on invalid
  configuration.
 ### CTX-07 — Distributed workload protocol and job planner
 **Goal:** Add the Python `DistributedWorkload` adapter protocol and coordinator
 planner bridge for registered workloads.
 **Depends on:** CTX-05, CTX-06.
 **Deliverables:** typed task-plan JSON; validator; planner registry; reducer
 registry; workload descriptions; no direct coordinator dependency in local
 chemistry helpers.
 **Acceptance criteria:**
 - unknown workload rejected before a job/task is written;
 - every plan payload is JSON-serializable and uses artifact references;
 - planner failure leaves no partial job/tasks transaction;
 - test fixture demonstrates planning a two-shard dummy workload;
 - reducer receives completed artifacts ordered by `chunk_index`.
 ### CTX-08 — Distributed similarity-search vertical slice
 **Goal:** Implement planner, worker execution adapter, reducer, and end-to-end
 tests for distributed exact top-k similarity search.
 **Depends on:** CTX-07.
 **Acceptance criteria:**
 - query ID is resolved before shards execute, or query SMILES is validated once;
 - shards are deterministic and contain valid TSV headers;
 - each shard produces local top-k CSV and reports invalid-SMILES counts;
 - reducer output matches local single-process SciMesh byte-for-byte apart from
  permitted elapsed-time metadata;
 - query molecule and duplicate canonical query SMILES are excluded;
 - tie order is deterministic across worker completion order;
 - test uses at least two workers and one retry.
 ### CTX-09 — Job reducer orchestration and final result API
 **Goal:** When all task results are complete, run the appropriate reducer once,
 persist a final artifact, update job state, and expose download/status.
 **Depends on:** CTX-07, CTX-08.
 **Acceptance criteria:**
 - only one reducer process may transition a job into `reducing`;
 - reducer is idempotent or protected by state/version transaction;
 - reducer failure marks job failed with a sanitized error;
 - `GET /jobs/{id}` reports counters and state correctly;
 - final artifact has stored checksum and downloadable URI;
 - integration test covers complete job lifecycle.
 ### CTX-10 — Distributed similarity-graph
 **Goal:** Implement block-pair planning, execution, and deterministic reduction
 for the exact sparse similarity graph.
 **Depends on:** CTX-08, CTX-09.
 **Acceptance criteria:**
 - each block pair is planned once with stable `(left_block, right_block)`;
 - diagonal and off-diagonal comparisons obey the pair invariant in section 7.2;
 - no task creates a dense matrix;
 - both `greater` and `less` threshold directions are preserved;
 - reducer detects duplicate unordered pairs and fails safely;
 - distributed output equals local brute-force output on a small fixture;
 - result is invariant to block size and worker completion order.
 ### CTX-11 — Minimal dashboard and operator views
 **Goal:** Add a small server-rendered or static HTML UI to inspect jobs, tasks,
 workers, and download final artifacts.
 **Depends on:** CTX-04, CTX-09.
 **Acceptance criteria:**
 - show worker name/capabilities/status/last heartbeat;
 - show job status and completed/total task progress;
 - show task attempt, lease owner, and sanitized error;
 - refresh with simple polling; no frontend framework required;
 - final result link is available only in `completed` state;
 - HTML escapes user-controlled values.
 ### CTX-12 — Reliability, security, and CI hardening
 **Goal:** Make the vertical slice safe to demo and difficult to regress.
 **Depends on:** CTX-06 through CTX-11.
 **Work items:**
 - CI jobs for Python tests, Go tests, `go vet`, migrations, and contract tests;
 - coordinator request-size limits and timeouts;
 - token configuration/rotation documentation;
 - structured logs and correlation IDs;
 - cleanup policy for failed worker directories and stale staging files;
 - metrics/status endpoint suitable for local monitoring;
 - retry/backoff test matrix;
 - release checklist and local two-worker demo script.
 **Acceptance criteria:**
 - clean checkout can run the documented demo;
 - all quality gates execute in CI;
 - no secret appears in logs/tests/errors;
 - failure/retry scenarios have automated coverage;
 - README contains architecture diagram, security caveat, and troubleshooting.
 ---
 ## 10. Suggested assignment bundles
 These bundles minimize overlap. Do not run tasks from the same bundle in
 parallel unless one engineer owns integration.
 | Bundle | Tasks | Recommended owner |
 | --- | --- | --- |
 | Contract and data | CTX-00, CTX-02, CTX-03 | Go/PostgreSQL engineer |
 | Coordinator API | CTX-01, CTX-04, CTX-05 | Go backend engineer |
 | Python transport | CTX-06 | Python worker engineer |
 | Distributed computation | CTX-07, CTX-08, CTX-10 | Scientific Python engineer |
 | Product surface | CTX-09, CTX-11 | Full-stack/backend engineer |
 | Quality gate | CTX-12 | DevOps/QA engineer |
 Suggested order for a small team:
 ```text
 Week 1: CTX-00 + CTX-01 + CTX-02
 Week 2: CTX-03 + CTX-04
 Week 3: CTX-05 + CTX-06
 Week 4: CTX-07 + CTX-08
 Week 5: CTX-09 + CTX-10
 Week 6: CTX-11 + CTX-12
 ```
 This is an ordering aid, not a deadline commitment. Start the next milestone
 only after its predecessor's acceptance criteria are demonstrably met.
 ---
 ## 11. Test strategy
 ### 11.1 Python unit tests
 - SMILES parsing, fingerprints, deterministic top-k and graph ordering;
 - Worker parameter allowlist and CLI argument mapping;
 - checksum verification and cross-origin auth stripping;
 - heartbeat rescheduling from returned TTL;
 - artifact upload manifest generation;
 - runner failure sanitization and cleanup behavior.
 ### 11.2 Go unit tests
 - configuration parser and error mapping;
 - storage-key sanitization;
 - request DTO validation;
 - service state transition guards;
 - reducer invocation selection.
 ### 11.3 PostgreSQL integration tests
 - migrations up/down;
 - concurrent `ClaimNextTask` without duplicate lease;
 - lease expiry/retry/exhaustion;
 - stale attempt conflict;
 - artifact ownership and task completion idempotency;
 - deterministic partial-result retrieval order.
 ### 11.4 Contract tests
 Run Python Worker tests against a real Go coordinator and PostgreSQL instance.
 At minimum prove:
 1. worker registers;
 2. worker claims one task;
 3. worker receives a renewed lease;
 4. input checksum is verified;
 5. result is uploaded and available from coordinator storage;
 6. completion persists the correct artifact;
 7. bad checksum or runner failure reaches `/failure`;
 8. a stale worker cannot complete after its lease has expired.
 ### 11.5 End-to-end tests
 - two-worker similarity-search run equals local CLI output;
 - retry one failed shard and still finish deterministically;
 - graph on small fixture has all and only brute-force threshold edges;
 - stopping a worker mid-task requeues its task after lease expiry;
 - final result remains downloadable after coordinator restart.
 ---
 ## 12. Review gates
 Before merging a task, reviewer checks:
 ### Every task
 - scope matches one CTX block;
 - tests cover new behavior and pass;
 - no unrelated refactor or generated data is committed;
 - errors are sanitized and parameters validated;
 - documentation/API contract updated when behavior changes.
 ### Worker changes
 - no SQL, DB credentials, or arbitrary command execution;
 - no worker-local URI persisted as a result;
 - artifact transfer is streamed and checksum verified;
 - heartbeat uses returned deadline;
 - cross-origin requests do not receive coordinator token.
 ### Coordinator changes
 - mutating actions are transactional;
 - lease/task ownership checked at every mutation;
 - PostgreSQL operations are parameterized;
 - result completion is idempotent;
 - artifact ownership and storage keys are validated;
 - state transitions cannot skip required intermediate conditions.
 ### Scientific workload changes
 - local reference result exists;
 - distributed result is compared to local result on fixtures;
 - reduction ordering is deterministic;
 - pair coverage/duplicate invariants are tested for graphs;
 - memory remains bounded as specified.
 ---
 ## 13. Deferred backlog
 Do not start these before CTX-12 is accepted.
 - Replace local artifact storage with S3/MinIO behind an `ArtifactStore` API.
 - Add worker labels/capacity-aware scheduling and concurrency > 1.
 - Add cancellation propagation to workers.
 - Add image outputs and final PDF reporting to job artifacts.
 - Add CV/video workloads using the same planner/runner/reducer contract.
 - Add observability export (Prometheus/OpenTelemetry).
 - Add per-user/project authorization and signed artifact URLs.
 - Add shard caching and content-addressed input deduplication.
 - Add job priority and fair scheduling.
 - Add a CLI for submitting and monitoring remote jobs.
 ---
 ## 14. Definition of the first usable distributed release
 The release is complete when all of the following are true:
 1. A user uploads a small ChEMBL TSV and starts a similarity-search job.
 2. The Go coordinator creates PostgreSQL job/task/artifact records.
 3. Two Python workers register, claim distinct shards, renew leases, and upload
   partial CSVs.
 4. The coordinator reduces partial results into a deterministic final CSV.
 5. The user observes progress and downloads the final CSV.
 6. Killing one worker requeues only its lease after expiry; the job still
   completes within its attempt budget.
 7. The same fixture run matches local SciMesh output.
 8. CI executes Python, Go, PostgreSQL, and contract tests successfully.
 Until these eight conditions are met, SciMesh is a promising set of components,
 not yet a complete distributed platform.
@@ -0,0 +1,56 @@
 # SciMesh Status
 **Updated:** 2026-07-23  
 **Branch baseline:** `planning` at `13f9a0b`
 ## Current state
 The local Python molecular workloads are implemented and tested. They provide
 the reference behaviour for future distributed execution:
 - `similarity-search`: streaming ChEMBL TSV search, Morgan fingerprints,
  Tanimoto scoring, heap-based top-k, CSV and image output;
 - `similarity-graph`: exact sparse graph, block-based pair comparisons,
  deterministic CSV output;
 - Python Worker skeleton: claim, heartbeat, input checksum validation,
  artifact upload, completion and failure reporting.
 The Go coordinator, PostgreSQL schema, coordinator artifact storage, planner,
 reducer, and end-to-end distributed execution are **not implemented yet**.
 ## Milestone tracker
 | CTX | Status | Notes |
 | --- | --- | --- |
 | CTX-00 API and error contract | Ready to implement | `docs/api-contract.md` created; needs owner review/freeze. |
 | CTX-01 Go coordinator bootstrap | Not started | Depends on CTX-00. |
 | CTX-02 PostgreSQL migrations | Not started | Depends on CTX-00 and CTX-01. |
 | CTX-03 Transactional queue | Not started | Depends on CTX-02. |
 | CTX-04 Worker registry and HTTP API | Not started | Depends on CTX-03. |
 | CTX-05 Artifact storage | Not started | Depends on CTX-02 and CTX-04. |
 | CTX-06 Python Worker live-contract alignment | Partially prepared | Worker skeleton exists; needs real Go contract tests. |
 | CTX-07 Distributed workload protocol | Not started | Depends on artifact and Worker contracts. |
 | CTX-08 Distributed similarity-search | Not started | Local reference exists. |
 | CTX-09 Reducer and final-result API | Not started | Depends on CTX-07 and CTX-08. |
 | CTX-10 Distributed similarity-graph | Not started | Local reference exists. |
 | CTX-11 Dashboard/operator view | Not started | Deferred until API and reducer work. |
 | CTX-12 Reliability, security, CI | Not started | Final milestone. |
 ## Next recommended assignment
 Assign **CTX-00** to the coordinator role in `.agents/coordinator.md`: review
 and freeze `docs/api-contract.md` against `PLAN.md`. Do not begin coordinator
 or Worker API implementation until the contract owner accepts it.
 ## Known constraints
 - Distributed execution is not available; use the local `scimesh` CLI.
 - No Go module, PostgreSQL migrations, runtime configuration, or integration
  environment exists yet.
 - Local worker unit tests do not prove interoperability with a live coordinator.
 ## Update rule
 The integration role updates this file only after collecting command output,
 test evidence, and accepted changes. State facts, revision hashes, blockers,
 and the next unblocked CTX task; do not mark work complete based on plans alone.
@@ -0,0 +1,151 @@
 # SciMesh Coordinator API Contract
 **Status:** draft, version 1. This document is the compatibility boundary
 between the Go coordinator and the Python Worker. Change it only in the same
 pull request as both implementation and contract tests.
 ## General rules
 - All worker endpoints require `Authorization: Bearer <token>`.
 - Times use UTC RFC 3339, for example `2026-07-23T12:05:00Z`.
 - JSON requests and responses use `application/json`.
 - `worker_id` and `attempt` identify a lease. The coordinator validates them
  transactionally on every task mutation.
 - A task becomes `completed` only after a coordinator-owned artifact is durable.
 - Identical repeated completion is successful; a different result for the same
  attempt is a conflict.
 ## Worker registration
 ```http
 POST /workers/register
 {"name":"lab-worker-01","capabilities":["similarity-search"],"cpu_count":8,"memory_mb":16384}
 ```
 Returns `200 OK`:
 ```json
 {"worker_id":"uuid","heartbeat_interval_seconds":15}
 ```
 ## Task lifecycle
 ### Claim
 ```http
 POST /tasks/claim
 {"worker_id":"uuid","capabilities":["similarity-search"],"max_concurrency":1}
 ```
 Returns `204 No Content` when no compatible task exists. A successful atomic
 claim returns `200 OK`:
 ```json
 {
  "task_id":"uuid",
  "attempt":1,
  "lease_expires_at":"2026-07-23T12:05:00Z",
  "workload":"similarity-search",
  "input":{"uri":"https://coordinator.example/tasks/uuid/input","sha256":"hex-sha256"},
  "parameters":{"query_id":"CHEMBL939","top_k":20}
 }
 ```
 The claim is one PostgreSQL transaction using `FOR UPDATE SKIP LOCKED`.
 ### Heartbeat
 ```http
 POST /tasks/{task_id}/heartbeat
 {"worker_id":"uuid","attempt":1}
 ```
 Returns `200 OK` and the renewed deadline:
 ```json
 {"lease_expires_at":"2026-07-23T12:10:00Z"}
 ```
 The Worker schedules its next heartbeat before half of the returned TTL.
 ### Input download
 `GET /tasks/{task_id}/input` returns the claimed task input. The Worker verifies
 its SHA-256 before execution. On a redirect to another origin, it removes the
 coordinator bearer token.
 ## Artifact upload
 ```http
 PUT /tasks/{task_id}/artifacts/{filename}
 Content-Type: text/csv
 X-Worker-ID: uuid
 X-Task-Attempt: 1
 <streamed bytes>
 ```
 The coordinator streams the body to storage, checks lease ownership, records
 the checksum and returns `201 Created`:
 ```json
 {
  "artifact_id":"uuid",
  "uri":"https://coordinator.example/artifacts/uuid/download",
  "sha256":"hex-sha256",
  "size_bytes":1234
 }
 ```
 The returned URI is the only URI the Worker may send in task completion.
 `worker://` and `file://` are invalid.
 ## Completion and failure
 ```http
 POST /tasks/{task_id}/result
 {
  "worker_id":"uuid",
  "attempt":1,
  "result":{
    "artifact_id":"uuid",
    "uri":"https://coordinator.example/artifacts/uuid/download",
    "sha256":"hex-sha256",
    "content_type":"text/csv"
  },
  "metrics":{"elapsed_seconds":12.4,"processed_rows":10000}
 }
 ```
 The coordinator returns `200`, `201`, or `202` for a valid completion. It must
 verify that the artifact belongs to that task and attempt before completing it.
 Use `POST /tasks/{task_id}/failure` only for a failed attempt:
 ```json
 {"worker_id":"uuid","attempt":1,"error_code":"ValueError","error_message":"input checksum mismatch"}
 ```
 Messages are sanitised: no token, traceback, absolute local path, or raw input.
 ## Error responses
 | Situation | Response |
 | --- | --- |
 | Invalid JSON, field, or parameter | `400 Bad Request` |
 | Missing or invalid authentication | `401 Unauthorized` / `403 Forbidden` |
 | Worker/attempt does not own an active lease | `409 Conflict` |
 | Artifact does not belong to the task/attempt | `409 Conflict` |
 | Same attempt, different completion manifest | `409 Conflict` |
 | Unexpected coordinator failure | `500` without internal details |
 ## Compatibility tests
 Contract tests must cover: registration, `204` claim, successful claim,
 heartbeat renewal, foreign worker and stale attempt conflicts, streamed upload,
 checksum mismatch, success after upload, failure through `/failure`, and
 idempotent completion.
@@ -0,0 +1,57 @@
 # Памятка для доработки SciMesh Worker
 Это короткие правила по итогам ревью первой версии воркера. Перед новой
 задачей прочитай также [PLAN.md](../PLAN.md) и документ задачи, который тебе
 дали.
 ## Что делать
 - Считай воркер клиентом. Очередью, статусами задач и PostgreSQL управляет
  только coordinator.
 - Сначала скачай и проверь входной файл по SHA-256, затем запусти расчёт.
 - Сохраняй файлы только в своей папке `task_id/attempt`.
 - После расчёта сначала загрузи результат через coordinator, затем отправляй
  результат задачи.
 - В `result` передавай URI, который вернул coordinator, плюс SHA-256 и тип
  файла.
 - При ошибке вызывай `/tasks/{id}/failure`; при успехе —
  `/tasks/{id}/result`.
 - Для heartbeat используй новое `lease_expires_at`, возвращённое coordinator.
 - Во все запросы, связанные с задачей, передавай `worker_id` и `attempt`.
 - Пиши тесты не только на успех: проверь ошибку runner, неверный checksum,
  ошибку загрузки, потерю lease и пустой результат.
 - Если поменял API, меняй одновременно Python-код, Go coordinator,
  документацию и тесты.
 ## Чего не делать
 - Не отправляй в coordinator пути вида `worker://...` или `file://...`.
  Coordinator не видит локальные файлы воркера.
 - Не используй `/result` для сообщения об ошибке.
 - Не считай, что проверок Python достаточно: coordinator обязан сам проверить
  владельца lease, attempt, срок lease и допустимость смены статуса.
 - Не повторяй любой HTTP-запрос вслепую. Повторные запросы должны быть
  безопасны: одинаковый `(task_id, attempt, worker_id)` не должен создавать
  дубликаты.
 - Не передавай bearer token на другой домен при редиректе и не пиши токены,
  traceback или полные локальные пути в логи.
 - Не помечай задачу завершённой, если артефакт не загружен надёжно.
 - Не меняй контракт API «по ощущениям». Сначала зафиксируй JSON, статусы и
  переходы состояний в документации.
 ## Мини-чеклист перед коммитом
 - [ ] Результат загружен в storage coordinator до `POST /result`.
 - [ ] Ошибка уходит в `POST /failure`, а не в `POST /result`.
 - [ ] Heartbeat возвращает новый срок lease, и код его сохраняет.
 - [ ] В запросах есть верные `worker_id` и `attempt`.
 - [ ] Нет `worker://`, `file://`, токенов и секретов в результатах или логах.
 - [ ] Добавлены тесты на новый сценарий и на ошибку.
 - [ ] `pytest` и проверка форматирования проходят.
 ## Короткий контекст для нейронки
 > SciMesh Worker — клиент coordinator, а не владелец очереди. Загружай
 > результат через coordinator до завершения задачи. Успех отправляй в
 > `/result`, ошибку — в `/failure`. Проверяй checksum, lease и attempt.
 > Меняя протокол, обновляй Python, Go, документацию и тесты в одном изменении.
Author	SHA1	Message	Date
Emil	abfda35170	Add team agent context and project contracts	2026-07-23 20:31:20 +03:00
Emil	13f9a0b494	Add repository contributor guidelines	2026-07-22 21:34:11 +03:00
Emil	69c34c9383	Simplify worker feedback in Russian	2026-07-22 19:40:11 +03:00
Emil	8a76b13759	Document worker implementation review lessons	2026-07-22 19:39:07 +03:00
Emil	e7aa0be22d	Add distributed platform implementation plan	2026-07-22 19:04:47 +03:00