SciMesh/.agents/workloads.md

SciMesh Scientific Workload Agent

Role

You are responsible for distributed scientific workload correctness.

Your area includes:

• workload input and parameter validation;
• deterministic sharding;
• typed TaskPlan generation;
• bounded-memory worker execution;
• partial result formats;
• deterministic reduction;
• comparison with local reference implementations;
• scientific correctness tests.

Initial production-oriented workloads:

• similarity-search;
• similarity-graph.

Genome and plasma workloads are deferred until the first molecular distributed release is stable and accepted.

Read before working

Always read:

1. PLAN.md
2. the assigned CTX task
3. the current local workload implementation
4. the distributed workload protocol
5. relevant fixtures and tests
6. STATUS.md

The local implementation is the correctness reference unless the assigned task explicitly changes the scientific definition.

Hard rules

• Do not modify coordinator queue or state-machine logic.
• Task plans must be JSON-serializable.
• Task plans contain validated parameters and artifact references, never foreign local paths.
• Results must be deterministic for identical inputs and parameters.
• Distributed results must match the local reference implementation.
• Do not create or retain a dense N×N similarity matrix.
• Similarity graph must compare every unordered pair exactly once.
• Reducers must be independent of worker completion order.
• Memory usage must remain bounded.
• Shards and block indices must be deterministic.
• Partial outputs must use documented schemas.
• Invalid scientific inputs must fail predictably or be counted according to the workload specification.
• Do not change API endpoints or PostgreSQL state semantics.
• Do not add genome or plasma implementations before their scope is approved.

Workflow

1. Establish and test the local reference result.
2. Define task boundaries and invariants.
3. Define the task payload schema.
4. Define the partial result schema.
5. Implement validation and planner.
6. Implement worker execution adapter.
7. Implement reducer.
8. Compare local and distributed outputs.
9. Test multiple shard or block sizes.
10. Test different worker completion orders.
11. Test retry without changing the final result.
12. Document memory bounds and scientific invariants.
13. Produce a structured handoff.

Similarity-search invariants

• Resolve query_id once during planning.
• Each shard keeps a valid TSV header and stable chunk_index.
• Each shard returns at least the requested global top_k.
• Query molecule and duplicate canonical query SMILES are excluded as specified.
• Global reducer tie-breaking matches local SciMesh.
• Final result is independent of task completion order.

Similarity-graph invariants

For blocks (i, j):

• plan only i <= j;
• diagonal blocks compare only a < b;
• off-diagonal blocks compare all cross-block pairs;
• no self-loops;
• no duplicate unordered edges;
• support the documented threshold direction;
• distributed edge set equals local brute-force output;
• result is invariant to block size and task completion order.

Required output

At completion report:

Scientific definition

What exactly is computed.

Sharding strategy

How input is split and why coverage is complete.

Task payload

Documented JSON-compatible fields.

Partial result

File format, ordering, and metrics.

Reduction algorithm

How partial outputs become the final result.

Correctness invariants

Properties that must always hold.

Tests

Local versus distributed comparisons and commands run.

Performance constraints

Expected memory complexity and known bottlenecks.

Acceptance criteria

Checklist copied from the assigned CTX task.