Emil_Shanaty/SciMesh
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add molwt-filter workload with default scaffold hooks

Browse Source
This commit is contained in:
Emil
2026-08-02 01:09:13 +03:00
parent bc76f386e5
commit 5c5a2af0a1
13 changed files with 661 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sdk-handoff.md
+2
View File
@@ -24,6 +24,8 @@ partial writer moved to `scimesh/workloads/search/core.py`; the partial
format is unchanged, so the Go reducer and UI keep working. The runner
resolves `query_id` per task and rejects plan-time `max_rows`.
**Default hooks + molwt-filter (2026-08-01):** `MapReduceWorkload` now provides default `partition_input` (row-bounded, header-preserving sharding for delimited inputs, `shard_rows` class attr) and default `reduce_partials` (`concatenate_partial_tables`, one header, byte-identical). A new built-in `molwt-filter@1.0.0` (`scimesh/workloads/molwt_filter/`) demonstrates the minimal authoring surface: only `compute_shard` is workload code. descriptor-batch dropped its now-redundant partition/reduce overrides.
**Authoring scaffold (2026-08-01):** `scimesh/sdk/batch.py` adds
`MapReduceWorkload` — the primary authoring surface for `core-batch-v1`. A
subclass declares identity/parameters/ports and three scientific hooks
docs/workload-sdk.md
+11 -2
View File
@@ -27,6 +27,10 @@ import the SDK and live outside it. The built-in SciMesh workloads are under
- `scimesh/workloads/search/` — SDK-built `similarity-search@1.0.0`;
- `scimesh/workloads/graph/` — SDK-built `similarity-graph@1.0.0`;
- `scimesh/workloads/descriptors/` — SDK-built `descriptor-batch@1.0.0`;
- `scimesh/workloads/molwt_filter/` — SDK-built `molwt-filter@1.0.0`, the
minimal authoring example: it only declares identity, parameters, ports,
and the `compute_shard` hook, using the scaffold's default sharding and
concatenation;
- `scimesh/workloads/library.py` — the built-in library wiring: a default
registry containing all three definitions and a runtime advertising their
capabilities;
@@ -272,8 +276,13 @@ class CountRowsWorkload(MapReduceWorkload):
```
The base class then provides `validate`, `plan`, `run`, `reduce`, and
`definition()`; the registry, negotiation, resource reservation, verification,
and the local conformance executor treat the result like any other workload:
`definition()`. For workloads whose map output is a simple filtered or
transformed table, the scaffold's defaults already cover partitioning
(row-bounded shards that keep the header) and reduction (concatenation with
one header), so only `compute_shard` has to be written — that is exactly what
the built-in `molwt-filter` workload does. The registry, negotiation, resource
reservation, verification, and the local conformance executor treat the
result like any other workload:
```python
from scimesh.sdk import (