Add product quality gates and milestone acceptance
This commit is contained in:
Emil Shanaty
@@ -45,6 +45,7 @@ Copy `assets/verifier.md.tmpl`. Rules:
|
||||
- Ask for scope, contract, and test-quality checks.
|
||||
- Require the full review file **first**, then a minimal `{task_id, verdict, summary, review_file}` return (`assets/verdict.schema.json`). The review file is the source of truth; a schema miss must not erase the verification work.
|
||||
- GUI: require an isolated Xvfb/display, screenshots at default and minimum sizes, and the exact commands used; shared-desktop screenshots are not evidence.
|
||||
- Product profile: require release-build performance numbers (p95 frame time, input latency, startup), the real UX scenario and a baseline comparison. A technical pass is not product acceptance; the user still gives the final verdict.
|
||||
|
||||
## Report
|
||||
|
||||
|
||||
@@ -12,6 +12,8 @@ Answer these before spawning:
|
||||
- **Interfaces.** Can you freeze signatures, data shapes, and file names before dispatch?
|
||||
- **Breadth.** Is the time dominated by many independent pieces, not one deep chain?
|
||||
|
||||
First decide the **quality profile** with the user: `prototype` (runnable spike, no performance promise) or `product` (performance, UX, baseline and human acceptance are mandatory). See `references/product-quality.md`. For UI/product work, prefer one implementer + one critic over parallel panel owners.
|
||||
|
||||
Budget:
|
||||
- `agents ≈ units + ceil(units / 5) verifiers + 1–2 integrators + 2–4 recon`; add retries and re-verifications explicitly. Set `max_agents` only after checking available tools and slot cap.
|
||||
- Tokens ≈ `agents × fixed overhead (10–25k) + context reads + output + reviews`. Treat this as an estimate, then calibrate from measured usage. A 3k-line file read by several agents can cost more than the code they write.
|
||||
@@ -96,6 +98,8 @@ Write the plan before any worker starts. Keep the plan proportionate: a small do
|
||||
- [ ] Files/modules over ~800 lines are split, or the task explicitly owns the integration cost.
|
||||
- [ ] Token budget, soft cap and verification reserve are recorded; units are merged/split to avoid fixed-overhead waste.
|
||||
- [ ] The plan uses vertical slices/checkpoints for a multi-milestone product.
|
||||
- [ ] Quality profile (`prototype`/`product`), performance budget, baseline scenarios and user-acceptance owner are recorded.
|
||||
- [ ] For product-profile GUI work, the plan names release-build measurements and the exact user scenario.
|
||||
- [ ] No path in two write scopes.
|
||||
- [ ] Contracts frozen for every cross-task interface.
|
||||
- [ ] Coverage matrix complete.
|
||||
@@ -183,13 +187,28 @@ For a GUI product, give each visual verifier its own Xvfb/display and project co
|
||||
|
||||
**Freeze before verify.** Once you start this wave, do not edit source, tests or docs. If you already started and then fixed something, mark the in-flight review stale and re-run that verifier on the new build. A verifier's verdict is only valid for the exact tree it saw.
|
||||
|
||||
**Technical verification is not product acceptance.** Verifiers check evidence; for product-profile work they must also measure release-build frame time/input latency, run the real UX scenario and compare with the baseline. If performance/UX was not measured, the product gate is `not_run`, not `pass`.
|
||||
|
||||
Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed.
|
||||
|
||||
Evidence standard: for every criterion, a command/artifact/citation/metric and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
|
||||
|
||||
On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run only the affected **delta** (diff + findings + acceptance commands), not the whole repository, and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run one final integrated verification pass on the frozen build.
|
||||
|
||||
## 6. Integrate
|
||||
## 6. Product gate
|
||||
|
||||
For `product`-profile work, run this before integration:
|
||||
|
||||
1. Build the release/optimized artifact. Debug numbers are provisional only.
|
||||
2. Measure p50/p95 frame time at the target size, input-to-visible latency, startup and idle CPU/memory on a realistic scene.
|
||||
3. Run the 3–5 step real user scenario, not just a screenshot.
|
||||
4. Compare with the baseline version on the same scenario; a regression is a failure.
|
||||
5. Give the user the artifact and ask for an explicit verdict: usable / usable with issues / not usable.
|
||||
6. Record measurements and the user verdict. Do not start the next milestone until accepted or explicitly waived.
|
||||
|
||||
If the result is functional but laggy or visually worse than the baseline, stop and fix the slice. Do not add features to compensate. See `references/product-quality.md`.
|
||||
|
||||
## 7. Integrate
|
||||
|
||||
One integrator (two only for disjoint areas) owns:
|
||||
- shared files and cross-cutting edits;
|
||||
@@ -199,7 +218,7 @@ One integrator (two only for disjoint areas) owns:
|
||||
|
||||
No parallel workers during integration. The integrator may read all task cards and reviews; it writes `.swarm/integration.md` with what changed and why. Freeze the tree while the final verification runs; any post-verification edit invalidates the affected evidence.
|
||||
|
||||
## 7. Close
|
||||
## 8. Close
|
||||
|
||||
Final review is against the **original request**, not the plan: a fresh agent checks the success criteria, the diff, and the evidence. Then the lead writes `.swarm/final-report.md` with these sections:
|
||||
|
||||
@@ -233,6 +252,10 @@ Final review is against the **original request**, not the plan: a fresh agent ch
|
||||
| Reports/reviews dominate input | oversized prose | cap to evidence lines; logs in artifacts |
|
||||
| Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots |
|
||||
| A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command |
|
||||
| Everything passes but the app is laggy | no performance budget; debug build used as evidence | measure release p95/input latency; fix or cut scope before features |
|
||||
| New version works but feels worse than the old one | no baseline comparison | define baseline scenarios; regression = failure |
|
||||
| Verifier passes, user says "not usable" | technical pass treated as product acceptance | product gate + explicit user verdict; fix before next milestone |
|
||||
| Agents add M1/M2 while M0 is rough | milestone gate skipped | freeze scope; accept M0 or explicitly waive before expanding |
|
||||
|
||||
## Time and cost
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Product quality gates
|
||||
|
||||
Functional correctness is necessary, not sufficient. A build can compile, pass tests and produce a
|
||||
screenshot while still being too slow, visually incoherent or unpleasant to use. Do not let
|
||||
`pass` on technical acceptance stand in for product acceptance.
|
||||
|
||||
## Prototype vs product
|
||||
|
||||
Choose the quality profile with the user before dispatch:
|
||||
|
||||
| Profile | Use when | Mandatory gates |
|
||||
|---|---|---|
|
||||
| **Prototype** | the user explicitly wants a rough idea, spike or throwaway demo | runnable artifact, honest limits, no performance promise |
|
||||
| **Product** | the artifact will be used, shipped, demoed to others or compared with an existing version | functional + performance + UX/visual + baseline + human acceptance |
|
||||
|
||||
Do not silently treat a prototype as a product. Do not demand product polish for a disposable spike.
|
||||
State the profile in the brief.
|
||||
|
||||
## Gates
|
||||
|
||||
### 1. Functional
|
||||
|
||||
- acceptance commands and edge cases pass;
|
||||
- the artifact runs, not just compiles;
|
||||
- the verifier can reproduce the checks;
|
||||
- no known crash/data-loss path.
|
||||
|
||||
### 2. Performance
|
||||
|
||||
Performance is part of correctness for interactive products. Define a budget in the brief before
|
||||
code, measure it on a **release/optimized** build, and record the environment.
|
||||
|
||||
Suggested starting points (adjust to the product, do not treat as universal):
|
||||
|
||||
- p95 frame time at the target window size: **≤16.7 ms** for 60 FPS, or a documented lower target;
|
||||
- input-to-visible response: **≤50 ms** for click/selection, **≤100 ms** for heavier actions;
|
||||
- startup to usable UI: **≤2 s** for a local desktop editor;
|
||||
- idle CPU and memory recorded; no runaway process when idle;
|
||||
- test with a realistic scene, not only an empty one (e.g. 100/1 000 entities).
|
||||
|
||||
Measure with a frame-time diagnostic/overlay, a scripted interaction, or an external profiler.
|
||||
Record p50/p95, resolution, GPU/CPU, build profile and commit. **Debug builds are not evidence of
|
||||
release performance**; if only debug is available, report the numbers as provisional and do not claim
|
||||
the budget is met.
|
||||
|
||||
If the budget is not met: reduce scope or fix the hot path before adding features. Do not hide a
|
||||
frame-time regression behind more UI.
|
||||
|
||||
### 3. UX / visual
|
||||
|
||||
For GUI and visual work:
|
||||
|
||||
- run a real 5-minute task, not just a screenshot;
|
||||
- check hierarchy, typography, spacing, contrast, focus, hover/active/disabled states;
|
||||
- check default, minimum and error/empty states;
|
||||
- check that the viewport is not blocked by panels and that resizing does not clip controls;
|
||||
- compare with the existing version/baseline on the same scenarios;
|
||||
- capture artifacts and record the exact commands.
|
||||
|
||||
“Looks fine” is not a verdict. Name what is readable, reachable, clipped or missing.
|
||||
|
||||
### 4. Baseline
|
||||
|
||||
If a previous version exists, define the scenarios where the new version must be **no worse**:
|
||||
startup, time to first edit, selection, drag, save, visual density. A regression against the baseline
|
||||
is a failure even if the new implementation has more features.
|
||||
|
||||
### 5. Human acceptance
|
||||
|
||||
A verifier can check evidence; it cannot decide that the product feels usable. After each milestone
|
||||
or vertical slice:
|
||||
|
||||
1. the lead gives the user a short runnable artifact and a 3–5 step scenario;
|
||||
2. the user runs it and answers: usable / usable with issues / not usable;
|
||||
3. issues become the next milestone, not a footnote in a report;
|
||||
4. **do not start the next milestone until the current one is accepted** (or explicitly waived).
|
||||
|
||||
Record the verdict in `state.json`/`final-report.md`. A detached verifier `pass` is not a substitute.
|
||||
|
||||
### 6. Scope
|
||||
|
||||
One milestone per session. Do not implement M1/M2 while M0 is still unpleasant. If the current slice is
|
||||
not usable, adding assets, play mode or export will not fix it.
|
||||
|
||||
## Stop rules
|
||||
|
||||
Stop and re-plan when any of these is true:
|
||||
|
||||
- functional acceptance passes but the product is visibly slower or worse than the baseline;
|
||||
- performance was not measured on an optimized build for a product task;
|
||||
- the user says “not usable” and the next step is more features instead of fixing the current slice;
|
||||
- the work has consumed half the milestone budget/time without a runnable artifact;
|
||||
- a UI slice has multiple parallel writers and the visual language is drifting;
|
||||
- the verifier can only produce a `pass` by ignoring performance/UX.
|
||||
|
||||
## Minimal product gate
|
||||
|
||||
> Release build, measured p95 frame time and input latency, 5-minute real scenario, baseline
|
||||
> comparison, explicit user verdict. If any is missing, the product gate is `not_run`, not `pass`.
|
||||
@@ -32,9 +32,11 @@ Useful ratios:
|
||||
|
||||
- Set a **hard budget** for the milestone and a **soft cap** at ~70%. At the soft cap, stop expanding
|
||||
scope; finish the current slice and verify.
|
||||
- Reserve **20–30%** for verification, integration and one round of fixes. Do not spend the whole
|
||||
budget on generation.
|
||||
- Budget per phase, not only per wave: recon, build, verify, integrate.
|
||||
- Reserve **20–30%** for verification, integration and one round of fixes, plus a **polish reserve**
|
||||
for product-profile GUI/UX work. Do not spend the whole budget on generation.
|
||||
- A budget is a ceiling, not a target. Do not stop a product task at the soft cap if the quality gate
|
||||
is unmet; stop at the hard cap or when the product gate passes. Cheapness is not quality.
|
||||
- Budget per phase, not only per wave: recon, build, verify, product gate, integrate.
|
||||
- Stop at the cap and report partial results plus unverified areas. Silent overspend is worse than a
|
||||
partial result.
|
||||
- Calibrate from the first measured milestone; do not copy invented numbers from a template.
|
||||
@@ -94,6 +96,7 @@ This preserves quality while avoiding repeated full reviews.
|
||||
| 20 tiny tasks | fixed overhead per child | merge into coherent units |
|
||||
| Reports/reviews as essays | output + lead input | schema + evidence lines |
|
||||
| Whole product in one wave | late integration, huge rework | vertical slices + checkpoints |
|
||||
| Cutting polish to stay under budget | functional but unusable product | protect a polish reserve; stop at the hard cap, not the soft cap |
|
||||
| Vague "max quality" with no budget | unbounded spend | hard/soft caps + reserve |
|
||||
|
||||
## Minimal operating rule
|
||||
|
||||
Reference in New Issue
Block a user