Add product quality gates and milestone acceptance
This commit is contained in:
Emil Shanaty
@@ -12,6 +12,8 @@ Answer these before spawning:
|
||||
- **Interfaces.** Can you freeze signatures, data shapes, and file names before dispatch?
|
||||
- **Breadth.** Is the time dominated by many independent pieces, not one deep chain?
|
||||
|
||||
First decide the **quality profile** with the user: `prototype` (runnable spike, no performance promise) or `product` (performance, UX, baseline and human acceptance are mandatory). See `references/product-quality.md`. For UI/product work, prefer one implementer + one critic over parallel panel owners.
|
||||
|
||||
Budget:
|
||||
- `agents ≈ units + ceil(units / 5) verifiers + 1–2 integrators + 2–4 recon`; add retries and re-verifications explicitly. Set `max_agents` only after checking available tools and slot cap.
|
||||
- Tokens ≈ `agents × fixed overhead (10–25k) + context reads + output + reviews`. Treat this as an estimate, then calibrate from measured usage. A 3k-line file read by several agents can cost more than the code they write.
|
||||
@@ -96,6 +98,8 @@ Write the plan before any worker starts. Keep the plan proportionate: a small do
|
||||
- [ ] Files/modules over ~800 lines are split, or the task explicitly owns the integration cost.
|
||||
- [ ] Token budget, soft cap and verification reserve are recorded; units are merged/split to avoid fixed-overhead waste.
|
||||
- [ ] The plan uses vertical slices/checkpoints for a multi-milestone product.
|
||||
- [ ] Quality profile (`prototype`/`product`), performance budget, baseline scenarios and user-acceptance owner are recorded.
|
||||
- [ ] For product-profile GUI work, the plan names release-build measurements and the exact user scenario.
|
||||
- [ ] No path in two write scopes.
|
||||
- [ ] Contracts frozen for every cross-task interface.
|
||||
- [ ] Coverage matrix complete.
|
||||
@@ -183,13 +187,28 @@ For a GUI product, give each visual verifier its own Xvfb/display and project co
|
||||
|
||||
**Freeze before verify.** Once you start this wave, do not edit source, tests or docs. If you already started and then fixed something, mark the in-flight review stale and re-run that verifier on the new build. A verifier's verdict is only valid for the exact tree it saw.
|
||||
|
||||
**Technical verification is not product acceptance.** Verifiers check evidence; for product-profile work they must also measure release-build frame time/input latency, run the real UX scenario and compare with the baseline. If performance/UX was not measured, the product gate is `not_run`, not `pass`.
|
||||
|
||||
Use `assets/verify.workflow.js` and `assets/verdict.schema.json`, not the full worker report schema. The verifier's detailed review **must be written to `.swarm/reviews/Vxx.md` before it returns**; the workflow return is only `{task_id, verdict, summary, review_file}`. Do not treat a workflow `null`/`no usable verdict` as a product failure if the review file exists — read the file, then re-run with the minimal schema if needed.
|
||||
|
||||
Evidence standard: for every criterion, a command/artifact/citation/metric and its observed result. `pass` without evidence is `not_run`. Accept `pass_with_risks` only with named risks and a follow-up task. Verifiers should copy or hash-check generated files rather than rewriting the tree; if a required gate script rewrites generated artifacts, report the hash delta.
|
||||
|
||||
On failure, create a fix task with the verifier's repro; do not send the verifier back to fix its own finding. After fixes, re-run only the affected **delta** (diff + findings + acceptance commands), not the whole repository, and mark the previous review stale; repeat until the remaining findings are minor/documented. Then run one final integrated verification pass on the frozen build.
|
||||
|
||||
## 6. Integrate
|
||||
## 6. Product gate
|
||||
|
||||
For `product`-profile work, run this before integration:
|
||||
|
||||
1. Build the release/optimized artifact. Debug numbers are provisional only.
|
||||
2. Measure p50/p95 frame time at the target size, input-to-visible latency, startup and idle CPU/memory on a realistic scene.
|
||||
3. Run the 3–5 step real user scenario, not just a screenshot.
|
||||
4. Compare with the baseline version on the same scenario; a regression is a failure.
|
||||
5. Give the user the artifact and ask for an explicit verdict: usable / usable with issues / not usable.
|
||||
6. Record measurements and the user verdict. Do not start the next milestone until accepted or explicitly waived.
|
||||
|
||||
If the result is functional but laggy or visually worse than the baseline, stop and fix the slice. Do not add features to compensate. See `references/product-quality.md`.
|
||||
|
||||
## 7. Integrate
|
||||
|
||||
One integrator (two only for disjoint areas) owns:
|
||||
- shared files and cross-cutting edits;
|
||||
@@ -199,7 +218,7 @@ One integrator (two only for disjoint areas) owns:
|
||||
|
||||
No parallel workers during integration. The integrator may read all task cards and reviews; it writes `.swarm/integration.md` with what changed and why. Freeze the tree while the final verification runs; any post-verification edit invalidates the affected evidence.
|
||||
|
||||
## 7. Close
|
||||
## 8. Close
|
||||
|
||||
Final review is against the **original request**, not the plan: a fresh agent checks the success criteria, the diff, and the evidence. Then the lead writes `.swarm/final-report.md` with these sections:
|
||||
|
||||
@@ -233,6 +252,10 @@ Final review is against the **original request**, not the plan: a fresh agent ch
|
||||
| Reports/reviews dominate input | oversized prose | cap to evidence lines; logs in artifacts |
|
||||
| Visual agents produce wrong/identical screenshots | shared desktop or portal capture | isolate per-agent Xvfb/config/project; use in-app screenshots |
|
||||
| A visual "pass" has no artifact | verifier skipped the visual tier | treat as `not_run`; require screenshot path + command |
|
||||
| Everything passes but the app is laggy | no performance budget; debug build used as evidence | measure release p95/input latency; fix or cut scope before features |
|
||||
| New version works but feels worse than the old one | no baseline comparison | define baseline scenarios; regression = failure |
|
||||
| Verifier passes, user says "not usable" | technical pass treated as product acceptance | product gate + explicit user verdict; fix before next milestone |
|
||||
| Agents add M1/M2 while M0 is rough | milestone gate skipped | freeze scope; accept M0 or explicitly waive before expanding |
|
||||
|
||||
## Time and cost
|
||||
|
||||
|
||||
Reference in New Issue
Block a user