Ticket #260: latency budget and SLO — verification
- Branch:
ticket/260-latency-budget - Base:
integration/prod-partials(66223e6) - Spec: #255, ticket #260
- Policy: latency-budget.md
- Verified: 2026-11-16 on Node 26.12.1 / OpenCode 2.2.7, Linux/tmux
What changed
src/v2/latency-budget.tsis the pure evaluator. It owns the reference and smoke profiles, nearest-rank percentiles, per-metric gate/report decisions, and the formatted report. First-token is always report-only.src/v2/latency-budget.test.tscovers the percentiles, pass/fail at each threshold, missing/invalid samples, the smoke profile, report-only first-token, the report profile, and a deliberate regression.scripts/lib/latency-driver.mjsis the deterministic patch-card fixture. It invokestask.delegate, reads the durablechanges.sincepatch after that version, applies the subtask changes to a client-side state map, and renders the realworkCardForTaskmapping. It never waits for a model token, so the fake provider's hold cannot dominate the card number.scripts/latency-budget.mjsevaluates a saved result with the pure evaluator and exits nonzero on a budget failure.scripts/live-latency-probe.shreports three metrics separately: gated rail keypress p95, gated patch-card p95, and report-only first-token. The default profile is the generoussmokePR threshold; the reference profile enforces 50 ms / 250 ms.ENSEMBLE_LATENCY_REGRESSION_MSinjects a deliberate regression. The result JSON goes toENSEMBLE_LATENCY_OUTfor CI artifacts..github/workflows/latency.ymlruns the smoke profile on pull requests and the reference profile nightly.npm run verifyruns the pure evaluator on every PR;npm run test:live:latency:referenceis the documented reference gate.package.jsonaddstest:live:latency:referenceandlatency:budget.
No TUI layout or behavior changed. The probe reuses the #256 harness readiness/artifact helpers and classifies rail/first-token wait timeouts as timing failures; a budget failure is a product failure and never retries.
Evidence
| Check | Result | Notes |
|---|---|---|
npm run verify | PASS | Typecheck + 2,007 unit tests |
npm run test:live:latency (smoke) | PASS | rail p95 15 ms, patch-card p95 38 ms, first-token 642 ms (report only) |
npm run test:live:latency:reference | PASS | rail p95 10 ms, patch-card p95 36 ms, first-token 672 ms (report only) |
ENSEMBLE_LATENCY_REGRESSION_MS=1000 reference | FAIL as designed | patch-card p95 1,051 ms > 250 ms; exit 1 |
node --test src/v2/latency-budget.test.ts | PASS | 13 evaluator tests, including the regression slice |
bash -n scripts/live-latency-probe.sh | PASS | No shell syntax errors |
npm run docs:check | PASS | Freshness check after the docs index update |
Baseline and thresholds
Measured on the reference host (Linux, Node 26.12.1, OpenCode 2.2.7, tmux, 140x50 TUI):
| Metric | Samples (ms) | p50 | p95 | Reference budget |
|---|---|---|---|---|
rail | 9, 9, 9, 9, 9, 9, 9, 10, 9, 9 | 9 | 10 | 50 |
patchCard | 36, 32, 33, 31, 33 | 33 | 36 | 250 |
firstToken | 672 | 672 | 672 | report only |
The earlier tmux card-appearance number (~1.9 s p50) was dominated by the client feed path and the fake model hold; it is not the patch-card budget metric. The deterministic fixture measures the RPC -> durable patch -> real card render path directly and leaves the TUI capture out of the gate.
Reference gate policy
- PRs:
npm run verify(pure evaluator) and thesmokelive job (rail 500 ms, patch-card 5000 ms). The smoke thresholds are generous because a shared PR runner is not a reference host. - Reference host/nightly:
npm run test:live:latency:reference(rail 50 ms, patch-card 250 ms). The nightly workflow pins Node 26.12.1 and OpenCode 2.2.7 and runs the preflight first. - Local measurement without a gate:
ENSEMBLE_LATENCY_PROFILE=report npm run test:live:latency. - Deliberate regression:
ENSEMBLE_LATENCY_PROFILE=reference ENSEMBLE_LATENCY_REGRESSION_MS=1000 npm run test:live:latencyexits 1.
Merge notes and conflict risk
integration/prod-partials was merged into the ticket branch before commit. The only shared files with ticket #261 (full deterministic E2E in CI) are the docs index, scripts/README.md, package.json, and CI. This ticket adds a separate latency.yml and does not edit verify.yml; #261 can fold the smoke and reference jobs into its matrix and remove the duplicate workflow when it lands. The probe script itself is additive.
Skips
- None. The reference and smoke runs used the isolated fake host; no real model or external service was required.
Privacy
- The result JSON contains fixture sample timings, counts, and generated fixture titles only.
- No conversation content, credentials, file bytes, or private paths are committed. CI failure artifacts use the #256 redaction path.