Skip to content

Ticket #260: latency budget and SLO — verification ​

  • Branch: ticket/260-latency-budget
  • Base: integration/prod-partials (66223e6)
  • Spec: #255, ticket #260
  • Policy: latency-budget.md
  • Verified: 2026-11-16 on Node 26.12.1 / OpenCode 2.2.7, Linux/tmux

What changed ​

  • src/v2/latency-budget.ts is the pure evaluator. It owns the reference and smoke profiles, nearest-rank percentiles, per-metric gate/report decisions, and the formatted report. First-token is always report-only.
  • src/v2/latency-budget.test.ts covers the percentiles, pass/fail at each threshold, missing/invalid samples, the smoke profile, report-only first-token, the report profile, and a deliberate regression.
  • scripts/lib/latency-driver.mjs is the deterministic patch-card fixture. It invokes task.delegate, reads the durable changes.since patch after that version, applies the subtask changes to a client-side state map, and renders the real workCardForTask mapping. It never waits for a model token, so the fake provider's hold cannot dominate the card number.
  • scripts/latency-budget.mjs evaluates a saved result with the pure evaluator and exits nonzero on a budget failure.
  • scripts/live-latency-probe.sh reports three metrics separately: gated rail keypress p95, gated patch-card p95, and report-only first-token. The default profile is the generous smoke PR threshold; the reference profile enforces 50 ms / 250 ms. ENSEMBLE_LATENCY_REGRESSION_MS injects a deliberate regression. The result JSON goes to ENSEMBLE_LATENCY_OUT for CI artifacts.
  • .github/workflows/latency.yml runs the smoke profile on pull requests and the reference profile nightly. npm run verify runs the pure evaluator on every PR; npm run test:live:latency:reference is the documented reference gate.
  • package.json adds test:live:latency:reference and latency:budget.

No TUI layout or behavior changed. The probe reuses the #256 harness readiness/artifact helpers and classifies rail/first-token wait timeouts as timing failures; a budget failure is a product failure and never retries.

Evidence ​

CheckResultNotes
npm run verifyPASSTypecheck + 2,007 unit tests
npm run test:live:latency (smoke)PASSrail p95 15 ms, patch-card p95 38 ms, first-token 642 ms (report only)
npm run test:live:latency:referencePASSrail p95 10 ms, patch-card p95 36 ms, first-token 672 ms (report only)
ENSEMBLE_LATENCY_REGRESSION_MS=1000 referenceFAIL as designedpatch-card p95 1,051 ms > 250 ms; exit 1
node --test src/v2/latency-budget.test.tsPASS13 evaluator tests, including the regression slice
bash -n scripts/live-latency-probe.shPASSNo shell syntax errors
npm run docs:checkPASSFreshness check after the docs index update

Baseline and thresholds ​

Measured on the reference host (Linux, Node 26.12.1, OpenCode 2.2.7, tmux, 140x50 TUI):

MetricSamples (ms)p50p95Reference budget
rail9, 9, 9, 9, 9, 9, 9, 10, 9, 991050
patchCard36, 32, 33, 31, 333336250
firstToken672672672report only

The earlier tmux card-appearance number (~1.9 s p50) was dominated by the client feed path and the fake model hold; it is not the patch-card budget metric. The deterministic fixture measures the RPC -> durable patch -> real card render path directly and leaves the TUI capture out of the gate.

Reference gate policy ​

  • PRs: npm run verify (pure evaluator) and the smoke live job (rail 500 ms, patch-card 5000 ms). The smoke thresholds are generous because a shared PR runner is not a reference host.
  • Reference host/nightly: npm run test:live:latency:reference (rail 50 ms, patch-card 250 ms). The nightly workflow pins Node 26.12.1 and OpenCode 2.2.7 and runs the preflight first.
  • Local measurement without a gate: ENSEMBLE_LATENCY_PROFILE=report npm run test:live:latency.
  • Deliberate regression: ENSEMBLE_LATENCY_PROFILE=reference ENSEMBLE_LATENCY_REGRESSION_MS=1000 npm run test:live:latency exits 1.

Merge notes and conflict risk ​

integration/prod-partials was merged into the ticket branch before commit. The only shared files with ticket #261 (full deterministic E2E in CI) are the docs index, scripts/README.md, package.json, and CI. This ticket adds a separate latency.yml and does not edit verify.yml; #261 can fold the smoke and reference jobs into its matrix and remove the duplicate workflow when it lands. The probe script itself is additive.

Skips ​

  • None. The reference and smoke runs used the isolated fake host; no real model or external service was required.

Privacy ​

  • The result JSON contains fixture sample timings, counts, and generated fixture titles only.
  • No conversation content, credentials, file bytes, or private paths are committed. CI failure artifacts use the #256 redaction path.

Released under the MIT License.