Skip to content

Ticket #256: live harness flake policy — verification ​

  • Branch: ticket/256-flake-policy
  • Base: integration/prod-partials (188fe5e, including #257/#258/#259/#264)
  • Spec: #255, ticket #256
  • Policy: flake-policy.md
  • Verified: 2026-11-16 on Node 26.12.1 / OpenCode 2.2.7

What changed ​

  • The wizard alignment check waits for step readiness (live_require_ready_text) and the typed field value (live_type) before Enter. Fixed sleeps around wizard transitions were removed; Enter is always a separate input event.
  • live_type observes the last four non-space characters of the typed text plus a pane change, so a wrapped long line still proves the field committed.
  • scripts/live-matrix.sh classifies a failed attempt from the harness failure.json record. Only a timing failure retries, once. Product, unknown, and infrastructure failures never retry. Skips keep the existing local-skip / require-mode-fail contract.
  • The harness writes redacted pane captures, logs, failure.json, and failure.txt under ENSEMBLE_LIVE_ARTIFACTS on failure. CI uploads the directory.
  • scripts/flake-quarantine.json entries require a linked issue and an owner; the matrix refuses to start with an invalid entry.
  • src/v2/flake-policy.ts owns the pure classifier, retry decision, quarantine validation, and report. src/v2/flake-policy.test.ts and src/v2/live-matrix.test.ts cover the policy and the matrix loop with stub scenarios.
  • The mixed-anchor assignment check reads the durable system message instead of the visible chat cell: a long streamed reply can scroll the older assignment line out of the viewport even though routing is correct. The Work rows, replies, and exact instruction remain asserted separately.
  • Header assertions use the compact #263 prefix (… · N active) because the merged three-pane width may drop the done segment while keeping the page and counts.

Evidence ​

CheckResultNotes
Alignment scenario, 20 consecutive runsPASS 20/20Merged tree; exact readiness/Enter block
ENSEMBLE_LIVE_REQUIRE=1 npm run test:live:fake:allPASSfake + empty suites; 0 retries, 0 failures; budget=ok
npm run verifyPASStypecheck + 1,989 unit tests
Synthetic timing failurePASSclassified timing; retried once; report names suite/attempt/classification/artifact
Synthetic product failurePASSclassified product; never retried
Invalid quarantinePASSmissing issue/owner refused before the run
Failure artifactsPASSredacted pane/log capture plus failure.json

Merge notes ​

integration/prod-partials was merged into the ticket branch. The only conflict was the docs/README.md verification index; both the integration entries and the flake-policy entries were kept. The merged header compaction (#263) required the live header assertions above to accept the compact form.

Skips ​

  • None in the final fake matrix. ENSEMBLE_LIVE_REQUIRE=1 makes any skip a failure.

Privacy ​

  • Artifacts are ephemeral local/CI diagnostics, not committed fixtures.
  • The harness redacts the live base, live home, live project, caller home, and the server password; credential-looking lines are dropped.
  • No conversation content, credentials, file bytes, or private paths are committed.

Released under the MIT License.