Skip to content

Ticket #261: full deterministic E2E in CI — verification ​

  • Branch: ticket/261-ci-matrix
  • Base: integration/prod-partials (4a84aa8, merge of #260)
  • Spec: #255, ticket #261
  • Host: Linux/tmux, Node 26.12.1, npm 12.2.1, OpenCode 2.2.7, tmux 3.5a, fd 10.2.0
  • Gate: npm run verify (typecheck + unit suite), npm run docs:check

What changed ​

  • verify.yml is split by cost into verify, live-fake, live-native, live-multilocation, and latency-smoke. Every live job runs in parallel, sets ENSEMBLE_LIVE_REQUIRE=1, writes redacted artifacts to $RUNNER_TEMP/ensemble-live-artifacts, and uploads them on failure. The workflow has a concurrency group that cancels superseded PR runs.
  • nightly.yml owns latency-reference (the strict 50/250 ms profile) and real (test:live with the real host catalog). The real job is opt-in via the real dispatch input or ENSEMBLE_NIGHTLY_REAL=1; the schedule is 17 4 * * * with workflow_dispatch available.
  • .github/actions/live-host/action.yml installs the pinned toolset once: Node, tmux, fd 10.2.0, OpenCode from the input version, and npm ci. Every live job in both workflows uses it, so the host pin cannot drift job by job.
  • verify.yml pins OpenCode 2.2.7, the latest host in SUPPORTED_OPENCODE_VERSIONS; compatibility.yml continues to run 2.2.6 and 2.2.7. The old latency.yml is removed: the smoke profile is in verify.yml and the reference gate is in nightly.yml.
  • src/v2/ci-workflows.ts is the pure job-graph checker. It owns the job budgets, suite ownership, host-pin check, and repository check. src/v2/ci-workflows.test.ts mutates workflow text to prove a missing suite, a duplicated suite, a missing require mode, a missing artifact upload, an over-budget or missing timeout, a stale host pin, a missing concurrency block, a non-reference nightly profile, a non-opt-in real job, a PR-triggered nightly workflow, and a hardcoded action version all fail.
  • scripts/live-lifecycle-fake.sh now matches the compact Work header (WORK ACTIVE │ N group(s) · N active) introduced by #263's 53-column card-mode floor. This is a test-harness assertion fix, not a product change; the group label still carries the active/done counts.

No product behavior changed. The skip-versus-fail contract is unchanged: local prerequisites skip, ENSEMBLE_LIVE_REQUIRE=1 turns a skip into a failure, and only a classified timing failure may retry once.

Job graph and time budget ​

PR and push (verify.yml) ​

JobSuitesTimeoutObserved
verifynpm run verify15 min~1 min
live-faketest:live:fake:all, test:live:sizes, test:live:lifecycle, test:live:subagent45 min~16 min
live-nativetest:live:native-first, test:live:vera, test:live:tool-surface20 min~3 min
live-multilocationtest:live:multilocation20 min~3 min
latency-smoketest:live:latency (smoke: 500 ms rail, 5000 ms card)20 min~3 min

The live jobs run in parallel, so the PR critical path is live-fake plus setup: target 20 minutes, budget 30 minutes wall clock. The deterministic suite sum is about 15 minutes locally (fake 419 s, lifecycle 334 s, sizes ~57 s, multilocation 71 s, subagent 19 s, native-first 10 s, Vera 19 s, tool-surface 5 s, latency 8 s); tool setup and npm ci add the rest.

Nightly (nightly.yml) ​

JobSuitesTimeoutObserved
latency-referencetest:live:latency (reference: 50 ms rail, 250 ms card)30 min~3 min
realtest:live with the real host catalog, opt-in30 min~3 min

Both nightly jobs together stay within a 60-minute wall-clock budget.

Kept separate ​

compatibility.yml runs the host probe and fake smoke on OpenCode 2.2.6 and 2.2.7; node-matrix.yml runs npm run verify on Node 22.23.4, 24.22.1, and 26.12.1; docs.yml runs npm run docs:check.

Evidence ​

All live commands ran with ENSEMBLE_LIVE_REQUIRE=1 on the isolated fake host.

CheckResultNotes
npm run verifyPASSstrict typecheck; 2,024 tests, 0 failed/skipped
npm run docs:checkPASScontract v7; OpenCode 2.2.6, 2.2.7; Node >=22.23.4
npm run test:live:preflight -- allPASSNode 26.12.1, OpenCode 2.2.7, tmux 3.5a, fd 10.2.0
npm run test:live:fake:allPASS419 s
npm run test:live:sizesPASS80x24/100x30/120x40/160x50 + NO_COLOR
npm run test:live:lifecyclePASS334 s after the compact-header assertion fix
npm run test:live:subagentPASS19 s
npm run test:live:native-firstPASS10 s
npm run test:live:veraPASS19 s
npm run test:live:tool-surfacePASSnative 6 tools vs legacy 11
npm run test:live:multilocationPASS71 s
npm run test:live:latencyPASS8 s, smoke profile
npm run test:live:latency:referencePASSrail p95 12 ms, patch-card p95 39 ms; first-token 680 ms report only

The CI job-graph check runs as part of npm test (17 cases, including the repository check). The workflow files were not executed on GitHub Actions from this worktree; the local suites above are the deterministic evidence.

First attempt ​

The first test:live:lifecycle run failed after 20 s waiting for WORK ACTIVE │ 1 group · 1 active · 0 done. The captured pane showed the compact post-#263 header (WORK ACTIVE │ 1 group · 1 active │ page 1/1) with the group label GROUP 1 · 1 active · 0 done directly below. The assertion expected the pre-#263 full header. The harness now matches the common prefix; the rerun passed in 334 s. No product code or layout changed.

Skips ​

  • test:live (real model) was not run: it is the opt-in nightly job and needs the runner's real catalog.
  • The compatibility matrix was not re-run locally: it needs OpenCode 2.2.6 in addition to the installed 2.2.7. The workflow and its probe/fake steps are unchanged.
  • GitHub Actions itself was not executed; the job graph is validated by the pure checker and the suites by the local runs above.

Privacy ​

No conversation content, credentials, file bytes, or private paths are committed. Failure artifacts use the #256 redaction path; the live fixtures carry IDs, counts, lengths, and digests only.

Merge notes and conflict risk ​

integration/prod-partials is merged into the ticket branch before commit. Shared files with the other partials: scripts/README.md, README.md, docs/README.md, and the workflow directory. latency.yml is deleted, so a later edit to that file conflicts by design; #260's smoke/reference jobs now live in verify.yml and nightly.yml. The lifecycle assertion change is a one-line-per-assertion edit in a file #263/#264 also touched; conflict risk is low but non-zero.

Released under the MIT License.