v5.9 T2 verification — CI fake-live guardrail
Status: verify and deterministic live suite passed locally; hosted CI not run Date: 2026-11-05 Ticket: #107 Parent specification: #105 Branch: feat/v59-107 Feature branch base: bf2c54b; integration merge commit: 27ae7f0
This record covers the CI live job, the pinned host toolset, and the skip-versus-fail contract. It does not close the parent specification or this ticket. No product runtime behavior changed.
Environment
| Item | Value |
|---|---|
| OS | Linux |
| Terminal host | tmux 3.5a |
| Node | 26.12.1 |
| npm | 12.2.1 |
| OpenCode | 2.2.5 |
fd | fdfind 10.2.0 |
| Git | 2.47.3 |
The repository still type-checks against @opencode/plugin 2.2.1. The live checks exercise the host binary, which is 2.2.5 locally and is the version the CI live job installs.
Delivered
| Acceptance area | Result |
|---|---|
| Verify job remains | .github/workflows/verify.yml keeps the verify job (npm ci, npm run verify) unchanged apart from sharing the pinned Node version. |
| Deterministic fake/empty suite in CI | A new live job runs on push and pull request and executes npm run test:live:fake:all (fake then empty catalog). |
| Pinned tool setup | The job pins Ubuntu 24.04, Node 26.12.1, OpenCode 2.2.5, and fd 10.2.0, and installs tmux from the runner image. |
| Missing tool fails, not skips | ENSEMBLE_LIVE_REQUIRE=1 turns harness skips into failures; the aggregate runner also treats a SKIP row as a failure. Local runs keep the skip-and-exit-zero contract. |
| Preflight | scripts/live-preflight.sh and npm run test:live:preflight resolve the binaries, print versions, fail on missing tools in require mode, and fail on a present-but-wrong pinned Node/OpenCode version. |
| Broken assertion fails the job | A temporary wrong TUI assertion made the require-mode matrix exit 1 with a FAIL summary; the file was restored immediately. |
| Real-model checks stay local | The live job runs only the fake and empty-catalog scenarios. test:live/test:live:all are not invoked in CI. |
| Documentation | scripts/README.md documents tool setup, runtime, the CI flow, the skip-vs-fail contract, and local duration; the runtime guide and CONTRIBUTING.md link the same contract. |
Commands
| Command | Result |
|---|---|
python3 -c 'import yaml; ...' .github/workflows/verify.yml | Pass — YAML parses; jobs are verify and live; triggers are push and pull_request. |
bash -n scripts/live-preflight.sh scripts/live-matrix.sh scripts/lib/live-harness.sh | Pass — no shell syntax errors. |
git diff --check | Pass — no whitespace errors. |
npm run verify | Pass — strict type check, then 666 tests passed, 0 failed, 0 skipped. |
ENSEMBLE_LIVE_REQUIRE=1 npm run test:live:preflight | Pass — node v26.12.1, opencode v2.2.5, tmux 3.5a, fdfind 10.2.0. |
ENSEMBLE_LIVE_REQUIRE=1 npm run test:live:fake:all | Pass — fake 167s and fake-empty 8s, 2 passed, 0 failed, 0 skipped, 0 not run in 175s. A second run passed in 176s (168s + 8s). |
Missing-tool require mode: HOME=<empty> PATH=<no opencode> ENSEMBLE_LIVE_REQUIRE=1 bash scripts/live-preflight.sh | Exit 1 — fails with opencode is not installed or not on PATH (required; ENSEMBLE_LIVE_REQUIRE=1). |
Missing-tool local mode: same command without ENSEMBLE_LIVE_REQUIRE | Exit 0 — prints a skipping line, preserving local behavior. |
Missing-tool require mode: same sandbox with bash scripts/live-matrix.sh fake | Exit 1 — FAIL fake, NOT RUN fake-empty, no green summary. |
Missing-tool local mode: same sandbox with bash scripts/live-matrix.sh fake | Exit 0 — two SKIP rows, 2 skipped, as before. |
Deliberately broken assertion: temporary DELIBERATELY-BROKEN-ASSERTION in scripts/live-check-fake.sh, then ENSEMBLE_LIVE_REQUIRE=1 bash scripts/live-matrix.sh fake | Exit 1 — FAIL fake: TUI did not show: Ensemble page; NOT RUN fake-empty. The file was restored with no diff. |
The local missing-tool and skip-mode checks used a temporary HOME without the nvm OpenCode binary while keeping the shell and system tools available. That exercises the same resolution path as an absent host tool; no real tool was removed or modified.
Intermediate run
The first ENSEMBLE_LIVE_REQUIRE=1 matrix run failed at the agent wizard (timed out after 10s waiting for agent title step) while the pane showed the next Agent role step. No assertion related to this ticket had run yet. The two subsequent full runs passed every scenario. This is recorded as an intermediate live interaction flake, not as evidence of a product regression; it was not retried automatically and no live assertion was weakened.
Not run
- Hosted GitHub Actions: not triggered or executed from this environment. The workflow syntax, job steps, and equivalent local commands were checked; this record does not claim a passing hosted CI run.
- Real-model live checks: intentionally not run for this ticket and not required in CI.
- macOS/Herdr: unchanged manual check; this work is Linux/CI tooling.