Skip to content

Ticket #241: Vera regression and cutover verification ​

  • Date: 2026-11-11
  • Branch: ticket/241-review-fixes (review-fix pass on ticket/241-vera-verification)
  • Base: integration/native-first-defaults at 514ac86 (includes #236–#240 and the original #241 merge)
  • Spec: parent #235; ticket #241
  • ADRs: 0006, 0011
  • Unit gate: npm run verify — strict typecheck plus 1,868 tests, 0 failed, 0 skipped
  • Live gate: npm run test:live:vera — seed 1/1, run 54/54, restart 18/18, exit 0
  • Host matrix: OpenCode 2.2.7, Node 26.12.1, npm 12.2.1, tmux 3.5a, Debian 13
  • Plugin: ensemble 0.1.0, shared-service version 3, contract version 6
  • Provider: isolated local ensemble-fake OpenAI-compatible provider; the real suite also ran on the local catalog, no external network

Scenario ​

scripts/live-vera-regression.sh (npm run test:live:vera) runs one isolated fake host three times against a real temp workspace:

  1. Seed. Create Vera (leader), Clark, Kakiko, and Phoenix with stored titles and roles, create a workspace whose memberIDs order differs from agent creation order, warm the group binding, and seed a stale v5 AGENTS.md block plus a project-owned line.
  2. Upgrade shape. Stop the host and remove only the persisted nativeFirst key from the isolated plugin store. The fixture reports row counts and the removed key only; it never reads or copies message content.
  3. Run. Restart, assert the absent value resolved to on and was persisted, assert the generated v6 roster table, drive the leader turn, and exercise rollback and restore.
  4. Restart. Restart again and assert durable turns, ledger, roster, and the no-completion result.

The explicit delegation uses the documented :tools legacy surface while native-first stays on: the native tool list deliberately omits ensemble_delegate, so the migration override is the only way one native turn can call it. Native ingestion, routing, the contact ledger, and completion stay on the native path.

Review-fix pass ​

The two-axis review of the #241 cutover produced these code and harness fixes. Behavior changes are covered by the unit/service suites; the live results below were re-recorded on this branch.

  • One native-first default. DEFAULT_NATIVE_FIRST_MODE is the single exported on constant used by DEFAULT_SETTINGS, nativeFirstMode, and the one-time store migration. An explicit persisted off/shadow is never rewritten; absent or invalid values migrate to the default.
  • Explicit-tool dedupe. After the existing idempotency lookup, delegate, handoff, and group-post/approval paths check the native turn's contacted ledger. A recipient already contacted in the turn gets a clear "already contacted this turn" result and no second child or approval.
  • Anchor validation. The pure anchor resolver now requires the candidate task to belong to the calling workspace and agent (including a workspace direct dm:<agent>:<workspace> turn). The group-channel rule that a workspace post must not borrow a DM turn remains a separate documented condition in send_message.
  • Anchor cleanup. The unused taskMaxHops field was removed from ActiveTurnAnchor; the contact ledger remains and is the dedupe source.
  • Typed contact source. NativeContactSource is a domain union; the durable ledger stores recipient IDs only and the source code stays in the native.contact.recorded log event.
  • One focus model. Card focus registers every rendered card at successive WORK_CARD_ROWS, matching the renderer's section.cards; one-card pagination makes this a no-visual-change agreement.
  • One command catalog. Command arity lives on CommandSpec; parsing and completions derive from COMMANDS. :native-first on changes only the mode and never clears an explicit toolSurface: legacy; :tools native is the separate restore.
  • Harness exactness. The Vera driver imports the product teamRosterTable renderer (run with --experimental-strip-types) for the expected table, keeps an independent leader-first/memberIDs ordering assertion, and applies the real nativeFirstCommand / toolsCommand patches through settings.update. The TUI key handler is a thin wrapper covered by src/v2/tui/commands.test.ts.
  • Default-on service coverage. A fresh store with no seeded settings starts on, persists the default, and registers the native surface.

The broad test:live:fake walkthrough predates default-on and asserts the legacy direct, DM-queue, and statusline surfaces. It now runs that scenario on the explicit legacy path; the Vera regression covers the default-on cutover, and the unit/service suites cover the default itself. This keeps the legacy assertions meaningful instead of rewriting them to the new surface.

The dispatch-budget probe inside test:live:multilocation also predates default-on and drives the explicit ensemble_handoff/ensemble_delegate tools, which the native surface deliberately does not register. It now pins nativeFirst: "off" before each scenario and waits for the mode to land, so the budget ledger is exercised on the legacy surface it was written for.

Regression assertions ​

Default-on restart from an unset store

  • Before restart: one v2/meta row, a persisted nativeFirst, exactly one removed key, and no nativeFirst key afterwards.
  • After restart: the store contains nativeFirst: "on", the diagnostic reports on, and agent.sessionBinding reports mode: "on".
  • The native.mode log reports mode: "on" and migrated: true.

Generated roster table

The real workspace AGENTS.md contains one v6 block and this exact table (fixture titles/roles; leader first and marked, then memberIDs order):

MemberTitleRole
Vera (leader)Vera LeadOwns the Vera regression and release sign-off.
ClarkRoute CheckerConfirms each mentioned member receives exactly one turn.
KakikoCompletion GuardChecks that no second completion turn reaches the leader.
PhoenixRoster KeeperKeeps the generated team roster exact.

The expected table is the product teamRosterTable renderer's exact output (imported by the driver under --experimental-strip-types), and a separate assertion parses the rendered member names to prove leader-first then memberIDs order. The project-owned line outside the Ensemble markers survives and the stale v5 text is gone.

One leader turn, no duplicate route, no completion bounce

ObservationResult
Leader native turns for the prompt1, workKind: "turn", phase replied
Successful execution events on the parent1
Parent user rows / final agent rows1 / 1
Final answer length76 chars (length only; no text recorded)
New work items3
Phoenix (explicit delegation)1 work item, workKind: "delegation", 0 peer turns
Clark (final-text mention)1 peer turn
Kakiko (final-text mention)1 peer turn
Parentage / dispatch groupevery route parented to the leader turn in its group
Contact ledger3 recipients in one read-back
native.route.suppressed1 already-routed for Phoenix
completion.skipped3 total: 1 parent-is-leader, 2 asker-is-leader
Completion tasks anywhere0
Leader-authored rows after the turn1 (the parent final row)

Rollback and restore

  • nativeFirst: "off" plus toolSurface: "legacy" reports off and runs one non-native legacy task to replied; the provider request carries all 11 legacy Ensemble tool schemas. The driver imports nativeFirstCommand and toolsCommand, validates each, and sends their patches through settings.update, so the live scenario exercises the same command catalog the TUI uses. The TUI key handler is a thin wrapper covered by src/v2/tui/commands.test.ts (#240).
  • Every durable native turn, the parent result row, and the three-entry contact ledger survive the rollback.
  • Restoring runs nativeFirstCommand(["on"]) and toolsCommand(["native"]); the mode command alone never clears an explicit legacy surface. The restore reports on and keeps the same turns and ledger.
  • After the second host restart: mode on, native sync live, parent phase replied, ledger unchanged, one successful execution event, one Phoenix work item, no completion task, no stuck turn, the exact roster table, one parent row, and one leader-authored row after the turn (the intentional legacy rollback row is excluded). Total work rows are unchanged.

AGENTS block prompt/token delta ​

Values use the documented chars/4 estimate. The generated Vera block is read from the real workspace file; the generic v6 value strips the roster section.

MeasurementCharsEst. tokens
Generated v6 block with the 4-member Vera roster4,5091,128
Generic v6 block without the roster section4,0581,015
Generated roster table36792
#238 fixed 12-member filled v6 block5,6671,417

The Vera block is 1,158 chars / 289 tokens smaller than #238's 12-member fixture because it carries four members, not twelve. The roster table costs 367 chars / 92 tokens in this workspace.

#240's default-on native surface is still the five-tool decision set; the rollback request measured the 11-tool legacy list. npm run benchmark:tools on this branch reports 1,206 → 461 tool-schema tokens (11 → 5 tools), 3,347 → 1,879 total prompt tokens, 1 → 0 sender tool calls, and 2 → 1 sender round-trips.

Work pane size matrix ​

npm run test:live:sizes passed at 80x24, 100x30, 120x40, and 160x50, including the NO_COLOR=1 check. The new assertions extend the existing matrix:

  • At 160x50, page one renders exactly four card rows; every top border starts at the Work content edge, ends at the content right edge, spans the full content width, and stands alone on its row.
  • At 120x40, 100x30, and the 80x24 drawer, the pane renders six/five compact rows, zero card borders, and no full-width card rows.

One targeted 160x50-only probe hit the pre-existing NO_COLOR still-motion assertion transiently (a spinner glyph was captured); the full matrix rerun passed all sizes and the NO_COLOR check.

Commands ​

CommandResult
npm run verifyPASS — strict typecheck; 1,868 tests, 0 failed, 0 skipped
npm run test:live:native-firstPASS — main 66/66, restart 18/18, 0 gaps, exit 0
npm run test:live:sizesPASS — 80x24, 100x30, 120x40, 160x50, NO_COLOR=1, exit 0
npm run test:live:veraPASS — seed 1/1, run 54/54, restart 18/18, exit 0
npm run test:live:fake:allPASS — fake 463s, fake-empty 9s; 2 passed, 0 failed, 0 skipped
npm run test:live:allPASS — fake 461s, fake-empty 8s, real 57s; 3 passed, 0 failed, 0 skipped
npm run test:live:multilocationPASS — 22/22 budget hops, 1 continuation, 1 exhausted group, 0 mismatches
npm run benchmark:toolsPASS — measurement only; values above

Skips ​

  • None. The real-model live suite ran on the local catalog and passed (test:live:all, real 57s); no scenario reported a skip.
  • The broad fake walkthrough runs the explicit legacy path as described in the review-fix section; it is a pass for that configured scenario, not a claim that the legacy assertions cover the default-on surface.

Privacy ​

The committed record contains fixture names, stored roster titles/roles, counts, lengths, versions, and token estimates only. It contains no prompt or answer text, conversation transcript, credentials, file bytes, or absolute private paths. The live scenario gates its driver outputs for the harness root, project directory, password, prompt markers, and answer markers; it gates service logs for the harness home paths and the prompt marker.

Notes ​

  • The original #241 cutover changed test infrastructure only. The review-fix pass adds narrow product changes (the canonical default constant, explicit tool dedupe, anchor validation, typed contact source, card-focus agreement, command arity, and the mode/tool-surface separation) plus the service and pure tests listed above.
  • The expected scripts/lib/native-first-driver.mjs conflict did not occur: the original branch started from integration at 7b405ad, which already contains #240's fresh-store default assertion, and that file was not modified. The review-fix branch is based on integration at 514ac86.
  • The fake walkthrough's explicit legacy setting is a fixture choice, not a product default: DEFAULT_NATIVE_FIRST_MODE remains on, and the Vera regression plus the default-on service test prove the effective default.

Released under the MIT License.