Skip to content

Single service per host and safe recovery (#151) ​

Status: verified locally on Linux/tmux; real post-fix 30-run recorded; gap-noise follow-up applied Date: 2026-11-08 Issue: #151 Spec: #146 ADR: 0003 Base: integration/single-service at 48263a6 (includes #147, #148, #149, #150 and the #146 review fixes) Review fixes: fix/spec-146-review Gap-noise follow-up: fix/feed-gap-noise Branch: verify/single-service Worktree: /home/tristan/dev/ensemble-151

This record covers the delivery evidence for the single-service registry (#147), lease-safe load/recovery (#148), restart failure truth (#149), and log attribution (#150):

  • One service.start for a host realm that loads the plugin at 13 locations.
  • Mock 100: 100 clean requests, 137/137 successful task rows, 0 task.failed, 0 failure records, 8 concurrent runs.
  • Real post-fix 30-request run: 30/30 requests, 39/39 task rows succeeded, one service start, no placeholders, no duplicate handoffs.
  • Lease-safe and restart truth tests pass.
  • The pre-fix real 30-request baseline is retained for contrast.
  • The real run exposed 180 recoverable feed.gap warnings; the follow-up makes recovered gaps debug summaries and keeps warnings only for real fallbacks.

Environment ​

ItemValue
OSLinux 6.12.113+deb13-amd64
Terminal hosttmux 3.5a
Nodev26.12.1
npm12.2.1
OpenCodeopencode v2.2.6
macOS/HerdrNot available; not run
Hosted CINot run

Commands and results ​

CommandResult
npm run verifyPass — strict type check; 1020 tests passed, 0 failed, 0 skipped, 0 todo, 28.0 s.
Mock 100 (ENSEMBLE_ROOT=/home/tristan/dev/ensemble-gapfix bash /tmp/opencode/mock100-setup-verify.sh)Pass — driver exit 0; 100 jobs, 137/137 tasks succeeded, 0 failed, 0 task.failed, 1 service.start, maxObservedRuns=8.
npm run test:live:fake:allPass — fake 194 s, fake-empty 9 s; 2 passed, 0 failed, 0 skipped, 0 not run (203 s total).
Real playground 30-request replay (live host, hot reload only)Pass, with gap-noise follow-up — 30/30 requests, 39/39 tasks, all attempt 1, 0 placeholders, 0 duplicate handoffs, 1 service.start, 0 task.failed, 0 failure records; exposed 180 recoverable feed.gap warnings, fixed in fix/feed-gap-noise.
Targeted restart/recovery tests (--test-name-pattern="ticket14[89]")Pass — 15 tests passed, 0 failed, 0 skipped (0.33 s): 7 ticket148, 8 ticket149.
Targeted feed-gap tests (src/v2/tui/feed-gaps.test.ts)Pass — 5 tests passed, 0 failed, 0 skipped: recovered-gap debug, fallback warn, burst coalescing, repeated hard-gap warn, trailing window close.
git diff --checkPass — exit 0, no whitespace errors.
Prototype tree check (git ls-files | grep -i prototype)Pass — no prototype files in the tree.

Mock 100 (isolated multi-location host) ​

The harness starts an isolated fake provider, isolated HOME/XDG/project tree, and a fresh OpenCode server whose plugin points at this worktree. The driver creates 4 channel workspaces (mock-a…mock-d), 8 agents, sets maxActiveRuns=8, maxDelegations=10, maxDepth=5, and sends 100 requests (34 easy, 33 medium, 33 hard). Fake-provider delays are 0 ms so the measured path is Ensemble orchestration.

Multi-location topology ​

server.log proves the host loaded the plugin once per location; the service log proves one service handled all of them:

ObservationCount
loading plugin (project + 4 channels + 8 agent homes)13
location services booted13
Server processes (run=)1
service.start in the service log1
service.stop0
Distinct service instanceID / threadID1 / 0

Task and log results ​

AssertionResult
Jobs sent100
Jobs with a failed root task0
Task rows (turn 69 + delegation 34 + handoff 34)137
Task outcomes / states137 succeeded / 137 replied
Attempts137 at attempt 1 (no retries)
Jobs with errors0
maxObservedRuns8
Settle time after the last send9.2 s
Service log rows926 (925 info, 1 error)
service.start / service.location1 / 1
task.failed0
failure.* records0
Failure sink (service-failures-*.jsonl)absent
Rows with role, instanceID, threadID926/926 (all role=service, one instance, threadID=0)

Event totals: task.created 137, task.claimed 137, task.transition 137, task.completed 137, message.appended 374, plus the 4 service rows above.

Latency (seconds, request to settled group) ​

Bucketnminmedianmeanp95max
Easy342.022.843.746.106.57
Medium332.833.454.055.705.78
Hard333.444.134.525.605.74
All1002.023.764.106.036.57

Artifacts ​

  • Results: /tmp/opencode/mock100-artifacts-1794118870/mock100-results.json
  • Harness output: /tmp/opencode/146-gapfix-mock100.log
  • Service log and server/fake logs: /tmp/opencode/mock100-artifacts-1794118870/
  • Prior review artifacts: /tmp/opencode/mock100-artifacts-1794117806/

Prototype evidence for #148's 13-instance acceptance ​

#148's acceptance asks for a mock 100 with 13 service instances (registry fix disabled or simulated) to be 100/100 with 0 task.failed. That run predates the merged code and is recorded here as prototype evidence, not as a run of the final tree:

ObservationResult
Service instances / service.start rows13 / 13
Plugin locations (project + 4 channels + 8 agents)13
Task rows137/137 succeeded, all at attempt 1
Jobs100 sent, 0 job errors
task.failed / failure records0 / 0
Failure sink (service-failures-*.jsonl)absent
Service log rows1,351 (role=service); instance/thread attribution absent in this pre-#150 build

The run shows lease-safe recovery is sufficient defense even when the registry does not collapse the 13 loads into one service. It produced 8→0 task failures compared with the pre-fix baseline.

Artifacts: /tmp/opencode/mock100-artifacts-1794109203/ (mock100-results.json, logs/service-20261108-3784961.jsonl). The merged tree was separately verified with the registry active: one service.start and 137/137 tasks in the main mock above. No prototype source remains in the tree.

Restart and recovery tests ​

node --test --experimental-strip-types --test-name-pattern="ticket14[89]" src/v2/service.test.ts → 15 passed, 0 failed, 0 skipped:

TicketTestResult
#148load leaves a live foreign lease working row untouchedpass
#148a live lease is never retired for attempts or a passed deadlinepass
#148an expired lease is requeued once and attempts rise only on the next claimpass
#148a working row with no claim and no local run is repairedpass
#148stop releases local claims and preserves foreign claimspass
#148streaming rows survive load only while their lease is livepass
#148a second service over shared storage never steals a live claimpass
#149restart requeue writes a failure attempt with the previous claim and leasepass
#149restart retirement writes a terminal failure recordpass
#149stream finalization reports settlement and result eligibility separatelypass
#149a recovered then successful task clears its restart errorpass
#149a fallback reply is retried as an empty answerpass
#149an exhausted fallback reply fails visibly without a resultpass
#149the request-23 shape never succeeds on a stale fallbackpass
#149an empty terminal stream settles as interrupted, never a completed placeholderpass

These cover the crash path by simulation: an unexpired foreign claim is never stolen, expiry requeues exactly once with attempts preserved, a clean stop releases only local claims, and a recovered success clears the restart error.

Review fixes (spec #146 code review) ​

FindingFixEvidence
Unsupported ctx.directoryservice.start reads the declared ctx.location.directory, falling back to cwdticket150: log rows identify the service instance, location, and thread
Acquire/stop raceacquire() re-checks the entry after its await; a stopping entry is never joinedan acquire already awaiting the entry never joins its teardown
Live rows retired earlyA live lease always wins over attempts/deadline; retire happens only after expiryticket148: a live lease is never retired for attempts or a passed deadline
Wrong retire labelsDeadline → timeout/deadline-exceeded; exhaustion → internal/attempts-exhausted; only recovery is restartticket149: restart retirement writes a terminal failure record
Requeue error leakRequeue writes failure evidence only; legacy transient strings are dropped on retireticket148: an expired lease ..., ticket149: restart retirement ...
Lost suppressed countWindow-close trailing summary plus flush() on logs.recent/stopa suppressed burst reports its total when the window closes, ticket150: feed resets ...
Private test seamsMaintenance is triggered through invoke; one narrow documented run hook replaces activeRuns accessticket148: ...expired lease..., ticket148: stop releases local claims...
Duplicated fixtures/blocksShared seedAgent/seedWorkspace fixtures; retireTask/requeueRecoveredTask helpersfull service.test.ts run
Duplicated changesSince shapeStore returns the canonical ChangesSince unionnpm run check, store.test.ts
Duplicated failure reasonTop-level reason derives from decision.reasonfailure-record.test.ts
Unused coalescer API / unreachable reasonForeign epochs use the delayed createRefreshCoordinator; reset reason is mandatory on resetsforeign-epoch patches coalesce one delayed snapshot...
Ambiguous stream finalizationfinalizeStream returns { message, eligible, reason }; callers gate resultMessageID explicitlyticket149: stream finalization reports settlement and result eligibility separately
Missing gap counterLogger stats count feed.gap; :logs shows feedGapsthe logger counts feed gaps..., :logs shows the feed-gap counter...
13-instance prototype evidenceRecorded explicitly above/tmp/opencode/mock100-artifacts-1794109203/
Real post-fix replay30-run recorded abovepass functionally
Recoverable gap noiseDebug summaries; warnings only for full-snapshot/exhausted/hard fallbacksfeed-gaps.test.ts

Real playground baseline (pre-fix) ​

Source: /tmp/opencode/playground-results.jsonl (33 appended rows; deduplicating by request index yields the 30-request run).

MetricResult
Requests30
Task rows42
Task rows succeeded42/42 (all replied)
Clean requests29/30
Corrupted request#23 (complex, $meow, incident-postmortem prompt)

Request 23 carried the pre-fix multi-instance restart damage: its turn task reached attempts 3 and retained "Ensemble restarted while this subtask was working."; the transcript contained the placeholder answers Reply completed. and Reply interrupted., plus a duplicated Handed to $xedo message at sequence 193. Per the spec, the run had four service instances. The remaining 29 requests had one attempt each and no placeholder or duplicate messages.

Real post-fix 30-request replay ​

Ran on the live playground after the single-service fixes were merged and the service module was hot-reloaded by the active host. This session did not restart the host.

MetricResult
Requests30/30 completed
Task rows39/39 replied/succeeded
Attemptsall at attempt 1
Placeholder results0
Duplicate completion handoffs0
service.start / service.location1 / 1
feed.reset1 (startup, reason epoch)
task.failed / failure records0 / 0

Gap-noise finding and fix ​

The same run logged 180 TUI feed.gap warnings in ~4 minutes (6 per ~10 s). The reasons were recoverable gap decisions whose from/version/ entries were served contiguously by changes.since; the service had no resets. These were successful catch-ups, not failures, so the warning level and per-event volume were misleading.

Follow-up fix/feed-gap-noise:

  • Every observed gap still increments the logger's feedGaps counter, so the true rate remains visible in diagnostics and :logs.
  • A burst collapses into one trailing summary per 10 s window.
  • A caught-up gap emits debug feed.gap.recovered with gaps, reasons, and fallbacks.
  • A full-snapshot fallback or exhausted recovery passes emits warn feed.gap.
  • reset, epoch, malformed, and other decisions keep their warning and fallback semantics.

Post-fix warning-rate smoke (10 requests) ​

After the gap-noise merge, the active host hot-reloaded the TUI (new mount 6185e96f-118c-467a-9cf9-47d0d9c01c6e; the service stayed on its single epoch). A mixed 10-request smoke was run:

MetricResult
Requests10/10 completed
Task rows13/13 replied/succeeded
Attemptsall at attempt 1
Placeholders / duplicate handoffs0 / 0
service.start (subject)1
Service task.failed / failure records0 / 0
Warn-level TUI feed.gap lines0

The new TUI logged only tui.mount; recovered gaps are debug-only and the feedGaps counter remains available in :logs. The service log for the smoke contains no warn/error rows.

Manual replay procedure (repeatable check) ​

Run this after the user deliberately restarts the playground host, or when checking the warning rate again. The full 30-run was not repeated after the gap-noise fix; the 10-request smoke above is the warning-rate check.

  1. Confirm one service for the restarted realm: count distinct instanceID values and service.start rows in the service log; expect one service start, even though the host loads the plugin at every location.
  2. Replay the 30 prompts against the live playground: python3 /tmp/opencode/playground-postfix.py /tmp/opencode/playground-prompts.json (append-only results land in /tmp/opencode/playground-postfix-results.jsonl).
  3. Audit the replay:
    • 30/30 requests sent with no send error; every task row state=replied.
    • 0 duplicate completion handoffs: each handoff.completion=true delivery appears once per correlation/recipient pair.
    • 0 placeholder results: no resultMessageID or agent result text is in FALLBACK_REPLIES (Reply completed., Reply interrupted., Reply interrupted before completion.).
    • <5 warn-level feed.gap lines. The feedGaps counter may be higher: it intentionally counts recovered gaps too. Recovered bursts should appear as debug feed.gap.recovered summaries.
    • 1 service.start / distinct service instanceID.
  4. Record the host version, command, counts, and artifact paths in this file. The functional 30-run above is already recorded; the warning-rate check must be repeated after the gap-noise fix.

Decision evidence ​

Invariant / acceptanceEvidenceResult
One service per host realm13 plugin loads / 13 location boots → 1 service.start, 1 instancepass
Lease-safe load never requeues live workticket148 live-lease and two-service testspass
Recovery after expiry or clean releaseticket148 expired-lease, repair, and stop-release testspass
No duplicate executiontwo-service test: one prompt, one attempt, one durable resultpass
Failure truth on restartticket149 failure-attempt/terminal-record/clear-error testspass
No placeholder successticket149 fallback and request-23 testspass
Mock 100 clean100/100 requests, 137/137 tasks, 0 task.failed, 0 failure recordspass
Real playground 30-run30/30 requests, 39/39 tasks, attempts 1, 0 placeholders, 0 duplicate handoffspass
Gap truthRecovered gaps are debug feed.gap.recovered; warnings only for real fallbackspass (targeted tests)
Gap warning rate after fix10-request mixed live smoke: 0 warn-level feed.gap, 13/13 taskspass
Attributable logs926/926 rows carry role, instance, and threadpass
npm run verify1020/1020 testspass
Documentationdocs/implementation.md already documents the registry and lease-safe recovery from #147/#148pass

Skips and limitations ​

  • Gap-noise fix live recheck: PASS. The 30-run predates the warning-level fix; a post-fix 10-request mixed smoke on the hot-reloaded TUI produced 0 warn-level feed.gap lines and 13/13 successful tasks. Recovered gaps are debug summaries; the full 30-run was not repeated after the gap fix.
  • macOS/Herdr: NOT RUN. No macOS host is available; the existing manual checklist remains authoritative there.
  • Hosted CI: NOT RUN.
  • Multi-process pumping remains unsupported by design (ADR 0003). The single-service registry is scoped to one host realm.
  • Mock artifacts are local to /tmp/opencode and are not committed.

Anomalies ​

  • One expected boot-time catalog race appears in the mock log: service.catalog.error — OpenCode reported no models for this project. — followed immediately by service.catalog.recovered. Work proceeded, the fake model was used for all 137 tasks, and no task failed.
  • No service.stop row appears in the mock log because the harness terminates the isolated host after the run instead of stopping the service cleanly. Clean-stop claim release is covered by ticket148: stop releases local claims and preserves foreign claims.

Released under the MIT License.