Skip to content

Structured logging (#145) ​

Status: verified locally on Linux/tmux; macOS/Herdr unexecuted Date: 2026-11-08 Issue: #145 Base: integration/ensemble-v7 at e6b6f06 Branch: feat/v7-logging

This record covers the operational logging layer:

  • One dependency-free logger with error/warn/info/debug/trace levels and service/task/retry/approval/message/feed/refresh/tui/host/failure categories.
  • A bounded 2,000-entry ring for live inspection plus per-process rotating JSONL sinks and a longer-retention failure sink.
  • Store observers for task, message, approval, and init lifecycle events. Lease heartbeats and stream checkpoints never create lines.
  • Self-contained failure records for each attempt, uncertain park, and terminal failure, including the error chain/stack, provider/model, run and session IDs, tool outcomes, recent message metadata, and the decision.
  • logs.recent, :logs, :logs failures, and :logs task <id>.
  • Content logging off by default; secrets redacted; files 0600 in a 0700 directory.
  • Asynchronous, batched writes with drop counters; write failures never fail work.

Environment ​

ItemValue
OSLinux 6.12.113+deb13-amd64
Terminal hosttmux 3.5a
Nodev26.12.1
OpenCodeopencode v2.2.6
macOS/HerdrNot available; not run

Commands and results ​

CommandResult
npm run verifyPass — strict type check; 974 passed, 0 failed, 0 skipped.
npm run test:live:fake:allPass — fake 195 s, fake-empty 8 s; 2 passed, 0 failed, 0 skipped. The fake walkthrough now opens :logs, asserts Logs · recent, sees service.start, confirms content=off, and closes with Esc.
Manual logger smokePass — writing a terminal failure produced service-*.jsonl and service-failures-*.jsonl with mode 0600; the failure line carried the error chain, model, tool outcome, decision, version, and epoch.
git diff --checkPass — no whitespace errors.

Behavioral evidence ​

BehaviorEvidenceResult
Level filtering, bounded ring, drop counterlogging levels filter noisy events and the ring stays boundedpass
Category/event/task/level filterslogging filters by task, category, event prefix, and levelpass
Credential redaction and structured error capturelogging redacts credentials and keeps errors structuredpass
JSONL sinks and 0600 permissionslogging writes JSONL main and failure files with private permissionspass
Rotation and retained failure sinklogging rotates files without blockingpass
Disabled logging writes nothingdisabled logging writes nothing and keeps no ringpass
Task created/transition/completed; heartbeat writes ignoredtask observer logs created, transition, completion, and ignores lease heartbeatspass
Failure and cancellation stay distincttask observer keeps failure and cancellation distinctpass
Message append/finalize without checkpoint spammessage observer logs append and finalize, and skips same-length streaming churnpass
Approval requested/resolvedapproval observer logs request and resolutionpass
Failure classification and self-contained recorda failure record is self-contained without conversation content, failure classification separates provider, timeout, and tool errorspass
Content gate on task/message textcontent logging attaches instruction and message text only when enabledpass
Terminal failure service pathticket145: a terminal task failure writes a self-contained failure record asserts failure.terminal, provider class, attempt 5/5, group, terminal decision, stack/message, and content=falsepass
Uncertain tool parkingticket145: an uncertain park carries tool outcomes and the approval decision asserts failure.parked, bash/failed/E_TOOL, and the approval IDpass
:logs filters and formatting:logs accepts only the documented filters, :logs maps filters onto the RPC input and title, log lines lead with time, level, and event…, failure lines surface the error message and decision…pass
Viewer presence precedenceticket126: presence keys cover help, approvals, and leader which-key without changing dialog precedencepass
Live viewer over the real service ringfake walkthrough :logs steppass

Sample failure record ​

The failure sink writes one JSON object per line. The core fields are:

json
{
  "ts": "2026-11-08T11:06:40.000Z",
  "level": "error",
  "cat": "failure",
  "event": "failure.terminal",
  "incidentID": "failure_...",
  "failureClass": "provider",
  "taskID": "t_1",
  "workspaceID": "w_1",
  "dispatchGroupID": "g_1",
  "title": "Fix parser",
  "attempts": 5,
  "error": {
    "name": "Error",
    "message": "quota exhausted",
    "code": "E_PROVIDER",
    "stack": "Error: quota exhausted …"
  },
  "model": { "providerID": "openai", "modelID": "gpt" },
  "runID": "run_1",
  "sessionID": "ses_1",
  "toolOutcomes": [{ "name": "bash", "status": "failed", "error": "exit 1" }],
  "lastMessages": [{ "messageID": "m_1", "author": "agent", "kind": "text", "chars": 12 }],
  "decision": { "type": "terminal", "reason": "attempts-exhausted", "attempt": 5, "maxAttempts": 5 },
  "version": 42,
  "epoch": 7
}

Task text, constraints, expected output, and message text are absent unless ENSEMBLE_LOG_CONTENT=1.

Operational notes ​

  • Service and TUI files are per-process (<role>-<date>-<pid>.jsonl), so a rotating sink is never shared across processes.
  • Main logs retain 10 MB × 5 files for 7 days; failure logs retain 20 MB × 5 files for 30 days.
  • Tests keep the ring but skip file sinks unless a test supplies an explicit directory, so npm test never writes into the user's state directory.
  • The logger drains every batch on close(); a shutdown cannot strand the tail of a burst.

Skips and limitations ​

  • macOS/Herdr: NOT RUN. The existing manual checklist remains the authority for macOS font coverage, color perception, and key ownership.
  • Hosted CI was not run.
  • The live fake scenario asserts the viewer and service ring, not a forced provider failure; failure records are exercised at the service seam.
  • test:live:sizes was not rerun for this ticket; the logs overlay mounts only on demand and does not change the base geometry.

Released under the MIT License.