Structured logging (#145)
Status: verified locally on Linux/tmux; macOS/Herdr unexecuted Date: 2026-11-08 Issue: #145 Base: integration/ensemble-v7 at e6b6f06 Branch: feat/v7-logging
This record covers the operational logging layer:
- One dependency-free logger with
error/warn/info/debug/tracelevels and service/task/retry/approval/message/feed/refresh/tui/host/failure categories. - A bounded 2,000-entry ring for live inspection plus per-process rotating JSONL sinks and a longer-retention failure sink.
- Store observers for task, message, approval, and init lifecycle events. Lease heartbeats and stream checkpoints never create lines.
- Self-contained failure records for each attempt, uncertain park, and terminal failure, including the error chain/stack, provider/model, run and session IDs, tool outcomes, recent message metadata, and the decision.
logs.recent,:logs,:logs failures, and:logs task <id>.- Content logging off by default; secrets redacted; files
0600in a0700directory. - Asynchronous, batched writes with drop counters; write failures never fail work.
Environment
| Item | Value |
|---|---|
| OS | Linux 6.12.113+deb13-amd64 |
| Terminal host | tmux 3.5a |
| Node | v26.12.1 |
| OpenCode | opencode v2.2.6 |
| macOS/Herdr | Not available; not run |
Commands and results
| Command | Result |
|---|---|
npm run verify | Pass — strict type check; 974 passed, 0 failed, 0 skipped. |
npm run test:live:fake:all | Pass — fake 195 s, fake-empty 8 s; 2 passed, 0 failed, 0 skipped. The fake walkthrough now opens :logs, asserts Logs · recent, sees service.start, confirms content=off, and closes with Esc. |
| Manual logger smoke | Pass — writing a terminal failure produced service-*.jsonl and service-failures-*.jsonl with mode 0600; the failure line carried the error chain, model, tool outcome, decision, version, and epoch. |
git diff --check | Pass — no whitespace errors. |
Behavioral evidence
| Behavior | Evidence | Result |
|---|---|---|
| Level filtering, bounded ring, drop counter | logging levels filter noisy events and the ring stays bounded | pass |
| Category/event/task/level filters | logging filters by task, category, event prefix, and level | pass |
| Credential redaction and structured error capture | logging redacts credentials and keeps errors structured | pass |
JSONL sinks and 0600 permissions | logging writes JSONL main and failure files with private permissions | pass |
| Rotation and retained failure sink | logging rotates files without blocking | pass |
| Disabled logging writes nothing | disabled logging writes nothing and keeps no ring | pass |
| Task created/transition/completed; heartbeat writes ignored | task observer logs created, transition, completion, and ignores lease heartbeats | pass |
| Failure and cancellation stay distinct | task observer keeps failure and cancellation distinct | pass |
| Message append/finalize without checkpoint spam | message observer logs append and finalize, and skips same-length streaming churn | pass |
| Approval requested/resolved | approval observer logs request and resolution | pass |
| Failure classification and self-contained record | a failure record is self-contained without conversation content, failure classification separates provider, timeout, and tool errors | pass |
| Content gate on task/message text | content logging attaches instruction and message text only when enabled | pass |
| Terminal failure service path | ticket145: a terminal task failure writes a self-contained failure record asserts failure.terminal, provider class, attempt 5/5, group, terminal decision, stack/message, and content=false | pass |
| Uncertain tool parking | ticket145: an uncertain park carries tool outcomes and the approval decision asserts failure.parked, bash/failed/E_TOOL, and the approval ID | pass |
:logs filters and formatting | :logs accepts only the documented filters, :logs maps filters onto the RPC input and title, log lines lead with time, level, and event…, failure lines surface the error message and decision… | pass |
| Viewer presence precedence | ticket126: presence keys cover help, approvals, and leader which-key without changing dialog precedence | pass |
| Live viewer over the real service ring | fake walkthrough :logs step | pass |
Sample failure record
The failure sink writes one JSON object per line. The core fields are:
json
{
"ts": "2026-11-08T11:06:40.000Z",
"level": "error",
"cat": "failure",
"event": "failure.terminal",
"incidentID": "failure_...",
"failureClass": "provider",
"taskID": "t_1",
"workspaceID": "w_1",
"dispatchGroupID": "g_1",
"title": "Fix parser",
"attempts": 5,
"error": {
"name": "Error",
"message": "quota exhausted",
"code": "E_PROVIDER",
"stack": "Error: quota exhausted …"
},
"model": { "providerID": "openai", "modelID": "gpt" },
"runID": "run_1",
"sessionID": "ses_1",
"toolOutcomes": [{ "name": "bash", "status": "failed", "error": "exit 1" }],
"lastMessages": [{ "messageID": "m_1", "author": "agent", "kind": "text", "chars": 12 }],
"decision": { "type": "terminal", "reason": "attempts-exhausted", "attempt": 5, "maxAttempts": 5 },
"version": 42,
"epoch": 7
}Task text, constraints, expected output, and message text are absent unless ENSEMBLE_LOG_CONTENT=1.
Operational notes
- Service and TUI files are per-process (
<role>-<date>-<pid>.jsonl), so a rotating sink is never shared across processes. - Main logs retain 10 MB × 5 files for 7 days; failure logs retain 20 MB × 5 files for 30 days.
- Tests keep the ring but skip file sinks unless a test supplies an explicit directory, so
npm testnever writes into the user's state directory. - The logger drains every batch on
close(); a shutdown cannot strand the tail of a burst.
Skips and limitations
- macOS/Herdr: NOT RUN. The existing manual checklist remains the authority for macOS font coverage, color perception, and key ownership.
- Hosted CI was not run.
- The live fake scenario asserts the viewer and service ring, not a forced provider failure; failure records are exercised at the service seam.
test:live:sizeswas not rerun for this ticket; the logs overlay mounts only on demand and does not change the base geometry.