Skip to content

Ticket #276: scale/soak and local diagnostics ​

Parent: #268. This record covers the repeatable soak harness, the tested operating envelope, the redacted support bundle, the no-telemetry stance, and the one leak fixed by the ticket. It does not claim that a short smoke or a nightly smoke is the default multi-hour gate.

What changed ​

  • scripts/soak-fake.sh starts an isolated fake host, creates agents and one-directory-per-workspace fixtures, sends real message.send work, and samples memory/task/message/queue/native/store/log growth into soak.jsonl. scripts/soak-sample.mjs reduces each diagnostic response to side-effect-free counters and discards the payload; scripts/soak-report.mjs applies the growth/settle gate.
  • Default full run: 14,400 s (4 h), enforced. Smoke: 180 s, observe-only. A nightly soak-smoke job uploads the record; the 4-hour gate stays manual/opt-in and is not a PR check.
  • scripts/support-bundle.mjs collects doctor.json, logs-summary.json, bounded warn/error/failure log rows, a log inventory, and a password-redacted service descriptor. Paths, credentials, titles, instructions, prompts, message text, results, patches/diffs, and file bytes are omitted or redacted.
  • docs/operations/scale-and-monitoring.md records the operating envelope and the no-telemetry policy.
  • NativeIngestionState now evicts the oldest settled per-session value above NATIVE_SESSION_STATE_LIMIT (512). A live execution, inbox, text, row, watchdog, or deferred child is never evicted.

Smoke observation (not a multi-hour pass) ​

Command: ENSEMBLE_SOAK_OUT=<soak-out> npm run soak:smoke

text
mode=smoke duration=180s sample=15s burst=2 agents=2 workspaces=1 enforce=0
workload done cycles=41 sent=102 failed=0
samples=13 duration=183s window=122s gate=observe-only
baseline rss=594.25MiB messages=100 tasks=37
tail     rss=626.42MiB messages=366 tasks=135
peak     rss=667.86MiB queue=2 native=0 runs=1
growth   rss=+949.22MiB/h messages=7849.18/h tasks=2891.8/h rss=120.93MiB/1k-msg
finding  rss-growth: above the 256MiB/hour absolute gate
finding  rss-per-message: above the 64MiB/1k-message gate
result   OBSERVED (not a release pass)

Interpretation: the queue settled every cycle and at the tail (queue=0, native=0), and no send/sample/TUI step failed. The service process was flat in the 356–400 MiB band; the TUI rose from 211 MiB to 238 MiB while it mounted and carried the first ~366 messages. The total RSS slope is dominated by that first-mount/TUI growth and does not represent a service leak. The absolute and normalized findings are observe-only and not evidence of a stable 4-hour slope; a 3-minute run cannot calibrate them. The last sample carried 388 messages + 143 tasks and a 301 KiB diagnostic payload, which is the scale signal behind the Tier 3 scoped-read gate.

The TUI growth direction was checked in a separate 20-minute run: it rose to 245 MiB while the full-snapshot path carried 966 rows, then fell to 220 MiB after the 1,000-message Tier 3 gate engaged (see below).

Sustained observation ​

Command (aggressive burst, observe-only): ENSEMBLE_SOAK_OUT=<soak-out> ENSEMBLE_SOAK_SAMPLE_SECONDS=15 ENSEMBLE_SOAK_BURST=8 ENSEMBLE_SOAK_CYCLE_PAUSE_SECONDS=2 bash scripts/soak-fake.sh --duration 420 --no-enforce

text
workload done cycles=80 sent=680 failed=0
samples=28 duration=413s window=336s gate=observe-only
baseline rss=734.51MiB messages=402 tasks=150
tail     rss=813.82MiB messages=2453 tasks=915
peak     rss=902.96MiB queue=4 native=0 runs=2
growth   rss=849.73MiB/h messages=21975/h tasks=8196.43/h rss=38.67MiB/1k-msg
finding  rss-growth: above the 256MiB/hour absolute gate
result   OBSERVED (not a release pass)

Interpretation: the normalized slope is 38.67 MiB per 1,000 durable messages, below the 64 MiB/1k gate. The absolute gate fired because this fixture deliberately ran at about 22k messages/hour (2,453 messages and 915 tasks in 7 minutes), far outside the default workload; it is a workload-ceiling signal, not evidence of a per-message leak. Service RSS oscillated 403–609 MiB and the TUI 222–364 MiB without a monotonic climb; the queue settled to 0 every cycle and native state stayed 0.

A separate 20-minute gentler run (burst 2) reached 1,637 messages and 603 tasks. TUI RSS rose while the full-snapshot path carried 966 rows, then fell from 245 MiB to 220 MiB after the 1,000-message Tier 3 gate engaged; service RSS stayed in a 410–464 MiB band. Its final report step was interrupted by an edit to the running harness (the sample series completed; the raw file was not kept), so it is supporting evidence, not the gate record.

Default multi-hour gate ​

Not run to completion for this ticket. The default gate is:

sh
npm run soak   # 4 h, enforced: fail above 256 MiB/hour, 64 MiB/1k messages,
               # or an unsettled tail

It is a nightly/opt-in gate. The nightly smoke (soak-smoke) proves the harness runs and uploads its record, but a smoke/nightly smoke is not a 4-hour pass. The next full run must be recorded here or in a follow-up verification record with duration, samples, baseline/tail, both growth slopes, peak queue/native/runs, and findings.

Other evidence ​

  • node --test --experimental-strip-types src/v2/soak-metrics.test.ts: diagnostic counters, warmup baseline, growth slope, observe-only vs enforced, queue/native settle findings.
  • node --test --experimental-strip-types src/v2/native-state.test.ts: 537 settled sessions stay bounded at 512; a live execution is never evicted.
  • node --test --experimental-strip-types src/v2/support-bundle.test.ts: path/credential redaction, content-key omission, log bounding, manifest, and a bundle with an unreachable service still containing useful redacted logs.
  • npm run verify: pass, 2,086 tests on Node 26.12.1 (includes the new soak, support-bundle, native-state, CI job-graph, and built-package gates). The operator-only soak-metrics.ts and support-bundle.ts are excluded from the published dist build, like doctor and data.
  • npm run docs:check: pass (contract v7; OpenCode 2.2.6, 2.2.7; Node >=22.23.4).

Accepted limits ​

  • Durable transcripts and dispatch-group rows are retained until workspace delete; there is no automatic transcript retention in this ticket. This is user data, not a leak. The scoped path keeps bootstrap payloads bounded; the data-retention ticket (#272) owns deletion/export policy.
  • Finished task rows purge after 24 h. The client keeps 240 patched messages per workspace and prunes invisible terminal Work rows.
  • The diagnostic RPC is a full-state support read and grows with history; it is not used for routine TUI refresh.

Telemetry ​

No telemetry, analytics, crash upload, or update ping. Logs are local and content-free by default (ENSEMBLE_LOG_CONTENT=1 is opt-in). Support bundles are local, user-run, redacted, and never uploaded by Ensemble.

Released under the MIT License.