perf(docker): stop loading ssh2, which allocates a 16 MiB WASM heap nobody uses

Profiling at the memory high-water mark - not at idle, which is what every earlier
round measured - found the largest single retainer in every snapshot was one V8
Context holding a 16.00 MiB ArrayBuffer. Expanding it through the dominator tree
named it: ssh2's crypto module, whose POLY1305_WASM_MODULE instantiates a
WebAssembly memory for ChaCha20-Poly1305 as soon as it is required.

Homarr has no SSH feature and does not depend on ssh2. The chain is
dockerode -> docker-modem -> ssh2, and docker-modem/lib/modem.js requires './ssh'
at module scope while using it in exactly one place, already guarded by
`protocol === 'ssh'`. A pnpm patch moves the require inside that branch. SSH-based
Docker hosts still work - require() is cached, so they pay it once - and every
socket-based install, which is everyone else, never loads it.

Measured over four alternated runs on a restored real backup, external memory:

                     before        after
  board loaded    25.4, 25.1    8.3, 11.2
  nav-01          28.5, 25.4   10.1,  8.6
  multi-tab       31.1, 35.8   13.0, 10.0
  post-soak       21.4, 21.5    5.5,  7.2

Consistently -15 to -18 MiB, ranges non-overlapping, and 30 fewer files loaded
(1470 -> 1440).

Peak RSS does NOT improve, and that is worth stating plainly: V8 sizes its heap
against available headroom, so freeing native memory lets it grow further before
collecting. What changes is the composition - a fixed 16 MiB of unusable WASM
buffer becomes headroom the app can actually use, which matters under a container
memory limit but does not move the peak number.

Also adds the instruments this needed. A heap snapshot can never explain a peak,
because V8 runs a full GC before serialising one and reports only survivors:

  - memory-probe.cjs gains in-process allocation sampling, on-demand snapshots via
    v8.writeHeapSnapshot, an smaps endpoint, and require-graph recording.
  - analyze-allocations.mjs diffs two stages to isolate what one stage allocated.
  - analyze-smaps-stages.mjs splits each stage's RSS into JS heap, JIT, native and
    file-backed pages.
  - analyze-heap-dominators.mjs can now expand the largest retainer, which is what
    identified ssh2. A plain graph walk escapes through __proto__ and is useless.

One trap recorded in the findings: taking the peak snapshot over CDP delivers it as
JS string chunks and added ~134 MiB of ArrayBuffers to the process being measured,
producing a fake 693 MiB reading. v8.writeHeapSnapshot streams from C++ instead, and
the probe now reports its own observerCostMiB so the cost of looking is visible.
This commit is contained in:
Thomas Camlong committed 2026-08-09 22:17:25 +02:00
1 parent 3bc25509cf
commit 3cc6ff33df
9 files changed
+598 -32

No files matched your search

+112
View File
@@ -609,6 +609,118 @@ This is also the worst case for the round-trip trade-off: that one event name co
sequential requests. It did not show up in board load latency (523 ms vs 526 ms median),
because the widget fetches after the board renders, and total CPU rose 4%.
## Dissecting the peak, not the idle
Everything above measures a drained process — boot idle, or post-soak. That is the wrong
moment: users complain about the high-water mark. Measuring it needs different instruments,
because the obvious one cannot work.
**A heap snapshot can never explain a peak.** V8 runs a full GC before serialising one, so by
construction it reports what survived. The 94 MiB of ArrayBuffers that drove this container to
500 MB was invisible in every snapshot taken. Three instruments were added instead:
| instrument | what it sees | blind to |
| --- | --- | --- |
| `HeapProfiler.startSampling` (in-process `inspector.Session`) | every V8-heap allocation, survivor or garbage, with a stack | anything outside the V8 heap |
| wrapped `Buffer.alloc`/`concat` | external buffer allocations, with a stack | V8-heap objects |
| `/proc/<pid>/smaps` at every stage | JS heap vs JIT vs native vs file-backed | anything below page granularity |
Together they cover the whole address space. Neither of the first two alone would have found
the Umami bug: the sampling profiler cannot see `Buffer` backing stores, which is precisely
where those 46 MiB lived.
### An observer effect worth recording
The first attempt took the peak snapshot over CDP, which delivers it as JS string chunks. For
a ~135 MB snapshot that added **~134 MiB of ArrayBuffers to the process being measured**, and
the stage right after it read 693.7 MiB. That is an artefact, not an application problem —
and it is exactly the kind of number that gets reported as a finding. Switching to
`v8.writeHeapSnapshot`, which streams from C++, removed it. The probe now reports its own
`observerCostMiB` so the cost of looking is visible rather than assumed to be zero.
### What the peak is actually made of
Peak here is `nav-01`, not the multi-tab stage — a GC lands during multi-tab and pulls RSS
*down*, so "most tabs open" is not the same as "most memory". Node RSS 357.8 MiB:
| component | MiB | notes |
| --- | --- | --- |
| live JS objects (`heapUsed`) | 175.7 | of which ~41 MiB is retained bundle source |
| node binary text | 65.7 | file-backed and shared; not in cgroup `anon` |
| native malloc | ~59 | see below |
| external buffers | 24.6 | |
| V8 committed but free | 12.3 | heapTotal − heapUsed |
| JIT code | 10.1 | grew from 1.9 MiB at boot |
Cross-checked rather than assumed: `heapTotal 188.0 − V8's 256 KiB pages 153.1 = 34.9 MiB`,
which matches `large_object_space` at 34.5 MiB. So the handful of very large anonymous
mappings (31.6, 20.7, 8.4 MiB) are V8's large-object space and external buffers — **not** the
runaway native allocator that their shape first suggested.
**The retained heap at peak is the same as at rest.** A post-GC snapshot at peak has 160.7 MiB
of shallow size against 157.7 MiB after the soak; strings 51.1 vs 50.1 MiB. Nothing
accumulates. The peak is transient churn plus pages V8 and malloc have not returned — which is
why memory "oscillates" and why soak growth is negative.
And the churn is small now: total V8-heap allocation across the entire run is **125 MiB**, 40%
of it node's own module loader reading source files. After the Umami fix the multi-tab burst
allocates **3.3 MiB**. There is no longer a churn problem on the JS side.
## The 16 MiB nobody was using: ssh2
The largest single retainer in every snapshot — at boot, at peak, and after the soak — was one
`system / Context` holding **16.03 MiB**. Expanding it through the dominator tree (a plain
graph walk escapes via `__proto__` and reports the whole heap) named it immediately:
```
=== inside the largest retainer (object:system / Context, 16.03 MiB) ===
16.00 MiB (indirect) -> object:ArrayBuffer
0.00 MiB POLY1305_WASM_MODULE -> object:Object
0.00 MiB ChaChaPolyCipherNative -> closure:ChaChaPolyCipherNative
0.00 MiB AESGCMCipherNative -> closure:AESGCMCipherNative
```
`ssh2`. Its crypto module instantiates a WebAssembly memory for ChaCha20-Poly1305 the moment
it is required. Homarr has no SSH feature and does not depend on ssh2; the chain is
`dockerode → docker-modem → ssh2`, and `docker-modem/lib/modem.js` requires `./ssh` at module
scope while using it in exactly one place, already guarded by `protocol === 'ssh'`.
Measured directly inside the image — requiring ssh2 and nothing else:
```
heapUsed +2.6 MiB external +0.9 MiB rss +14.0 MiB
```
Fixed with a pnpm patch moving that one require inside the branch that uses it. SSH-based
Docker hosts still work — `require` is cached, so they pay it once — and socket-based installs,
which is everyone else, never load it.
### The same question asked of every dependency
Since one package was hiding 14 MiB behind a require, all 173 were measured the same way: each
required in a fresh process, RSS delta recorded. Packages costing more than 2 MiB, cross-checked
against whether the server actually loads them:
| package | RSS to require | loaded at runtime? |
| --- | --- | --- |
| drizzle-orm | 124.3 MiB | only via subpaths — the root import loads every dialect, so this figure overstates it |
| @kubernetes/client-node | 66.9 MiB | **no** — correctly lazy |
| dockerode | 28.8 MiB | yes (includes docker-modem 16.1 and ssh2 14.3) |
| **mysql2** | **20.9 MiB** | **yes — 77 files, on a SQLite install** |
| sharp | 20.4 MiB | no |
| openid-client / jose | 16.1 / 15.5 MiB | no |
| @grpc/grpc-js | 14.6 MiB | yes, 61 files (via dockerode) |
| **pg** | **9.4 MiB** | **yes — 13 files, on a SQLite install** |
These are not additive — dockerode's figure contains ssh2's — but the ranking is what matters.
**mysql2 and pg are loaded on every SQLite install.** `db/drivers/index.ts` statically imports
all three driver wrappers and picks one at call time, so importing the module executes all
three. Not yet fixed: `packages/db/index.ts` does `export const db = createDb(schema)` at
module scope, so `createDb` must stay synchronous and cannot use `await import()`. The viable
route is deferring the external `require` inside each wrapper. Both dialects have testcontainer
migration tests, so it is verifiable — it is queued, not dismissed.
## Why 24 MiB of code is resident before the first request
The module cache is a flat list with no parentage, so "what pulled this in" needed the