Profiling at the memory high-water mark - not at idle, which is what every earlier
round measured - found the largest single retainer in every snapshot was one V8
Context holding a 16.00 MiB ArrayBuffer. Expanding it through the dominator tree
named it: ssh2's crypto module, whose POLY1305_WASM_MODULE instantiates a
WebAssembly memory for ChaCha20-Poly1305 as soon as it is required.
Homarr has no SSH feature and does not depend on ssh2. The chain is
dockerode -> docker-modem -> ssh2, and docker-modem/lib/modem.js requires './ssh'
at module scope while using it in exactly one place, already guarded by
`protocol === 'ssh'`. A pnpm patch moves the require inside that branch. SSH-based
Docker hosts still work - require() is cached, so they pay it once - and every
socket-based install, which is everyone else, never loads it.
Measured over four alternated runs on a restored real backup, external memory:
before after
board loaded 25.4, 25.1 8.3, 11.2
nav-01 28.5, 25.4 10.1, 8.6
multi-tab 31.1, 35.8 13.0, 10.0
post-soak 21.4, 21.5 5.5, 7.2
Consistently -15 to -18 MiB, ranges non-overlapping, and 30 fewer files loaded
(1470 -> 1440).
Peak RSS does NOT improve, and that is worth stating plainly: V8 sizes its heap
against available headroom, so freeing native memory lets it grow further before
collecting. What changes is the composition - a fixed 16 MiB of unusable WASM
buffer becomes headroom the app can actually use, which matters under a container
memory limit but does not move the peak number.
Also adds the instruments this needed. A heap snapshot can never explain a peak,
because V8 runs a full GC before serialising one and reports only survivors:
- memory-probe.cjs gains in-process allocation sampling, on-demand snapshots via
v8.writeHeapSnapshot, an smaps endpoint, and require-graph recording.
- analyze-allocations.mjs diffs two stages to isolate what one stage allocated.
- analyze-smaps-stages.mjs splits each stage's RSS into JS heap, JIT, native and
file-backed pages.
- analyze-heap-dominators.mjs can now expand the largest retainer, which is what
identified ssh2. A plain graph walk escapes through __proto__ and is useless.
One trap recorded in the findings: taking the peak snapshot over CDP delivers it as
JS string chunks and added ~134 MiB of ArrayBuffers to the process being measured,
producing a fake 693 MiB reading. v8.writeHeapSnapshot streams from C++ instead, and
the probe now reports its own observerCostMiB so the cost of looking is visible.
Profiling the container end to end found one allocation site responsible for
most of the peak: a single 46.3 MiB JSON body, assembled from 3,946 socket
chunks, inside undici's fullyReadBody. That function holds the chunk list and
the joined copy at the same instant, so a body of size N costs about 2N.
The source was two Umami calls asking for pageSize=100000, with
getMultiEventTimeSeriesAsync running them for every tracked event name
concurrently via Promise.all. getEventNamesAsync was downloading every raw
event of the last 30 days purely to collect a handful of distinct name strings.
- getEventNamesAsync now reads the aggregated /metrics?type=event endpoint.
- getWebsiteEventTimeSeriesAsync pages 5,000 records at a time, folding each
page into the bucket map before requesting the next.
- getMultiEventTimeSeriesAsync runs series sequentially.
Instances that return aggregated {x,y} buckets still answer in one request, an
instance that accepts pageSize but ignores page is detected rather than
double-counted, and hitting the page ceiling logs instead of silently
undercounting.
Alternated runs, 2 per side, on a restored real backup:
peak container 498.9 -> 398.5 MiB (-20%)
peak node RSS 452.4 -> 353.3 MiB (-22%)
peak during 4 tabs 498.5 -> 356.3 MiB (-29%)
peak arrayBuffers 95.8 -> 6.6 MiB (-93%)
largest allocation 46.4 -> 2.4 MiB (-95%)
Costs +4% CPU from the extra round trips; board load latency unchanged.
Post-soak steady state and boot idle are unchanged within run-to-run spread.
Adds the attribution tooling this needed: attribute-memory.mts reconciles
cgroup -> processes -> address-space mappings -> V8 spaces -> loaded code,
memory-probe.cjs is an injectable diagnostics preload (never shipped) that
tracks buffer and fetch allocation sites, analyze-heap-dominators.mjs computes
real retained sizes, and compare-stress-runs.mjs reports alternated runs with
their spread.
Also corrects an earlier claim in PHASE2-FINDINGS.md: the reported "7.86 MiB of
duplicate chunk sources" was an artefact of V8 truncating string contents to
~1 KiB in heap snapshots. There are zero byte-identical duplicates among all
3,055 server JS files. The real doubling is character width - one codepoint
above U+00FF forces a whole source string to two bytes per character, which
costs 9.45 MiB across 80 loaded files.
Adds 6 tests; this integration had none.
Thomas Camlong