perf(docker): stop loading ssh2, which allocates a 16 MiB WASM heap nobody uses

Profiling at the memory high-water mark - not at idle, which is what every earlier
round measured - found the largest single retainer in every snapshot was one V8
Context holding a 16.00 MiB ArrayBuffer. Expanding it through the dominator tree
named it: ssh2's crypto module, whose POLY1305_WASM_MODULE instantiates a
WebAssembly memory for ChaCha20-Poly1305 as soon as it is required.

Homarr has no SSH feature and does not depend on ssh2. The chain is
dockerode -> docker-modem -> ssh2, and docker-modem/lib/modem.js requires './ssh'
at module scope while using it in exactly one place, already guarded by
`protocol === 'ssh'`. A pnpm patch moves the require inside that branch. SSH-based
Docker hosts still work - require() is cached, so they pay it once - and every
socket-based install, which is everyone else, never loads it.

Measured over four alternated runs on a restored real backup, external memory:

                     before        after
  board loaded    25.4, 25.1    8.3, 11.2
  nav-01          28.5, 25.4   10.1,  8.6
  multi-tab       31.1, 35.8   13.0, 10.0
  post-soak       21.4, 21.5    5.5,  7.2

Consistently -15 to -18 MiB, ranges non-overlapping, and 30 fewer files loaded
(1470 -> 1440).

Peak RSS does NOT improve, and that is worth stating plainly: V8 sizes its heap
against available headroom, so freeing native memory lets it grow further before
collecting. What changes is the composition - a fixed 16 MiB of unusable WASM
buffer becomes headroom the app can actually use, which matters under a container
memory limit but does not move the peak number.

Also adds the instruments this needed. A heap snapshot can never explain a peak,
because V8 runs a full GC before serialising one and reports only survivors:

  - memory-probe.cjs gains in-process allocation sampling, on-demand snapshots via
    v8.writeHeapSnapshot, an smaps endpoint, and require-graph recording.
  - analyze-allocations.mjs diffs two stages to isolate what one stage allocated.
  - analyze-smaps-stages.mjs splits each stage's RSS into JS heap, JIT, native and
    file-backed pages.
  - analyze-heap-dominators.mjs can now expand the largest retainer, which is what
    identified ssh2. A plain graph walk escapes through __proto__ and is useless.

One trap recorded in the findings: taking the peak snapshot over CDP delivers it as
JS string chunks and added ~134 MiB of ArrayBuffers to the process being measured,
producing a fake 693 MiB reading. v8.writeHeapSnapshot streams from C++ instead, and
the probe now reports its own observerCostMiB so the cost of looking is visible.

PHASE2-FINDINGS.md

@@ -609,6 +609,118 @@ This is also the worst case for the round-trip trade-off: that one event name co
 sequential requests. It did not show up in board load latency (523 ms vs 526 ms median),
 because the widget fetches after the board renders, and total CPU rose 4%.
 
+## Dissecting the peak, not the idle
+
+Everything above measures a drained process — boot idle, or post-soak. That is the wrong
+moment: users complain about the high-water mark. Measuring it needs different instruments,
+because the obvious one cannot work.
+
+**A heap snapshot can never explain a peak.** V8 runs a full GC before serialising one, so by
+construction it reports what survived. The 94 MiB of ArrayBuffers that drove this container to
+500 MB was invisible in every snapshot taken. Three instruments were added instead:
+
+| instrument | what it sees | blind to |
+| --- | --- | --- |
+| `HeapProfiler.startSampling` (in-process `inspector.Session`) | every V8-heap allocation, survivor or garbage, with a stack | anything outside the V8 heap |
+| wrapped `Buffer.alloc`/`concat` | external buffer allocations, with a stack | V8-heap objects |
+| `/proc/<pid>/smaps` at every stage | JS heap vs JIT vs native vs file-backed | anything below page granularity |
+
+Together they cover the whole address space. Neither of the first two alone would have found
+the Umami bug: the sampling profiler cannot see `Buffer` backing stores, which is precisely
+where those 46 MiB lived.
+
+### An observer effect worth recording
+
+The first attempt took the peak snapshot over CDP, which delivers it as JS string chunks. For
+a ~135 MB snapshot that added **~134 MiB of ArrayBuffers to the process being measured**, and
+the stage right after it read 693.7 MiB. That is an artefact, not an application problem —
+and it is exactly the kind of number that gets reported as a finding. Switching to
+`v8.writeHeapSnapshot`, which streams from C++, removed it. The probe now reports its own
+`observerCostMiB` so the cost of looking is visible rather than assumed to be zero.
+
+### What the peak is actually made of
+
+Peak here is `nav-01`, not the multi-tab stage — a GC lands during multi-tab and pulls RSS
+*down*, so "most tabs open" is not the same as "most memory". Node RSS 357.8 MiB:
+
+| component | MiB | notes |
+| --- | --- | --- |
+| live JS objects (`heapUsed`) | 175.7 | of which ~41 MiB is retained bundle source |
+| node binary text | 65.7 | file-backed and shared; not in cgroup `anon` |
+| native malloc | ~59 | see below |
+| external buffers | 24.6 | |
+| V8 committed but free | 12.3 | heapTotal − heapUsed |
+| JIT code | 10.1 | grew from 1.9 MiB at boot |
+
+Cross-checked rather than assumed: `heapTotal 188.0 − V8's 256 KiB pages 153.1 = 34.9 MiB`,
+which matches `large_object_space` at 34.5 MiB. So the handful of very large anonymous
+mappings (31.6, 20.7, 8.4 MiB) are V8's large-object space and external buffers — **not** the
+runaway native allocator that their shape first suggested.
+
+**The retained heap at peak is the same as at rest.** A post-GC snapshot at peak has 160.7 MiB
+of shallow size against 157.7 MiB after the soak; strings 51.1 vs 50.1 MiB. Nothing
+accumulates. The peak is transient churn plus pages V8 and malloc have not returned — which is
+why memory "oscillates" and why soak growth is negative.
+
+And the churn is small now: total V8-heap allocation across the entire run is **125 MiB**, 40%
+of it node's own module loader reading source files. After the Umami fix the multi-tab burst
+allocates **3.3 MiB**. There is no longer a churn problem on the JS side.
+
+## The 16 MiB nobody was using: ssh2
+
+The largest single retainer in every snapshot — at boot, at peak, and after the soak — was one
+`system / Context` holding **16.03 MiB**. Expanding it through the dominator tree (a plain
+graph walk escapes via `__proto__` and reports the whole heap) named it immediately:
+
+```
+=== inside the largest retainer (object:system / Context, 16.03 MiB) ===
+     16.00 MiB  (indirect) -> object:ArrayBuffer
+      0.00 MiB  POLY1305_WASM_MODULE -> object:Object
+      0.00 MiB  ChaChaPolyCipherNative -> closure:ChaChaPolyCipherNative
+      0.00 MiB  AESGCMCipherNative -> closure:AESGCMCipherNative
+```
+
+`ssh2`. Its crypto module instantiates a WebAssembly memory for ChaCha20-Poly1305 the moment
+it is required. Homarr has no SSH feature and does not depend on ssh2; the chain is
+`dockerode → docker-modem → ssh2`, and `docker-modem/lib/modem.js` requires `./ssh` at module
+scope while using it in exactly one place, already guarded by `protocol === 'ssh'`.
+
+Measured directly inside the image — requiring ssh2 and nothing else:
+
+```
+heapUsed  +2.6 MiB     external  +0.9 MiB     rss  +14.0 MiB
+```
+
+Fixed with a pnpm patch moving that one require inside the branch that uses it. SSH-based
+Docker hosts still work — `require` is cached, so they pay it once — and socket-based installs,
+which is everyone else, never load it.
+
+### The same question asked of every dependency
+
+Since one package was hiding 14 MiB behind a require, all 173 were measured the same way: each
+required in a fresh process, RSS delta recorded. Packages costing more than 2 MiB, cross-checked
+against whether the server actually loads them:
+
+| package | RSS to require | loaded at runtime? |
+| --- | --- | --- |
+| drizzle-orm | 124.3 MiB | only via subpaths — the root import loads every dialect, so this figure overstates it |
+| @kubernetes/client-node | 66.9 MiB | **no** — correctly lazy |
+| dockerode | 28.8 MiB | yes (includes docker-modem 16.1 and ssh2 14.3) |
+| **mysql2** | **20.9 MiB** | **yes — 77 files, on a SQLite install** |
+| sharp | 20.4 MiB | no |
+| openid-client / jose | 16.1 / 15.5 MiB | no |
+| @grpc/grpc-js | 14.6 MiB | yes, 61 files (via dockerode) |
+| **pg** | **9.4 MiB** | **yes — 13 files, on a SQLite install** |
+
+These are not additive — dockerode's figure contains ssh2's — but the ranking is what matters.
+
+**mysql2 and pg are loaded on every SQLite install.** `db/drivers/index.ts` statically imports
+all three driver wrappers and picks one at call time, so importing the module executes all
+three. Not yet fixed: `packages/db/index.ts` does `export const db = createDb(schema)` at
+module scope, so `createDb` must stay synchronous and cannot use `await import()`. The viable
+route is deferring the external `require` inside each wrapper. Both dialects have testcontainer
+migration tests, so it is verifiable — it is queued, not dismissed.
+
 ## Why 24 MiB of code is resident before the first request
 
 The module cache is a flat list with no parentage, so "what pulled this in" needed the

patches/docker-modem@5.0.7.patch

@@ -0,0 +1,27 @@
+diff --git a/lib/modem.js b/lib/modem.js
+index b45a798d63647d55e0193fd7e82c2e36785d9b5f..d2dbf42969fc0ebf39b30c1a27373c33c2d20333 100644
+--- a/lib/modem.js
++++ b/lib/modem.js
+@@ -3,7 +3,6 @@ var querystring = require('querystring'),
+   fs = require('fs'),
+   path = require('path'),
+   url = require('url'),
+-  ssh = require('./ssh'),
+   HttpDuplex = require('./http_duplex'),
+   debug = require('debug')('modem'),
+   utils = require('./utils'),
+@@ -263,8 +262,13 @@ Modem.prototype.buildRequest = function (options, context, data, callback) {
+   var connectionTimeoutTimer;
+   var finished = false;
+ 
++  // Loaded here rather than at module scope: './ssh' pulls in ssh2, whose crypto module
++  // instantiates a ~16 MiB WebAssembly memory for ChaCha20-Poly1305 as soon as it is
++  // required. Measured at +14 MiB RSS for a require alone, paid by every Homarr install
++  // even though Docker is reached over a socket and this branch never runs. require() is
++  // cached, so SSH connections pay it once and behave exactly as before.
+   var opts = self.protocol === 'ssh' ? Object.assign(options, {
+-    agent: ssh(Object.assign({}, self.sshOptions, {
++    agent: require('./ssh')(Object.assign({}, self.sshOptions, {
+       'host': self.host,
+       'port': self.port,
+       'username': self.username,

pnpm-lock.yaml

@@ -585,6 +585,7 @@ overrides:
 
 patchedDependencies:
   '@types/node-unifi': 5e6ae51e2a17a7f9729bfa30b0eb3d0842a5810ac6db47603ab4a6efa1ed84c5
+  docker-modem@5.0.7: 4a1c50dc8b726af921a34a77c4942a768bed229123ab116baa06c4e62ad6815c
   trpc-to-openapi: 2ca3c16af0fcca0c736697ad4fe553a14f794524fa9ce0d5c3e8ee4aea76090c
 
 importers:
@@ -23646,7 +23647,7 @@ snapshots:
     dependencies:
       yaml: 2.9.0
 
-  docker-modem@5.0.7(supports-color@10.2.2):
+  docker-modem@5.0.7(patch_hash=4a1c50dc8b726af921a34a77c4942a768bed229123ab116baa06c4e62ad6815c)(supports-color@10.2.2):
     dependencies:
       debug: 4.4.3(supports-color@10.2.2)
       readable-stream: 3.6.2
@@ -23660,7 +23661,7 @@ snapshots:
       '@balena/dockerignore': 1.0.2
       '@grpc/grpc-js': 1.12.5
       '@grpc/proto-loader': 0.7.13
-      docker-modem: 5.0.7(supports-color@10.2.2)
+      docker-modem: 5.0.7(patch_hash=4a1c50dc8b726af921a34a77c4942a768bed229123ab116baa06c4e62ad6815c)(supports-color@10.2.2)
       protobufjs: 7.4.0
       tar-fs: 2.1.4
     transitivePeerDependencies:

pnpm-workspace.yaml

@@ -233,6 +233,7 @@ overrides:
   "vite@>=5.0.0 <=5.4.18": ">=8.1.4"
 patchedDependencies:
   "@types/node-unifi": "patches/@types__node-unifi.patch"
+  docker-modem@5.0.7: patches/docker-modem@5.0.7.patch
   "trpc-to-openapi": "patches/trpc-to-openapi.patch"
 
 allowUnusedPatches: true

scripts/benchmarks/analyze-allocations.mjs

@@ -0,0 +1,99 @@
+// Reports what allocated memory during a stress run, by site and by owner — and, given two
+// stages, what a single stage allocated.
+//
+//   node scripts/benchmarks/analyze-allocations.mjs <dir> [fromStage] [toStage]
+//   node scripts/benchmarks/analyze-allocations.mjs .bench/stress/peak nav-03 07-multi-tab
+//
+// This is the counterpart to a heap snapshot and answers the opposite question. A snapshot is
+// taken after a forced GC, so it shows only what *survived*; the churn that produces a memory
+// peak is by definition gone by then. The sampling heap profiler records an allocation stack
+// every N bytes regardless of survival, so a peak driven by transient garbage — which is what
+// this container's peak turned out to be — shows up here and nowhere else.
+//
+// Sampling totals are cumulative since boot, so a single stage is only meaningful as the
+// difference between two captures. Without stage arguments this reports the cumulative
+// profile, which is dominated by boot-time module loading.
+import { readFileSync, existsSync } from "node:fs";
+import path from "node:path";
+
+const [, , directory, fromStage, toStage] = process.argv;
+if (!directory) throw new Error("usage: analyze-allocations.mjs <dir> [fromStage] [toStage]");
+
+const load = (stage) => {
+  const file = path.join(directory, `${stage}.allocations.json`);
+  if (!existsSync(file)) throw new Error(`missing ${file}`);
+  const parsed = JSON.parse(readFileSync(file, "utf8"));
+  if (parsed.disabled) throw new Error("sampling was not enabled (HOMARR_PROBE_SAMPLE_ALLOCATIONS)");
+  return parsed;
+};
+
+const MiB = (bytes) => (bytes / 1048576).toFixed(2);
+
+/** Groups a frame to something a developer could act on: a package, a chunk, or node itself. */
+const ownerOf = (key) => {
+  const url = key.slice(key.indexOf("|") + 1);
+  if (!url || url.startsWith("undefined")) return "unattributed (no script)";
+  if (/^node:/.test(url)) {
+    if (/cjs\/loader|modules\//.test(url)) return "node: CommonJS module loader (reading source)";
+    return "node internals";
+  }
+  const dependency = /\/app\/node_modules\/((?:@[^/]+\/)?[^/]+)\//.exec(url);
+  if (dependency) return `node_modules: ${dependency[1]}`;
+  if (url.includes("/.next/server/chunks/")) {
+    const chunk = /chunks\/(?:ssr\/)?([^/:]+)/.exec(url);
+    return `chunk ${chunk ? chunk[1] : "?"}`;
+  }
+  if (url.includes("/.next/server/app/")) return "route bundle";
+  if (url.includes("/probe/")) return "the probe itself (diagnostics)";
+  return "other";
+};
+
+const shorten = (key) =>
+  key.replace("|", "  <- ").replace("/app/apps/nextjs/.next/server/", "").replace("/app/node_modules/", "");
+
+let label;
+let total;
+let sites;
+
+if (fromStage && toStage) {
+  const before = load(fromStage);
+  const after = load(toStage);
+  const previous = new Map(before.sites.map((site) => [site.key, site.bytes]));
+  label = `allocated between "${fromStage}" and "${toStage}"`;
+  total = after.total - before.total;
+  sites = after.sites
+    .map((site) => ({ ...site, bytes: site.bytes - (previous.get(site.key) ?? 0) }))
+    .filter((site) => site.bytes > 0)
+    .sort((a, b) => b.bytes - a.bytes);
+} else {
+  const profile = load(fromStage ?? "06-final");
+  label = `cumulative since boot (${fromStage ?? "06-final"})`;
+  total = profile.total;
+  sites = profile.sites;
+}
+
+const covered = sites.reduce((sum, site) => sum + site.bytes, 0);
+console.log(`${label}: ${MiB(total)} MiB total, ${MiB(covered)} MiB attributed across ${sites.length} sites\n`);
+
+const byOwner = new Map();
+for (const site of sites) {
+  const owner = ownerOf(site.key);
+  const acc = byOwner.get(owner) ?? { bytes: 0, sites: 0 };
+  acc.bytes += site.bytes;
+  acc.sites++;
+  byOwner.set(owner, acc);
+}
+
+console.log("=== allocation volume by owner ===");
+console.log(`${"MiB".padStart(9)} ${"share".padStart(7)} ${"sites".padStart(6)}  owner`);
+for (const [owner, acc] of [...byOwner.entries()].sort((a, b) => b[1].bytes - a[1].bytes).slice(0, 18)) {
+  const share = covered === 0 ? "-" : `${((acc.bytes / covered) * 100).toFixed(1)}%`;
+  console.log(`${MiB(acc.bytes).padStart(9)} ${share.padStart(7)} ${String(acc.sites).padStart(6)}  ${owner}`);
+}
+
+console.log(`\n=== top individual allocation sites ===`);
+for (const site of sites.slice(0, 16)) {
+  const share = covered === 0 ? "-" : `${((site.bytes / covered) * 100).toFixed(1)}%`;
+  console.log(`\n  ${MiB(site.bytes)} MiB (${share})  ${shorten(site.key).slice(0, 110)}`);
+  for (const frame of (site.stack ?? []).slice(1, 4)) console.log(`      from ${shorten(frame).slice(0, 106)}`);
+}

scripts/benchmarks/analyze-heap-dominators.mjs

@@ -198,6 +198,17 @@ const label = (o) => {
   return `${type}:${name}`;
 };
 
+/** The property name a parent reaches a child through — that is what names the contents. */
+const edgeLabelBetween = (from, to) => {
+  for (let e = firstEdge[from]; e < firstEdge[from + 1]; e++) {
+    if (edges[e * eStride + eTo] / nStride !== to) continue;
+    const type = edgeTypes[edges[e * eStride + eType]];
+    const raw = edges[e * eStride + eName];
+    return type === "element" || type === "hidden" ? `[${raw}]` : (strings[raw] ?? String(raw));
+  }
+  return "(indirect)";
+};
+
 const candidates = [];
 for (let i = 0; i < reachableCount; i++) {
   if (i === rootPost) continue;
@@ -228,6 +239,26 @@ for (const i of candidates) {
   if (shown.length >= topN) break;
 }
 
+// Expanding the single largest retainer is usually the whole point: "a Context retains 16 MiB"
+// is not actionable until its contents are named. Only the dominator tree can do this — a
+// plain graph walk escapes through __proto__ into the entire heap and reports nonsense.
+if (shown.length) {
+  const top = shown[0];
+  const childrenOfTop = [];
+  for (let i = 0; i < reachableCount; i++) {
+    if (i !== top && dom[i] === top) childrenOfTop.push(i);
+  }
+  childrenOfTop.sort((a, b) => retained[b] - retained[a]);
+  const topNode = postOrderNodes[top];
+  console.log(`\n=== inside the largest retainer (${label(topNode)}, ${MiB(retained[top])} MiB) ===`);
+  console.log(`  it dominates ${childrenOfTop.length} direct children; heaviest:`);
+  for (const child of childrenOfTop.slice(0, 20)) {
+    const node = postOrderNodes[child];
+    const edgeName = edgeLabelBetween(topNode, node);
+    console.log(`    ${MiB(retained[child]).padStart(8)} MiB  ${edgeName} -> ${label(node).slice(0, 78)}`);
+  }
+}
+
 // Grouping by constructor over retained size shows which *kind* of thing dominates, which
 // is what a shallow histogram cannot express.
 const byName = new Map();

scripts/benchmarks/analyze-smaps-stages.mjs

@@ -0,0 +1,121 @@
+// Breaks each stage's RSS into JS heap, JIT code, native malloc and file-backed pages.
+//
+//   node scripts/benchmarks/analyze-smaps-stages.mjs .bench/stress/<label>
+//
+// `process.memoryUsage()` accounts for the JS heap and external buffers and stops there. On a
+// loaded container that leaves a large unexplained remainder — measured at ~152 MiB of a
+// 352 MiB peak — which is invisible to every JS-level tool. Only /proc/<pid>/smaps
+// distinguishes what that remainder is, and only a per-stage capture shows which part of it
+// grows under load.
+//
+// Anonymous mappings are classified by signature: V8 allocates its heap in 256 KiB pages
+// (kRegularPageSize) and nothing else in the process allocates at that granularity in bulk,
+// while pthread reserves ~8 MiB per thread stack and touches almost none of it.
+import { readFileSync, readdirSync } from "node:fs";
+import path from "node:path";
+
+const directory = process.argv[2];
+if (!directory) throw new Error("usage: analyze-smaps-stages.mjs <stress output dir>");
+
+const V8_PAGE_BYTES = 262144;
+const THREAD_STACK_BYTES = 8196 * 1024;
+
+const parse = (text) => {
+  const mappings = [];
+  let current;
+  for (const line of text.split("\n")) {
+    const header = /^([0-9a-f]+)-([0-9a-f]+)\s+(\S{4})\s+\S+\s+\S+\s+\S+\s*(.*)$/.exec(line);
+    if (header) {
+      if (current) mappings.push(current);
+      const start = Number.parseInt(header[1], 16);
+      const end = Number.parseInt(header[2], 16);
+      current = { size: end - start, rss: 0, perms: header[3], file: (header[4] ?? "").trim() };
+      continue;
+    }
+    if (!current) continue;
+    const rss = /^Rss:\s+(\d+) kB/.exec(line);
+    if (rss) current.rss = Number(rss[1]) * 1024;
+  }
+  if (current) mappings.push(current);
+  return mappings;
+};
+
+const bucketOf = (mapping) => {
+  const executable = mapping.perms.includes("x");
+  if (mapping.file === "[stack]" || mapping.size === THREAD_STACK_BYTES) return "thread stacks";
+  if (mapping.file === "[heap]") return "malloc brk heap";
+  if (mapping.file.startsWith("[")) return "kernel";
+  if (mapping.file) {
+    if (/\.node$/.test(mapping.file)) return "native addons (file-backed)";
+    if (/\/node$/.test(mapping.file)) return "node binary (file-backed)";
+    if (/\.so/.test(mapping.file)) return "shared libraries (file-backed)";
+    return "other file-backed";
+  }
+  if (executable) return "JIT code";
+  if (mapping.size === V8_PAGE_BYTES) return "V8 heap pages";
+  return "native malloc / buffers";
+};
+
+const ORDER = ["01-boot", "02-post-restore", "03-post-login", "04-board", "nav-", "07-multi-tab", "05-after", "soak-", "06-final"];
+const rank = (name) => {
+  const index = ORDER.findIndex((prefix) => name.startsWith(prefix));
+  return index === -1 ? ORDER.length : index;
+};
+
+const stages = readdirSync(directory)
+  .filter((name) => name.endsWith(".smaps.json"))
+  .map((name) => name.replace(".smaps.json", ""))
+  .sort((a, b) => rank(a) - rank(b) || a.localeCompare(b));
+if (!stages.length) throw new Error(`no *.smaps.json in ${directory}`);
+
+const MiB = (bytes) => (bytes / 1048576).toFixed(1);
+const BUCKETS = [
+  "V8 heap pages",
+  "native malloc / buffers",
+  "JIT code",
+  "node binary (file-backed)",
+  "shared libraries (file-backed)",
+  "native addons (file-backed)",
+  "thread stacks",
+  "malloc brk heap",
+];
+
+const rows = [];
+for (const stage of stages) {
+  const payload = JSON.parse(readFileSync(path.join(directory, `${stage}.smaps.json`), "utf8"));
+  if (!payload.smaps) continue;
+  const totals = new Map();
+  let rss = 0;
+  for (const mapping of parse(payload.smaps)) {
+    rss += mapping.rss;
+    const bucket = bucketOf(mapping);
+    totals.set(bucket, (totals.get(bucket) ?? 0) + mapping.rss);
+  }
+  rows.push({ stage, rss, totals });
+}
+
+const header = ["stage".padEnd(16), "RSS".padStart(8), ...BUCKETS.map((name) => name.slice(0, 11).padStart(12))].join(" ");
+console.log(`\n=== RSS composition per stage, MiB ===`);
+console.log(header);
+console.log("-".repeat(header.length));
+for (const row of rows) {
+  const cells = BUCKETS.map((bucket) => MiB(row.totals.get(bucket) ?? 0).padStart(12));
+  console.log([row.stage.padEnd(16), MiB(row.rss).padStart(8), ...cells].join(" "));
+}
+
+const first = rows[0];
+const peak = rows.reduce((best, row) => (row.rss > best.rss ? row : best), rows[0]);
+console.log(`\n=== boot ("${first.stage}") vs peak ("${peak.stage}") ===`);
+console.log(`${"bucket".padEnd(32)} ${"boot".padStart(9)} ${"peak".padStart(9)} ${"growth".padStart(9)}`);
+for (const bucket of BUCKETS) {
+  const from = first.totals.get(bucket) ?? 0;
+  const to = peak.totals.get(bucket) ?? 0;
+  if (from === 0 && to === 0) continue;
+  const delta = to - from;
+  console.log(
+    `${bucket.padEnd(32)} ${MiB(from).padStart(9)} ${MiB(to).padStart(9)} ${((delta >= 0 ? "+" : "") + MiB(delta)).padStart(9)}`,
+  );
+}
+console.log(
+  `${"TOTAL RSS".padEnd(32)} ${MiB(first.rss).padStart(9)} ${MiB(peak.rss).padStart(9)} ${("+" + MiB(peak.rss - first.rss)).padStart(9)}`,
+);

scripts/benchmarks/memory-probe.cjs

@@ -105,6 +105,112 @@ globalThis.__homarrProbe = {
   moduleLoadList: () => process.moduleLoadList.slice(),
 };
 
+/**
+ * Opt-in allocation sampling, and on-demand heap snapshots, over an in-process inspector
+ * session.
+ *
+ * A heap snapshot cannot explain a peak. V8 runs a full GC before serialising one, so by
+ * construction it shows what survived, never the transient churn that *is* the peak — the
+ * 94 MiB of ArrayBuffers that drove this container to 500 MB were invisible in every
+ * snapshot taken. The sampling heap profiler is the tool that sees them: it records an
+ * allocation stack every N bytes allocated, whether or not the object survives.
+ *
+ * An in-process `inspector.Session` is used rather than SIGUSR1 plus a debug port, so this
+ * needs no signal, opens no socket, and works while the process is under load.
+ *
+ * Enable with HOMARR_PROBE_SAMPLE_ALLOCATIONS=<sampling interval in bytes>.
+ */
+const inspector = require("inspector");
+let session = null;
+const ensureSession = () => {
+  if (session) return session;
+  session = new inspector.Session();
+  session.connect();
+  return session;
+};
+const postAsync = (method, params) =>
+  new Promise((resolve, reject) =>
+    ensureSession().post(method, params, (error, result) => (error ? reject(error) : resolve(result))),
+  );
+
+const samplingInterval = Number(process.env.HOMARR_PROBE_SAMPLE_ALLOCATIONS ?? 0);
+if (samplingInterval > 0) {
+  postAsync("HeapProfiler.enable")
+    .then(() => postAsync("HeapProfiler.startSampling", { samplingInterval }))
+    .then(() => console.log(`[memory-probe] allocation sampling every ${samplingInterval} bytes`))
+    .catch((error) => console.error(`[memory-probe] could not start sampling: ${error.message}`));
+
+  /**
+   * Flattens the profile's call tree to self-bytes per allocation site.
+   *
+   * Sites are keyed by frame and aggregated over the *whole* tree, not just the largest
+   * nodes, so the reported totals actually sum to the profile total — a truncated list
+   * silently drops most of the bytes and makes every percentage wrong.
+   *
+   * `getSamplingProfile` reports cumulative bytes since sampling began, which mixes boot with
+   * load. Capturing at two stages and subtracting is what isolates a single stage, so the key
+   * is kept stable and machine-comparable for exactly that purpose.
+   */
+  globalThis.__homarrProbe.allocations = async () => {
+    const { profile } = await postAsync("HeapProfiler.getSamplingProfile");
+    const sites = new Map();
+    let total = 0;
+    const walk = (node, ancestry) => {
+      const frame = node.callFrame;
+      const key = `${frame.functionName || "(anonymous)"}|${frame.url}:${frame.lineNumber + 1}`;
+      const stack = [key, ...ancestry];
+      const selfSize = node.selfSize ?? 0;
+      if (selfSize > 0) {
+        total += selfSize;
+        const acc = sites.get(key) ?? { bytes: 0, samples: 0, stack: stack.slice(0, 6) };
+        acc.bytes += selfSize;
+        acc.samples++;
+        sites.set(key, acc);
+      }
+      for (const child of node.children ?? []) walk(child, stack);
+    };
+    walk(profile.head, []);
+    return {
+      total,
+      sites: [...sites.entries()]
+        .map(([key, acc]) => ({ key, ...acc }))
+        .sort((a, b) => b.bytes - a.bytes)
+        .slice(0, 400),
+    };
+  };
+}
+
+/**
+ * Writes a heap snapshot to a path inside the container. Exposed so the harness can grab one
+ * at a chosen moment — in particular while tabs are still open, which is a different heap
+ * from the post-soak one every earlier snapshot captured.
+ */
+globalThis.__homarrProbe.snapshot = (destination) => {
+  // v8.writeHeapSnapshot streams to the file from C++. The CDP route
+  // (HeapProfiler.takeHeapSnapshot) delivers the snapshot as JS string chunks instead, and
+  // for a ~135 MB snapshot that briefly added ~134 MiB of ArrayBuffers to the very process
+  // being measured — an observer effect large enough to be mistaken for an application
+  // problem, which is exactly what happened on the first attempt.
+  const before = process.memoryUsage();
+  v8.writeHeapSnapshot(destination);
+  const after = process.memoryUsage();
+  return {
+    written: destination,
+    bytes: fs.statSync(destination).size,
+    // Reported so the cost of observing is visible rather than assumed to be zero.
+    observerCostMiB: Math.round(((after.rss - before.rss) / 1048576) * 10) / 10,
+  };
+};
+
+/** Address-space breakdown at the moment of capture; only /proc shows native vs JS vs code. */
+globalThis.__homarrProbe.maps = () => {
+  try {
+    return { smaps: fs.readFileSync(`/proc/${process.pid}/smaps`, "utf8") };
+  } catch (error) {
+    return { error: String(error && error.message) };
+  }
+};
+
 /**
  * Opt-in recording of the CommonJS require graph.
  *
@@ -284,36 +390,49 @@ if (probePort > 0) {
   const probe = globalThis.__homarrProbe;
   require("http")
     .createServer((request, response) => {
-      let body;
-      try {
-        // The raw file list is served separately: it is long, and every stage of a stress
-        // run fetches the summary, so keeping it out of the default payload keeps those
-        // captures small while still allowing a boot-vs-loaded diff on demand.
-        body =
-          request.url === "/files"
-            ? JSON.stringify(probe.moduleFiles())
-            : request.url === "/buffers"
-              ? JSON.stringify(probe.buffers ? probe.buffers() : { disabled: true })
-              : request.url === "/fetches"
-                ? JSON.stringify(probe.fetches ? probe.fetches() : { disabled: true })
-                : request.url === "/requires"
-                  ? JSON.stringify(probe.requireGraph ? probe.requireGraph() : { disabled: true })
-                  : JSON.stringify({
-                usage: probe.usage(),
-                spaces: probe.spaces(),
-                heap: probe.heap(),
-                modules: probe.modules(),
-                largestModules: probe.largestModules(25),
-                uptime: process.uptime(),
-                execArgv: process.execArgv,
-              });
-      } catch (error) {
-        response.writeHead(500, { "content-type": "application/json" });
-        response.end(JSON.stringify({ error: String(error && error.message) }));
-        return;
-      }
-      response.writeHead(200, { "content-type": "application/json" });
-      response.end(body);
+      // Several endpoints are async (the inspector ones), so every route resolves through a
+      // promise rather than some returning a value and some a pending Promise object.
+      const [route, query] = (request.url ?? "/").split("?");
+      const resolve = async () => {
+        switch (route) {
+          // The raw file list is served separately: it is long, and every stage of a stress
+          // run fetches the summary, so keeping it out of the default payload keeps those
+          // captures small while still allowing a boot-vs-loaded diff on demand.
+          case "/files":
+            return probe.moduleFiles();
+          case "/buffers":
+            return probe.buffers ? probe.buffers() : { disabled: true };
+          case "/fetches":
+            return probe.fetches ? probe.fetches() : { disabled: true };
+          case "/requires":
+            return probe.requireGraph ? probe.requireGraph() : { disabled: true };
+          case "/allocations":
+            return probe.allocations ? await probe.allocations() : { disabled: true };
+          case "/maps":
+            return probe.maps();
+          case "/snapshot":
+            return await probe.snapshot(new URLSearchParams(query).get("path") ?? "/tmp/peak.heapsnapshot");
+          default:
+            return {
+              usage: probe.usage(),
+              spaces: probe.spaces(),
+              heap: probe.heap(),
+              modules: probe.modules(),
+              largestModules: probe.largestModules(25),
+              uptime: process.uptime(),
+              execArgv: process.execArgv,
+            };
+        }
+      };
+      resolve()
+        .then((value) => {
+          response.writeHead(200, { "content-type": "application/json" });
+          response.end(JSON.stringify(value));
+        })
+        .catch((error) => {
+          response.writeHead(500, { "content-type": "application/json" });
+          response.end(JSON.stringify({ error: String(error && error.message) }));
+        });
     })
     // NODE_OPTIONS applies to every node process in the container, including the DB
     // migration step and any `docker exec node` the harness runs, so all of them load this

scripts/benchmarks/stress-restore.mts

@@ -46,6 +46,10 @@ const soakIntervalMs = Number(process.env.STRESS_SOAK_INTERVAL_MS ?? 20_000);
 const keepContainer = process.env.STRESS_KEEP_CONTAINER === "true";
 /** Dump a browser heap snapshot here (analyse with analyze-react-fibers.mjs). */
 const clientSnapshotPath = process.env.STRESS_CLIENT_SNAPSHOT;
+/** Dump a *server* heap snapshot taken at peak, with all tabs still open. */
+const peakSnapshotPath = process.env.STRESS_PEAK_SNAPSHOT
+  ? path.resolve(process.env.STRESS_PEAK_SNAPSHOT)
+  : undefined;
 /**
  * Extra `docker run` wiring, so a diagnostics-only preload can be injected into a
  * shipped image without rebuilding it. Used to mount scripts/benchmarks at /probe and
@@ -115,6 +119,54 @@ const captureProbeAsync = async (name: string) => {
   if (!fetches.includes('"disabled"')) {
     await writeFile(path.join(outputDirectory, `${name}.fetches.json`), fetches);
   }
+  // Allocation totals are cumulative since boot, so a single capture cannot separate "the
+  // app loaded its code" from "four tabs churned through memory". Capturing at every stage
+  // makes those subtractable.
+  const allocations = await fetchProbeAsync("/allocations");
+  if (!allocations.includes('"disabled"')) {
+    await writeFile(path.join(outputDirectory, `${name}.allocations.json`), allocations);
+  }
+  // The address-space map is the only place JS heap, JIT code, native malloc and file-backed
+  // pages are distinguishable, and `rss - heapTotal - external` grew by ~85 MiB between boot
+  // and peak with no JS-side explanation. Captured every stage because which stage is the
+  // true high-water mark is not known in advance — it is not always the multi-tab one.
+  const maps = await fetchProbeAsync("/maps");
+  if (!maps.includes('"error"')) {
+    await writeFile(path.join(outputDirectory, `${name}.smaps.json`), maps);
+  }
+};
+
+/**
+ * Grabs the two artefacts that only make sense at the high-water mark, with load still on:
+ * a heap snapshot (what peak memory *retains*) and the cumulative allocation profile (what
+ * peak memory *churned through*, which no snapshot can show because V8 GCs before it
+ * serialises one). Set STRESS_PEAK_SNAPSHOT to a host path to keep the snapshot.
+ */
+const capturePeakAsync = async () => {
+  if (!probePort) return;
+  // Mappings first, and before any snapshot: this is the only reading that separates JS heap
+  // from native, JIT and file-backed pages, and it must reflect the loaded process rather
+  // than the process plus whatever the act of snapshotting allocated.
+  const maps = await fetchProbeAsync("/maps");
+  if (!maps.includes('"error"')) {
+    await writeFile(path.join(outputDirectory, "peak.smaps.json"), maps);
+    log("peak: address-space map captured");
+  }
+  const allocations = await fetchProbeAsync("/allocations");
+  if (!allocations.includes('"disabled"')) {
+    await writeFile(path.join(outputDirectory, "peak.allocations.json"), allocations);
+    log("peak: allocation profile captured");
+  }
+  if (!peakSnapshotPath) return;
+  const inContainer = "/tmp/peak.heapsnapshot";
+  const result = await fetchProbeAsync(`/snapshot?path=${encodeURIComponent(inContainer)}`);
+  if (result.includes('"error"')) {
+    log(`peak: snapshot failed — ${result.slice(0, 160)}`);
+    return;
+  }
+  await execFileAsync("docker", ["cp", `${containerName}:${inContainer}`, peakSnapshotPath]);
+  await execFileAsync("docker", ["exec", containerName, "rm", "-f", inContainer]).catch(() => undefined);
+  log(`peak: heap snapshot saved to ${peakSnapshotPath}`);
 };
 
 const captureAsync = async (name: string) => {
@@ -512,6 +564,9 @@ const main = async () => {
       }
       log(`multi-tab: clicked ${widgetInteractions.clicked}/${widgetInteractions.available} widget controls`);
       await captureAsync("07-multi-tab");
+      // Taken here, with every tab still open, because this is the high-water mark. Every
+      // earlier snapshot was of a drained process and could not show what peak memory holds.
+      await capturePeakAsync();
     }
 
     if (clientSnapshotPath) await captureClientSnapshotAsync(freshPage, clientSnapshotPath);