perf(docker): stop loading ssh2, which allocates a 16 MiB WASM heap nobody uses
Profiling at the memory high-water mark - not at idle, which is what every earlier
round measured - found the largest single retainer in every snapshot was one V8
Context holding a 16.00 MiB ArrayBuffer. Expanding it through the dominator tree
named it: ssh2's crypto module, whose POLY1305_WASM_MODULE instantiates a
WebAssembly memory for ChaCha20-Poly1305 as soon as it is required.
Homarr has no SSH feature and does not depend on ssh2. The chain is
dockerode -> docker-modem -> ssh2, and docker-modem/lib/modem.js requires './ssh'
at module scope while using it in exactly one place, already guarded by
`protocol === 'ssh'`. A pnpm patch moves the require inside that branch. SSH-based
Docker hosts still work - require() is cached, so they pay it once - and every
socket-based install, which is everyone else, never loads it.
Measured over four alternated runs on a restored real backup, external memory:
before after
board loaded 25.4, 25.1 8.3, 11.2
nav-01 28.5, 25.4 10.1, 8.6
multi-tab 31.1, 35.8 13.0, 10.0
post-soak 21.4, 21.5 5.5, 7.2
Consistently -15 to -18 MiB, ranges non-overlapping, and 30 fewer files loaded
(1470 -> 1440).
Peak RSS does NOT improve, and that is worth stating plainly: V8 sizes its heap
against available headroom, so freeing native memory lets it grow further before
collecting. What changes is the composition - a fixed 16 MiB of unusable WASM
buffer becomes headroom the app can actually use, which matters under a container
memory limit but does not move the peak number.
Also adds the instruments this needed. A heap snapshot can never explain a peak,
because V8 runs a full GC before serialising one and reports only survivors:
- memory-probe.cjs gains in-process allocation sampling, on-demand snapshots via
v8.writeHeapSnapshot, an smaps endpoint, and require-graph recording.
- analyze-allocations.mjs diffs two stages to isolate what one stage allocated.
- analyze-smaps-stages.mjs splits each stage's RSS into JS heap, JIT, native and
file-backed pages.
- analyze-heap-dominators.mjs can now expand the largest retainer, which is what
identified ssh2. A plain graph walk escapes through __proto__ and is useless.
One trap recorded in the findings: taking the peak snapshot over CDP delivers it as
JS string chunks and added ~134 MiB of ArrayBuffers to the process being measured,
producing a fake 693 MiB reading. v8.writeHeapSnapshot streams from C++ instead, and
the probe now reports its own observerCostMiB so the cost of looking is visible.
This commit is contained in:
9 files changed
+598
-32
No files matched your search
@@ -105,6 +105,112 @@ globalThis.__homarrProbe = {
|
||||
moduleLoadList: () => process.moduleLoadList.slice(),
|
||||
};
|
||||
|
||||
/**
|
||||
* Opt-in allocation sampling, and on-demand heap snapshots, over an in-process inspector
|
||||
* session.
|
||||
*
|
||||
* A heap snapshot cannot explain a peak. V8 runs a full GC before serialising one, so by
|
||||
* construction it shows what survived, never the transient churn that *is* the peak — the
|
||||
* 94 MiB of ArrayBuffers that drove this container to 500 MB were invisible in every
|
||||
* snapshot taken. The sampling heap profiler is the tool that sees them: it records an
|
||||
* allocation stack every N bytes allocated, whether or not the object survives.
|
||||
*
|
||||
* An in-process `inspector.Session` is used rather than SIGUSR1 plus a debug port, so this
|
||||
* needs no signal, opens no socket, and works while the process is under load.
|
||||
*
|
||||
* Enable with HOMARR_PROBE_SAMPLE_ALLOCATIONS=<sampling interval in bytes>.
|
||||
*/
|
||||
const inspector = require("inspector");
|
||||
let session = null;
|
||||
const ensureSession = () => {
|
||||
if (session) return session;
|
||||
session = new inspector.Session();
|
||||
session.connect();
|
||||
return session;
|
||||
};
|
||||
const postAsync = (method, params) =>
|
||||
new Promise((resolve, reject) =>
|
||||
ensureSession().post(method, params, (error, result) => (error ? reject(error) : resolve(result))),
|
||||
);
|
||||
|
||||
const samplingInterval = Number(process.env.HOMARR_PROBE_SAMPLE_ALLOCATIONS ?? 0);
|
||||
if (samplingInterval > 0) {
|
||||
postAsync("HeapProfiler.enable")
|
||||
.then(() => postAsync("HeapProfiler.startSampling", { samplingInterval }))
|
||||
.then(() => console.log(`[memory-probe] allocation sampling every ${samplingInterval} bytes`))
|
||||
.catch((error) => console.error(`[memory-probe] could not start sampling: ${error.message}`));
|
||||
|
||||
/**
|
||||
* Flattens the profile's call tree to self-bytes per allocation site.
|
||||
*
|
||||
* Sites are keyed by frame and aggregated over the *whole* tree, not just the largest
|
||||
* nodes, so the reported totals actually sum to the profile total — a truncated list
|
||||
* silently drops most of the bytes and makes every percentage wrong.
|
||||
*
|
||||
* `getSamplingProfile` reports cumulative bytes since sampling began, which mixes boot with
|
||||
* load. Capturing at two stages and subtracting is what isolates a single stage, so the key
|
||||
* is kept stable and machine-comparable for exactly that purpose.
|
||||
*/
|
||||
globalThis.__homarrProbe.allocations = async () => {
|
||||
const { profile } = await postAsync("HeapProfiler.getSamplingProfile");
|
||||
const sites = new Map();
|
||||
let total = 0;
|
||||
const walk = (node, ancestry) => {
|
||||
const frame = node.callFrame;
|
||||
const key = `${frame.functionName || "(anonymous)"}|${frame.url}:${frame.lineNumber + 1}`;
|
||||
const stack = [key, ...ancestry];
|
||||
const selfSize = node.selfSize ?? 0;
|
||||
if (selfSize > 0) {
|
||||
total += selfSize;
|
||||
const acc = sites.get(key) ?? { bytes: 0, samples: 0, stack: stack.slice(0, 6) };
|
||||
acc.bytes += selfSize;
|
||||
acc.samples++;
|
||||
sites.set(key, acc);
|
||||
}
|
||||
for (const child of node.children ?? []) walk(child, stack);
|
||||
};
|
||||
walk(profile.head, []);
|
||||
return {
|
||||
total,
|
||||
sites: [...sites.entries()]
|
||||
.map(([key, acc]) => ({ key, ...acc }))
|
||||
.sort((a, b) => b.bytes - a.bytes)
|
||||
.slice(0, 400),
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes a heap snapshot to a path inside the container. Exposed so the harness can grab one
|
||||
* at a chosen moment — in particular while tabs are still open, which is a different heap
|
||||
* from the post-soak one every earlier snapshot captured.
|
||||
*/
|
||||
globalThis.__homarrProbe.snapshot = (destination) => {
|
||||
// v8.writeHeapSnapshot streams to the file from C++. The CDP route
|
||||
// (HeapProfiler.takeHeapSnapshot) delivers the snapshot as JS string chunks instead, and
|
||||
// for a ~135 MB snapshot that briefly added ~134 MiB of ArrayBuffers to the very process
|
||||
// being measured — an observer effect large enough to be mistaken for an application
|
||||
// problem, which is exactly what happened on the first attempt.
|
||||
const before = process.memoryUsage();
|
||||
v8.writeHeapSnapshot(destination);
|
||||
const after = process.memoryUsage();
|
||||
return {
|
||||
written: destination,
|
||||
bytes: fs.statSync(destination).size,
|
||||
// Reported so the cost of observing is visible rather than assumed to be zero.
|
||||
observerCostMiB: Math.round(((after.rss - before.rss) / 1048576) * 10) / 10,
|
||||
};
|
||||
};
|
||||
|
||||
/** Address-space breakdown at the moment of capture; only /proc shows native vs JS vs code. */
|
||||
globalThis.__homarrProbe.maps = () => {
|
||||
try {
|
||||
return { smaps: fs.readFileSync(`/proc/${process.pid}/smaps`, "utf8") };
|
||||
} catch (error) {
|
||||
return { error: String(error && error.message) };
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Opt-in recording of the CommonJS require graph.
|
||||
*
|
||||
@@ -284,36 +390,49 @@ if (probePort > 0) {
|
||||
const probe = globalThis.__homarrProbe;
|
||||
require("http")
|
||||
.createServer((request, response) => {
|
||||
let body;
|
||||
try {
|
||||
// The raw file list is served separately: it is long, and every stage of a stress
|
||||
// run fetches the summary, so keeping it out of the default payload keeps those
|
||||
// captures small while still allowing a boot-vs-loaded diff on demand.
|
||||
body =
|
||||
request.url === "/files"
|
||||
? JSON.stringify(probe.moduleFiles())
|
||||
: request.url === "/buffers"
|
||||
? JSON.stringify(probe.buffers ? probe.buffers() : { disabled: true })
|
||||
: request.url === "/fetches"
|
||||
? JSON.stringify(probe.fetches ? probe.fetches() : { disabled: true })
|
||||
: request.url === "/requires"
|
||||
? JSON.stringify(probe.requireGraph ? probe.requireGraph() : { disabled: true })
|
||||
: JSON.stringify({
|
||||
usage: probe.usage(),
|
||||
spaces: probe.spaces(),
|
||||
heap: probe.heap(),
|
||||
modules: probe.modules(),
|
||||
largestModules: probe.largestModules(25),
|
||||
uptime: process.uptime(),
|
||||
execArgv: process.execArgv,
|
||||
});
|
||||
} catch (error) {
|
||||
response.writeHead(500, { "content-type": "application/json" });
|
||||
response.end(JSON.stringify({ error: String(error && error.message) }));
|
||||
return;
|
||||
}
|
||||
response.writeHead(200, { "content-type": "application/json" });
|
||||
response.end(body);
|
||||
// Several endpoints are async (the inspector ones), so every route resolves through a
|
||||
// promise rather than some returning a value and some a pending Promise object.
|
||||
const [route, query] = (request.url ?? "/").split("?");
|
||||
const resolve = async () => {
|
||||
switch (route) {
|
||||
// The raw file list is served separately: it is long, and every stage of a stress
|
||||
// run fetches the summary, so keeping it out of the default payload keeps those
|
||||
// captures small while still allowing a boot-vs-loaded diff on demand.
|
||||
case "/files":
|
||||
return probe.moduleFiles();
|
||||
case "/buffers":
|
||||
return probe.buffers ? probe.buffers() : { disabled: true };
|
||||
case "/fetches":
|
||||
return probe.fetches ? probe.fetches() : { disabled: true };
|
||||
case "/requires":
|
||||
return probe.requireGraph ? probe.requireGraph() : { disabled: true };
|
||||
case "/allocations":
|
||||
return probe.allocations ? await probe.allocations() : { disabled: true };
|
||||
case "/maps":
|
||||
return probe.maps();
|
||||
case "/snapshot":
|
||||
return await probe.snapshot(new URLSearchParams(query).get("path") ?? "/tmp/peak.heapsnapshot");
|
||||
default:
|
||||
return {
|
||||
usage: probe.usage(),
|
||||
spaces: probe.spaces(),
|
||||
heap: probe.heap(),
|
||||
modules: probe.modules(),
|
||||
largestModules: probe.largestModules(25),
|
||||
uptime: process.uptime(),
|
||||
execArgv: process.execArgv,
|
||||
};
|
||||
}
|
||||
};
|
||||
resolve()
|
||||
.then((value) => {
|
||||
response.writeHead(200, { "content-type": "application/json" });
|
||||
response.end(JSON.stringify(value));
|
||||
})
|
||||
.catch((error) => {
|
||||
response.writeHead(500, { "content-type": "application/json" });
|
||||
response.end(JSON.stringify({ error: String(error && error.message) }));
|
||||
});
|
||||
})
|
||||
// NODE_OPTIONS applies to every node process in the container, including the DB
|
||||
// migration step and any `docker exec node` the harness runs, so all of them load this
|
||||
|
||||
Reference in new issue
Block a user