From ab5b8dad431fda744994984e80974995342c3a37 Mon Sep 17 00:00:00 2001 From: Thomas Camlong Date: Mon, 10 Aug 2026 09:37:10 +0200 Subject: [PATCH] docs(findings): attribute the native bucket at peak, and record two disproven theories Closes the last unexplained part of the peak footprint. The 118.8 MiB 'other anonymous' smaps bucket is now mostly accounted for: V8 large-object space 34.5, external buffers 24.6, and ~38 MiB of V8 parser/compiler zone memory that was freed back to malloc but never returned to the kernel by musl - peak_malloced_memory goes 4.6 -> 42.2 MiB at board load while current malloced_memory stays at 1.0. Two theories tested and rejected rather than asserted: - 'Fewer loaded files means less compiler zone memory, so shipping less server JS pays twice.' Wrong. peak_malloced_memory is identical (42.1-42.5) across the preloadEntriesOnStart A/B despite 310 fewer files. The spike is a fixed cost of the board route, not a per-file one. Claim withdrawn. - Orphaned zlib streams. Next documents this leak class and the fix is absent from this build, and a peak snapshot showed 15 Gzip objects still present at idle - which read as accumulation. getActiveResourcesInfo() reports Zlib=0 both at baseline and after 300 deliberately aborted gzip requests, RSS flat. Snapshot objects are not live native handles, and nginx buffers responses so Next never sees the mid-response disconnect that orphans a stream. Adds a /handles endpoint to the probe, which is what settled the second one. --- PHASE2-FINDINGS.md | 50 +++++++++++++++++++++++++++++ scripts/benchmarks/memory-probe.cjs | 15 +++++++++ 2 files changed, 65 insertions(+) diff --git a/PHASE2-FINDINGS.md b/PHASE2-FINDINGS.md index 82e59cfc4..809a9b507 100644 --- a/PHASE2-FINDINGS.md +++ b/PHASE2-FINDINGS.md @@ -844,6 +844,56 @@ changed and the comparison is void.** Flags that cannot be used this way at all: `--optimize-for-size`, `--no-opt` and `--lite-mode` are rejected by Node with "not allowed in NODE_OPTIONS". +## Closing the last gap: what the native bucket at peak actually is + +At peak the `smaps` bucket "other anonymous" holds **118.8 MiB** and was the least understood +part of the footprint. It is now mostly accounted for: + +| component | MiB | how it was established | +| --- | --- | --- | +| V8 `large_object_space` | 34.5 | `heapTotal 188.0 − the 256 KiB pages 153.1 = 34.9`, matching the space report | +| `external` — Buffers/ArrayBuffers | 24.6 | `process.memoryUsage()` | +| freed-but-unreturned V8 zone memory | ~38 | `peak_malloced_memory` 4.6 → **42.2 MiB** at the board-loaded stage, while *current* `malloced_memory` stays at 1.0 MiB | +| unattributed native malloc | ~22 | remainder | + +The zone figure is the interesting one. V8 mallocs ~38 MiB of parser/compiler zone memory while +compiling the board's modules, frees it back to malloc, and **musl never returns it to the +kernel** — so it stays resident as anonymous memory with nothing alive in it. + +### A prediction that was wrong + +The obvious inference — loading fewer files means less compiler zone memory, so "ship less +server JS" pays twice — **does not hold**. `peak_malloced_memory` measured across the +`preloadEntriesOnStart` A/B: + +| stage | before ×2 | after ×2 | +| --- | --- | --- | +| boot idle | 4.5, 4.5 | 4.5, 4.5 | +| post-login | 4.5, 4.5 | 4.5, 4.5 | +| board loaded | 42.1, 42.3 | 42.1, 42.1 | + +Identical, despite 310 fewer files being loaded. The 38 MiB spike is a **fixed cost incurred at +board load**, not a per-file one — one large parse or compile, not the sum of many. So the +"pays twice" claim is withdrawn; `preloadEntriesOnStart` saves retained source and live heap, and +does nothing for the native bucket. + +### The zlib-orphan hypothesis: disproven here + +Next's own source documents a real leak class: a zlib stream left open by a client disconnect is +pinned by its native handle, survives GC, and permanently retains ~256 KiB of deflate state. +`releaseCompressionStream`, the fix, is **not present** in this Next build, and a peak heap +snapshot showed 16 `Zlib` and 15 `Gzip` objects — 14 still there at idle, which looked exactly +like accumulation. + +It is not. `process.getActiveResourcesInfo()` reports **Zlib=0**, both at baseline and after +**300 deliberately aborted gzip requests**, with RSS flat at 136.5 MiB. Those snapshot entries +are heap objects, not live handles — an important distinction, since a snapshot happily lists +objects with no live native resource behind them. nginx buffers the response from Next, so Next +never observes the mid-response client disconnect that creates an orphan in the first place. + +Recorded because the initial reading of the snapshot was wrong in a way that would have shipped a +fix for a leak this deployment does not have. + ## Measured but not pursued diff --git a/scripts/benchmarks/memory-probe.cjs b/scripts/benchmarks/memory-probe.cjs index ed46075a7..33faf7a27 100644 --- a/scripts/benchmarks/memory-probe.cjs +++ b/scripts/benchmarks/memory-probe.cjs @@ -202,6 +202,19 @@ globalThis.__homarrProbe.snapshot = (destination) => { }; }; +/** + * Live libuv/Node handles by kind. + * + * Native handles are invisible to both the JS heap and the allocation profiler, yet a zlib + * stream left open by a client disconnect is pinned by its handle, survives GC, and holds + * ~256 KiB of deflate state. Counting handles is the only cheap way to watch that accumulate. + */ +globalThis.__homarrProbe.handles = () => { + const counts = {}; + for (const kind of process.getActiveResourcesInfo()) counts[kind] = (counts[kind] ?? 0) + 1; + return counts; +}; + /** Address-space breakdown at the moment of capture; only /proc shows native vs JS vs code. */ globalThis.__homarrProbe.maps = () => { try { @@ -410,6 +423,8 @@ if (probePort > 0) { return probe.allocations ? await probe.allocations() : { disabled: true }; case "/maps": return probe.maps(); + case "/handles": + return probe.handles(); case "/snapshot": return await probe.snapshot(new URLSearchParams(query).get("path") ?? "/tmp/peak.heapsnapshot"); default: