docs(perf): split the findings into per-subject markdown files
PHASE2-FINDINGS.md had grown past 1,200 lines mixing server memory, client rendering, methodology and a long list of rejected experiments, which made it unusable as a reference. Split under docs/perf/: README.md headline results and the change-by-change ratio table SERVER-MEMORY.md what the container's RAM is made of, layer by layer CLIENT-PERFORMANCE.md dashboard cost in Chrome: DOM, fibers, images, paint REJECTED.md ideas implemented, measured and abandoned, with numbers METHODOLOGY.md the tools, and the measurement traps that gave wrong answers PHASE2-FINDINGS.md is now an index so existing links keep working. METHODOLOGY.md is the part worth keeping: every wrong number in this investigation came from an instrument rather than the application, and each trap is written down - snapshots cannot explain peaks, V8 truncates snapshot strings to ~1 KiB, snapshot objects are not live handles, measuring without a forced GC measures V8's growth, taking a CDP snapshot added 134 MiB to the process being measured, per-component React attribution silently returns nothing on a production build, a fixed port can measure a leftover container, and a server-side change that moves a client-side metric means the workload changed.
This commit is contained in:
6 files changed
+1120
-1215
No files matched your search
+21
-1215
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,100 @@
|
||||
# Dashboard performance in Chrome
|
||||
|
||||
Measured on the busiest board of a restored real backup: 28 items, 16 widget kinds.
|
||||
|
||||
## Where the board's cost actually is
|
||||
|
||||
`scripts/benchmarks/measure-widget-cost.mts` attributes DOM nodes and React fibers to each widget by
|
||||
walking the fiber subtree hanging off its DOM node. Totals do not tell you what to fix; this does.
|
||||
|
||||
Board totals: **9,293 DOM nodes, 8,165 event listeners, 86.3 MiB JS heap, 28 widgets.**
|
||||
|
||||
| nodes | each | fibers | each | n | widget kind |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **2,951** | 2,951 | **8,387** | 8,387 | 1 | **mediaReleases** |
|
||||
| 1,156 | 1,156 | 3,231 | 3,231 | 1 | beszelSystemStats |
|
||||
| 451 | 451 | 1,286 | 1,286 | 1 | beszelSystemTable |
|
||||
| 393 | 393 | 1,204 | 1,204 | 1 | mediaRequests-requestList |
|
||||
| 313 | 313 | 669 | 669 | 1 | beszelSystemGrid |
|
||||
| 265 | 265 | 900 | 900 | 1 | downloads |
|
||||
| 251 | 251 | 1,195 | 1,195 | 1 | calendar |
|
||||
| 177 | 177 | 390 | 390 | 1 | dockerContainers |
|
||||
| 88 | 8 | 528 | 48 | 11 | app |
|
||||
| 62 | 21 | 297 | 99 | 3 | umami |
|
||||
|
||||
**One widget is 44% of the board's DOM.** `mediaReleases` contains **1,935 Mantine `Box`
|
||||
components** on its own.
|
||||
|
||||
## Why: it renders 13× more than the tile can show
|
||||
|
||||
Measured directly on that widget:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| tile height | **539 px** |
|
||||
| content height | **7,206 px** |
|
||||
| cards rendered | **80** |
|
||||
| cards visible | **7** |
|
||||
| nodes per card | 34 |
|
||||
| `<img>` elements | 80 |
|
||||
| `<svg>` / `<path>` | 160 / 560 |
|
||||
|
||||
The component is `releases.map(...)` with **no cap** — every release the integrations return becomes
|
||||
a card. So 73 of 80 cards are entirely off-screen, costing ~2,480 DOM nodes, ~7,600 fibers, and a
|
||||
poster plus a CSS backdrop image each. Decoded bitmaps are far larger in memory than their files, and
|
||||
this is the expensive part.
|
||||
|
||||
### The fix
|
||||
|
||||
`content-visibility: auto` with `contain-intrinsic-size` on each card, plus `loading="lazy"` on the
|
||||
poster. The browser then skips layout, paint, raster **and background-image fetch** for off-screen
|
||||
cards. Nothing leaves the DOM, so scrolling and in-page search are unchanged; the intrinsic size stops
|
||||
the scrollbar jumping.
|
||||
|
||||
Three repeats per side, identical restored data:
|
||||
|
||||
| metric | before | after | |
|
||||
| --- | --- | --- | --- |
|
||||
| image requests | 135, 135, 135 | **55, 55, 56** | **−59%** |
|
||||
| image bytes (declared) | 12.27 MiB | **8.41 MiB** | −31% |
|
||||
| layout objects | 8346, 7815, 7815 | **5117, 4696, 4636** | **−41%** |
|
||||
|
||||
**Not established, stated because a single run suggested otherwise:** JS heap did not clearly improve
|
||||
— 46–88 MiB before against 74–86 MiB after, a 2× spread on the before side. This is expected, since
|
||||
decoded image memory lives **outside** `JSHeapUsedSize`, so the saving from 81 avoided image fetches
|
||||
is invisible to that metric. Task duration and time-to-settled overlap and support no conclusion.
|
||||
|
||||
## Idle churn is widget-driven, and legitimate
|
||||
|
||||
An idle board commits **73–96 times per minute**, which earlier rounds attributed to the App Router
|
||||
re-rendering once a second. That was wrong — see the correction in
|
||||
[REJECTED.md](./REJECTED.md#coalescing-beszel-subscription-updates--measured-exactly-zero) and
|
||||
[METHODOLOGY.md](./METHODOLOGY.md).
|
||||
|
||||
| board | items | idle commits/min |
|
||||
| --- | --- | --- |
|
||||
| `hey` | 0 | **0.0** |
|
||||
| `footbol` | 3 | **0.0** |
|
||||
| `default` | 28 | **76.0** |
|
||||
|
||||
A 3-item board commits zero times per minute. Attributed by DOM mutation
|
||||
(`measure-widget-churn.mts`), three Beszel widgets account for **82%** of it — but the real rate is
|
||||
**1.22 React commits per second** for the whole board at **13.3 DOM mutations per commit**. That is
|
||||
three live-monitoring widgets refreshing about once a second, which is what they are for. Not a
|
||||
defect.
|
||||
|
||||
## Earlier client wins
|
||||
|
||||
| change | effect |
|
||||
| --- | --- |
|
||||
| one shared board context menu instead of one per item | **−23% listeners, −31% fibers** |
|
||||
| `useTimeAgo` comparing before `setState` | correctness fix; idle commits unchanged, recorded as *not* a measured win |
|
||||
|
||||
## What remains on the client
|
||||
|
||||
- **`beszelSystemStats` at 1,156 nodes / 3,231 fibers** is the next largest single widget. Its cost is
|
||||
chart SVG (`<g>×317`, `<path>×98`), which is harder to reduce without changing what it draws.
|
||||
- **`documents` reads 18–32** depending on the run. Not yet investigated; each document carries its
|
||||
own overhead.
|
||||
- **The same map-everything pattern** appears in `mediaRequests-requestList` (393 nodes),
|
||||
`beszelSystemTable` (451) and `downloads` (265). Smaller instances of the widget fixed above.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Methodology, and the traps that produced wrong answers
|
||||
|
||||
Every wrong number in this investigation came from an instrument, not from the application. They are
|
||||
recorded here because each one looked plausible at the time, and several survived long enough to be
|
||||
published before being caught.
|
||||
|
||||
## The harness
|
||||
|
||||
`scripts/benchmarks/stress-restore.mts` boots a fresh container, restores a supplied backup
|
||||
**through the onboarding restore UI** (including typing "I understand"), signs in, opens boards
|
||||
across several tabs, clicks into widgets, reloads, then soaks idle — sampling memory, CPU, browser
|
||||
heap, DOM nodes and listeners at each stage.
|
||||
|
||||
Each run proves it is actually behind a login rather than reading a public page: it asserts a
|
||||
cookie-less request to the private board is **denied**, requires the Auth.js session cookie after
|
||||
sign-in, and reports whether it modified the restored data at all.
|
||||
|
||||
The backup is passed by path via `STRESS_BACKUP_ZIP` and never enters the repository — it carries
|
||||
real integration secrets. `.gitignore` also refuses `homarr-backup*.zip` and loose `*.sqlite`.
|
||||
|
||||
## Tools
|
||||
|
||||
| tool | answers |
|
||||
| --- | --- |
|
||||
| `attribute-memory.mts` | cgroup → processes → address-space mappings → V8 spaces → loaded code, each layer reconciled against the one above |
|
||||
| `memory-probe.cjs` | injected `--require` preload: V8 spaces, module cache, buffer/fetch allocation sites, require graph, smaps, handles. Diagnostics only, never shipped |
|
||||
| `analyze-heap-dominators.mjs` | real retained sizes via a dominator tree, and expansion of the largest retainer |
|
||||
| `analyze-allocations.mjs` | allocation volume by site, diffable between two stages |
|
||||
| `analyze-smaps-stages.mjs` | RSS split into JS heap, JIT, native and file-backed, per stage |
|
||||
| `analyze-require-graph.mjs` | what pulled each module in |
|
||||
| `compare-stress-runs.mjs` | alternated runs, medians **and per-run spread** |
|
||||
| `measure-widget-cost.mts` | per-widget DOM nodes, fibers, heaviest components |
|
||||
| `measure-board-paint.mts` | image requests, bytes, style/layout/task time, layout objects |
|
||||
| `measure-widget-churn.mts` | idle churn attributed to a widget via DOM mutations |
|
||||
|
||||
## Metric choice
|
||||
|
||||
**cgroup `anon`** is the server-memory figure. It is non-reclaimable and is what a container limit
|
||||
acts on.
|
||||
|
||||
**`memory.current` is not used for comparisons.** It includes page cache and swung **192–261 MiB
|
||||
for the same image**, and 60+ MiB between runs of a byte-identical build.
|
||||
|
||||
**`heapUsed` is not server memory.** It is one component. An early version of this work reported a
|
||||
`heapUsed` improvement as though it were the whole result.
|
||||
|
||||
## The traps
|
||||
|
||||
**A heap snapshot cannot explain a peak.** V8 runs a full GC before serialising one, so it reports
|
||||
survivors only. The 94 MiB of transient ArrayBuffers that drove this container past 500 MB never
|
||||
appeared in a single snapshot. Peaks need an allocation profiler and wrapped allocators.
|
||||
|
||||
**V8 truncates snapshot string contents to ~1 KiB.** Grouping snapshot strings by content therefore
|
||||
compares only each string's first kilobyte. This produced a published claim of "7.86 MiB of literal
|
||||
duplicate chunk sources"; hashing the real files found **zero** byte-identical duplicates among all
|
||||
3,055 server JS files.
|
||||
|
||||
**Snapshot objects are not live native handles.** A snapshot showed 15 `Gzip` objects still present
|
||||
at idle, which read as leaked zlib streams pinned by native handles — a real leak class that Next
|
||||
documents. `process.getActiveResourcesInfo()` reported **Zlib=0**, before and after 300 deliberately
|
||||
aborted gzip requests.
|
||||
|
||||
**Measuring memory without forcing a GC measures V8's growth, not the module's cost.** A
|
||||
require-cost sweep reported `drizzle-orm` at 124.3 MiB; with `global.gc()` before each reading the
|
||||
same measurement was a fraction of that. Two database drivers measured 21.4 MiB, not the ~30 MiB the
|
||||
naive sum implied.
|
||||
|
||||
**The act of measuring can dominate the measurement.** Taking a heap snapshot over CDP delivers it
|
||||
as JS string chunks: for a ~135 MB snapshot that added **~134 MiB of ArrayBuffers to the process
|
||||
being measured**, and the next stage read 693.7 MiB. `v8.writeHeapSnapshot` streams from C++
|
||||
instead, and the probe now reports its own `observerCostMiB`.
|
||||
|
||||
**Per-component React attribution requires a profiling build.** `fiber.actualDuration` is only
|
||||
populated by one, so on a production build every fiber fails the "did it re-render" test and the
|
||||
attribution tables come back **empty** — which reads as "nothing re-rendered" rather than "this
|
||||
build cannot tell you". This produced a published claim that the App Router re-rendered once a
|
||||
second. The tool now says which case it is.
|
||||
|
||||
**When a server-side change moves a client-side metric, the workload changed.** `--jitless` looked
|
||||
like −24% peak RSS, −14% CPU and 30% *faster* board loads. Three of those are impossible for a
|
||||
server flag. The client had **80% fewer event listeners**: the page had simply under-rendered.
|
||||
|
||||
**A fixed port can silently measure the wrong container.** A readiness check passed against a
|
||||
leftover container on the same port while the container under test had not started. Ephemeral ports
|
||||
and asserting the container's own identity are the fix.
|
||||
|
||||
**A completed build is not a built image.** A `docker builder prune` running concurrently killed a
|
||||
BuildKit session; the build failed and the next step ran against a stale tag. Verify the tag exists.
|
||||
|
||||
**Boards differ, so measure the right one.** Peak memory is at `nav-01`, not the multi-tab stage — a
|
||||
GC lands during multi-tab and pulls RSS down. Measuring the obvious stage understates peak by
|
||||
37 MiB.
|
||||
|
||||
## Reproducing
|
||||
|
||||
```bash
|
||||
STRESS_IMAGE=<image> \
|
||||
STRESS_BACKUP_ZIP=~/Downloads/<backup>.zip \
|
||||
STRESS_BOARDS="default,defaultv2" STRESS_TABS_PER_BOARD=2 \
|
||||
STRESS_NAV_ITERATIONS=3 STRESS_SOAK_MS=120000 \
|
||||
node --experimental-strip-types scripts/benchmarks/stress-restore.mts
|
||||
|
||||
node scripts/benchmarks/compare-stress-runs.mjs .bench/stress <before-prefix> <after-prefix>
|
||||
```
|
||||
|
||||
For attribution, add the probe:
|
||||
|
||||
```bash
|
||||
STRESS_MOUNT="$PWD/scripts/benchmarks:/probe:ro" \
|
||||
STRESS_ENV="HOMARR_PROBE_PORT=9333,HOMARR_PROBE_TRACK_BUFFERS=32768" \
|
||||
STRESS_NODE_OPTIONS="--require /probe/memory-probe.cjs"
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
# Homarr performance investigation
|
||||
|
||||
Measured work on Homarr's memory and rendering cost, split by subject. Every figure here comes from
|
||||
a run of `scripts/benchmarks/stress-restore.mts` against a **restored real backup** — 4 boards, 51
|
||||
widgets, 11 integrations, driven through an actual login — not from a synthetic or empty container.
|
||||
|
||||
| document | subject |
|
||||
| --- | --- |
|
||||
| [SERVER-MEMORY.md](./SERVER-MEMORY.md) | What the container's RAM is made of, and the changes that reduced it |
|
||||
| [CLIENT-PERFORMANCE.md](./CLIENT-PERFORMANCE.md) | Dashboard rendering cost in Chrome: DOM, fibers, images, paint |
|
||||
| [REJECTED.md](./REJECTED.md) | Ideas that were implemented and measured, then abandoned — with the numbers |
|
||||
| [METHODOLOGY.md](./METHODOLOGY.md) | The tools, and the measurement traps that produced wrong answers |
|
||||
|
||||
## Headline result
|
||||
|
||||
Server RAM, merge-base `06724cfeb` vs this branch, two alternated runs per side, reported as cgroup
|
||||
**anonymous** memory because that is what counts against a container limit and cannot be reclaimed:
|
||||
|
||||
| stage | before | after | change |
|
||||
| --- | --- | --- | --- |
|
||||
| **peak** | 376.6 MiB | **279.3 MiB** | **−26%** |
|
||||
| boot idle | 130.5 MiB | **62.4 MiB** | **−52%** |
|
||||
| after sign-in | 247.4 MiB | **142.9 MiB** | **−42%** |
|
||||
| dashboard loaded | 293.4 MiB | **170.4 MiB** | **−42%** |
|
||||
| 4 tabs, 2 boards | 366.5 MiB | **258.2 MiB** | **−30%** |
|
||||
| after 2 min idle soak | 323.7 MiB | **200.8 MiB** | **−38%** |
|
||||
|
||||
No before/after range overlaps at any stage. The cost is **+18% CPU** (solid, non-overlapping).
|
||||
|
||||
## The changes that did it, by ratio
|
||||
|
||||
| change | size | effect |
|
||||
| --- | --- | --- |
|
||||
| `--max-semi-space-size=4` | **1 line** | −19% idle, −21% peak |
|
||||
| `preloadEntriesOnStart: false` | **1 line** | −52% anon at idle, 310 fewer files even at full load |
|
||||
| Umami: stop fetching 100k raw events per refresh | 1 file | −26% peak, −93% transient buffers |
|
||||
| `serverExternalPackages` (mysql2/pg + 8 more) | 1 block | −21 MiB idle |
|
||||
| ssh2's 16 MiB WASM heap (docker-modem patch) | 1 patch | −15 to −18 MiB external |
|
||||
| `content-visibility` on media-release cards | 4 lines | −59% image requests, −41% layout objects |
|
||||
|
||||
## What is left, and why it is hard
|
||||
|
||||
One lever has real headroom and it pays twice. Roughly **41 MiB of retained bundle source** and
|
||||
**~38 MiB of V8 compiler zone memory** both scale with how much JavaScript the server loads and
|
||||
compiles, and the largest single contributor is a **16.10 MiB subtree reached by every API route** —
|
||||
the tRPC router importing every integration eagerly. Reducing that reduces both.
|
||||
|
||||
Everything else has been measured and closed: see [REJECTED.md](./REJECTED.md).
|
||||
@@ -0,0 +1,128 @@
|
||||
# Ideas measured and abandoned
|
||||
|
||||
Each of these was implemented or configured, built, and measured. They are recorded with their
|
||||
numbers so nobody repeats them, and because several looked like wins until the measurement came in.
|
||||
|
||||
## `--jitless` — looked like the biggest win of the entire investigation, and was fake
|
||||
|
||||
V8's `--jitless` removes the optimising compiler and all generated machine code. Its effect on code
|
||||
memory is real and unambiguous: **code space 11.6 → 0.0 MiB**. The stress figures looked
|
||||
spectacular: peak node RSS −24%, peak container −36%, CPU −14%, median board load **30% faster**.
|
||||
|
||||
Three of those are impossible. The flag was applied to the **server**, so it cannot make the browser
|
||||
lighter, and removing an optimising compiler does not make code faster.
|
||||
|
||||
| | normal | jitless |
|
||||
| --- | --- | --- |
|
||||
| DOM nodes | 8,256 / 8,280 | **5,442 / 5,442** |
|
||||
| JS event listeners | 7,781 / 7,784 | **1,570 / 1,570** |
|
||||
| browser JS heap | 50.0 / 49.5 MiB | 28.1 / 28.2 MiB |
|
||||
|
||||
Boards still reported 28/28 widgets ready, so the coarse check passed — but the page carries **80%
|
||||
fewer event listeners**. Much of the UI never mounted. The memory "win" is an application doing less
|
||||
work, and the CPU and latency "improvements" are the same artefact.
|
||||
|
||||
## `--max-opt=2` (cap V8 at Maglev) — null result, and it closes the door
|
||||
|
||||
Since full jitless is unusable, capping the *tier* while still generating code was the obvious next
|
||||
try. `--max-opt` is rejected by `NODE_OPTIONS` and accepted on the command line.
|
||||
|
||||
| stage | full | Maglev-capped |
|
||||
| --- | --- | --- |
|
||||
| peak anon | 276.0 MiB | 282.8 MiB (+2%) |
|
||||
| boot idle | 64.0 MiB | 72.3 MiB (+13%) |
|
||||
| dashboard loaded | 186.1 MiB | 179.2 MiB (−4%) |
|
||||
| 4 tabs | 270.8 MiB | 238.5 MiB (−12%) |
|
||||
| after soak | 199.0 MiB | 214.3 MiB (+8%) |
|
||||
| CPU | 28.1 s | 27.8 s |
|
||||
|
||||
Render integrity was verified explicitly this time: listeners **7,784 in all four runs**, DOM nodes
|
||||
within 40. So unlike jitless the page rendered fully and the comparison is valid — there is simply no
|
||||
consistent win.
|
||||
|
||||
The mechanism closes it permanently. Measured under `--max-opt=2`, `peak_malloced_memory` at board
|
||||
load is **42.1 MiB, identical to Turbofan**. Maglev's zones are just as large. Capping the tier does
|
||||
not reduce zone memory; only not compiling does, and that breaks the page.
|
||||
|
||||
## jemalloc — the hypothesis was backwards
|
||||
|
||||
~38 MiB of V8 compiler zone memory is freed back to malloc and never returned to the kernel by musl,
|
||||
so an allocator that decays unused spans back to the OS looked like the answer. Tested on the
|
||||
**byte-identical image** with only `HOMARR_MALLOC` differing, asserting the "Using jemalloc" log
|
||||
line.
|
||||
|
||||
| stage | musl | jemalloc |
|
||||
| --- | --- | --- |
|
||||
| boot idle | 62.4 MiB | **80.3 MiB (+29%)** |
|
||||
| after sign-in | 144.9 MiB | **170.6 MiB (+18%)** |
|
||||
| dashboard loaded | 214.8 MiB | 184.4 MiB (−14%) |
|
||||
| 4 tabs | 259.9 MiB | **319.6 MiB (+23%)** |
|
||||
| after soak | 215.5 MiB | **279.6 MiB (+30%)** |
|
||||
| **CPU** | 28.0 s | **24.5 s (−13%)** |
|
||||
|
||||
jemalloc does not release freed spans sooner; it deliberately retains more of them (thread-local
|
||||
caches, time-based dirty-page decay) and spends that memory on speed. **musl is already the frugal
|
||||
choice**, so "swap the allocator" is not a RAM direction. If CPU ever becomes the binding
|
||||
constraint, this is a measured 13% for two lines.
|
||||
|
||||
## Lazy-loading mysql2 and pg — 21.4 MiB, three broken attempts
|
||||
|
||||
Both load on every SQLite install, because `db/drivers/index.ts` imports all three driver wrappers
|
||||
statically and picks one at call time. The prize is 21.4 MiB. Three attempts, each caught only by
|
||||
booting the real image:
|
||||
|
||||
1. `createRequire(import.meta.url)` is `undefined` in the CommonJS `migrate.cjs` bundle. It threw at
|
||||
load and broke startup **for every dialect, including SQLite**.
|
||||
2. With that fixed, MySQL failed with `MODULE_NOT_FOUND` on `drizzle-orm/mysql2/index.cjs` — a
|
||||
dynamic require resolves through the package's `require` condition to a `.cjs` build that Next's
|
||||
tracing never copies into the standalone output.
|
||||
3. Leaving drizzle's dialect static defeats the purpose: `drizzle-orm/mysql2` imports `mysql2`
|
||||
itself, so the client loads anyway. Verified — the built image still loaded all 77 mysql2 files.
|
||||
|
||||
The remaining routes are worse than the problem: forcing drizzle's `.cjs` into the image loads a
|
||||
**second copy of drizzle's core** beside the ESM one (straddled class identities), and the
|
||||
`require(esm)` route works only via a hardcoded absolute path, so a drizzle upgrade would break
|
||||
*startup* on two dialects. Structural blocker: `createDb` must stay synchronous because
|
||||
`packages/db/index.ts` does `export const db = createDb(schema)` at module scope and `migrate.cjs` is
|
||||
CommonJS.
|
||||
|
||||
## Coalescing Beszel subscription updates — measured exactly zero
|
||||
|
||||
Three Beszel widgets account for 82% of idle DOM churn, so batching their per-record `setState` calls
|
||||
looked like an easy win. Implemented, built, measured:
|
||||
|
||||
| | before | after |
|
||||
| --- | --- | --- |
|
||||
| DOM mutations / 45 s | 733 | 748 |
|
||||
| React commits / 45 s | **55** | **55** |
|
||||
|
||||
Identical. Beszel emits roughly one record per second and the flush interval was also one second, so
|
||||
each flush batched exactly one record — arithmetic that should have come before the implementation.
|
||||
|
||||
It also exposed a unit error: **DOM mutations are not re-renders**. The real rates are **1.22 React
|
||||
commits per second** for the whole board and **13.3 DOM mutations per commit**. So the stats widget's
|
||||
"10.4 mutations/second" is about 0.8 *renders* per second: three live-monitoring widgets each
|
||||
refreshing about once a second, which is what they exist to do. **73 commits/min is the feature
|
||||
working, not a defect.**
|
||||
|
||||
## Next's image optimiser — 18.5 MiB shipped, 0 MiB of RAM
|
||||
|
||||
sharp plus `@img` is 18.5 MiB in the image and costs **+9.1 MiB RSS** once `/_next/image` is hit. But
|
||||
`shouldUseNextImage={true}` appears nowhere, so the logo always renders a plain `<img>`, and across
|
||||
three full 4-tab stress runs sharp **never loads**. `images: { unoptimized: true }` would save no RAM
|
||||
at all. The 9.1 MiB is conditional on visiting the About or Kubernetes pages; the 18.5 MiB is an
|
||||
image-size observation, not a memory one.
|
||||
|
||||
## Smaller null results
|
||||
|
||||
| idea | measured | verdict |
|
||||
| --- | --- | --- |
|
||||
| base64 images in cache / TanStack Query / Redis | **0.01 MiB** (2 data-URLs), Redis totals 2.17 MB, `image-proxy` stores URL + headers only | the theory that started this — not a factor |
|
||||
| orphaned zlib streams | `Zlib=0` handles before and after 300 aborted gzip requests | Next's leak class does not occur here; nginx buffers the response so Next never sees the disconnect |
|
||||
| V8 native contexts (2 → 10 under load) | **0.25 MiB each**, ~2.0 MiB total | too small to chase; sized before investigating |
|
||||
| Redis 6 → 2 client consolidation | 139.3 vs 139.2 MiB idle | inside noise, reverted. The `pubSub:last:*` TTL was kept as a real unbounded-growth fix |
|
||||
| V8 old-space cap | removing it entirely matched the capped build at every stage | a drift ceiling, not a reduction. Kept as one line |
|
||||
| `--max-semi-space-size=2` (vs 4) | no improvement | 4 is the sweet spot |
|
||||
| `--optimize-for-size`, `--no-opt`, `--lite-mode` | rejected by Node in `NODE_OPTIONS` | not available as a runtime knob |
|
||||
| in-image Redis / nginx removal | 8.1 / 9.2 MiB RSS (3.1 PSS for nginx) | real but small, and both are load-bearing |
|
||||
| ASCII-escaping server bundles | **9.45 MiB** — one char above U+00FF forces a whole source string to 2 bytes/char, proven at 2.00 vs 1.00 bytes/char | genuine, but it means rewriting minified **vendor** output at build time; deserves its own PR |
|
||||
@@ -0,0 +1,711 @@
|
||||
# Server memory: what the container's RAM is made of
|
||||
|
||||
Measured with `scripts/benchmarks/stress-restore.mts` on a restored real backup. See
|
||||
[METHODOLOGY.md](./METHODOLOGY.md) for the tools and the measurement traps, and
|
||||
[REJECTED.md](./REJECTED.md) for everything that was tried and abandoned.
|
||||
|
||||
## Metric choice
|
||||
|
||||
`memory.current` moved between **192 and 261 MiB for the same baseline image** across runs,
|
||||
because cgroup v2 counts reclaimable page cache. At idle that split was 144 MiB anon vs
|
||||
77 MiB page cache. Everything below therefore reports **anonymous memory**, which was
|
||||
stable to ±2.5 MiB across repeated runs of the same image.
|
||||
|
||||
## Attribution — this is the part that matters
|
||||
|
||||
Built cumulatively from the baseline image, one change at a time:
|
||||
|
||||
| config | boot idle anon | verdict |
|
||||
| --- | --- | --- |
|
||||
| baseline | 139.2 | — |
|
||||
| + integrations barrel diet | 139.5 | **no idle effect** |
|
||||
| + Redis clients 6 → 2 | 139.3 | **no idle effect** |
|
||||
| + `serverExternalPackages: mysql2, pg` | **118.6** | **−21 MiB — the entire idle win** |
|
||||
| branch (everything) | 115.2 / 99.0 | no further gain beyond the above |
|
||||
|
||||
| change | diff size | idle | under load |
|
||||
| --- | --- | --- | --- |
|
||||
| `serverExternalPackages` + `mysql2`/`pg` | **1 line** | **−21 MiB** | −25 MiB |
|
||||
| integrations barrel diet | ~10 files, +119 lines | ~0 | −40 MiB (±25 noise) |
|
||||
| Redis clients 6 → 2 | 4 files | ~0 | ~0 |
|
||||
| V8 heap cap (`HOMARR_MAX_OLD_SPACE_SIZE=512`) | 1 line | ~0 | ~0 |
|
||||
| cache invalidation + `next-cache-handler` | ~30 files, 66 call sites, ~290 lines | **0** | **0** |
|
||||
|
||||
### Why one config line beats the whole module-graph refactor
|
||||
|
||||
`packages/core/src/infrastructure/db/drivers/index.ts` statically imports all three
|
||||
drivers, so `mysql2` and `pg` are in the graph even when `DB_DRIVER=better-sqlite3`.
|
||||
Bundled, their full transitive graphs are inlined into the boot-loaded server chunks;
|
||||
marked external they become a `require()` that the sqlite path never reaches. Verified
|
||||
inside the images:
|
||||
|
||||
| image | server chunks | `mysql2` in `node_modules` |
|
||||
| --- | --- | --- |
|
||||
| baseline | 108,336 KB | no — it was inlined into the chunks |
|
||||
| + external pkgs | 94,936 KB | yes — traced, loaded only on demand |
|
||||
| branch | 98,084 KB | yes |
|
||||
|
||||
This is fix-catalog **F6 ("load only the configured DB driver")** achieved by one config
|
||||
line, after the dynamic-`import()` attempt was reverted for breaking Vitest.
|
||||
|
||||
### The V8 heap cap is a safety ceiling, not a reduction
|
||||
|
||||
`branch-cap4096` (cap effectively removed) matched `branch` at every stage — idle 99.0 vs
|
||||
115.2, board loaded 230.2 vs 228.1, after stress 362.6 vs 358.8. The workload never
|
||||
approaches a 512 MB old space, so the cap cannot reduce steady-state usage. It is still
|
||||
worth keeping as one line: baseline's soak hit a 543.8 MiB anon sample, and capped
|
||||
baseline topped out at 347.5 MiB, which is exactly the unbounded-drift protection
|
||||
issue #5301 needs. It should not be described as a memory saving.
|
||||
|
||||
### The cache layer is provably dead code
|
||||
|
||||
On the current branch state there are **0** `"use cache"` directives, **0** `cacheTag()`,
|
||||
**0** `cacheLife()`, **0** `unstable_cache`, and no tagged `fetch` in application source —
|
||||
and `cacheHandlers`/`cacheComponents` have both been removed from `next.config.ts`.
|
||||
`packages/next-cache-handler` is referenced only by a `package.json` dependency line;
|
||||
nothing imports it. The 66 remaining `invalidate*` call sites call `revalidateTag` on tags
|
||||
nothing ever registers, with no handler to receive them, inside a `try/catch` that
|
||||
swallows the resulting out-of-scope error. Zero functional effect, ~30 files of churn.
|
||||
|
||||
## What is actually hogging the memory
|
||||
|
||||
Tested directly against the hypothesis that cached base64 images were bloating
|
||||
TanStack Query / Redis. **They are not** — measured on a loaded 4-tab session:
|
||||
|
||||
| suspect | measured | verdict |
|
||||
| --- | --- | --- |
|
||||
| base64 data-URLs in server heap | **0.01 MiB** (2 strings) | not a factor |
|
||||
| base64 blobs in browser heap | **0.01 MiB** (7 strings) | not a factor |
|
||||
| Redis total | **2.17 MB** | not a factor |
|
||||
| image bytes in Redis | **none** — `image-proxy` stores only URL + headers | not a factor |
|
||||
| React Query `gcTime` | already 5 min, not 24 h | already addressed |
|
||||
|
||||
What the server heap is actually made of (141 MiB used, 259 MiB RSS):
|
||||
|
||||
| bucket | size | share of heap |
|
||||
| --- | --- | --- |
|
||||
| **bundled JS source retained as strings** | **41.4 MiB** | **~29%** |
|
||||
| other strings (translations, ids, misc) | 10.3 MiB | 7% |
|
||||
| URLs | 2.0 MiB | 1% |
|
||||
| everything else (objects, code, maps) | ~87 MiB | 62% |
|
||||
|
||||
**The memory is the code, not cached data.** That is why every cache-tuning idea in the
|
||||
fix catalog measured at zero, and why `serverExternalPackages` — which removes code from
|
||||
the boot-loaded bundle — was the only lever that moved idle memory.
|
||||
|
||||
> **Correction (later round).** An earlier version of this section claimed "7.86 MiB is
|
||||
> literal duplicate copies" of identical chunk sources. That figure was wrong. It came from
|
||||
> grouping heap-snapshot strings by content, but **V8 truncates string contents to ~1 KiB
|
||||
> when serialising a snapshot**, so the comparison only ever saw each string's first
|
||||
> kilobyte and reported near-identical chunks as identical. Hashing the real files found
|
||||
> **zero byte-identical duplicates among all 3,055 `.next/server/**/*.js` files (77.6 MiB)**.
|
||||
> The chunks that looked like copies are distinct files that happen to begin with the same
|
||||
> vendored module. The real doubling has a different cause — see "Why the source costs
|
||||
> twice what it should" below.
|
||||
|
||||
`@ioredis/commands` is now external, which removes it from all 10 chunks. Recorded as
|
||||
**bundle hygiene, not a memory win**: idle went 89.7 → 89.4 MiB, inside noise, and
|
||||
board-loaded/peak differences were smaller than the run-to-run spread (one "before" run
|
||||
read 220.6 MiB against 169–173 for identical code).
|
||||
|
||||
`got` and `@octokit/request-error` were deliberately **not** externalised. They are only
|
||||
transitive dependencies here, and a check inside the image confirmed `got` is **not
|
||||
traced** into `node_modules` — externalising it would have turned a bundled module into a
|
||||
runtime `require` that resolves to nothing, trading zero measured gain for a possible
|
||||
`MODULE_NOT_FOUND` on a code path the benchmarks never hit.
|
||||
|
||||
### Where the remaining headroom is
|
||||
|
||||
Nothing further is available from caches, Redis, or process trimming. The two real levers
|
||||
left are both large:
|
||||
|
||||
1. **Ship less server JavaScript.** ~29% of the heap is retained bundle source. Every
|
||||
package moved out of the boot graph pays twice — smaller chunks and less retained
|
||||
source. This is the same lever that produced the −37% idle result.
|
||||
2. **Stop the App Router re-rendering every second** (see above). That governs the
|
||||
browser side, which is where the 650 MB peak lives.
|
||||
|
||||
## Full attribution — every megabyte, top to bottom
|
||||
|
||||
Everything above measured *totals*. This section measures *composition*: each layer is
|
||||
reconciled against the one above it, so nothing hides in a rounding gap. Tooling:
|
||||
`scripts/benchmarks/attribute-memory.mts` (cgroup → processes → mappings → V8 spaces →
|
||||
loaded code) with `scripts/benchmarks/memory-probe.cjs` injected as a `--require` preload.
|
||||
|
||||
The probe is diagnostics-only and is never in a shipped image: the harness mounts
|
||||
`scripts/benchmarks` at `/probe` and sets `NODE_OPTIONS=--require /probe/memory-probe.cjs`.
|
||||
|
||||
### Layer 1 — the container is one process
|
||||
|
||||
| process | RSS at steady state |
|
||||
| --- | --- |
|
||||
| **next-server** | **188.1 MiB (95%)** |
|
||||
| redis-server | 4.8 MiB |
|
||||
| nginx (2 procs) | 3.7 MiB |
|
||||
| shells | 1.3 MiB |
|
||||
|
||||
Redis and nginx are rounding errors. Every question about Homarr's memory is a question
|
||||
about one Node process.
|
||||
|
||||
### Layer 2 — where that process's RSS lives
|
||||
|
||||
Measured from `/proc/<pid>/smaps`. Anonymous mappings cannot be told apart by size, so they
|
||||
are classified by signature: V8 allocates its heap in **256 KiB pages** (`kRegularPageSize`),
|
||||
and pthread reserves ~8 MiB per thread stack.
|
||||
|
||||
At boot idle (139 MiB RSS):
|
||||
|
||||
| region | RSS | what it is |
|
||||
| --- | --- | --- |
|
||||
| **node binary (code)** | **54.3 MiB** | the Node runtime's own text pages — **file-backed and shared**, so it counts in RSS but *not* in cgroup `anon` |
|
||||
| V8 heap pages (256 KiB each) | 47.8 MiB | the JS heap |
|
||||
| native malloc / small V8 spaces | 28.2 MiB | musl arenas, Buffers |
|
||||
| V8 code range / trusted space | 4.8 MiB | |
|
||||
| JIT code (anonymous executable) | 1.8 MiB | machine code V8 compiled |
|
||||
| better_sqlite3.node | 2.0 MiB | the only native addon that matters |
|
||||
| thread stacks | 0.1 MiB resident | 48 MiB *reserved*, essentially untouched |
|
||||
|
||||
**This explains the single most confusing number in the whole exercise:** RSS 139 MiB but
|
||||
cgroup `anon` only 89 MiB. The 54 MiB gap is the Node binary itself. It is reclaimable page
|
||||
cache shared with every other Node process on the host, so it is real RSS but not a real
|
||||
cost — which is exactly why `anon` is the metric this PR reports.
|
||||
|
||||
Address space reserved but not committed: **16.8 GiB**. V8 claims its cage up front. It is
|
||||
not memory.
|
||||
|
||||
### Layer 3 — inside the JS heap
|
||||
|
||||
| V8 space | boot idle | after real use |
|
||||
| --- | --- | --- |
|
||||
| old_space | 42.0 MiB | **97.2 MiB** |
|
||||
| large_object_space | 22.7 MiB | 27.3 MiB |
|
||||
| code_space + code_large | 1.9 MiB | 10.9 MiB |
|
||||
| trusted_space | 3.9 MiB | 8.1 MiB |
|
||||
| new_space | 0.0 MiB | 0.2 MiB |
|
||||
| external (Buffers) | 4.1 MiB | 22.0 MiB |
|
||||
|
||||
### Layer 4 — what those objects *are*
|
||||
|
||||
From a post-GC heap snapshot at steady state (143.8 MiB heapUsed, 1.39M nodes):
|
||||
|
||||
| bucket | size | share of heap |
|
||||
| --- | --- | --- |
|
||||
| **bundled JS source retained as strings** | **41.0 MiB** | **28%** |
|
||||
| ArrayBuffer backing stores | 17.6 MiB | 12% |
|
||||
| object property/element arrays | 16.4 MiB | 11% |
|
||||
| other strings (translations, ids) | 10.3 MiB | 7% |
|
||||
| closures | 5.7 MiB | 4% |
|
||||
| plain objects | 4.5 MiB | 3% |
|
||||
| shapes/maps/descriptors | 6.6 MiB | 5% |
|
||||
| compiled code metadata | ~12 MiB | 8% |
|
||||
|
||||
Confirmed again, with a dominator-tree analysis this time
|
||||
(`scripts/benchmarks/analyze-heap-dominators.mjs`, real retained sizes rather than shallow
|
||||
histograms): the largest single retainers are `code:` nodes named after chunk files, each
|
||||
holding **0.5–1.4 MiB**, i.e. the Script objects keeping their own source alive. V8 retains
|
||||
a script's source string for as long as any function in it might still be lazily compiled.
|
||||
|
||||
**The memory is the code, not cached data.**
|
||||
|
||||
### Why the source costs twice what it should
|
||||
|
||||
`.next/server` chunks loaded at boot are 21.7 MiB on disk, but the heap holds 32.4 MiB of
|
||||
source strings. The gap is not duplication — it is character width.
|
||||
|
||||
A V8 string is either one byte per character (Latin-1) or two. **One character above U+00FF
|
||||
anywhere in a file forces the entire source string to two bytes.**
|
||||
|
||||
| loaded JS | files | on disk | heap cost |
|
||||
| --- | --- | --- | --- |
|
||||
| pure Latin-1 | 1,025 | 12.25 MiB | 12.25 MiB |
|
||||
| contains ≥1 char > U+00FF | **80** | 9.45 MiB | **18.91 MiB** |
|
||||
|
||||
Predicted total 31.16 MiB against 32.43 MiB measured — within 4%.
|
||||
|
||||
Proven directly rather than inferred, by reading one chunk as shipped and again with its
|
||||
non-Latin-1 characters replaced (identical length):
|
||||
|
||||
```
|
||||
_049nb9n._.js 538,519 chars, 2 of them non-Latin-1
|
||||
as shipped : 1053 KiB per copy -> 2.00 bytes/char
|
||||
ascii variant : 526 KiB per copy -> 1.00 bytes/char
|
||||
```
|
||||
|
||||
**Two characters cost 527 KiB.** The offenders are almost all trivia in dependencies:
|
||||
|
||||
| chunk | non-ASCII chars | the actual characters |
|
||||
| --- | --- | --- |
|
||||
| `_049nb9n._.js` (+2 siblings) | 2 | `process.versions.icu?"✅":"Y "` |
|
||||
| `[root-of-the-server]__1ipwvlq._.js` | 13,141 | Next's log symbols `○ ⨯ ⚠` |
|
||||
| `_1ew9xg8._.js` (+ssr twin) | 70,175 | a UTF-8 self-test containing `💩` |
|
||||
| zod chunks ×4 | 13,137 | `✖ ${i.message}` |
|
||||
| `app-page-turbo.runtime.prod.js` | 13 | an ellipsis `…` |
|
||||
| `[root-of-the-server]__1xyzm2f._.js` | 13 | **an em-dash in Homarr's own MCP instructions** |
|
||||
|
||||
Total recoverable: **~9.45 MiB of heap** (≈10% of idle `anon`), by emitting server bundles
|
||||
with `\uXXXX` escapes instead of raw characters. Not implemented here — it means rewriting
|
||||
minified vendor output at build time, and the safety argument needs to be made properly
|
||||
before that ships. Recorded as a measured, mechanistically understood opportunity.
|
||||
|
||||
## The peak: one integration was allocating 46 MiB per refresh
|
||||
|
||||
The steady state above is ~237 MiB RSS. The *peak* during a 4-tab session was 429 MiB RSS /
|
||||
539 MiB container, and that peak is a completely different phenomenon.
|
||||
|
||||
Per-stage V8 accounting (`scripts/benchmarks/analyze-probe-stages.mjs`):
|
||||
|
||||
| stage | rss | old_space | external | **arrayBuffers** |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| boot idle | 138.6 | 42.0 | 4.1 | 0.3 |
|
||||
| board loaded | 254.9 | 95.3 | 22.9 | 3.0 |
|
||||
| nav ×3 | 361.9 | 122.5 | 24.7 | 4.8 |
|
||||
| **4 tabs, 2 boards** | **429.3** | **152.6** | **114.3** | **94.3** |
|
||||
| after 90 s idle | 237.0 | 97.2 | 22.0 | **2.0** |
|
||||
|
||||
`arrayBuffers` goes **0.3 → 94.3 → 2.0**. It is entirely transient — which is precisely why
|
||||
the app appears to "oscillate between 835 MB and 400 MB" and never settles at the high mark.
|
||||
Soak growth was **−102 MiB/hour**: there is no leak.
|
||||
|
||||
Tracking large `Buffer` allocations by stack (`HOMARR_PROBE_TRACK_BUFFERS`) put **48.3 MiB
|
||||
in 16 allocations at one site, with a single allocation of 46.3 MiB**:
|
||||
|
||||
```
|
||||
T (/app/apps/nextjs/.next/server/chunks/_1q3xovl._.js:92:32374)
|
||||
process.processTicksAndRejections
|
||||
```
|
||||
|
||||
That function is undici's `fullyReadBody` — the machinery behind `Response.json()`:
|
||||
|
||||
```js
|
||||
async function T(A,e,t){ let r=[],s=0; for(;;){ let{done:i,value:o}=await A.read();
|
||||
if(i) return void e(Buffer.concat(r,s)); r.push(o), s+=o.length } }
|
||||
```
|
||||
|
||||
It holds the chunk list *and* the joined copy at the same moment, so a body of size N costs
|
||||
~2N at the instant it completes. Capturing the bytes identified the response immediately:
|
||||
|
||||
```
|
||||
46.3 MiB assembled from 3,946 socket chunks
|
||||
{"data":[{"id":"1251f6db…","websiteId":"f133f10c…","sessionId":"9e02d227…","createdAt":"2026-08-09T…
|
||||
```
|
||||
|
||||
**Umami.** And the cause was in plain source — two call sites requesting
|
||||
`pageSize: "100000"`, i.e. up to 100,000 raw event records:
|
||||
|
||||
1. `getEventNamesAsync` downloaded every raw event of the last 30 days **to collect a
|
||||
handful of distinct event name strings**.
|
||||
2. `getWebsiteEventTimeSeriesAsync` downloaded them to bucket them into a time series.
|
||||
3. `getMultiEventTimeSeriesAsync` ran (2) for **every tracked event name concurrently** via
|
||||
`Promise.all`, multiplying the resident pages by the number of events.
|
||||
|
||||
A board with 3 Umami widgets and several tracked events therefore had several ~46 MiB bodies
|
||||
in flight at once, each briefly doubled, plus ~100k parsed objects per response landing in
|
||||
old_space. That is the 500–880 MB.
|
||||
|
||||
### The fix
|
||||
|
||||
| call site | before | after |
|
||||
| --- | --- | --- |
|
||||
| `getEventNamesAsync` | list 100k raw events, collect distinct names | aggregated `/metrics?type=event`, one row per name |
|
||||
| `getWebsiteEventTimeSeriesAsync` | one page of up to 100k records | pages of 1,000, folded into buckets and released |
|
||||
| `getMultiEventTimeSeriesAsync` | `Promise.all` over all event names | sequential |
|
||||
|
||||
The paging path keeps the existing dual-shape handling: instances that return aggregated
|
||||
`{x,y}` buckets still answer in one request, and an instance that accepts `pageSize` but
|
||||
ignores `page` is detected (repeated first record id) and stopped rather than double-counted.
|
||||
Hitting the page ceiling logs a warning instead of silently undercounting.
|
||||
|
||||
Covered by `packages/integrations/src/umami/test/umami-integration.spec.ts` (6 tests, new —
|
||||
there were none).
|
||||
|
||||
### Result
|
||||
|
||||
Alternated runs, 2 before and 2 after, same harness and workload, `before` = branch HEAD
|
||||
`f707f6433`, `after` = HEAD + this change. Per-run values are given because the harness's
|
||||
own spread is what decides whether a difference means anything.
|
||||
|
||||
| metric | before | after | change | before spread / after spread |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **peak container** | 498.9 MiB | **398.5 MiB** | **−20%** | [461.6, 536.3] / [387.0, 410.0] |
|
||||
| **peak node RSS** | 452.4 MiB | **353.3 MiB** | **−22%** | [432.7, 472.2] / [355.6, 350.9] |
|
||||
| **peak during 4 tabs** | 498.5 MiB | **356.3 MiB** | **−29%** | [460.8, 536.3] / [341.8, 370.8] |
|
||||
| **peak arrayBuffers** | 95.8 MiB | **6.6 MiB** | **−93%** | [95.7, 95.8] / [6.4, 6.7] |
|
||||
| largest single allocation | 46.4 MiB | **2.4 MiB** | −95% | identical in both runs |
|
||||
| post-soak steady state | 324.2 MiB | 322.5 MiB | −1% | [337.7, 310.7] / [280.7, 364.4] |
|
||||
| boot idle | 119.4 MiB | 125.8 MiB | +5% | [130.0, 108.8] / [152.8, 98.8] |
|
||||
| median board load | 526 ms | 523 ms | −1% | [389, 663] / [401, 645] |
|
||||
| CPU over the run | 19.5 s | 20.4 s | +4% | [19.6, 19.4] / [20.3, 20.4] |
|
||||
|
||||
Every peak metric separates cleanly — the two sides' ranges do not overlap. Two rows do
|
||||
**not** support a claim and are listed so they are not read as one:
|
||||
|
||||
- **post-soak steady state is inconclusive.** One "after" run finished at 364.4 MiB, above
|
||||
a "before" run at 310.7. This change targets a transient spike, so a steady-state effect
|
||||
was not expected.
|
||||
- **boot idle is noise.** The spread is ±20 MiB on *identical* code (before: 130.0 vs
|
||||
108.8), which is wider than the apparent difference. Umami's code is not even loaded at
|
||||
boot; it arrives when a board containing the widget renders.
|
||||
|
||||
The cost is **+4% CPU**, from turning one large request into several smaller ones. Board
|
||||
load latency did not regress.
|
||||
|
||||
A prediction worth recording, because it confirms the mechanism rather than just the
|
||||
outcome: at `pageSize=1000` the largest allocation measured 0.7 MiB, and at 5,000 it
|
||||
measured 2.4 MiB — 5× the page size, 3.4× the bytes, exactly as a per-page bound implies.
|
||||
|
||||
### What the real instance revealed
|
||||
|
||||
Widget readiness is identical before and after — 28/28 on the board carrying the three
|
||||
Umami widgets, and the same single pre-existing error on the other board — so nothing broke.
|
||||
But the container log from the "after" runs contains something the "before" runs could not
|
||||
have produced:
|
||||
|
||||
```
|
||||
warn: Event time series hit the page ceiling, counts are a lower bound
|
||||
module="umami-integration" eventName="Add app" maxRecords="100000"
|
||||
```
|
||||
|
||||
That site really does have more than 100,000 events for one event name in the window. **The
|
||||
old code had exactly the same ceiling** — `pageSize=100000` is 100k records — and the 46.3 MiB
|
||||
body it received was that ceiling being hit. The difference is that it truncated in silence
|
||||
and reported the undercount as fact. The ceiling is unchanged; it is now visible.
|
||||
|
||||
This is also the worst case for the round-trip trade-off: that one event name costs 20
|
||||
sequential requests. It did not show up in board load latency (523 ms vs 526 ms median),
|
||||
because the widget fetches after the board renders, and total CPU rose 4%.
|
||||
|
||||
## Dissecting the peak, not the idle
|
||||
|
||||
Everything above measures a drained process — boot idle, or post-soak. That is the wrong
|
||||
moment: users complain about the high-water mark. Measuring it needs different instruments,
|
||||
because the obvious one cannot work.
|
||||
|
||||
**A heap snapshot can never explain a peak.** V8 runs a full GC before serialising one, so by
|
||||
construction it reports what survived. The 94 MiB of ArrayBuffers that drove this container to
|
||||
500 MB was invisible in every snapshot taken. Three instruments were added instead:
|
||||
|
||||
| instrument | what it sees | blind to |
|
||||
| --- | --- | --- |
|
||||
| `HeapProfiler.startSampling` (in-process `inspector.Session`) | every V8-heap allocation, survivor or garbage, with a stack | anything outside the V8 heap |
|
||||
| wrapped `Buffer.alloc`/`concat` | external buffer allocations, with a stack | V8-heap objects |
|
||||
| `/proc/<pid>/smaps` at every stage | JS heap vs JIT vs native vs file-backed | anything below page granularity |
|
||||
|
||||
Together they cover the whole address space. Neither of the first two alone would have found
|
||||
the Umami bug: the sampling profiler cannot see `Buffer` backing stores, which is precisely
|
||||
where those 46 MiB lived.
|
||||
|
||||
### An observer effect worth recording
|
||||
|
||||
The first attempt took the peak snapshot over CDP, which delivers it as JS string chunks. For
|
||||
a ~135 MB snapshot that added **~134 MiB of ArrayBuffers to the process being measured**, and
|
||||
the stage right after it read 693.7 MiB. That is an artefact, not an application problem —
|
||||
and it is exactly the kind of number that gets reported as a finding. Switching to
|
||||
`v8.writeHeapSnapshot`, which streams from C++, removed it. The probe now reports its own
|
||||
`observerCostMiB` so the cost of looking is visible rather than assumed to be zero.
|
||||
|
||||
### What the peak is actually made of
|
||||
|
||||
Peak here is `nav-01`, not the multi-tab stage — a GC lands during multi-tab and pulls RSS
|
||||
*down*, so "most tabs open" is not the same as "most memory". Node RSS 357.8 MiB:
|
||||
|
||||
| component | MiB | notes |
|
||||
| --- | --- | --- |
|
||||
| live JS objects (`heapUsed`) | 175.7 | of which ~41 MiB is retained bundle source |
|
||||
| node binary text | 65.7 | file-backed and shared; not in cgroup `anon` |
|
||||
| native malloc | ~59 | see below |
|
||||
| external buffers | 24.6 | |
|
||||
| V8 committed but free | 12.3 | heapTotal − heapUsed |
|
||||
| JIT code | 10.1 | grew from 1.9 MiB at boot |
|
||||
|
||||
Cross-checked rather than assumed: `heapTotal 188.0 − V8's 256 KiB pages 153.1 = 34.9 MiB`,
|
||||
which matches `large_object_space` at 34.5 MiB. So the handful of very large anonymous
|
||||
mappings (31.6, 20.7, 8.4 MiB) are V8's large-object space and external buffers — **not** the
|
||||
runaway native allocator that their shape first suggested.
|
||||
|
||||
**The retained heap at peak is the same as at rest.** A post-GC snapshot at peak has 160.7 MiB
|
||||
of shallow size against 157.7 MiB after the soak; strings 51.1 vs 50.1 MiB. Nothing
|
||||
accumulates. The peak is transient churn plus pages V8 and malloc have not returned — which is
|
||||
why memory "oscillates" and why soak growth is negative.
|
||||
|
||||
And the churn is small now: total V8-heap allocation across the entire run is **125 MiB**, 40%
|
||||
of it node's own module loader reading source files. After the Umami fix the multi-tab burst
|
||||
allocates **3.3 MiB**. There is no longer a churn problem on the JS side.
|
||||
|
||||
## The 16 MiB nobody was using: ssh2
|
||||
|
||||
The largest single retainer in every snapshot — at boot, at peak, and after the soak — was one
|
||||
`system / Context` holding **16.03 MiB**. Expanding it through the dominator tree (a plain
|
||||
graph walk escapes via `__proto__` and reports the whole heap) named it immediately:
|
||||
|
||||
```
|
||||
=== inside the largest retainer (object:system / Context, 16.03 MiB) ===
|
||||
16.00 MiB (indirect) -> object:ArrayBuffer
|
||||
0.00 MiB POLY1305_WASM_MODULE -> object:Object
|
||||
0.00 MiB ChaChaPolyCipherNative -> closure:ChaChaPolyCipherNative
|
||||
0.00 MiB AESGCMCipherNative -> closure:AESGCMCipherNative
|
||||
```
|
||||
|
||||
`ssh2`. Its crypto module instantiates a WebAssembly memory for ChaCha20-Poly1305 the moment
|
||||
it is required. Homarr has no SSH feature and does not depend on ssh2; the chain is
|
||||
`dockerode → docker-modem → ssh2`, and `docker-modem/lib/modem.js` requires `./ssh` at module
|
||||
scope while using it in exactly one place, already guarded by `protocol === 'ssh'`.
|
||||
|
||||
Measured directly inside the image — requiring ssh2 and nothing else:
|
||||
|
||||
```
|
||||
heapUsed +2.6 MiB external +0.9 MiB rss +14.0 MiB
|
||||
```
|
||||
|
||||
Fixed with a pnpm patch moving that one require inside the branch that uses it. SSH-based
|
||||
Docker hosts still work — `require` is cached, so they pay it once — and socket-based installs,
|
||||
which is everyone else, never load it.
|
||||
|
||||
### The same question asked of every dependency
|
||||
|
||||
Since one package was hiding 14 MiB behind a require, all 173 were measured the same way: each
|
||||
required in a fresh process, RSS delta recorded. Packages costing more than 2 MiB, cross-checked
|
||||
against whether the server actually loads them:
|
||||
|
||||
| package | RSS to require | loaded at runtime? |
|
||||
| --- | --- | --- |
|
||||
| drizzle-orm | 124.3 MiB | only via subpaths — the root import loads every dialect, so this figure overstates it |
|
||||
| @kubernetes/client-node | 66.9 MiB | **no** — correctly lazy |
|
||||
| dockerode | 28.8 MiB | yes (includes docker-modem 16.1 and ssh2 14.3) |
|
||||
| **mysql2** | **20.9 MiB** | **yes — 77 files, on a SQLite install** |
|
||||
| sharp | 20.4 MiB | no |
|
||||
| openid-client / jose | 16.1 / 15.5 MiB | no |
|
||||
| @grpc/grpc-js | 14.6 MiB | yes, 61 files (via dockerode) |
|
||||
| **pg** | **9.4 MiB** | **yes — 13 files, on a SQLite install** |
|
||||
|
||||
These are not additive — dockerode's figure contains ssh2's — but the ranking is what matters.
|
||||
|
||||
> **Correction.** The RSS figures in the table above were measured without forcing a GC
|
||||
> between readings, so they include V8 growing its heap opportunistically rather than the real
|
||||
> cost of the module. Re-measured with `global.gc()` before each reading, the two unused
|
||||
> database drivers cost **21.4 MiB RSS / 4.1 MiB heap**, not the ~30 MiB the naive sum implied
|
||||
> — and `drizzle-orm`'s 124.3 MiB was almost entirely this artefact. The ssh2 conclusion is
|
||||
> unaffected: it was confirmed independently by the before/after external-memory drop.
|
||||
|
||||
### mysql2 and pg load on every SQLite install — and it was not worth fixing
|
||||
|
||||
`db/drivers/index.ts` statically imports all three driver wrappers and picks one at call time,
|
||||
so importing the module executes all three. Confirmed at runtime: mysql2 (77 files) and pg
|
||||
(13 files) are resident on a SQLite install. Re-measured properly, the prize is **21.4 MiB**.
|
||||
|
||||
Three attempts, each caught by testing the real image rather than by typecheck:
|
||||
|
||||
1. `createRequire(import.meta.url)` — this module is bundled into two entry points, and
|
||||
`db/migrations/<dialect>/migrate.cjs` is **CommonJS**, where `import.meta.url` compiles to
|
||||
`undefined`. `createRequire` threw `ERR_INVALID_ARG_VALUE` at load and took down migrations,
|
||||
and therefore startup, **for every dialect including SQLite**. Fixed by basing resolution on
|
||||
`process.cwd()`.
|
||||
2. With that fixed, MySQL failed with `MODULE_NOT_FOUND` on
|
||||
`drizzle-orm/mysql2/index.cjs`. A dynamic `require` resolves through the package's `require`
|
||||
condition to a `.cjs` build, and Next's tracing only ever copied the ESM `index.js` into the
|
||||
standalone output.
|
||||
3. Leaving drizzle's dialect as a static import defeats the purpose entirely:
|
||||
`drizzle-orm/mysql2` itself does `from "mysql2"`, so the client loads anyway. Verified — the
|
||||
built image still loaded all 77 mysql2 files.
|
||||
|
||||
That leaves two routes, and both are worse than the problem. Forcing drizzle's `.cjs` into the
|
||||
image via `outputFileTracingIncludes` would load **a second copy of drizzle's core** next to
|
||||
the ESM one for MySQL users, so class identities and `is()` checks would straddle two module
|
||||
instances. Requiring the ESM file works under Node 24's `require(esm)` — but only via an
|
||||
absolute path (`/app/node_modules/drizzle-orm/mysql2/index.js`); both bare specifiers fail,
|
||||
so it would hardcode drizzle's internal file layout and a drizzle upgrade would break
|
||||
*startup* for MySQL and Postgres users.
|
||||
|
||||
**Backed out.** 21.4 MiB is not worth a startup-failure class of risk on two dialects that
|
||||
cannot be exercised in normal development. The blocker to record is structural: `createDb`
|
||||
must stay synchronous because `packages/db/index.ts` does `export const db = createDb(schema)`
|
||||
at module scope and `migrate.cjs` is CommonJS, so neither `await import()` nor top-level await
|
||||
is available. Making the database handle async would unblock it.
|
||||
|
||||
## Why 24 MiB of code is resident before the first request
|
||||
|
||||
The module cache is a flat list with no parentage, so "what pulled this in" needed the
|
||||
require graph recorded as the loader resolves it (`HOMARR_PROBE_TRACK_REQUIRES=1`, analysed
|
||||
by `scripts/benchmarks/analyze-require-graph.mjs`). At boot idle, having served only
|
||||
`/api/health/ready`: **2,004 edges over 1,121 nodes, 23.97 MiB of source**.
|
||||
|
||||
| importer | subtree |
|
||||
| --- | --- |
|
||||
| `apps/nextjs/server.js` (everything) | 23.83 MiB / 1,087 files |
|
||||
| `next/dist/server/load-components.js` | 21.75 MiB / 796 files |
|
||||
| **`[root-of-the-server]__0g06wx7._.js`** | **16.10 MiB / 460 files** |
|
||||
| next's own framework code, exclusively | 2.16 MiB / 315 files |
|
||||
|
||||
Two things follow, and the second is the one that matters:
|
||||
|
||||
1. **Next loads all 74 app route bundles at startup**, not on first request. ~~That is
|
||||
framework behaviour, not something this codebase sets.~~ **That was wrong** — see below.
|
||||
2. **One subtree of 16.10 MiB / 460 files is reached by the tRPC/API routes *and* by
|
||||
`instrumentation`** — the cron tasks and websocket server. Exclusive attribution for
|
||||
instrumentation is therefore **0 bytes**: everything it touches, an API route touches
|
||||
too. Only 1.10 MiB / 119 files is common to *every* route, so this is not one universal
|
||||
barrel — it is specifically the API surface pulling in every integration and widget
|
||||
request handler.
|
||||
|
||||
That kills the obvious fix. Deferring `instrumentation` would free nothing, because Next
|
||||
preloads `/api/[...trpc]` at boot and that route reaches the same graph. Cutting the shared
|
||||
subtree itself requires the tRPC router to stop referencing every integration eagerly — the same
|
||||
lever the integrations barrel diet started, with roughly 16 MiB of source still behind it.
|
||||
|
||||
### Correction: preloading *is* configurable, and turning it off is the largest idle win here
|
||||
|
||||
The claim above that route preloading is "framework behaviour, not something this codebase sets"
|
||||
was wrong. Next documents `experimental.preloadEntriesOnStart: false` for exactly this, and the
|
||||
guide describes it as preventing the server from preloading page modules into memory at startup.
|
||||
Found by reading Next's own memory-usage guide rather than by inference — which is the lesson.
|
||||
|
||||
Deterministic per-stage measurements, identical across both runs on each side:
|
||||
|
||||
| stage | source loaded | files | heapUsed |
|
||||
| --- | --- | --- | --- |
|
||||
| boot idle | 24.0 → **10.6 MiB** | 1114 → **706** | 70.7 → **44.8 MiB** (−37%) |
|
||||
| post-login | 27.9 → **19.3 MiB** | 1215 → **884** | 123.8 → **109.8 MiB** |
|
||||
| board loaded | 29.0 → **21.7 MiB** | 1323 → **1006** | run-to-run noise |
|
||||
| multi-tab | 29.2 → **22.9 MiB** | 1325 → **1015** | 139.7 → **127.8 MiB** |
|
||||
| post-soak | 29.2 → **22.9 MiB** | 1325 → **1015** | 142.5 → **127.4 MiB** |
|
||||
|
||||
The last two rows are the important ones: **310 files and 6.3 MiB of source stay unloaded even
|
||||
after a full multi-tab session**, so this is not merely deferred cost — routes a session never
|
||||
visits are never paid for at all. Live heap is lower at every stage.
|
||||
|
||||
Peak node RSS does not move (350.3 → 346.1, spreads overlap), for the same reason the ssh2 chop
|
||||
did not move it: V8 sizes its heap against available headroom, so a smaller live set does not
|
||||
shrink peak RSS. Median board load is +3% with overlapping spreads, so the first-hit cost this
|
||||
trades for did not show at this workload. Full E2E suite passes 8/8.
|
||||
|
||||
Also from that guide, and **not** applicable here: `webpackMemoryOptimizations`,
|
||||
`serverSourceMaps`, `enablePrerenderSourceMaps` and disabling the Webpack cache all address
|
||||
*build-time* memory, and this build uses Turbopack.
|
||||
|
||||
### `--jitless`: rejected, and the reason generalises
|
||||
|
||||
V8's `--jitless` removes the optimising compiler and all generated machine code. It applies
|
||||
through `NODE_OPTIONS` and its effect on code memory is real and unambiguous: **code space
|
||||
11.6 → 0.0 MiB**. The stress figures looked spectacular — peak node RSS −24%, peak container
|
||||
−36%, CPU −14%, median board load 30% *faster*.
|
||||
|
||||
Three of those are impossible. The flag was applied to the **server**, so it cannot make the
|
||||
browser lighter, and removing an optimising compiler does not make code faster. The client
|
||||
metrics show what actually happened:
|
||||
|
||||
| | normal | jitless |
|
||||
| --- | --- | --- |
|
||||
| DOM nodes | 8,256 / 8,280 | **5,442 / 5,442** |
|
||||
| JS event listeners | 7,781 / 7,784 | **1,570 / 1,570** |
|
||||
| browser JS heap | 50.0 / 49.5 MiB | 28.1 / 28.2 MiB |
|
||||
| widget controls available | 244 | 216 |
|
||||
|
||||
Boards still reported 28/28 widgets ready, so the coarse readiness check passed — but the page
|
||||
carries **80% fewer event listeners**. Much of the UI never mounted. The memory "win" is an app
|
||||
doing less work, and the CPU and latency "improvements" are the same artefact.
|
||||
|
||||
The generalisable tell: **when a server-side change moves a client-side metric, the workload
|
||||
changed and the comparison is void.**
|
||||
|
||||
Flags that cannot be used this way at all: `--optimize-for-size`, `--no-opt` and `--lite-mode`
|
||||
are rejected by Node with "not allowed in NODE_OPTIONS".
|
||||
|
||||
## Closing the last gap: what the native bucket at peak actually is
|
||||
|
||||
At peak the `smaps` bucket "other anonymous" holds **118.8 MiB** and was the least understood
|
||||
part of the footprint. It is now mostly accounted for:
|
||||
|
||||
| component | MiB | how it was established |
|
||||
| --- | --- | --- |
|
||||
| V8 `large_object_space` | 34.5 | `heapTotal 188.0 − the 256 KiB pages 153.1 = 34.9`, matching the space report |
|
||||
| `external` — Buffers/ArrayBuffers | 24.6 | `process.memoryUsage()` |
|
||||
| freed-but-unreturned V8 zone memory | ~38 | `peak_malloced_memory` 4.6 → **42.2 MiB** at the board-loaded stage, while *current* `malloced_memory` stays at 1.0 MiB |
|
||||
| unattributed native malloc | ~22 | remainder |
|
||||
|
||||
The zone figure is the interesting one. V8 mallocs ~38 MiB of parser/compiler zone memory while
|
||||
compiling the board's modules, frees it back to malloc, and **musl never returns it to the
|
||||
kernel** — so it stays resident as anonymous memory with nothing alive in it.
|
||||
|
||||
### A prediction that was wrong
|
||||
|
||||
The obvious inference — loading fewer files means less compiler zone memory, so "ship less
|
||||
server JS" pays twice — **does not hold**. `peak_malloced_memory` measured across the
|
||||
`preloadEntriesOnStart` A/B:
|
||||
|
||||
| stage | before ×2 | after ×2 |
|
||||
| --- | --- | --- |
|
||||
| boot idle | 4.5, 4.5 | 4.5, 4.5 |
|
||||
| post-login | 4.5, 4.5 | 4.5, 4.5 |
|
||||
| board loaded | 42.1, 42.3 | 42.1, 42.1 |
|
||||
|
||||
Identical, despite 310 fewer files being loaded. The 38 MiB spike is a **fixed cost incurred at
|
||||
board load**, not a per-file one — one large parse or compile, not the sum of many. So the
|
||||
"pays twice" claim is withdrawn; `preloadEntriesOnStart` saves retained source and live heap, and
|
||||
does nothing for the native bucket.
|
||||
|
||||
### The zlib-orphan hypothesis: disproven here
|
||||
|
||||
Next's own source documents a real leak class: a zlib stream left open by a client disconnect is
|
||||
pinned by its native handle, survives GC, and permanently retains ~256 KiB of deflate state.
|
||||
`releaseCompressionStream`, the fix, is **not present** in this Next build, and a peak heap
|
||||
snapshot showed 16 `Zlib` and 15 `Gzip` objects — 14 still there at idle, which looked exactly
|
||||
like accumulation.
|
||||
|
||||
It is not. `process.getActiveResourcesInfo()` reports **Zlib=0**, both at baseline and after
|
||||
**300 deliberately aborted gzip requests**, with RSS flat at 136.5 MiB. Those snapshot entries
|
||||
are heap objects, not live handles — an important distinction, since a snapshot happily lists
|
||||
objects with no live native resource behind them. nginx buffers the response from Next, so Next
|
||||
never observes the mid-response client disconnect that creates an orphan in the first place.
|
||||
|
||||
Recorded because the initial reading of the snapshot was wrong in a way that would have shipped a
|
||||
fix for a leak this deployment does not have.
|
||||
|
||||
## The 38 MiB of compiler zones: attributed, then found to be unaddressable
|
||||
|
||||
The largest single item in the peak breakdown that was never explained is now explained, and then
|
||||
closed off.
|
||||
|
||||
**It is V8's optimising compiler.** `peak_malloced_memory` jumps from 4.5 MiB to 42.1 MiB the
|
||||
moment a board's modules are compiled, while *current* `malloced_memory` stays at 1.0 MiB — memory
|
||||
V8 takes for compilation, frees back to malloc, and that musl never returns to the kernel. Proven
|
||||
by re-reading the `--jitless` runs, where the spike **never happens at all**:
|
||||
|
||||
| stage | normal | `--jitless` |
|
||||
| --- | --- | --- |
|
||||
| boot idle | 4.5 MiB | 4.5 MiB |
|
||||
| **board loaded** | **42.1 MiB** | **4.5 MiB** |
|
||||
| multi-tab | 42.7 MiB | 4.5 MiB |
|
||||
|
||||
`--jitless` itself is unusable (it leaves the page under-rendered — see above), but V8 can cap the
|
||||
*tier* instead of removing code generation: `--max-opt=3` is the default Turbofan, `2` stops at
|
||||
Maglev, `1` at Sparkplug. It is rejected by `NODE_OPTIONS` but accepted on the command line, so it
|
||||
was wired behind `HOMARR_MAX_OPT` and A/B'd on one image.
|
||||
|
||||
**`--max-opt=2` is a null result for memory:**
|
||||
|
||||
| stage | full | capped at Maglev |
|
||||
| --- | --- | --- |
|
||||
| peak anon | 276.0 MiB | 282.8 MiB (+2%) |
|
||||
| boot idle | 64.0 MiB | 72.3 MiB (+13%) |
|
||||
| board loaded | 186.1 MiB | 179.2 MiB (−4%) |
|
||||
| 4 tabs | 270.8 MiB | 238.5 MiB (−12%) |
|
||||
| after soak | 199.0 MiB | 214.3 MiB (+8%) |
|
||||
| CPU | 28.1 s | 27.8 s (unchanged) |
|
||||
|
||||
Render integrity was checked explicitly this time, because that is how `--jitless` was caught:
|
||||
listeners were **7,784 in all four runs** and DOM nodes within 40 of each other, so unlike jitless
|
||||
the page rendered fully and the comparison is valid. There is simply no consistent memory win —
|
||||
some stages better, some worse, peak unchanged.
|
||||
|
||||
The mechanism says why, and it is worth recording as a closed door. Measured directly with the
|
||||
probe under `--max-opt=2`:
|
||||
|
||||
```
|
||||
01-boot-idle peak_malloced= 4.5 MiB
|
||||
03-post-login peak_malloced= 4.5 MiB
|
||||
04-board-loaded peak_malloced= 42.1 MiB <- identical to Turbofan
|
||||
```
|
||||
|
||||
**Maglev's zone allocation is just as large as Turbofan's.** Capping the tier does not reduce zone
|
||||
memory; only not compiling at all does, and that breaks the page. So this 38 MiB is not reachable
|
||||
through V8 configuration — it is proportional to how much JavaScript V8 has to compile, which puts
|
||||
it back under the one remaining server-side lever: ship less server code. Reverted.
|
||||
Reference in new issue
Block a user