Thomas Camlong
c73b139dbd
PHASE2-FINDINGS.md had grown past 1,200 lines mixing server memory, client rendering, methodology and a long list of rejected experiments, which made it unusable as a reference. Split under docs/perf/: README.md headline results and the change-by-change ratio table SERVER-MEMORY.md what the container's RAM is made of, layer by layer CLIENT-PERFORMANCE.md dashboard cost in Chrome: DOM, fibers, images, paint REJECTED.md ideas implemented, measured and abandoned, with numbers METHODOLOGY.md the tools, and the measurement traps that gave wrong answers PHASE2-FINDINGS.md is now an index so existing links keep working. METHODOLOGY.md is the part worth keeping: every wrong number in this investigation came from an instrument rather than the application, and each trap is written down - snapshots cannot explain peaks, V8 truncates snapshot strings to ~1 KiB, snapshot objects are not live handles, measuring without a forced GC measures V8's growth, taking a CDP snapshot added 134 MiB to the process being measured, per-component React attribution silently returns nothing on a production build, a fixed port can measure a leftover container, and a server-side change that moves a client-side metric means the workload changed.
113 lines
6.1 KiB
Markdown
113 lines
6.1 KiB
Markdown
# Methodology, and the traps that produced wrong answers
|
||
|
||
Every wrong number in this investigation came from an instrument, not from the application. They are
|
||
recorded here because each one looked plausible at the time, and several survived long enough to be
|
||
published before being caught.
|
||
|
||
## The harness
|
||
|
||
`scripts/benchmarks/stress-restore.mts` boots a fresh container, restores a supplied backup
|
||
**through the onboarding restore UI** (including typing "I understand"), signs in, opens boards
|
||
across several tabs, clicks into widgets, reloads, then soaks idle — sampling memory, CPU, browser
|
||
heap, DOM nodes and listeners at each stage.
|
||
|
||
Each run proves it is actually behind a login rather than reading a public page: it asserts a
|
||
cookie-less request to the private board is **denied**, requires the Auth.js session cookie after
|
||
sign-in, and reports whether it modified the restored data at all.
|
||
|
||
The backup is passed by path via `STRESS_BACKUP_ZIP` and never enters the repository — it carries
|
||
real integration secrets. `.gitignore` also refuses `homarr-backup*.zip` and loose `*.sqlite`.
|
||
|
||
## Tools
|
||
|
||
| tool | answers |
|
||
| --- | --- |
|
||
| `attribute-memory.mts` | cgroup → processes → address-space mappings → V8 spaces → loaded code, each layer reconciled against the one above |
|
||
| `memory-probe.cjs` | injected `--require` preload: V8 spaces, module cache, buffer/fetch allocation sites, require graph, smaps, handles. Diagnostics only, never shipped |
|
||
| `analyze-heap-dominators.mjs` | real retained sizes via a dominator tree, and expansion of the largest retainer |
|
||
| `analyze-allocations.mjs` | allocation volume by site, diffable between two stages |
|
||
| `analyze-smaps-stages.mjs` | RSS split into JS heap, JIT, native and file-backed, per stage |
|
||
| `analyze-require-graph.mjs` | what pulled each module in |
|
||
| `compare-stress-runs.mjs` | alternated runs, medians **and per-run spread** |
|
||
| `measure-widget-cost.mts` | per-widget DOM nodes, fibers, heaviest components |
|
||
| `measure-board-paint.mts` | image requests, bytes, style/layout/task time, layout objects |
|
||
| `measure-widget-churn.mts` | idle churn attributed to a widget via DOM mutations |
|
||
|
||
## Metric choice
|
||
|
||
**cgroup `anon`** is the server-memory figure. It is non-reclaimable and is what a container limit
|
||
acts on.
|
||
|
||
**`memory.current` is not used for comparisons.** It includes page cache and swung **192–261 MiB
|
||
for the same image**, and 60+ MiB between runs of a byte-identical build.
|
||
|
||
**`heapUsed` is not server memory.** It is one component. An early version of this work reported a
|
||
`heapUsed` improvement as though it were the whole result.
|
||
|
||
## The traps
|
||
|
||
**A heap snapshot cannot explain a peak.** V8 runs a full GC before serialising one, so it reports
|
||
survivors only. The 94 MiB of transient ArrayBuffers that drove this container past 500 MB never
|
||
appeared in a single snapshot. Peaks need an allocation profiler and wrapped allocators.
|
||
|
||
**V8 truncates snapshot string contents to ~1 KiB.** Grouping snapshot strings by content therefore
|
||
compares only each string's first kilobyte. This produced a published claim of "7.86 MiB of literal
|
||
duplicate chunk sources"; hashing the real files found **zero** byte-identical duplicates among all
|
||
3,055 server JS files.
|
||
|
||
**Snapshot objects are not live native handles.** A snapshot showed 15 `Gzip` objects still present
|
||
at idle, which read as leaked zlib streams pinned by native handles — a real leak class that Next
|
||
documents. `process.getActiveResourcesInfo()` reported **Zlib=0**, before and after 300 deliberately
|
||
aborted gzip requests.
|
||
|
||
**Measuring memory without forcing a GC measures V8's growth, not the module's cost.** A
|
||
require-cost sweep reported `drizzle-orm` at 124.3 MiB; with `global.gc()` before each reading the
|
||
same measurement was a fraction of that. Two database drivers measured 21.4 MiB, not the ~30 MiB the
|
||
naive sum implied.
|
||
|
||
**The act of measuring can dominate the measurement.** Taking a heap snapshot over CDP delivers it
|
||
as JS string chunks: for a ~135 MB snapshot that added **~134 MiB of ArrayBuffers to the process
|
||
being measured**, and the next stage read 693.7 MiB. `v8.writeHeapSnapshot` streams from C++
|
||
instead, and the probe now reports its own `observerCostMiB`.
|
||
|
||
**Per-component React attribution requires a profiling build.** `fiber.actualDuration` is only
|
||
populated by one, so on a production build every fiber fails the "did it re-render" test and the
|
||
attribution tables come back **empty** — which reads as "nothing re-rendered" rather than "this
|
||
build cannot tell you". This produced a published claim that the App Router re-rendered once a
|
||
second. The tool now says which case it is.
|
||
|
||
**When a server-side change moves a client-side metric, the workload changed.** `--jitless` looked
|
||
like −24% peak RSS, −14% CPU and 30% *faster* board loads. Three of those are impossible for a
|
||
server flag. The client had **80% fewer event listeners**: the page had simply under-rendered.
|
||
|
||
**A fixed port can silently measure the wrong container.** A readiness check passed against a
|
||
leftover container on the same port while the container under test had not started. Ephemeral ports
|
||
and asserting the container's own identity are the fix.
|
||
|
||
**A completed build is not a built image.** A `docker builder prune` running concurrently killed a
|
||
BuildKit session; the build failed and the next step ran against a stale tag. Verify the tag exists.
|
||
|
||
**Boards differ, so measure the right one.** Peak memory is at `nav-01`, not the multi-tab stage — a
|
||
GC lands during multi-tab and pulls RSS down. Measuring the obvious stage understates peak by
|
||
37 MiB.
|
||
|
||
## Reproducing
|
||
|
||
```bash
|
||
STRESS_IMAGE=<image> \
|
||
STRESS_BACKUP_ZIP=~/Downloads/<backup>.zip \
|
||
STRESS_BOARDS="default,defaultv2" STRESS_TABS_PER_BOARD=2 \
|
||
STRESS_NAV_ITERATIONS=3 STRESS_SOAK_MS=120000 \
|
||
node --experimental-strip-types scripts/benchmarks/stress-restore.mts
|
||
|
||
node scripts/benchmarks/compare-stress-runs.mjs .bench/stress <before-prefix> <after-prefix>
|
||
```
|
||
|
||
For attribution, add the probe:
|
||
|
||
```bash
|
||
STRESS_MOUNT="$PWD/scripts/benchmarks:/probe:ro" \
|
||
STRESS_ENV="HOMARR_PROBE_PORT=9333,HOMARR_PROBE_TRACK_BUFFERS=32768" \
|
||
STRESS_NODE_OPTIONS="--require /probe/memory-probe.cjs"
|
||
```
|