Two things this settles.
First, the end-to-end number. Every Phase 2 result so far was an incremental A/B against
the build immediately before it, which cannot be summed. Rebuilt the merge-base
(06724cfeb) from a clean worktree and ran 2+2 alternated against HEAD, reporting cgroup
anonymous memory - the figure that counts against a container limit and cannot be
reclaimed:
stage before after
peak anon 376.6 -> 279.3 MiB (-26%)
boot idle 130.5 -> 62.4 MiB (-52%)
after sign-in 247.4 -> 142.9 MiB (-42%)
dashboard 293.4 -> 170.4 MiB (-42%)
4 tabs 366.5 -> 258.2 MiB (-30%)
after soak 323.7 -> 200.8 MiB (-38%)
peak node RSS 430.7 -> 336.2 MiB (-22%)
No before/after range overlaps at any stage. Costs: CPU 23.5 -> 27.7s (+18%, solid,
non-overlapping). Median board load reads +30% but is NOT established - individual loads
were [506,383,1233] and [848,431,574] before against [780,808,515] and [621,579,1004]
after, heavily overlapping with no cold-then-warm pattern, so six samples per side cannot
separate a real regression from the variance.
compare-stress-runs.mjs now leads with anonMiB, which the harness recorded all along but
the report was ignoring in favour of memory.current - a figure that includes page cache
and swings 60+ MiB between runs of a byte-identical image.
Second, jemalloc. The largest attributed-but-unfixed item is ~38 MiB of V8 zone memory
that musl frees to its own pools and never returns to the kernel, so an allocator that
decays spans back to the OS looked like the answer. Tested on the byte-identical image
with only HOMARR_MALLOC differing, asserting the "Using jemalloc" log line:
boot idle 62.4 -> 80.3 MiB (+29%)
after soak 215.5 -> 279.6 MiB (+30%)
4 tabs 259.9 -> 319.6 MiB (+23%)
CPU 28.0 -> 24.5 s (-13%)
The hypothesis was backwards. jemalloc keeps more freed memory on purpose - thread caches
plus time-based dirty-page decay - and spends it on speed. 13% cheaper CPU, 18-30% more
resident memory. Reverted, and it closes off "swap the allocator" as a RAM direction:
musl is already the frugal choice here.
Profiling the container end to end found one allocation site responsible for
most of the peak: a single 46.3 MiB JSON body, assembled from 3,946 socket
chunks, inside undici's fullyReadBody. That function holds the chunk list and
the joined copy at the same instant, so a body of size N costs about 2N.
The source was two Umami calls asking for pageSize=100000, with
getMultiEventTimeSeriesAsync running them for every tracked event name
concurrently via Promise.all. getEventNamesAsync was downloading every raw
event of the last 30 days purely to collect a handful of distinct name strings.
- getEventNamesAsync now reads the aggregated /metrics?type=event endpoint.
- getWebsiteEventTimeSeriesAsync pages 5,000 records at a time, folding each
page into the bucket map before requesting the next.
- getMultiEventTimeSeriesAsync runs series sequentially.
Instances that return aggregated {x,y} buckets still answer in one request, an
instance that accepts pageSize but ignores page is detected rather than
double-counted, and hitting the page ceiling logs instead of silently
undercounting.
Alternated runs, 2 per side, on a restored real backup:
peak container 498.9 -> 398.5 MiB (-20%)
peak node RSS 452.4 -> 353.3 MiB (-22%)
peak during 4 tabs 498.5 -> 356.3 MiB (-29%)
peak arrayBuffers 95.8 -> 6.6 MiB (-93%)
largest allocation 46.4 -> 2.4 MiB (-95%)
Costs +4% CPU from the extra round trips; board load latency unchanged.
Post-soak steady state and boot idle are unchanged within run-to-run spread.
Adds the attribution tooling this needed: attribute-memory.mts reconciles
cgroup -> processes -> address-space mappings -> V8 spaces -> loaded code,
memory-probe.cjs is an injectable diagnostics preload (never shipped) that
tracks buffer and fetch allocation sites, analyze-heap-dominators.mjs computes
real retained sizes, and compare-stress-runs.mjs reports alternated runs with
their spread.
Also corrects an earlier claim in PHASE2-FINDINGS.md: the reported "7.86 MiB of
duplicate chunk sources" was an artefact of V8 truncating string contents to
~1 KiB in heap snapshots. There are zero byte-identical duplicates among all
3,055 server JS files. The real doubling is character width - one codepoint
above U+00FF forces a whole source string to two bytes per character, which
costs 9.45 MiB across 80 loaded files.
Adds 6 tests; this integration had none.
Thomas Camlong