Lighthouse: Insane memory consumption while syncing Medalla

[dankrad]

My lighthouse node is currently consuming 54.6 GB of memory syncing Medalla. This morning it locked up my 64 GB machine due to consuming all memory. There must be some sort of memory leak, because it constantly increases its memory consumption until it runs out.

[dankrad]

I'm on commit 46a06069c605842111d68b85b3e9aeea1974e202 (HEAD -> master, tag: v0.3.4, origin/master, origin/HEAD)

[realbigsean]

@dankrad Can you try running with this branch: #1916 and adding the flag --auto-compact-db=false

[dankrad]

Should I re-sync form the start to verify, as when I restart memory consumption seems normal to start with

[realbigsean]

If you don't mind, that would be helpful :)

[michaelsproul]

Thanks for trying the experimental fix @dankrad. Out of interest, what SSD does this machine have, and are there a lot of other processes writing to disk? Rough numbers from iotop would be useful. My current theory about the compaction is that on slow disks it causes a lot of writes to be buffered in memory, which gets worse and worse as more compactions run and the disk can’t keep up (we _were_ running every finalization)

[dankrad]

Currently at height ~500,000 and it seems much better. However memory usage is still 8.7 GB so not at all trivial. Disk writes according to iotop are varying between 10-100MB/s, pretty much all due to lighthouse -- no other processes with significant IO. That feels like a lot to me as well...

[michaelsproul]

Going to close this as resolved, because it seems it was just the queue backlog and the compaction wreaking havoc. Please feel free to re-open @dankrad if you notice any more RAM spikes.