Lighthouse: Memory usage increases significantly during non-finality

[emansipater]

Description

Lighthouse hit a peak of 10.09 GiB of RAM usage during the recent non-finality period on Medalla

Version

I was on v0.2.12 (via dappnode package) when I hit 10.09 GiB though I have since upgraded to v0.2.13-c0e76d2 (again via dappnode) and usage is again rising rapidly on a similar looking curve (originally it took ~5 days to reach the peak)

Present Behaviour

RAM consumption rises significantly while the chain is failing to finalize, easily exceeding the amounts of RAM that are recommended in most guides for running beacon nodes. Obviously could be related to #1782

Expected Behaviour

RAM consumption should remain within the recommended specs unless there is a significant performance advantage to using more RAM during non-finality, in which case the recommended specs should be changed

Steps to resolve

Unknown

[michaelsproul]

I'm working on this as part of #800 and #1769. Once those issues are resolved we can reassess memory usage to see if it's within acceptable limits and close this issue.

[raymonddurk]

I see both #800 and #1769 were closed over a week ago and there have been two releases since. Have these been addressed in 0.3.1 or 0.3.2 or is this pending for a future release still?

[michaelsproul]

Yes! Both 0.3.1 and 0.3.2 should be a lot gentler on RAM. I think around 5.5GB should be the highest it gets to now

[paulhauner]

Closing this since reducing memory usage during non-finality is a well-known issue and this issue doesn't identify and specific bug we need to fix. Thanks for raising it and providing detail :)

[dankrad]

I am currently syncing Medalla with lighthouse and seeing absolutely insane memory consumption -- currently at almost 40 GB. This morning it locked up my 64 GB machine, and I had to reboot.
It does seem like there is some sort of memory leak that warrants attention

[dankrad]

df@pascal:~/test/lighthouse/target/release$ ps aux | grep lighthouse
df          3610  131 64.5 47115528 42285820 pts/1 Sl+ 00:07 1065:54 ./lighthouse bn