Haskell-ide-engine: Memory leaks every time a file is saved

[AlexeyRaga]

I am using a recently recent commit of HIE (0f697c8919747effca54be8a9b494896aea7d947) and I have noticed a strange behaviour:

• In a very small project (just a few modules)
• Open one module in HIE and observe how much RAM does it take. For a module with less ~200 lines it takes ~290MB.
• Without changing anything (add a space and remove it), save this module.
• Repeat.

Every time I save a file for no reason memory footprint increases by a significant number 20MB-40MB

I am not sure if there is a limit, but by saving no changes multiple times I was easily able to get it to consume 1GB, which is a lot for saving a 200 lines module a few times with no changes.

[robrix]

With d794e596ae2647fe5101d28769ac82c2fe5604b5 I’m currently sitting at 2.18 GB, and it’s no longer responding to messages 😕

[nponeccop]

It's a memory leak, so it's not merely a performance problem.

[lorenzo]

The problem is known https://lukelau.me/haskell/posts/leak/

@AlexeyRaga you mean that this is a memory leak introduced recently?

[AlexeyRaga]

@lorenzo I don't know when it has been introduced TBH. I just happened to have a monitor opened and noticed...

[ProofOfKeags]

Just compiled 0.12.0.0 and this appears to be an issue here as well.

[mpickering]

@CaptJakk How do I reproduce this issue?

[ProofOfKeags]

Here's my best guess

Note: this is on MacOS

1. check out 0.12.0.0
2. build with stack ./install.hs stack-hie-8.6.3
3. open project in VSCode (may happen elsewhere, but this is how I did it)
4. watch the memory swell slowly without interaction until it stabilizes (Activity Monitor)
5. save a file, watch memory jump

My project that this was occurring on was about 140 modules, and it was consuming 16G of memory (on a 16G system).

[jneira]

A benchmark would be very, very helpful

[ProofOfKeags]

How would I go about getting one of these?

[ProofOfKeags]

How can I use the profiler to help find the leak? I'd like to be able to help out with this if possible as this is the sole reason why I can't use HIE.

[fendor]

@CaptJakk
First, start with a series of action that causes the space leak. If you do find such a sequence of actions, record for reproducibility. I think there is -c in hie (or something similar) that can be replayed with lsp-test: https://github.com/bubba/lsp-test
If you can, share this, so other people can tuck in too. Hopefully, the space leak is reproducible with lsp-test.

Or: compile with profiling enabled, see https://github.com/mpickering/haskell-ide-engine/pull/37 for details and record the eventlog or something similar and use https://github.com/mpickering/eventlog2html to see which data-structures are using a lot of memory.
IIRC, mpickering said that hie uses a lot of lists which seemed to be reasonable. Just as a warning.

mpickering did a video on how to find space leaks: https://www.youtube.com/watch?v=PL8Wjdt0cKo&t

Also: Make sure you are using latest master, not some old version of hie.
Also, before spending some time in this debugging adventure, try the pr #1126 and see if the memory leaks have improved or even been solved. Maybe some problems are already solved.

[bubba]

This might also be useful https://lukelau.me/haskell/posts/leak

On 27 Oct 2019, at 00:38, fendor [e-mail] wrote:


@CaptJakk
First, start with a series of action that causes the space leak. If you do find such a sequence of actions, record for reproducibility. I think there is -c in hie (or something similar) that can be replayed with lsp-test: https://github.com/bubba/lsp-test
If you can, share this, so other people can tuck in too. Hopefully, the space leak is reproducible with lsp-test.

Or: compile with profiling enabled, see mpickering#37 for details and record the eventlog or something similar and use https://github.com/mpickering/eventlog2html to see which data-structures are using a lot of memory.
IIRC, mpickering said that hie uses a lot of lists which seemed to be reasonable. Just as a warning.

mpickering did a video on how to find space leaks: https://www.youtube.com/watch?v=PL8Wjdt0cKo&t

Also: Make sure you are using latest master, not some old version of hie.
Also, before spending some time in this debugging adventure, try the pr #1126 and see if the memory leaks have improved or even been solved. Maybe some problems are already solved.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[poscat0x04]

Can reproduce on Archlinux running hie 0.13.0

[Ailrun]

Can reproduce on Ubuntu with hie 0.13.0

[fendor]

Please try again with the current master! Now that #1126 has been merged we expect less leaking.