Haskell-ide-engine: Memory characteristics questions

Created on 15 Dec 2017 · 10Comments · Source: haskell/haskell-ide-engine

I have a stack project, which defines a bunch of simple executables (~20). I use atom with atom-ide-ui as a client to hie. Now hie's memory usage starts out quite normal (~150MB), but as I open more of the executables source files the memory usage shows an alarming growth rate reaching over 2GB of usage when most of them are opened and after a while it can more than double that again (What activities increase the memory usage is a little unclear, but the usage increases over time (not indefinitely, but enough to force me to allocate more memory for my vm)). If I close some of the open files the usage drops again.

It isn't a problem per se (The benefits are worth the cost), but it does make me wonder:

Does the structure of a project affect the memory usage? My executables share a bunch of dependencies, but this does not seem to help hie, since each file adds a whole lot of memory, compared to the amount of information in it.
What causes the memory usage to increase like that over time? I know that I am not adding enough new information to the code to warrant it.

question performance

Source

sniperrifle2004

👍5

Most helpful comment

For those having issues with high memory usage: You can use bash wrapper for HIE to set some memory limits with RTS flags. It just crashed (VSCode plugin restarts it automatically) when the limit is reached. Not a solution, but at least prevents from reaching the stage at which it makes my OS unresponsive.

#!/bin/sh

export HIE_SERVER_PATH=`which hie`

if [ "X" = "X$HIE_SERVER_PATH" ]; then
  echo "Content-Length: 100\r\n\r"
  echo '{"jsonrpc":"2.0","id":1,"error":{"code":-32099,"message":"Cannot find hie.exe in the path"}}'
  exit 1
fi

hie  +RTS -c -M4G -K4G -A16M -RTS --lsp $@

kfigiela on 27 Apr 2018

👍8

All 10 comments

I have the same problem, a simple stack project with two executable (several hundreds of LOC in total), the memory goes up to 4~5 GiB over time and CPU goes up to 40%~60% regularly, make the OS unusable, had to uninstall.

qwfy on 10 Jan 2018

I have same problem. Memory go up 8GB RAM if I keep hie server running for a day
I use MacOS High Sierra 10.13.2 and Visual Code plugin, with latest master commit

screen shot 2018-02-01 at 12 19 33 pm

hgiasac on 1 Feb 2018

I can confirm this as I am having the same issue here. As memory grows, there is heavy CPU usage allocated by kernel_task on macOS (however, fans are not spinning like crazy). It probably has sth to do with low level stuff such as memory allocation as whole OS gets very laggy. For sure it is not an out of memory condition that causes OS being non responsive. As soon as I kill hie, things immediately get back to normal. Since I am working in not so small project, I need to restart hie every hour or two. I will be happy to provide any debug info required for debugging.

kfigiela on 26 Feb 2018

There is at least one memory leak in GHC itself which causes memory usage to rise over time.

It is possible there are more in HIE too.

We have not been able to track this down yet, though.

alanz on 11 Mar 2018

I am experiencing high memory usage as well on Fedora 27. I am using VS code (because it seemed like the recommended editor for HIE) and I cannot get HIE to use less than 700 MB of RAM, and I've seen it exceed 8 Gigs.

DevJac on 9 Apr 2018

#!/bin/sh

export HIE_SERVER_PATH=`which hie`

if [ "X" = "X$HIE_SERVER_PATH" ]; then
  echo "Content-Length: 100\r\n\r"
  echo '{"jsonrpc":"2.0","id":1,"error":{"code":-32099,"message":"Cannot find hie.exe in the path"}}'
  exit 1
fi

hie  +RTS -c -M4G -K4G -A16M -RTS --lsp $@

kfigiela on 27 Apr 2018

👍8

kfigiela on 8 Jun 2018

I have a machine with 8 GB of memory in total and having HIE consume 50% or more makes it unusable. Therefore, I'd argue that before moving onto implementing more new or fixing existing functionality, fixing this memory usage is a high priority. People won't be using HIE at all if it keeps on bringing their machines down.

This is what hie --version gives:

Version 0.2.2.0, Git revision f73e4698fabf44667f6bf3165d9fac8d091d2437 (1640 commits) x86_64 ghc-8.2.2

Let me know if I can provide any debugging information. I'd be happy to help.

mdimjasevic on 15 Aug 2018

👍2

I've got a medium-small (33kloc) codebase that I'm working on and HIE along with its child processes are currently sitting at about 13GB of RAM, which along with firefox is nearly pegging both my main RAM and my swap. I'd also like to pile on this issue, since it's basically making HIE unusable for me without frequent restarts

I'm happy to run some memory profiles as well over the weekend, but unfortunately can't share the codebase