Ngrok: Unknown, big memory-leak

[KnutHelland]

Hello,

I don't know if it is a good idea to post this very unspecified bug report here. But I can't see any other issues about memleaks.

I'm running ngrok on my Raspberry Pi (http://weather.knuthelland.com/) and the ngrok-process eats memory for every http-request. After approx. 1000 (small) requests (I've not counted), the RaspPi goes down because ngrok uses 100% memory.

ngrok version = 1.6 (I think this is the latest?)

For now I've just created a daemon monitoring the process and restarting it when memory grows over 20%. If this bug is completely unknown, I can do some more research. Are there any easy way to dump a memory-report for go tool pprof?

[inconshreveable]

This is a well known issue, unfortunately. Although I don't have a fix in the works, i do have a workaround for you because the memory leaks are isolated to ngrok's UI components:

First, run ngrok with -log=stdout
This will disable the console UI.

Second, in you ngrok config file at ~/.ngrok add:

inspect_addr: disabled

This will disable the web inspector.

I plan on fixing this in the next iteration of ngrok, but for now, this should at least let you get stability.

Let me know if it helps.

On Apr 4, 2014, at 1:02 AM, Knut Helland [email protected] wrote:

Hello,

I don't know if it is a good idea to post this very unspecified bug report here. But I can't see any other issues about memleaks.

I'm running ngrok on my Raspberry Pi (http://weather.knuthelland.com/) and the ngrok-process eats memory for every http-request. After approx. 1000 (small) requests (I've not counted), the RaspPi goes down because ngrok uses 100% memory.

ngrok version = 1.6 (I think this is the latest?)

For now I've just created a daemon monitoring the process and restarting it when memory grows over 20%. If this bug is completely unknown, I can do some more research. Are there any easy way to dump a memory-report for go tool pprof?

—
Reply to this email directly or view it on GitHub.

[jmoody]

Can inspect_addr disabled be incorporated on the command line?

I am using a LaunchAgent on MacOS and I want to avoid using ~/.ngrok file in the agent.

[KnutHelland]

This is the part of the source code that parses the args, and no, there is no way to set inspect_addr in command line.

However you can specify another path for the config-file, so you don't have to save it in your home folder. ngrok -config="some/other/path" [...]

It would be pretty easy to change the behaviour if you can recompile the program. Change line 76 here to set "disabled" instead of "127.0.0.1:4040" and you should have what you want. This page gives some hints about compiling.

[jmoody]

@KnutHelland Perfect and thanks. I was going to check on the -config option, but got distracted by the World Cup.

[cool-RR]

I had this issue too, and the workaround worked for me. But I need it to work without a config file. Can you please enable setting inspect_addr: disabled on the command line? This would make this bug less urgent.

[cool-RR]

Also: The -log=stdout flag causes a lot of log to be written to the console. I'd like to avoid that, I'd like only brief output from ngrok. (e.g. success or failure.) How can I do that?

[inconshreveable]

there's no great way to do that right now. the best answer is to redirect it to /dev/null (NUL on windows):

ngrok -log=stdout 80 > NUL

[cool-RR]

And that would possibly silence critical errors, wouldn't it?

[inconshreveable]

it would, although the program will of course exit with an appropriate return code which you can check. you might also grep at the desired log level

[cool-RR]

Hmm, waiting for the return code would result in an Entscheidungsproblem, since a correct run is supposed to stay running... I'll try to come up with something. Thanks!

[insekt]

I had this issue too. ngrok v1.7 (ARM). Workaround seems to be working.
Is there any chance to fix it?

[inconshreveable]

this is fixed in ngrok 2.x. no plans to fix in 1.x, but will leave it open if anyone wants to fix

[niaiwomattt]

This problem can be solved in the 2.x version? This is a serious problem.

[insekt]

I'm still using 1.x version because of self-hosting option. So if it's not hard to fix then it would be great.

[dapao9999]

i had this issue too,ngrok eat all of my memory

[tobischulz]

ngrok v1 client eats all the memory at a 3gb fileupload on a 4gb machine so ngrok client crashed before upload complete. Is there anyway to fix it or should i not use ngrok for uploading larger files?

ngrok2 dosn´t got that bug, but on this machine i would like to use selfhosted ngrok.

[insekt]

@dapao9999 @tobischulz
you could try to disable web inspector in config file.
inspect_addr: disabled

It's work for me. I'm using version 1 because it allows selfhosting.

[tobischulz]

@insekt thank you, that helps! but ngrok1 client still eating 2 gb on a 3 gb uploaded file. Is this a problem by the clients hdd? so the server is sending all the bytes and the client writes it to the hdd but can´t handle all the bytes are coming cause hdd wait?

[insekt]

@tobischulz doesn't know :( could you describe you case in detail, I'll try to reproduce it.

[tobischulz]

@insekt

• i´m using nextcloud on a local server with the ngrokv1 client.
• ngrokv1 server is hosted on a digitalocean droplet (512mb).
• Im uploading a file >= 3 gb to nextcloud over ngrok url.

You can see the ram is getting full on the local server until ngrokv1 client crashes. inspector is disabled by config file that let the client be alive some time longer till crash.

[insekt]

@tobischulz will try to repeat your case

[cedricve]

Having the same issue with forwarding a MJPEG stream (30FPS), it looks like some memory isn't released properly. I have hacked this with a restart policy running in the background, which makes sure the ngrok process is still running and isn't over a % threshold of memory. (using @KnutHelland workaround). This library is a great implementation, but v1.x is highly unstable.