Puma: Memory Leak in rails app with ruby 2.3.0 and Puma 2.15.3

[ThomasCelen]

Hi all,

For some time I've been having a memory leak in my application. The current setup runs with Ruby 2.3.0, Rails 4.2 and Puma Webserver puma 2.15.3 It is recommended by Heroku to use Puma, however a lot of debugging has lead me to believe it's actual Puma Webserver which is the cause of all the troubles. During debugging I've removed most of the gems, code etc but the issue kept showing up, RAM Memory increasing until an out of memory showed up. This is why I'm 99,999% sure the leak isn't from my application. Next I installed "Thin" Webserver on my Development system and also this was not a solution. Then I also upgraded my ruby version to 2.3.0 and the leak was immediately solved, RAM usage stayed ok.

Due to this evolution, I upgraded my production system to Ruby 2.3.0 (thinking it was only a ruby version issue) but still I'm having the memory leak. I'm now at the point that it seems to be a combination of my Puma Webserver, together with Ruby 2.3.0...

Does anyone of you have any experience with Puma giving memory leaks with Ruby 2.3.0.

Thanks in advance,
T

UPDATE:
my puma.rb

workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

[evanphx]

Is there a way you can replicate the issue for me? There are 2 ways: either
provide me an app that leaks in a repo for me to run, or show that the
hello.ru trivial rack app leaks.
On Mon, Jun 13, 2016 at 2:42 AM ThomasCelen [email protected]
wrote:

Hi all,

For some time I've been having a memory leak in my application. The
current setup runs with Ruby 2.3.0, Rails 4.2 and Puma Webserver puma
2.15.3 It is recommended by Heroku to use Puma, however a lot of debugging
has lead me to believe it's actual Puma Webserver which is the cause of all
the troubles. During debugging I've removed most of the gems, code etc but
the issue kept showing up, RAM Memory increasing until an out of memory
showed up. This is why I'm 99,999% sure the leak isn't from my application.
Next I installed "Thin" Webserver on my Development system and also this
was not a solution. Then I also upgraded my ruby version to 2.3.0 and the
leak was immediately solved, RAM usage stayed ok.

Due to this evolution, I upgraded my production system to Ruby 2.3.0
(thinking it was only a ruby version issue) but still I'm having the memory
leak. I'm now at the point that it seems to be a combination of my Puma
Webserver, together with Ruby 2.3.0...

Does anyone of you have any experience with Puma giving memory leaks with
Ruby 2.3.0.

Thanks in advance,
T

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/puma/puma/issues/1000, or mute the thread
https://github.com/notifications/unsubscribe/AAAAB1ajo4kVTcaph6A-3xrj_8LJtvRCks5qLSYGgaJpZM4I0J_o
.

[ThomasCelen]

Hi, I added you to a test repository. It's an older version where ruby isn't yet upgraded to 2.3.0 but besides that it's unfortunately also leaking...

[ThomasCelen]

hi @evanphx , have you been able to spot any issue yet?
Many thanks for your help!

[ThomasCelen]

hi @evanphx , can you please share if you already looked into my issue? If not I will attempt Unicorn webserver as I'm really convinced it's Puma.. The only thing I'm doubtful of is that some library is loaded incorrectly and causes the memory leak.

Cheers,
T

[bailsman]

Hi Thomas. Is it a private repository? Are you able to produce a minimal app that still leaks and share it publicly in this issue?

[evanphx]

@ThomasCelen Sorry for the delay. I don't see the repository... Where is it?

[ThomasCelen]

@codehotter and @evanphx I added you both as collaborators to my test repository.. https://github.com/ThomasCelen/test_app/invitations
Can you please help me find the issue? I've been tracking this for months and the only thing I can figure is that, unfortunately, it's a Puma issue in combination with Ruby.

[evanphx]

@ThomasCelen Do you have any steps to reproduce the issue?

[ThomasCelen]

@evanphx just running the app will already trigger the leak actually so it's not difficult to replicate..

[evanphx]

@ThomasCelen Just leave it idle? Do I need to send it request? To what URLs?

[ThomasCelen]

@evanphx Indeed, just "rails s" to boot the application, run localhost:3000 (don't think there are any big DB migrations that are needed") and then the app automatically starts increasing RAM.. I've been tracking it with "puma worker killer", though I've never been able to trace the source.

[evanphx]

@ThomasCelen Alrighty, I'm looking now.

[evanphx]

@ThomasCelen I'm still looking at the issue, but in the mean time, can you disable preload and move your puma_worker_killer out of the initializer and into the puma config file detailed here: https://github.com/schneems/puma_worker_killer

[evanphx]

@ThomasCelen do you have any stats/graphs of the memory usage you're seeing? I've running some tests and I do see it increase in small amounts but it appears to drop back down once a GC is done.

[evanphx]

For example:

[90336] PumaWorkerKiller: Consuming 169.87890625 mb with master and 2 workers
T_OBJECT=18 T_STRING=194 T_ARRAY=120 T_HASH=12 T_DATA=68 T_IMEMO=30 
[90336] PumaWorkerKiller: Consuming 169.8828125 mb with master and 2 workers
T_OBJECT=9 T_STRING=109 T_ARRAY=63 T_HASH=7 T_DATA=34 T_IMEMO=15 
[90336] PumaWorkerKiller: Consuming 169.94921875 mb with master and 2 workers
T_OBJECT=9 T_STRING=109 T_ARRAY=63 T_HASH=7 T_DATA=34 T_IMEMO=15 
[90336] PumaWorkerKiller: Consuming 152.5703125 mb with master and 2 workers
[90336] PumaWorkerKiller: Consuming 135.046875 mb with master and 2 workers
T_OBJECT=18 T_STRING=194 T_ARRAY=120 T_HASH=12 T_DATA=68 T_IMEMO=30 
[90336] PumaWorkerKiller: Consuming 135.08203125 mb with master and 2 workers
T_OBJECT=9 T_STRING=109 T_ARRAY=63 T_HASH=7 T_DATA=34 T_IMEMO=15 
[90336] PumaWorkerKiller: Consuming 127.75390625 mb with master and 2 workers

[evanphx]

@schneems he is running PumaWorkerKiller from config/initializers with preload on. Pretty sure that should absolutely NOT be done, correct?

[ThomasCelen]

@evanphx I'll run my test environment for a while without Puma_Auto_tune so that I can show you a screenshot of my memory usage. Thanks already for your checks.

[ThomasCelen]

@evanphx This is a screenshot from 24hrs ago. Please know that Puma_Worker_killer is still turned on here. During the time frame of the screenshot there was very little traffic on the app. At the end you can see that it goes out of memory. This was when I was trying to change some data in the admin.

[jjb]

@ThomasCelen that looks like pretty typical behavior of a rails app

the thrashing around and then slow growth could be from a lot of things

to get a more focused look at the "real" memory growth, try this:

• set the worker count to 1 (you may of course need to scale the # of dynos accordingly)
• heroku config:set RUBY_GC_HEAP_GROWTH_FACTOR=1.10 (reference: https://devcenter.heroku.com/articles/ruby-memory-use#gc-tuning)

leave that running for a day or two, the graph should be more informative.

and may as well update to ruby 2.3.1, if possible.

[jjb]

(and also perhaps remove PumaWorkerKiller)

[evanphx]

@ThomasCelen Please update to 2.3.1 and see what the memory looks like there. There are definitely slow memory growth issues with older versions of ruby and they should be corrected in 2.3.

[TomLiu]

@ThomasCelen have you found out what the problem is?

[ce07c3]

@ThomasCelen I'm seeing this (Puma 3.6.0, Ruby 2.3.1, Rails 5.0.0.1):

Pretty similar to your memory growth between 1AM and 4AM, occurring once an hour. What could this be? Using New Relic, I was able to see that it coincides with Ruby GC runs:

@evanphx could this be related to a GC memory leak? My code doesn't run once an hour peridiocally...or maybe Rails does something? Or Puma? I'm out of suggestions, even ❓

[schneems]

That looks like normal growth

[ce07c3]

@schneems the same growth happens on idle dynos.

Why do the large memory jumps cooincide with the GC spiking? I have no recurring tasks, no "unusual calls" once an hour either.

[ce07c3]

And most importantly, shouldn't the GC reduce memory? I am open for suggestions on what to do; profiling, benchmarking, ....

[schneems]

Here's a primer http://www.schneems.com/2015/05/11/how-ruby-uses-memory.html

No, GC should not reduce memory. Up to Ruby 2.3 it will not free aggressively, or not enough to matter for most apps. It also takes a larger memory overhead before it would start to reclaim memory. There is a GC setting for 2.4 that can be set, but it won't impact this use case.

Even if the app isn't doing any thing there are still event loops that create objects and do things like set timeouts, wait for incoming requests, etc. Not just from puma but other places in your code or libraries can run loops in new threads.

Your seeing memory grow and then level off. It's totally normal.

[ce07c3]

Thanks! Will read up on it.

[ce07c3]

Thanks a bunch, will read up on it.

21 okt. 2016 kl. 17:11 skrev Richard Schneeman [email protected]:

Here's a primer http://www.schneems.com/2015/05/11/how-ruby-uses-memory.html

No, GC should not reduce memory. Up to Ruby 2.3 it will not free aggressively, or not enough to matter for most apps. It also takes a larger memory overhead before it would start to reclaim memory. There is a GC setting for 2.4 that can be set, but it won't impact this use case.

Even if the app isn't doing any thing there are still event loops that create objects and do things like set timeouts, wait for incoming requests, etc. Not just from puma but other places in your code or libraries can run loops in new threads.

Your seeing memory grow and then level off. It's totally normal.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[nateberkopec]

@ThomasCelen's graph looks normal to me.

@ce07c3 - do you have some "RUBY_" environment variables set? Your graph shows you have no free heap slots, which is abnormal. I'm guessing you have some GC variables tuned to incorrect values, probably RUBY_GC_HEAP_GROWTH_FACTOR.