Sidekiq: Memory issues in sidekiq-pro (3.4.5)

[fatcatt316]

Ruby version:
2.2.5
(Tested and same issue still happens in 2.4.1)

Sidekiq / Pro / Enterprise version(s):
sidekiq-pro (3.4.5)

Rails version:
5.0.1

Please include your initializer and any error message with the full backtrace.

Sidekiq::Client.reliable_push! unless Rails.env.test?

Sidekiq.configure_server do |config|
  config.redis = {
    url: Settings.redis_hosts.sidekiq,
    namespace: 'joe'
  }

  config.reliable_fetch!
  config.reliable_scheduler!
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: Settings.redis_hosts.sidekiq,
    namespace: 'joe'
  }
end

No error messages, just memory bloat up to 2GB.

Are you using an old version? No.

Have you checked the changelog to see if your issue has been fixed in a later version? Yes.

---

I have a Sidekiq Worker that's loading about 150,000 objects from the database (in batches of 100), and writing to one file.

When I run the code that's in the worker's perform method in rails console, my memory usage goes up a tiny bit, but not much at all (the Ruby process goes up to about 280MB).

But if I call that worker's perform_async method, and let Sidekiq pick it up, memory usage goes _way_ up, to almost 2GB.

Calling this same method to run _now_ (with WorkerClass.new.perform(blah), memory usage isn't too bad at all (Ruby process tops out at about 300MB).

I know you specifically said not to create an issue about memory leaks, but since the only difference I can find is that this happens when run asynchronously through Sidekiq, I figured it'd be worth seeing if you could help or offer any advice.

[mperham]

You can try Rails 5.0.2 but I don't have further ideas except running a memory profiler and tracking it down. If you can give me a small, reproducible app, I'd be happy to investigate.

[fatcatt316]

@mperham upgrading to Rails 5.0.2 doesn't seem to have done the trick.

I should be able to get you a small, reproducible app soon! And yep, I've been doing some memory profiling, and will keep doing so 👍

[mperham]

Closing, feel free to open a new issue if you get something concrete to investigate. 👍

[fatcatt316]

@mperham in case you're interested, I think I figured out a solution. I think the problem is I was reading many, many rows, but never writing to the DB (just to a file), so the cache wasn't getting cleared.

Disabling the cache for this section of code seems to have cut the memory usage way down.

This comment (and thread) helped me figure this out.

Hopefully this helps someone in the future if they run into the same thing I did!