Mastodon: Reduce Sidekiq memory usage 2-4x

[mperham]

Hi, Sidekiq author here. I see a lot of complaints from people about Mastodon's Sidekiq memory usage. I'd recommend you read this blog post and switch to jemalloc in your Docker image. I've heard from a dozen users that the memory reduction is "miraculous". You install jemalloc and then use LD_PRELOAD to override the default glibc malloc when executing Ruby.

https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
https://github.com/jemalloc/jemalloc/wiki/Getting-Started

[Gargron]

@mperham Hey! I'm a big fan of Sidekiq. However, it's not so simple. We use Alpine Linux in Docker, and apparently jemalloc does not work well on Alpine: https://bugs.ruby-lang.org/issues/13524

[mperham]

@Gargron It appears to be fixed with Alpine 3.7 and jemalloc 5.0.1. I could repro the segfault before, can't with those versions.

[Gargron]

Hmm, we had to go back from 3.7 to 3.6 because 3.7 introduced random segfaults into Sidekiq! (#6734) :sweat_smile:

[mperham]

Oy. I would bump to 2.4.4 first before giving up. It's common for MRI to get fixes for this type of thing and 2.3.0 is several years old.

[saper]

Please take into consideration that Alpine has a very small default thread stack size (https://wiki.musl-libc.org/functional-differences-from-glibc.html#Thread-stack-size https://github.com/voidlinux/void-packages/issues/4147).

A story with RSpec:

https://www.madetech.com/blog/a-tale-in-adopting-alpine-linux-for-docker-problems-we-faced-with-rspec-testing

libuv (and therefore node) avoid the problem since node 8.x by increasing the stack size explicitly https://github.com/libuv/libuv/pull/1526/files#diff-391600498c36f644bc9439a0b05a12f7R194

It is possible to apply a quick work around with LD_PRELOAD (https://github.com/jubel-han/dockerfiles/blob/master/node/Dockerfile) and see if that helps.

[mperham]

Since you aren't using glibc, it's not clear to me that memory bloat is a problem. glibc has terrible bloat but Alpine uses musl libc which I assume has a different allocator. Closing.

[nightpool]

I'm pretty sure we still see memory leaks in sidekiq under docker but I can't speak to it myself, i run mastodon baremetal.

(not saying this is your fault, just saying it's maybe still good to look into the jemalloc thing)

maybe @0xa could give some more insight here? I know they OOM kill docker containers pretty often on mastodon.social. unsure about sidekiq specifically.

[mperham]

I've pitched the idea to have Ruby use jemalloc by default. Like any major change to a large project, it's a steep uphill climb, especially when we can't easily reproduce the problem. I'm going to look into building a custom Rails app which will reproduce the bloat and show that jemalloc helps it.