Mastodon: After upgrading to 3.1.3, much more enqueued workers.

[HolgerHuo]

My instance used to have a load average under 0.3 0.3 0.3 and there's hardly any CPU usage by Ruby and Bundle when nobody accessed to my site.

However, there are now many enqueued jobs in sidekiq and the CPU load is high. Is that normal after upgrading?

[Gargron]

The upgrade cannot could have had any effect on this, it must be a coincidence. All of the displayed job types have existed in Mastodon for a very long time and the upgrade itself does not enqueue anything, only user or inter-server activity can do that.

[HolgerHuo]

I see that. But it did happen right after the upgrade and we seem unable to toot anymore, the enqueued tasks are increasing dramatically. Does it have anything to do with #13372 ?

[HolgerHuo]

4248 I think it's quite similar to this problem but I didn't get what you mean by systemd service

[HolgerHuo]

Hi, it turned out that I was using an outdated Redis server and the problem has been solved. Thanks a lot.

[jplock]

I'm seeing this too after upgrading to 3.1.3. Is there a specific version of Redis that is required now? I am running 3.0.6.

[HolgerHuo]

It requires Redis 4.0.0 at least now. Just use yum update redis or any other commands according to your distribution and restart mastodon-sidekiq, everything would work as usual~

[draft13]

I also had the same experience. I am running Ubuntu 16.04 and had to remove the packaged version 3.something and had to download and compile Redis 4 from source. Once that was done the queue started fresh and began to work.

[saper]

It requires Redis 4.0.0 at least now.

If this is the case indeed, it should be a big warning in the release notes. Was that intended?

[Gargron]

I've added a warning to the release notes just now. Sorry I haven't done so earlier -- the sidekiq update went completely out of my head. If anyone has a snippet on how to update Redis on Ubuntu 16.04, I'll include that as well, as it happens, I don't actually know where you could get it from except compiling from source (which is more complicated). At the very least, official docs have been recommending Ubuntu 18.04 for a good while so hopefully a lot of people aren't affected.

[draft13]

I followed this write up: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-redis-on-ubuntu-16-04

I can vouch that the compile portion works well, I have yet to setup the systemd service. I will do that in a bit (currently it is running from the command line in a tmux session). I also tell you that doing a do-release-upgrade also doesn't fix the problem, as it introduced a slew of other errors.

I ended up reverting to a snapshot I created prior to running the release upgrade, and roll with updating my own redis for a while.

[Gargron]

Yeah, upgrading 16.04 to 18.04 is a mess. Although 16.04 is nearing EOL, so don't know if I can actually recommend sticking to it longer...

[drsebro]

_If anyone has a snippet on how to update Redis on Ubuntu 16.04, I'll include that as well, as it happens, I don't actually know where you could get it from except compiling from source (which is more complicated). At the very least, official docs have been recommending Ubuntu 18.04 for a good while so hopefully a lot of people aren't affected._

this could help:

add-apt-repository ppa:chris-lea/redis-server
apt update && apt upgrade

the reason why I have to use ubuntu 16.04 is my vserver provider, I have a chip one :)

[HolgerHuo]

Thanks a lot for all your help!