Mastodon: I'm losing, one by one, federation with anothers instances

[Miaourt]

Hello, I'm the admin of niu.moe.
I'm running into a very strange probleme on my docker-based installation, compiled on the server (few images changed for the /about page), on master / stables :
We lose federation with instances we where... federating fine like a few days befores... and theses are all "biggest" first, we can't follow/read recent post from mastodon.social / mastodon.xyz / mstdn.io / and surely many other that I'm not aware of.

Troubleshooting :
When I try to follow someone from theses instance, I'm waiting a bunch of time before getting an 422 error: "422 Remote account could not be resolved".
Theses instances (or at leat mstdn.io) won't blocked me, logs aren't showing anythings but timeouts...
My firewall isn't filtering outcoming ports, incoming authorized ones are http/https and ssh (80/443 and 22)
My nginx config is the following one : https://pastebin.com/iVWSRpuB
My docker compose is this : https://pastebin.com/AbvQM10Q
I'm... really lost.
If there is some rake commands to "reset/recreate a bunch of things" without losing followings/toots/content... I will happily use thoses :/ at this rate we are going to be... "alone".
Oh, and my mastodon:daily take some _hours_ to complete, mostly the "mastodon:push:refresh" task...

EDIT following some discussion on my post : https://niu.moe/@Technowix/375342

• Nothing that really pop out in the logs, only post/get/head/... requests that success/sometime fail...
• Other peoples can see my toots, but I can't see their...
• I have already restarted the server / network / proxy / rebuild the container many times / make sure that every bit of the network is fine....

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [x] This bug happens on a tagged release and on master (If you're a user, don't worry about this).

[kaniini]

A ton of people are seeing this, it's not just you. It appears to be a problem with the way PuSH notifications are being processed.

[iamgreaser]

I run a single-user instance and I don't think my outgoing messages are being delivered. Last notification was from 11 hours ago.

[kaniini]

In my case, I mostly recovered the instance by running mastodon:push:refresh by hand.

I suspect that in this case @Technowix is seeing some sort of IPv6 issue on his mastodon:push:refresh task.

I think, in general, there is some sort of bug with the way mastodon:push:refresh is handled still, but in my case, I was able to fix my instance by running it by hand. It would certainly be nice if those tasks were documented on the various serverless guides.

[ashfurrow]

@Technowix Just to confirm, you have run the mastodon:daily and/or mastodon:push:refresh rake tasks, eh?

[kaniini]

@ashfurrow his description says:

Oh, and my mastodon:daily take some hours to complete, mostly the "mastodon:push:refresh" task...

I'm pretty sure it is an IPv6-related issue, it smells like that to me.

[ashfurrow]

Cool cool, just wanted to cover the basics first :wink:

[Miaourt]

Thank you both for interest, I have already tried to disable ipv6 completely from my server (not DNS tonight....) but nothing changed... :/ you think removing the dns entry might change something? Bunch of instances have ipv6 and no problems at all...
Oh, and @kaniini, I already launched by hand and waited push:refresh to complete like 4/5 time in a day, nothing changed (and that why I know this takes hours)

[Miaourt]

After some help on my instances, some pointed me on http://qiita.com/koteitan/items/fadbe900c88481888917 and this "fix" caugth my intereset :
https://github.com/clworld/mastodon/commit/608a314d79f19b98e17051e1d516b927cdd1762b

I might have not be running proprely the "refresh task" for a while before (the cron might haven't worked at all ?) and the token can be renewed because he totally expired...

(I'm not good enought to apply the fix by myself, I afraid)

[Miaourt]

I'm trying to apply the fix provided, with a good backup behind...

[mjankowski]

If you are having this issue, can you let us know exactly what release (commit sha or tagged release) you are running on your instance?

I suspect that anyone running a commit either at or after this one - https://github.com/tootsuite/mastodon/commit/d4f7f11c3c8f4c117394feccca9bbe537f59524d - but before this one - https://github.com/tootsuite/mastodon/commit/322cbf83c8ce92ff7cd4c26265a36060b37f134f - might have seen some issues with the daily maintenance tasks.

Those commits are both less than 24 hours old, so you'd likely have to be running very recent code and running non-tagged releases.

There might very well be other issues here, of course - but just want to rule that out.

[Miaourt]

I was having the issue way before this commit, before 1.2.2 to be exact.
I was hoping this would solve be itself but...
I'm running on 322cbf83c8ce92ff7cd4c26265a36060b37f134f

[Miaourt]

I changed my ip/server also, and ssl certificates, but I don't hink this was the problem (no HPKP here).
If this can help :|

[Miaourt]

So, I isolated an "usual error"

sidekiq_1    | 2017-04-26T17:27:14.849Z 1 TID-gneiax2wc WARN: {"context":"Job raised exception","job":{"class":"ThreadResolveWorker","args":[386892,"https://mastodon.social/users/polarity/updates/2143199"],"retry":false,"queue":"pull","jid":"856874823d1e9551750cad07","created_at":1493227624.8000734,"enqueued_at":1493227624.8001473},"jobstr":"{\"class\":\"ThreadResolveWorker\",\"args\":[386892,\"https://mastodon.social/users/polarity/updates/2143199\"],\"retry\":false,\"queue\":\"pull\",\"jid\":\"856874823d1e9551750cad07\",\"created_at\":1493227624.8000734,\"enqueued_at\":1493227624.8001473}"}
sidekiq_1    | 2017-04-26T17:27:14.849Z 1 TID-gneiax2wc WARN: HTTP::TimeoutError: Read timed out after 10 seconds
sidekiq_1    | 2017-04-26T17:27:14.849Z 1 TID-gneiax2wc WARN: /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/null.rb:62:in `rescue in rescue_readable'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/null.rb:59:in `rescue_readable'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/per_operation.rb:31:in `connect_ssl'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/null.rb:35:in `start_tls'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/connection.rb:158:in `start_tls'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/connection.rb:44:in `initialize'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/httplog-0.99.2/lib/httplog/adapters/http.rb:42:in `initialize'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/client.rb:60:in `new'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/client.rb:60:in `perform'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/httplog-0.99.2/lib/httplog/adapters/http.rb:21:in `block (2 levels) in <class:Client>'
sidekiq_1    | /usr/local/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/httplog-0.99.2/lib/httplog/adapters/http.rb:20:in `block in <class:Client>'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/client.rb:41:in `request'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/chainable.rb:12:in `head'
sidekiq_1    | /mastodon/app/services/fetch_atom_service.rb:7:in `call'
sidekiq_1    | /mastodon/app/services/fetch_remote_status_service.rb:6:in `call'
sidekiq_1    | /mastodon/app/workers/thread_resolve_worker.rb:10:in `perform'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:167:in `execute_job'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:139:in `block (5 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/rails.rb:90:in `block in call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.2/lib/active_support/execution_wrapper.rb:85:in `wrap'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/rails.rb:89:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:135:in `block (4 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-unique-jobs-5.0.0/lib/sidekiq_unique_jobs/server/middleware.rb:15:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb:10:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:134:in `block (3 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb:32:in `with_context'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:183:in `stats'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:131:in `block in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:126:in `process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:82:in `process_one'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:70:in `run'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:17:in `watchdog'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:26:in `block in safe_thread'

sidekiq_1    | 2017-04-26T17:37:05.193Z 1 TID-gnehwp9k4 WARN: {"context":"Job raised exception","job":{"class":"Pubsubhubbub::ConfirmationWorker","args":[523,"subscribe","47f08d3091de2786dfff90ccb09e6ce6",""],"retry":false,"queue":"push","jid":"bebbc8a1708f5f2b49fc0081","created_at":1493228205.1498992,"enqueued_at":1493228205.1499686},"jobstr":"{\"class\":\"Pubsubhubbub::ConfirmationWorker\",\"args\":[523,\"subscribe\",\"47f08d3091de2786dfff90ccb09e6ce6\",\"\"],\"retry\":false,\"queue\":\"push\",\"jid\":\"bebbc8a1708f5f2b49fc0081\",\"created_at\":1493228205.1498992,\"enqueued_at\":1493228205.1499686}"}
sidekiq_1    | 2017-04-26T17:37:05.193Z 1 TID-gnehwp9k4 WARN: HTTP::TimeoutError: Read timed out after 20 seconds
sidekiq_1    | 2017-04-26T17:37:05.193Z 1 TID-gnehwp9k4 WARN: /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/null.rb:62:in `rescue in rescue_readable'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/null.rb:59:in `rescue_readable'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/per_operation.rb:31:in `connect_ssl'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/timeout/null.rb:35:in `start_tls'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/connection.rb:158:in `start_tls'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/connection.rb:44:in `initialize'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/httplog-0.99.2/lib/httplog/adapters/http.rb:42:in `initialize'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/client.rb:60:in `new'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/client.rb:60:in `perform'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/httplog-0.99.2/lib/httplog/adapters/http.rb:21:in `block (2 levels) in <class:Client>'
sidekiq_1    | /usr/local/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/httplog-0.99.2/lib/httplog/adapters/http.rb:20:in `block in <class:Client>'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/client.rb:41:in `request'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/http-2.2.1/lib/http/chainable.rb:19:in `get'
sidekiq_1    | /mastodon/app/workers/pubsubhubbub/confirmation_worker.rb:19:in `perform'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:167:in `execute_job'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:139:in `block (5 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/rails.rb:90:in `block in call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.2/lib/active_support/execution_wrapper.rb:85:in `wrap'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/rails.rb:89:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:135:in `block (4 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-unique-jobs-5.0.0/lib/sidekiq_unique_jobs/server/middleware.rb:15:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/logging.rb:10:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:134:in `block (3 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/logging.rb:32:in `with_context'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:183:in `stats'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:131:in `block in process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:126:in `process'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:82:in `process_one'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/processor.rb:70:in `run'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:17:in `watchdog'
sidekiq_1    | /mastodon/vendor/bundle/ruby/2.4.0/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:26:in `block in safe_thread'

I can ping to mastodon.social, I can resolve the domain, ...
They seem to be both ssl related, or somewhat... I comparend all my ssl config with others admins, nothing should be wrong :/
And yes, my .env.production file have the LOCAL_HTTPS=true.

[Miaourt]

So, after some try, all the problems seem to have disappear when disabling the voxility's anti-ddos... i'm closing this for now.

[ashfurrow]

I'm glad you got it figured out! And thank you for follow up with the solution, I'm sure it'll be helpful for others 🙇