Sidekiq: Sidekiq hangs using Rails 6 app with "classic" autoloader

[langsharpe]

Ruby version: ruby 2.6.2p47 (2019-03-13 revision 67232) [x86_64-darwin18]
Sidekiq version(s): 5.2.7

Steps to reproduce:

• Create a new rails app using Rails 6.0.0.rc2
• Add Sidekiq and create some jobs (I used the instructions here https://github.com/mperham/sidekiq/wiki/Getting-Started#rails)
• Run bundle exec sidekiq
• Watch as it completes the jobs
• Switch to the classic autoloader (and a spring stop)
• Create more jobs
• Run bundle exec sidekiq
• Watch as is starts jobs, but never completes any.

Here is a test app based on the instructions above: https://github.com/langsharpe/test_sidekiq_activetext

I believe that ActionText is responsible. If you don't include ActionText this problem doesn't occur. Our use case is an existing app we upgraded to Rails 6. The problem doesn't occur when setting the concurrency to 1. I'm reporting here because I can't reproduce the problem without using sidekiq.

Here is a gist of the output of the process after sending the TTIN signal
https://gist.githubusercontent.com/langsharpe/8ca50bca283f863e09c226b0b88a3f57/raw/f8d6343fe2e388b6b2a16988dd281b29c57c332f/Sidekiq.log

Please let me know if this is unreproducible, or if there is anything else I can do to help diagnose the problem.

[mperham]

Agreed, this looks like a Rails issue, not a Sidekiq issue. Action Text might have a thread safety issue.

On Jul 26, 2019, at 00:13, Lang Sharpe [e-mail] wrote:

Ruby version: ruby 2.6.2p47 (2019-03-13 revision 67232) [x86_64-darwin18]
Sidekiq version(s): 5.2.7

Steps to reproduce:

Create a new rails app using Rails 6.0.0.rc2
Add Sidekiq and create some jobs (I used the instructions here https://github.com/mperham/sidekiq/wiki/Getting-Started#rails)
Run bundle exec sidekiq
Watch as it completes the jobs
Switch to the classic autoloader (and a spring stop)
Create more jobs
Run bundle exec sidekiq
Watch as is starts jobs, but never completes any.
Here is a test app based on the instructions above: https://github.com/langsharpe/test_sidekiq_activetext

I believe that ActionText is responsible. If you don't include ActionText this problem doesn't occur. Our use case is an existing app we upgraded to Rails 6. The problem doesn't occur when setting the concurrency to 1. I'm reporting here because I can't reproduce the problem without using sidekiq.

Here is a gist of the output of the process after sending the TTIN signal
https://gist.githubusercontent.com/langsharpe/8ca50bca283f863e09c226b0b88a3f57/raw/f8d6343fe2e388b6b2a16988dd281b29c57c332f/Sidekiq.log

Please let me know if this is unreproducible, or if there is anything else I can do to help diagnose the problem.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[mperham]

Out of curiousity, why are you using the classic autoloader?

[langsharpe]

We are upgrading a Rails app from v5, but haven't done the work to make it compatible with Zeitwerk yet.

I don't have a reproducible case without Sidekiq so I'm hesitant to open an issue on Rails. I've tried the Puma server with multiple threads (which appears to be the default) and haven't had an issue.

[mperham]

In that case, perhaps @matthewd might know of an issue?

[matthewd]

@georgeclaghorn https://github.com/rails/rails/blob/1e5c0a7836840be03c370d121c1cdefc1034595e/actiontext/lib/action_text/engine.rb#L42 (and surrounds) seems a bit odd to me, so maybe related?

From the backtraces this looks to be a Kernel#autoload vs Rails Reloader deadlock, entirely within our code. I suspect it's just much easier to trigger with Sidekiq because you need several jobs to be started at ~exactly the same time, without having any occur before that.

I haven't immediately spotted what would need to change to fix it, but I do think it's ours to fix.

@langsharpe please do open an issue on rails/rails.

[mperham]

Closing this, please follow up there and ping me if there's anything Sidekiq can do. Right now I document the classic loader as not working with Sidekiq 6.0 but that's not on purpose -- I just don't know what the problem is and don't know how to debug it.

[choonggg]

For anyone that missed the rails thread, set concurrency to 1 in development for a temp fix. Took me awhile to find out.

[jmaino]

Hi everyone - I've had similar situations and found this thread.
I, like others potentially, cheated while upgrading and as-of 6.0 I began to see jobs stuck in busy as well. Moving to "load_defaults 6.0" seemed to work for me until I removed my old new_framework_defaults.rb (which I didn't think I needed anymore). Then the problem reappeared. Restored the file and traced to see what might have exposed this for me.

In my case I have it narrowed down to:
ActiveSupport.to_time_preserves_timezone = false

If the above is set, I see no hangs. If I comment this out or revert to the default true as-of 5.0, then I see the jobs stuck in the busy state again.

Perhaps I have something else wrongly or inconsistently configured - but wanted to share this in case it sheds any light on what might be the root cause and how to resolve it.

ruby 2.7.0
rails 6.0.2.2
sidekiq 6.0.6

[nachoal]

This was driving me crazy, followed the temporary fix that @choonggg has and it is now working, will try to debug it now that I know that this is coming from classic load vs the gem that I was trying to setup.