Sidekiq: Sidekiq workers hung up until all the workers finished their jobs

[nextofsearch]

Ruby version: 2.3.1
Sidekiq / Pro / Enterprise version(s): Sidekiq

Hi,

Sidekiq has worked without any problem for sometime but something odd happened today. The system launched 15 workers at 11:30 and later 10 more workers around 11:40 (25 concurrency per process and just 1 sidekiq process). The log tells me that 15 workers finished their jobs in around 6 minutes and the other 10 workers launched at 11:40 finished their jobs in 2 secs. But all the workers didn't finish until 11:45 somehow while it's quite clear that they processed all the jobs according to the log.

I copied a log below:

2016-07-19T02:30:14.794Z 14784 TID-on2r4 PushNotificationJobSidekiqWorker JID-e5b5607b869847a6517bb826 INFO: start
2016-07-19T02:30:20.158Z 14784 TID-on2r4 PushNotificationJobSidekiqWorker JID-e5b5607b869847a6517bb826 INFO: PushNotificationJobSidekiqWorker Done ! :: push_notification_job_id=2679222, market_app_id=1312, successed_device_market_app_count=100, time_diff = 5.363284346s
2016-07-19T02:45:46.253Z 14784 TID-on2r4 PushNotificationJobSidekiqWorker JID-e5b5607b869847a6517bb826 INFO: done: 931.459 sec

One thing to note is that #11 ~ #15 workers launched at 11:30 didn't finish their jobs until 11:45:46, when other workers stopped.

25 jobs were enqueued separately. It seems that the sidekiq process was hung up somehow.

[mperham]

No clue, there's not much to go on - you'll need to gather a lot more data.

[mperham]

You can use the TTIN signal to dump thread backtraces to see if they are all hung up on a lock, e.g.

[mperham]

Please reopen if you get more data that points to a Sidekiq issue.

[nextofsearch]

@mperham If I use TTIN signal, will it actually kill the hung threads? Or will it just dump thread backtraces and keep working threads untouched?

[mperham]

https://github.com/mperham/sidekiq/wiki/Signals#ttin

[nextofsearch]

I read it already but I am still confused. Does this actually kill all the threads in the process? In my case, the threads are hung but not dead and they seem to wait until all other threads finish so I don't want to kill the threads.

Sidekiq will respond to SIGTTIN by printing backtraces for all threads in the process to the logfile. This is useful for debugging if you have a Sidekiq process that appears to be dead or stuck.

[mperham]

Sidekiq will respond to SIGTTIN by printing backtraces for all threads in the process to the logfile.

That's it. I'm confused why you'd think it would kill anything.

[nextofsearch]

@mperham Thank you so much! I assumed "Kill" command will kill something. :)

[mperham]

Of course, that makes perfect sense. 'Kill' is terribly named!

On Aug 4, 2016, at 06:49, Dano Lee [email protected] wrote:

@mperham Thank you so much! I assumed "Kill" command will kill something. :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.