Sidekiq: Stuck Enqueued workers

[krzkrzkrz]

Once a day or so, sidekiq just stops processing jobs. The number of enqueued jobs just pile up to a few thousand and don't get processed. As if they are just sitting there.. This is happening on our production server. However, the minute I visit Sidekiq's dashboard (https://foobar.com/sidekiq), the queue starts processing. Very strange.

Should I have some kind of background monitoring process like god or monit, to monitor and restart if the Sidekiq process crashed or went idle?

I've looked at https://github.com/mperham/sidekiq/issues/1993 but not sure if they are entirely related. On issue 1993, they have suggested to update celluloid to 0.16, which I will try later on today. Curent version is celluloid (0.15.2), apparently has some locking issues.

In the meantime, anyone else come across this issue? Any feedback will be appreciated.

I am using sidekiq (3.2.5)

[mperham]

Can you post your Gemfile.lock?

On Oct 14, 2014, at 20:03, Christian Fazzini [email protected] wrote:

Once a day or so, sidekiq just stops processing jobs. The number of enqueued jobs just pile up to a few thousand and don't get processed. As if they are just sitting there.. This is happening on our production server. However, the minute I visit Sidekiq's dashboard (https://foobar.com/sidekiq), the queue starts processing. Very strange.

Should I have some kind of background monitoring process like god or monit, to monitor and restart if the Sidekiq process crashed or went idle?

I've looked at #1993 but not sure if they are entirely related. On issue 1993, they have suggested to update celluloid to 0.16, which I will try later on today. Curent version is celluloid (0.15.2), apparently has some locking issues.

In the meantime, anyone else come across this issue? Any feedback will be appreciated.

I am using sidekiq (3.2.5)

—
Reply to this email directly or view it on GitHub.

[mperham]

Can you give me ssh access to the machine when the workers get stuck?

On Oct 14, 2014, at 20:03, Christian Fazzini [email protected] wrote:

Once a day or so, sidekiq just stops processing jobs. The number of enqueued jobs just pile up to a few thousand and don't get processed. As if they are just sitting there.. This is happening on our production server. However, the minute I visit Sidekiq's dashboard (https://foobar.com/sidekiq), the queue starts processing. Very strange.

Should I have some kind of background monitoring process like god or monit, to monitor and restart if the Sidekiq process crashed or went idle?

I've looked at #1993 but not sure if they are entirely related. On issue 1993, they have suggested to update celluloid to 0.16, which I will try later on today. Curent version is celluloid (0.15.2), apparently has some locking issues.

In the meantime, anyone else come across this issue? Any feedback will be appreciated.

I am using sidekiq (3.2.5)

—
Reply to this email directly or view it on GitHub.

[krzkrzkrz]

@mperham https://gist.github.com/4d2ad902c1813876f8d1 - Gemfile.lock
Unfortunately, I cannot give you SSH access, as this is our production environment

[mperham]

Then please post two TTIN outputs from the logs: one when the process is stuck and one after it is unstuck.

[mperham]

BTW your Gemfile.lock says you are using Sidekiq 2.17.4, not 3.2.5.

[krzkrzkrz]

Sorry @mperham its https://gist.github.com/09efa886d5a09e676da2

[krzkrzkrz]

What I will normall see in the logs is this:

2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: Error fetching message: Connection timed out
2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-16T03:32:18.053Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-16T03:32:18.054Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/sidekiq-3.2.5/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/sidekiq-3.2.5/lib/sidekiq.rb:72:in `redis'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/sidekiq-3.2.5/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/sidekiq-3.2.5/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/sidekiq-3.2.5/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/sidekiq-3.2.5/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize'
2014-10-16T03:32:18.055Z 21031 TID-o67nk ERROR: /home/deploy/.bundler/foobar_production/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'
2014-10-16T03:32:21.857Z 21031 TID-o67nk INFO: Redis is online, 3.8021864891052246 sec downtime

It seems Redis is timing out ever so often, and then it comes back on. I also realized our redis server was sitting on a cluster with 213 mb. Today I will try to update the cluster to 1.5gb and see if that changes anything

[krzkrzkrz]

@mperham I only see TID in the sidekiq.log which log are you referring to for the TTIN?

[mperham]

See Signals wiki page. kill -TTIN $sidekiq_pid

On Wed, Oct 15, 2014 at 8:53 PM, Christian Fazzini <[email protected]

wrote:

@mperham https://github.com/mperham I only see TID in the sidekiq.log
which log are you referring to for the TTIN?

—
Reply to this email directly or view it on GitHub
https://github.com/mperham/sidekiq/issues/2003#issuecomment-59310862.

[krzkrzkrz]

ps -aux | grep sidekiq # returns the pid
kill -TTIN 21044

returns nothing...

[mperham]

Now look in your logs for the TTIN output....

On Oct 15, 2014, at 21:56, Christian Fazzini [email protected] wrote:

ps -aux | grep sidekiq # returns the pid
kill -TTIN 21044

returns nothing...

—
Reply to this email directly or view it on GitHub.

[mperham]

Signals don't "return" anything. They signal to the process to do something. Sidekiq dumps backtraces for all threads to the log when it gets a TTIN signal. I need to see what the threads are doing when stuck and not stuck to determine any difference.

[rkamun1]

Thanks @krzkrzkrz for posting this!! @mperham I'll gladly give you ssh access to my server when this happens to me(every day it happens sporadically, and I can't figure out the common factor). I have been trying to debug this problem for a couple of months now. It seem like sidekiq just falls asleep.

[mperham]

In #2012, they are seeing Sidekiq freezes due to DNS resolution problems. Make sure you have proper timeouts set up on your Net::HTTP connections or use require 'resolv-replace' to ensure a bad DNS server can't hang your process.

[krzkrzkrz]

@mperham https://gist.github.com/ccb74e6c91e69c5bb70f the sidkiq log when I issue the kill -TTIN command

[krzkrzkrz]

As opposed to #2012

Their sidekiq.log returned several getaddrinfo, which could indicate a DNS issue. On my end, sidekiq received the kill signal, and then rebooted fine. As you can see, it started a background job (AggregateRawStatsWorker) and completed the job fine.

2014-10-27T04:15:46.689Z 21211 TID-lqo9k AggregateRawStatsWorker JID-ef7c8d48bb1b55ce0ab1e514 INFO: start
2014-10-27T04:15:46.695Z 21211 TID-lqo9k AggregateRawStatsWorker JID-ef7c8d48bb1b55ce0ab1e514 INFO: done: 0.007 sec

I can try the resolv-replace approach, but don't know how far that will get me...

[mperham]

Your TTIN output seems to indicate that your Sidekiq process was completely quiet, i.e. there was no work to be done. This is normal.

[krzkrzkrz]

Should I run this when the issue arises? If so, will I lose all queued jobs?

[krzkrzkrz]

I am curious, when accessing Sidekiq's dashboard. Does Sidekiq check if the background process is running and if it isn't, attempts to run it? This certainly seems like the case. But would like to verify.

I suspect that the Sidekiq process stops (for whatever reason), and nothing is there to "wake it up" again. If so, perhaps monit/upstart could be the solution here. Your thoughts @mperham ?

[mperham]

Run the TTIN command when the process is stuck. Restarting the process doesn't solve the underlying issue.

The Web UI and the sidekiq process are independent. The Web UI reads current heartbeat data out of Redis.

Don't use monit, use Inspeqtor. It's better because I designed it to be better than monit. :+1:

[rkamun1]

When I run TTIN, I get -bash: kill: (28966) - No such process

[mperham]

@sirBertly That error is pretty self-explanatory. That's not a valid PID. You either typed it wrong or the process crashed and died.

[rkamun1]

Hmmm. But it sees the process
ps -aux | grep sidekiq
deploy 28966 0.0 0.1 10464 928 pts/0 S+ 19:03 0:00 grep --color=auto sidekiq

[seuros]

That the Pid of the grep process.

[rkamun1]

Ah..ok, so my sidekiq dies....@krzkrzkrz is this the same thing that's been happening to you?

[mperham]

@sirBertly You should be using Upstart (if you are on Ubuntu) to restart Sidekiq in the case of a crash.

https://github.com/mperham/sidekiq/tree/master/examples/upstart/manage-one

[rkamun1]

Hi @mperham I do have both those files on my server.

[krzkrzkrz]

@sirBertly today the jobs were "stuck" again. 20k jobs or so. Going onto Sidekiq's dashboard, made Sidekiq start the jobs again.

@mperham it seems the Web UI (Sidekiq's dashboard) starts the job processing again though.

I'll have to wait until tomorrow to see if the sidekiq process is running or not, via pa -aux. If the process isn't running, then something must have made it crash, and if so, then a monitoring tool like monit, god or even inspeqtor _could_ be the solution.

I believe having a supervisor process is also mentioned somewhere in the wiki.

[krzkrzkrz]

Today the jobs were "stuck" again. ps aux | grep sidekiq returned:

deploy   21150 16.5 13.4 1011200 228724 ?      Sl   Oct29 162:40 sidekiq 3.2.5 foo_production [5 of 5 busy]

Which means the process was running, never crashed, but "stuck"! Going on the dashboard started working on the queue. i.e. the queue started to go down.

Frustrating, because the logs dont really indicate anything. @mperham any other ideas I should try?

@sirBertly how are things on your end?

[rkamun1]

@krzkrzkrz ok, looks like yours is a little different from mine. My sidekiq process seems to crash. My response to ps aux | grep sidekiq is deploy 29839 0.0 0.1 10464 928 pts/0 S+ 10:54 0:00 grep --color=auto sidekiq.

I haven't started it lately as I wanted to troubleshoot when I get some time. My stacktrace looks as follows: This is from my last deploy(when the USR1 was issued and sidekiq restarted) to when it spun out...

I really cannot figure it out. I have a monit recipe that's supposed to restart sidekiq every time redis restarts, that doesn't seem to help either. I haven't tried inspeqtor. This is next on the list...

2014-10-22T14:26:57.671Z 10343 TID-osgrcv86s INFO: Received USR1, no longer accepting new work
2014-10-22T14:26:57.679Z 10343 TID-osgsjyj5w INFO: Shutting down 25 quiet workers
2014-10-22T14:27:51.516Z 10343 TID-osgrcv58o ERROR: Error fetching message: Connection timed out
2014-10-22T14:27:51.529Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T14:27:51.529Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T14:27:51.530Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T14:27:51.530Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T14:27:51.530Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T14:27:51.530Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T14:27:51.530Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T14:27:51.531Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T14:27:51.531Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T14:27:51.531Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T14:27:51.531Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T14:27:51.531Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T14:27:51.533Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T14:27:51.533Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T14:27:51.534Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T14:27:51.535Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T14:27:51.535Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T14:27:51.535Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T14:27:51.535Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T14:27:51.535Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T14:27:51.539Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T14:28:06.051Z 10343 TID-osgrcv58o INFO: Redis is online, 14.511500358581543 sec downtime
2014-10-22T14:28:12.924Z 10343 TID-osgrcv58o ERROR: Error fetching message: Connection timed out
2014-10-22T14:28:12.940Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T14:28:12.940Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T14:28:12.940Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T14:28:12.940Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T14:28:12.940Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T14:28:12.940Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T14:28:12.941Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T14:28:12.942Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T14:28:12.943Z 10343 TID-osgrcv58o ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T14:28:12.943Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T14:28:12.943Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T14:28:12.943Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T14:28:12.955Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T14:28:12.973Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T14:28:12.977Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T14:28:12.977Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T14:28:12.977Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T14:28:12.977Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T14:28:12.977Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T14:28:12.978Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T14:28:12.978Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T14:28:12.978Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T14:28:12.978Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T14:28:12.978Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T14:28:12.978Z 10343 TID-osgrcv58o ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T14:29:04.093Z 13985 TID-otkukaecg INFO: [Sidetiq] Sidetiq v0.6.3 - Copyright (c) 2012-2013, Tobias Svensson <[email protected]>
2014-10-22T14:29:04.095Z 13985 TID-otkukaecg INFO: [Sidetiq] Sidetiq is covered by the 3-clause BSD license.
2014-10-22T14:29:04.095Z 13985 TID-otkukaecg INFO: [Sidetiq] See LICENSE and http://opensource.org/licenses/BSD-3-Clause for licensing details.
2014-10-22T14:29:04.095Z 13985 TID-otkukaecg INFO: [Sidetiq] Sidetiq::Supervisor start
2014-10-22T14:29:04.097Z 13985 TID-otkv6at0g INFO: [Sidetiq] Sidetiq::Actor::Clock id: 70024619219700 initialize
2014-10-22T14:29:04.099Z 13985 TID-otkv6acdk INFO: [Sidetiq] Sidetiq::Actor::Handler id: 70024619206380 initialize
2014-10-22T14:29:04.099Z 13985 TID-otkv6a6yk INFO: [Sidetiq] Sidetiq::Actor::Handler id: 70024619192580 initialize
2014-10-22T14:29:05.686Z 13985 TID-otkukaecg INFO: Booting Sidekiq 3.2.4 with redis options {:url=>"redis://localhost:6379/3", :namespace=>"sidekiq"}
2014-10-22T14:29:10.528Z 13985 TID-otkukaecg INFO: Running in ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
2014-10-22T14:29:10.529Z 13985 TID-otkukaecg INFO: See LICENSE and the LGPL-3.0 for licensing details.
2014-10-22T14:29:10.529Z 13985 TID-otkukaecg INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
2014-10-22T18:52:43.862Z 13985 TID-otkukyl0w ERROR: Error fetching message: Connection timed out
2014-10-22T18:52:43.882Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T18:52:43.885Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T18:52:43.886Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T18:52:43.887Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T18:52:43.888Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T18:52:43.888Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T18:52:45.067Z 13985 TID-otkukyl0w INFO: Redis is online, 1.1789309978485107 sec downtime
2014-10-22T18:53:27.736Z 13985 TID-otkukyl0w ERROR: Error fetching message: Connection timed out
2014-10-22T18:53:27.760Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T18:53:27.760Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T18:53:27.761Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T18:53:27.762Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T18:53:27.762Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T18:53:27.786Z 13985 TID-otkukyl0w ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T18:53:27.792Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T18:53:27.794Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T18:53:27.797Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T18:53:27.798Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T18:53:27.798Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T18:53:27.799Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T18:53:27.799Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T18:53:27.799Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T18:53:27.799Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T18:53:27.800Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T18:53:27.800Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T18:53:27.801Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T18:53:27.801Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T18:53:27.802Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T18:53:27.802Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T18:53:27.802Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T18:53:29.024Z 13985 TID-otkukyl0w INFO: Redis is online, 1.2213859558105469 sec downtime
2014-10-22T18:55:15.080Z 13985 TID-otkukyl0w ERROR: Error fetching message: Connection timed out
2014-10-22T18:55:15.109Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T18:55:15.110Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T18:55:15.111Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T18:55:15.112Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T18:55:15.120Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T18:55:15.120Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T18:55:15.120Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T18:55:15.120Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T18:55:20.820Z 13985 TID-otkukyl0w INFO: Redis is online, 5.6997129917144775 sec downtime
2014-10-22T18:55:35.966Z 13985 TID-otkukyl0w ERROR: Error fetching message: Connection timed out
2014-10-22T18:55:35.985Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T18:55:35.985Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T18:55:35.989Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T18:55:35.989Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T18:55:35.989Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T18:55:35.989Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T18:55:35.990Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T18:55:35.990Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T18:55:35.990Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T18:55:35.999Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T18:55:41.482Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T18:55:41.492Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T18:55:41.502Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T18:55:41.503Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T18:55:41.505Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T18:55:41.506Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T18:55:41.506Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T18:55:41.506Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T18:55:41.506Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T18:55:41.507Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T18:55:49.057Z 13985 TID-otkukyl0w INFO: Redis is online, 7.5493223667144775 sec downtime
2014-10-22T18:56:56.944Z 13985 TID-otkukyl0w ERROR: Error fetching message: Connection timed out
2014-10-22T18:56:56.973Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:55:in `rescue in _read_from_socket'
2014-10-22T18:56:56.974Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
2014-10-22T18:56:56.974Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:41:in `gets'
2014-10-22T18:56:56.974Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/connection/ruby.rb:273:in `read'
2014-10-22T18:56:56.974Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:238:in `block in read'
2014-10-22T18:56:56.974Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:226:in `io'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:237:in `read'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `block in call'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:207:in `block (2 levels) in process'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:329:in `ensure_connected'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
2014-10-22T18:56:56.975Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:185:in `block in call_with_timeout'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:253:in `with_socket_timeout'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis/client.rb:184:in `call_with_timeout'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1059:in `block in _bpop'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /home/deploy/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1056:in `_bpop'
2014-10-22T18:56:56.976Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-3.1.0/lib/redis.rb:1101:in `brpop'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:392:in `call_with_namespace'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:289:in `method_missing'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `block in retrieve_work'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in `with'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq.rb:72:in `redis'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:102:in `retrieve_work'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:37:in `block in fetch'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/util.rb:15:in `watchdog'
2014-10-22T18:56:56.978Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:33:in `fetch'
2014-10-22T18:56:56.979Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/sidekiq-3.2.4/lib/sidekiq/fetch.rb:44:in `block (2 levels) in fetch'
2014-10-22T18:56:56.979Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
2014-10-22T18:56:56.979Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
2014-10-22T18:56:56.979Z 13985 TID-otkukyl0w ERROR: /var/www/mobi-ads/shared/bundle/ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2014-10-22T18:56:58.058Z 13985 TID-otkukyl0w INFO: Redis is online, 1.0787773132324219 sec downtime 

[oleander]

@sirBertly It looks like Sidekiq can't correct to you redis server. Make sure you can connect to it using redis-cli -h localhost -p 6379 -n 3.

[rkamun1]

Yes. When I type in that command it connects:
~$ redis-cli -h localhost -p 6379 -n 3.
localhost:6379[3]>

[rkamun1]

The confusing thing is that response to ps aux | grep sidekiq is
deploy 5398 0.0 0.1 10464 928 pts/0 S+ 23:39 0:00 grep --color=auto sidekiq

[oleander]

@sirBertly What happens if you try starting sidekiq manually? bundle exec sidekiq --verbose --environment production -C config/sidekiq.yml,

[rkamun1]

hmmm, I should be able to run this from the current folder, right? I get:
WARNING:root:could not open file '/etc/apt/sources.list.d/passenger.list' The program 'bundle' is currently not installed. You can install it by typing: sudo apt-get install bundler

I am so sure I have run bundle exec rails c from this folder before

[oleander]

How do you start sidekiq now?

[rkamun1]

It starts up when I do a deploy. I have require 'capistrano/sidekiq' in my Capfile

[oleander]

That's you problem then. You don't know what capistrano does when it deploys. Figure that out first and try to start sidekiq in the same way. Then fix whats missing/doesn't work. When that's done, adjust your deploy script to reflect you previous findings.

[rkamun1]

ohhh....cool! Thanks for that!

[rkamun1]

@oleander Yup! That works, thanks again! @krzkrzkrz I had noticed that sidekiq craps out when it loses connection to redis, for whatever reason. I solved this with a monitor. I'm using monit, sorry @mperham, I had it installed before I found out about inspector. I'll probably change this later..

/etc/monit/conf.d/redis.conf

check process redis-server with pidfile "/var/run/redis/redis-server.pid" 
start program = "/etc/init.d/redis-server start" 
stop program = "/etc/init.d/redis-server stop" 
if failed host 127.0.0.1 port 6379 then restart if 5 restarts within 5 cycles then timeout

and

/etc/monit/conf.d/sidekiq.conf

check process sidekiq with pidfile /var/www/mobi-ads/current/tmp/pids/sidekiq.pid 
depends on redis-server 
start program = "/usr/bin/sudo start sidekiq index=0" 
stop program = "/usr/bin/sudo stop sidekiq index=0" 
if totalmem is greater than 500 MB for 2 cycles then restart # eating up memory? 
if 3 restarts within 5 cycles then timeout

[seuros]

@sirBertly I advice you to try Inspeqtor

[rkamun1]

@seuros Yes! That's the plan. I had monit in place and I want to monitor this first and make sure that I actually fixed my issues before tinkering with new stuff.

[oleander]

@sirBertly You're working from the wrong direction. You should figure out why redis crashes instead of just restarting it when it does. Redis shouldnt crash (unless maybe if you run out of memory). For this you don't need capistrano nor monit. Just start everything manually and see what happens.

[krzkrzkrz]

So I did some debugging. I initially thought there was something wrong with the Redis cluster Sidekiq was connecting to.

I ran the following command on another instance:

do echo -e "\n" ; date; src/redis-cli -h <redis_cluster_url> -p 6379 time; sleep 1; done > /tmp/elc-check.txt 2>&1 &

The above command will run in the background every second and connect to Redis, and will log the output to /tmp/elc-check.txt file. The content of this file is essential to get to the root cause of the issue.
I kept this running and waited for a timeout to happen with your sidekiq process. Once a Sidekiq timeout occurs, I can cross check the timestamp of the timeout with the log file /tmp/elc-check.txt generated by the above command.

After 24 hours, I checked Sidekiq's dashboard, and saw 20k jobs "stuck". Not moving / not processing. I checked Sidekiq's log file and didn't see any errors.

On the other instance, I checked elc-check.txt for any timeout or errors and none were reported. Which indicates that the Redis cluster was running fine.

This is definitely an issue with Sidekiq. Moreover, is the fact that it isn't reporting any significant errors in log/sidekiq.log is concerning

[krzkrzkrz]

It might be worth mentioning, I am also using gem 'sidekiq-middleware'. Not sure if this is causing any issues

[mperham]

@krzkrzkrz One fellow's "stuck" Sidekiq was fixed by adding require 'resolv-replace' to his sidekiq.rb initializer. Have you tried that?

[rkamun1]

@krzkrzkrz I added require 'resolve-replace' in an initializer file and monit, though everyone advices to use inspector

[rkamun1]

*Inspeqtor

[krzkrzkrz]

Issue still happens with require 'resolv-replace' in config/initializers/resolv_replace.rb

The logs dont show anything useful, where do we even begin? Such a frustrating issue

[mperham]

@krzkrzkrz I have no advice. Maybe start here: https://www.youtube.com/watch?v=VV7b7fs4VI8

[krzkrzkrz]

I am using Ruby 2.0, would Ruby 2.1 make a difference?

[rkamun1]

Are you using inspeqtor or something similar?
PS: I'm using ruby 2.1.2

[krzkrzkrz]

No inspeqtor, monit, etc. My sidekiq process doesn't crash. Sidekiq logs don't say anything about the process crashing. And the apps connection to the Redis host is also fine. My case is different from your @sirBertly

[oleander]

@krzkrzkrz Could you run kill -TTIN $sidekiq_pid and post the output?

[krzkrzkrz]

@oleander when shoudl I run this? When the queue is stuck? Or when there is a queue that is being processsed?

Also, will this destroy all currently queued jobs? As in, will I lose these jobs permanently?

[arrowcircle]

Hi! I have the same problem.
Sometimes workers just stop processing jobs. If I restart sidekiq(kill -9 or any other signal) and start it manually - it will process jobs for some time and then stucks again.

I use ruby 2.1.3, gemfile and -TTIN are in gist https://gist.github.com/arrowcircle/9fbc9e840cff4a532fac
I use sidekiq-limit-fetch only.

What I doing wrong?

[oleander]

@arrowcircle Try removing sidekiq-limit-fetch. I used the gem before and it resulted in an array of problems, one being unprocessed jobs.

[arrowcircle]

@oleander, I updated sidekiq to 3.2.6 and now it works perfect. sidekiq-limit-fetch don't cause any problems

[sarmiena]

I'm also having the same issue. Mine happens very frequently, however. Processes jobs just fine, then all of a sudden gets stuck in enqueued and starts piling up.

scheduled => enqueued (gets stuck here for a few mins) => busy => completed

All the while, I have 15 workers that aren't busy.

    sidekiq (3.1.3)
      celluloid (>= 0.15.2)
      connection_pool (>= 2.0.0)
      json
      redis (>= 3.0.6)
      redis-namespace (>= 1.3.1)
    sidekiq-limit_fetch (2.3.0)
      sidekiq (>= 2.6.5, < 4.0)
    sidekiq-status (0.5.1)
      sidekiq (>= 2.7, < 3.4)
    sidekiq_mailer (0.0.8)
      actionmailer (>= 3.0)
      activesupport (>= 3.0)
      sidekiq (>= 2.3)

:concurrency: 15
:verbose: true
:pidfile: ./tmp/pids/sidekiq.pid
:queues:
  - [impression_caps, 2]
  - [calendar_items, 1]
  - [fb_sync, 1]
  - [carets, 3]
  - [default, 2]
  - [push_notifications, 2]
  - [mailer, 2]
  - [drip_mailer, 4]
  - [modified_records, 1]
  - [es_import, 3]
  - [distributed_company, 1]
  - [reminders, 1]
  - [csv_export, 1]
:limits:
  default: 5
  carets: 5
  impression_caps: 2
  calendar_items: 5
  push_notifications: 5
  mailer: 5
  drip_mailer: 5
  modified_records: 3
  es_import: 3
  distributed_company: 3
  reminders: 5
  csv_export: 3

[mperham]

I can't help without the TTIN output.

On Dec 6, 2014, at 17:03, aldo sarmiento [email protected] wrote:

I'm also having the same issue. Mine happens very frequently, however. Processes jobs just fine, then all of a sudden gets stuck in enqueued and starts piling up.

scheduled => enqueued (gets stuck here for a few mins) => busy => completed

All the while, I have 15 workers that aren't busy.

sidekiq (3.1.3)
  celluloid (>= 0.15.2)
  connection_pool (>= 2.0.0)
  json
  redis (>= 3.0.6)
  redis-namespace (>= 1.3.1)
sidekiq-limit_fetch (2.3.0)
  sidekiq (>= 2.6.5, < 4.0)
sidekiq-status (0.5.1)
  sidekiq (>= 2.7, < 3.4)
sidekiq_mailer (0.0.8)
  actionmailer (>= 3.0)
  activesupport (>= 3.0)
  sidekiq (>= 2.3)

---

:concurrency: 15
:verbose: true
:pidfile: ./tmp/pids/sidekiq.pid
:queues:

• [impression_caps, 2]
• [calendar_items, 1]
• [fb_sync, 1]
• [carets, 3]
• [default, 2]
• [push_notifications, 2]
• [mailer, 2]
• [drip_mailer, 4]
• [modified_records, 1]
• [es_import, 3]
• [distributed_company, 1]
• [reminders, 1]
• [csv_export, 1]
  :limits:
  default: 5
  carets: 5
  impression_caps: 2
  calendar_items: 5
  push_notifications: 5
  mailer: 5
  drip_mailer: 5
  modified_records: 3
  es_import: 3
  distributed_company: 3
  reminders: 5
  csv_export: 3
  —
  Reply to this email directly or view it on GitHub.

[krzkrzkrz]

@mperham kill -TTIN $sidekiq_pid on queued workers will kill the jobs permanently?

[mperham]

No, it sends the TTIN signal to the sidekiq process.

[sarmiena]

@mperham My bad about the TTIN. Anyhow I figured out that the issue was celluloid @ version 16. Backed it to 15.2 and working fine again.

[mperham]

Upgrade sidekiq. 3.3.0 works great with 16.

On Dec 11, 2014, at 11:59, aldo sarmiento [email protected] wrote:

@mperham My bad about the TTIN. Anyhow I figured out that the issue was celluloid @ version 16. Backed it to 15.2 and working fine again.

—
Reply to this email directly or view it on GitHub.

[krzkrzkrz]

Upgrading, will update this thread in a few hours with findings

[krzkrzkrz]

Nope, no difference. Issue still happens. I even tried removing sidekiq-middleware and replaced it with https://github.com/mhenrixon/sidekiq-unique-jobs still the same.... I am thinking of changing hosts (currently using AWS Elastic-Cache) to see if its a remote server concern

[krzkrzkrz]

I think we found where the issue is. Apparently there was a rake task that was causing high CPU spikes for more than an hour. The task involved pulling a few thousand records from the DB and running a worker for each.

[alexfalkowski]

I'm seeing the same issue and running kill -TTIN is not showing anything in my logs. This started happening after I upgraded to Ruby 2.2.0. Strange :(

[chinshr]

@krzkrzkrz Really? But why did the jobs start processing when you visited Dashboard? I seem to have the same problem, but restarting sidekiq seems to work for me. What is the sidekiq version you are running on right now?

[krzkrzkrz]

@chinshr Yea that's the only odd result about this. It seems like the issue happens when several hundred jobs are put into the queue at a time.

They start re-processing once you visit the Dashboard. Perhaps this strange behavior is due to high CPU spikes over a constant period of time. Some process' act funny in situations like this. Not sure if Sidekiq could have handled this better.

The way forward for now, is to execute workers in batches...

[smasry]

Also seeing this with Ruby 2.2.0 and Celluloid 0.16.0
Ran a kill -TTID ... looks like read timeouts

[chinshr]

@krzkrzkrz Thanks. My jobs are getting processed without problems (>200k/day) after figuring out that setting reaping_frequency on a MySQL connection will actually create a MySQL segfault and halt the sidekiq process. Removing reaping_frequency setting solved my problem along with making my jobs thread safe.

[jobygoude]

I have the same problem in production (AWS, ruby 2.0.0p481, sidekiq (3.3.2), celluloid (0.16.0), redis (2.8.4) ). I run kill -TTIN. Can you tell me if you detect the anomaly in log file.
https://gist.github.com/jobygoude/95789aacfbe558aa7539
Thank you in advance for your support.

[mperham]

@jobygoude I'd recommend upgrading to Ruby 2.2.

[alexfalkowski]

It didn't get much better for me when I upgraded to Ruby 2.2, it still got stuck. What I did to fix it is wrap a mutex around my job and that seemed to get it going again. I need to spend more time trying to diagnose why.

[mperham]

BTW I recently discovered that therubyracer gem can cause complete lockups of an MRI process. If you are using it within your Sidekiq jobs, it's quite likely the cause of stuck processes.

I always update the Problems wiki page with the latest advice on dealing with stuck processes. https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting#frozen-workers

[imgarylai]

Hi,

I have the same issue in production. I try to reproduce on my local machine, but everything works...
This my environment variable.

$ ruby -v
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]
$ redis-cli -v
redis-cli 3.0.7
$ rails -v
Rails 4.2.5

sidekiq version:  v4.1.1

This is result when I did $ kill -TTIN.
https://gist.github.com/imgarylai/cfe934f84855ef3b824c

Here is my Gemfile.lock
https://gist.github.com/imgarylai/fa25bff4a191bc755d20

[jobygoude]

Hello,

do you use rubyracer gem in your project ? If yes, i advise you to use nodejs instead.

[imgarylai]

@jobygoude ,yes, I use rubyracer in my project. Okay, I will try this. Do you know what cause this problem?

[imgarylai]

@jobygoude After I change rubyracer to nodejs. The problem still exist.
Here is what I see on the web UI:

Inside the rollbar page:

| Job | Argument |
| --- | --- |
| Rollbar::Delay::Sidekiq | please look in gist: https://gist.github.com/imgarylai/3f3b74f6cb113f58c3da |

When I found this problem, it seems that sidekiq was dead.

$ ps aux | grep sidekiq
ubuntu   26723  0.0  0.0  10432   668 pts/1    S+   16:06   0:00 grep --color=auto sidekiq
# redis still work
$ ps aux | grep redis
redis     1151  0.1  0.3  41228  7380 ?        Ssl  06:52   0:46 /usr/bin/redis-server 127.0.0.1:6379
ubuntu   26861  0.0  0.0  10432   672 pts/1    S+   16:11   0:00 grep --color=auto redis

So, I couldn't try $ kill -TTIN again.

I don't have any idea to to...

[mperham]

@imgarylai Please don't resurrect old issues. Open your own new issue and we'd be glad to help.

[imgarylai]

@mperham sorry for that. I will open a new issue. Thank you.