Sidekiq: Heroku deployment - RuntimeError - ERR max number of clients reached

Created on 9 Apr 2012  Â·  40Comments  Â·  Source: mperham/sidekiq

Hi there

Firstly, please excuse me for posting on GH Issues. Because I am not so sure if this is an issue of Sidekiq or just something I did wrong on my setup.

I've deployed a simple Rails 3.2.2 app with Sidekiq Monitor app mount to /sidekiq. I did not have any custom configuration, all are default Sidekiq.

My heroku has 2 worker processes:

$ heroku ps
192-168-1-3:spree-thurley macbookair$ heroku ps
Process   State            Command                               
--------  ---------------  ------------------------------------  
web.1     up for 9m        bundle exec puma -p $PORT -C ./con..  
worker.1  up for 45s   bundle exec sidekiq                   
worker.2  up for 13s       bundle exec sidekiq

Initially the both 2 sidekiq processes are up and running normally. When I browse /sidekiq, I got error in heroku logs:

2012-04-09T11:00:41+00:00 heroku[worker.2]: Starting process with command `bundle exec sidekiq`
2012-04-09T11:00:42+00:00 heroku[worker.2]: State changed from starting to up
2012-04-09T11:00:51+00:00 app[worker.2]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)
2012-04-09T11:00:51+00:00 app[worker.2]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 INFO: Booting sidekiq 0.11.1 with Redis at guppy.redistogo.com:9366
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 INFO: Running in ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-linux]
2012-04-09T11:01:02+00:00 app[worker.2]: ERR max number of clients reached
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:47:in `call'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:24:in `connect'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:247:in `ensure_connected'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:435:in `block in sadd'
2012-04-09T11:01:02+00:00 app[worker.2]: /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:434:in `sadd'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/processor.rb:27:in `block in initialize'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.9.1/lib/connection_pool.rb:47:in `with'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq.rb:60:in `redis'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/util.rb:54:in `redis'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/processor.rb:27:in `initialize'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/calls.rb:56:in `dispatch'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:212:in `block in handle_message'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/task.rb:45:in `block in initialize'
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 ERROR: Manager#processor_died died
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 ERROR: task was terminated
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 ERROR: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/task.rb:23:in `suspend'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:45:in `call'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor_proxy.rb:76:in `method_missing'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid.rb:115:in `new_link'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/manager.rb:95:in `block in processor_died'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/util.rb:42:in `watchdog'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/manager.rb:91:in `processor_died'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:225:in `handle_exit_event'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:146:in `block in run'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/task.rb:45:in `block in initialize'

and my worker process status turn to crashed:

192-168-1-3:spree-thurley macbookair$ heroku ps
Process   State            Command                               
--------  ---------------  ------------------------------------  
web.1     up for 9m        bundle exec puma -p $PORT -C ./con..  
worker.1  crashed for 5m   bundle exec sidekiq                                     
worker.2  crashed for 34s  bundle exec sidekiq

and my web process complains about max number of clients reached:

2012-04-09T11:01:11+00:00 app[web.1]: 
2012-04-09T11:01:11+00:00 app[web.1]: 
2012-04-09T11:01:11+00:00 app[web.1]: Started GET "/sidekiq/queues/default" for 115.64.25.106 at 2012-04-09 11:01:11 +0000
2012-04-09T11:01:11+00:00 app[web.1]: RuntimeError - ERR max number of clients reached:
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:47:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:24:in `connect'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:247:in `ensure_connected'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:308:in `block in lrange'
2012-04-09T11:01:11+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:307:in `lrange'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/web.rb:127:in `block (2 levels) in <class:Web>'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.9.1/lib/connection_pool.rb:47:in `with'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq.rb:60:in `redis'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/web.rb:127:in `block in <class:Web>'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1212:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1212:in `block in compile!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:785:in `[]'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:785:in `block (3 levels) in route!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:801:in `route_eval'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:785:in `block (2 levels) in route!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:822:in `block in process_route'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:820:in `catch'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:820:in `process_route'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:784:in `block in route!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:783:in `each'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:783:in `route!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:886:in `dispatch!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:719:in `block in call!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:871:in `block in invoke'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:871:in `catch'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:871:in `invoke'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:719:in `call!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:705:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/web.rb:26:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/xss_header.rb:22:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/path_traversal.rb:16:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/json_csrf.rb:17:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/base.rb:47:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/xss_header.rb:22:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/nulllogger.rb:9:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/head.rb:9:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1334:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1416:in `synchronize'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1334:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.3/lib/journey/router.rb:68:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.3/lib/journey/router.rb:56:in `each'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.3/lib/journey/router.rb:56:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/routing/route_set.rb:594:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/warden-1.1.1/lib/warden/manager.rb:35:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/warden-1.1.1/lib/warden/manager.rb:34:in `catch'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/warden-1.1.1/lib/warden/manager.rb:34:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/head.rb:14:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/flash.rb:242:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/cookies.rb:338:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/query_cache.rb:64:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:443:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `_run__1803640746170885601__call__1928046963732201738__callbacks'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `__run_callback'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/rack/logger.rb:26:in `call_app'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/rack/logger.rb:16:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/request_id.rb:22:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/static.rb:61:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:49:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/engine.rb:479:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/application.rb:220:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/railtie/configurable.rb:30:in `method_missing'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/commonlogger.rb:20:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/server.rb:370:in `handle_request'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/server.rb:243:in `process_client'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/server.rb:157:in `block in run'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/thread_pool.rb:92:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:   /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/thread_pool.rb:92:in `block in spawn_thread'
2012-04-09T11:01:11+00:00 heroku[router]: GET smooth-leaf-8675.herokuapp.com/sidekiq/queues/default dyno=web.1 queue=0 wait=0ms service=65ms status=500 bytes=30
2012-04-09T11:01:11+00:00 app[web.1]: cache: [GET /sidekiq/queues/default] miss
2012-04-09T11:01:11+00:00 app[web.1]: 115.64.25.106 - - [09/Apr/2012 11:01:11] "GET /queues/default HTTP/1.1" 500 30 0.0585
2012-04-09T11:05:12+00:00 heroku[worker.1]: State changed from crashed to created
2012-04-09T11:05:12+00:00 heroku[worker.1]: State changed from created to starting
2012-04-09T11:05:23+00:00 heroku[worker.1]: Starting process with command `bundle exec sidekiq`
2012-04-09T11:05:24+00:00 heroku[worker.1]: State changed from starting to up
2012-04-09T11:05:34+00:00 app[worker.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)
2012-04-09T11:05:34+00:00 app[worker.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)

I am new to Sidekiq and Heroku so please let me know if I do something stupid here.

picture runlevel5

Most helpful comment

@kbighorse Did you specify the config file path for the worker in your Procfile?

worker: bundle exec sidekiq -C config/sidekiq.yml

either that, or specify the number of connections directly:

worker: bundle exec sidekiq -c 3
picture fixr on 6 Oct 2012
👍11 ❤4 🎉2

All 40 comments

You've hit the max number of Redis connections allowed by your plan: https://addons.heroku.com/redistogo

On 9 Apr 2012, at 04:10, Trung Lê[email protected] wrote:

Hi there

Firstly, please excuse me for posting on GH Issues. Because I am not so sure if this is an issue of Sidekiq or just something I did wrong on my setup.

I've deployed a simple Rails 3.2.2 app with Sidekiq Monitor app mount to /sinatra. I did not have any custom configuration, all are default Sidekiq.

My heroku has 2 worker processes:

$ heroku ps
192-168-1-3:spree-thurley macbookair$ heroku ps
Process   State            Command                               
--------  ---------------  ------------------------------------  
web.1     up for 9m        bundle exec puma -p $PORT -C ./con..  
worker.1  up for 45s   bundle exec sidekiq                   
worker.2  up for 13s       bundle exec sidekiq

Initially the both 2 sidekiq processes are up and running normally. When I browse /sidekiq, I got error in heroku logs:

2012-04-09T11:00:41+00:00 heroku[worker.2]: Starting process with command `bundle exec sidekiq`
2012-04-09T11:00:42+00:00 heroku[worker.2]: State changed from starting to up
2012-04-09T11:00:51+00:00 app[worker.2]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)
2012-04-09T11:00:51+00:00 app[worker.2]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 INFO: Booting sidekiq 0.11.1 with Redis at guppy.redistogo.com:9366
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 INFO: Running in ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-linux]
2012-04-09T11:01:02+00:00 app[worker.2]: ERR max number of clients reached
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:47:in `call'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:24:in `connect'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:247:in `ensure_connected'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:435:in `block in sadd'
2012-04-09T11:01:02+00:00 app[worker.2]: /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:434:in `sadd'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/processor.rb:27:in `block in initialize'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.9.1/lib/connection_pool.rb:47:in `with'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq.rb:60:in `redis'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/util.rb:54:in `redis'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/processor.rb:27:in `initialize'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/calls.rb:56:in `dispatch'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:212:in `block in handle_message'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/task.rb:45:in `block in initialize'
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 ERROR: Manager#processor_died died
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 ERROR: task was terminated
2012-04-09T11:01:02+00:00 app[worker.2]: 2012-04-09T11:01:02Z 1 TID-ue0i8 ERROR: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/task.rb:23:in `suspend'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:45:in `call'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor_proxy.rb:76:in `method_missing'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid.rb:115:in `new_link'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/manager.rb:95:in `block in processor_died'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/util.rb:42:in `watchdog'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/manager.rb:91:in `processor_died'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:225:in `handle_exit_event'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/actor.rb:146:in `block in run'
2012-04-09T11:01:02+00:00 app[worker.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.10.0/lib/celluloid/task.rb:45:in `block in initialize'

and my worker process status turn to crashed:

192-168-1-3:spree-thurley macbookair$ heroku ps
Process   State            Command                               
--------  ---------------  ------------------------------------  
web.1     up for 9m        bundle exec puma -p $PORT -C ./con..  
worker.1  crashed for 5m   bundle exec sidekiq                                     
worker.2  crashed for 34s  bundle exec sidekiq

and my web process complains about max number of clients reached:

2012-04-09T11:01:11+00:00 app[web.1]: 
2012-04-09T11:01:11+00:00 app[web.1]: 
2012-04-09T11:01:11+00:00 app[web.1]: Started GET "/sidekiq/queues/default" for 115.64.25.106 at 2012-04-09 11:01:11 +0000
2012-04-09T11:01:11+00:00 app[web.1]: RuntimeError - ERR max number of clients reached:
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:47:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:24:in `connect'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:247:in `ensure_connected'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:308:in `block in lrange'
2012-04-09T11:01:11+00:00 app[web.1]:    /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:307:in `lrange'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/web.rb:127:in `block (2 levels) in <class:Web>'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.9.1/lib/connection_pool.rb:47:in `with'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq.rb:60:in `redis'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/web.rb:127:in `block in <class:Web>'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1212:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1212:in `block in compile!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:785:in `[]'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:785:in `block (3 levels) in route!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:801:in `route_eval'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:785:in `block (2 levels) in route!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:822:in `block in process_route'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:820:in `catch'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:820:in `process_route'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:784:in `block in route!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:783:in `each'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:783:in `route!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:886:in `dispatch!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:719:in `block in call!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:871:in `block in invoke'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:871:in `catch'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:871:in `invoke'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:719:in `call!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:705:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sidekiq-0.11.1/lib/sidekiq/web.rb:26:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/xss_header.rb:22:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/path_traversal.rb:16:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/json_csrf.rb:17:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/base.rb:47:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.2.0/lib/rack/protection/xss_header.rb:22:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/nulllogger.rb:9:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/head.rb:9:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1334:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1416:in `synchronize'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1334:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.3/lib/journey/router.rb:68:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.3/lib/journey/router.rb:56:in `each'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.3/lib/journey/router.rb:56:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/routing/route_set.rb:594:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/warden-1.1.1/lib/warden/manager.rb:35:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/warden-1.1.1/lib/warden/manager.rb:34:in `catch'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/warden-1.1.1/lib/warden/manager.rb:34:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/head.rb:14:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/flash.rb:242:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/cookies.rb:338:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/query_cache.rb:64:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:443:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `_run__1803640746170885601__call__1928046963732201738__callbacks'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `__run_callback'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/rack/logger.rb:26:in `call_app'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/rack/logger.rb:16:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/request_id.rb:22:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.2/lib/action_dispatch/middleware/static.rb:61:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:49:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/engine.rb:479:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/application.rb:220:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/railtie/configurable.rb:30:in `method_missing'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/commonlogger.rb:20:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/server.rb:370:in `handle_request'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/server.rb:243:in `process_client'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/server.rb:157:in `block in run'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/thread_pool.rb:92:in `call'
2012-04-09T11:01:11+00:00 app[web.1]:    /app/vendor/bundle/ruby/1.9.1/gems/puma-1.1.1/lib/puma/thread_pool.rb:92:in `block in spawn_thread'
2012-04-09T11:01:11+00:00 heroku[router]: GET smooth-leaf-8675.herokuapp.com/sidekiq/queues/default dyno=web.1 queue=0 wait=0ms service=65ms status=500 bytes=30
2012-04-09T11:01:11+00:00 app[web.1]: cache: [GET /sidekiq/queues/default] miss
2012-04-09T11:01:11+00:00 app[web.1]: 115.64.25.106 - - [09/Apr/2012 11:01:11] "GET /queues/default HTTP/1.1" 500 30 0.0585
2012-04-09T11:05:12+00:00 heroku[worker.1]: State changed from crashed to created
2012-04-09T11:05:12+00:00 heroku[worker.1]: State changed from created to starting
2012-04-09T11:05:23+00:00 heroku[worker.1]: Starting process with command `bundle exec sidekiq`
2012-04-09T11:05:24+00:00 heroku[worker.1]: State changed from starting to up
2012-04-09T11:05:34+00:00 app[worker.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)
2012-04-09T11:05:34+00:00 app[worker.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/01/04/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/config/environment.rb:5)

I am new to Sidekiq and Heroku so please let me know if I do something stupid here.

Reply to this email directly or view it on GitHub:
https://github.com/mperham/sidekiq/issues/117

mperham picture mperham on 9 Apr 2012
👍6

@joneslee85 @mperham is right here.

I solved it by having just one worker and using a sidekiq.yml where the concurrency is about 25 connections.

The connection pool on the redis to go mini I think it about 50 connections.

https://github.com/mperham/sidekiq/blob/master/examples/config.yml

ivanacostarubio picture ivanacostarubio on 9 Apr 2012

Sidekiq server defaults to (concurrency + 2) redis connections per process. Sidekiq client running in your Rails process defaults to 5 connections per process.

mperham picture mperham on 9 Apr 2012
👍5

@mperham Thanks, I got it resolved.

runlevel5 picture runlevel5 on 10 Apr 2012

I'm using Redistogo Nano on Heroku. After clicking around the web client, takes a few clicks, but eventually get the max clients reached error.

# config/sidekiq.yml

# https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting
:concurrency:  3

Here's the log output: https://gist.github.com/3843341

Is there anything else I should post?

kbighorse picture kbighorse on 6 Oct 2012

@kbighorse Did you specify the config file path for the worker in your Procfile?

worker: bundle exec sidekiq -C config/sidekiq.yml

either that, or specify the number of connections directly:

worker: bundle exec sidekiq -c 3
fixr picture fixr on 6 Oct 2012
👍11 ❤4 🎉2

I also have this problem on Heroku with Redisto Nano, even with concurrency set to 1.

kendagriff picture kendagriff on 7 Oct 2012

Me too :(

sudden-break picture sudden-break on 12 Oct 2012

web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq -e production -c 3

this is Procfile I have with running sidekiq and Redistogo Nano. I had to specify environment '-e production' to make it work on heroku

mikecheverda picture mikecheverda on 12 Oct 2012

Hi, I'm getting the same problem here...

I copied Redis-to-Go conf file and run Redis locally with the same restrictions, 10 max connections.
Watching the log I noticed Sidekiq opens (concurrency + 2) Redis Connections, as @mperham said, plus 1 new connection to every new job (each "perform_async" call).
The point is after the job code have been executed the connection still active until the Redis close it as "idle client", long time after.

Is it right?

uchoaaa picture uchoaaa on 16 Oct 2012

@uchoaaa Yes, that's the client connection. Set the client pool size to 1, as explained in the wiki:

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end
mperham picture mperham on 16 Oct 2012

If you have a max of 10 connections, you want:

  • Set the client connection pool size to 1. You will need 1 connection per web process.
  • Assuming you have one Sidekiq process, set the server connection pool size to whatever you have left.

If you have 3 unicorn processes, you can do this:

# three unicorns = 3 connections
Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end
# so one sidekiq can have 7 connections
Sidekiq.configure_server do |config|
  config.redis = { :size => 7 }
end

You will want to use concurrency about 10 or so since you only have 5 connections for the workers (remember, 2 connections are required for the server internals).

NOTE This assumes that your application is using Redis for Sidekiq only. If you are using Redis for other things, you'll need to ensure those connections are accounted for also.

mperham picture mperham on 16 Oct 2012
👍4

Adding the pool size in an initializer seemed to fix this. Thanks! I'll post back if it in fact doesn't...

kendagriff picture kendagriff on 16 Oct 2012

Thanks @mperham, it seems ok now.

One more question, if I set the pool size to 1 and concurrency to 5 Sidekiq Client will create 5 threads and all 5 threads will share only one Redis connection, right? If so, this scenario could generate some issues, like race condition?
(actually I'm trying to understand the the relation of connections pool and concurrency :-)

uchoaaa picture uchoaaa on 17 Oct 2012

The client is your Rails app process. It is single threaded so you only need one connection per process. Concurrency has nothing to do with the client. The client does default to 5 connections per process, only because ActiveRecord's database connection pool also defaults to 5. In practice, :size => 1 should be ok.

The Sidekiq server is multi-threaded and runs many worker threads. The concurrency option adjusts the number of worker threads in the server process. The server will never allow less than 3 connections because you need 2 for sidekiq and at least one to share for the workers. If you set concurrency to 5, the server will default to creating 7 redis connections but you can set the connection pool size as low as 3 if necessary.

mperham picture mperham on 17 Oct 2012
❤8

@mperham man, you are awesome! Thanks, it's so clear now!

uchoaaa picture uchoaaa on 17 Oct 2012

Sorry, I'm still a bit confused, lots of "assumings", "ifs" and "at leasts" in this thread. Here is what is currently producing the max clients error for me:

Procfile:

web:    bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -c 5

config/initializers/sidekiq.yml:

# three unicorns = 3 connections
Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

# so one sidekiq can have 7 connections
Sidekiq.configure_server do |config|
  config.redis = { :size => 7 }
end

Ran 2 perform_async calls with 2 hits to /sidekiq, then reloaded /sidekiq in the web UI 2 more times and got "Internal Server Error" and the log shows the max clients error reached. Also got the same result with concurrency set to 3.

Maybe someone could distinguish the following nouns for me in the case of a Unicorn/Redistogo setup with 10 max connections: connection, thread (concurrency seems to be somewhat the same thing here?), worker, process. Which are static, and what causes which to increase or decrease as workers are run and the web UI is hit? And what happens when you say heroku scale worker=2?

kbighorse picture kbighorse on 18 Oct 2012

Ah, just checked my config/unicorn.rb and I have worker_processes=4, not 3 like I had thought. Trying now with 3. Still would love a noun distinction. How about this:

(Unicorn worker process) * (Client size) + (Server size) > 10 # error!

kbighorse picture kbighorse on 18 Oct 2012

Change :size => 7 to something lower like 4 or 5. Does that fix it?

RedisToGo limits your connections. All that matters are the # of processes and the size of the connection pool within those processes. Workers = Threads and they share the connections within the pool so you don't want a pool size of 2 when using concurrency of 25 as those 25 threads/workers will all fight over 2 connections.

worker=2 means you are increasing your server process count, so you need to multiple your server connection pool size by that amount. Pool size of 7 means 14 total connections used by 2 sidekiq processes.

mperham picture mperham on 18 Oct 2012

@kbighorse You're basically right: (Unicorn process count * client pool size) + (sidekiq process count * server pool size)

mperham picture mperham on 18 Oct 2012

Awesome, so everything looks good with these settings;

Redistogo Nano, 10 connections

config/unicorn.rb:

...
worker_processes 3
...

config/initializers/sidekiq.yml:

# three unicorns = 3 connections
Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

# so one sidekiq can have 5 connections
Sidekiq.configure_server do |config|
  config.redis = { :size => 5 }
end

Procfile:

web:    bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -c 5

Summary:

connection limit: 10
unicorn processes: 3
web dynos: 1
worker dynos (sidekiq server processes): 1
client pool size: 1
server pool size: 5
concurrency: 5

If I'm not maximizing my resources, let me know, right now I'm satisfied that it works. I'm guessing I could do server pool size of 6 and unicorn worker_processes of 4, which would give me 4 client connections + 6 server connections = 10. I do forget what increasing what the concurrency would do, though.

kbighorse picture kbighorse on 19 Oct 2012
👍1

Concurrency gives you more workers = doing more things at the same time. They fight over the redis connections though so you generally want concurrency no more than (server pool size - 2) * 2 or so, which works out to 6 in your case. So concurrency of 5 or 6 sounds right in your case.

mperham picture mperham on 19 Oct 2012

Just wanted to mention I've made a small tool to do the above calculations for you. Maybe it's more something for the wiki, but here it is http://manuel.manuelles.nl/sidekiq-heroku-redis-calc/

manuelvanrijn picture manuelvanrijn on 15 Nov 2012
❤10

Just a heads up, there is also a limit on AR connections to pg as well.

On Nov 15, 2012, at 5:48 PM, Manuel van Rijn <[email protected]notifications@github.com>
wrote:

Just wanted to mention I've made a small tool to do the above calculations for you. Maybe it's more something for the wiki, but here it is http://manuel.manuelles.nl/sidekiq-heroku-redis-calc/

—
Reply to this email directly or view it on GitHubhttps://github.com/mperham/sidekiq/issues/117#issuecomment-10402732.

seivan picture seivan on 15 Nov 2012

thanks @seivan I'll add a link how to fix this to the tool

manuelvanrijn picture manuelvanrijn on 15 Nov 2012

Thanks guys. Setting the initializer with a pool size of 5 worked for me. 7 seemed to max it.

Sidekiq.configure_server do |config|
config.redis = { :size => 5 }
end

DruRly picture DruRly on 16 Dec 2012

Is there a good document that explains all this? I can't seem to get a combination of settings that works for me. I'm running 'thin' on Heroku with just one worker process. Whether it works or not seems random.

procfile:

worker: bundle exec sidekiq -e production -c 3
web: bundle exec rails server thin -p $PORT -e $RACK_ENV

sidekiq.yml:

Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end

Sidekiq.configure_server do |config|
config.redis = { :size => 5 }
end

FlopTheNuts picture FlopTheNuts on 23 Dec 2012

I have the exact same issue as @FlopTheNuts. I don't fully understand how all the parts combine together, but based on what I've read the settings that FlopTheNuts has should work and they don't I get ERR max number of clients reached constantly, along with this:

The max pool size is currently 5; consider increasing it.

I have Redis mini with 50 connections and the CleaDB Drift plan with 30 connections to the Mysql database.

Redis server = 4
Redis client = 1
Sidekiq pool = 5
dynos running in a single job = 3
concurrency (bundle exec sidekiq -c 10) = 10

Mysql connections should be ok 3 * 5 = 15 but I don't know about the rest.

I am running Rails 3.2 with Thin so no mutlithreads.

I've been trying all sorts of combinations that should work according to what I've read but none of them do.

Even setting the concurrency value to 6 which @mperham says follows this formula: server pool size -2 * 2

Sidekiq.configure_server do |config|
  config.redis = {:size => 8} 
  ENV['DATABASE_URL'] = "#{database_url}?pool=5" 
bundle exec sidekiq -c 6

Constantly gives me the following error and lots of failed jobs that never get run again:

"error_message"=>"could not obtain a database connection within 5 seconds (waited 5.06928721 seconds). The max pool size is currently 5; consider increasing it.", "error_class"=>"ActiveRecord::ConnectionTimeoutError", "failed_at"=>2013-06-05 09:37:22 UTC, "retry_count"=>0}

Any help would be greatly appreciated I'm going crazy with this.

thanks.

kakubei picture kakubei on 5 Jun 2013

I'm not sure if you got to the bottom of this, but for me, I had to
allocate another worker on Heroku.

On Wed, Jun 5, 2013 at 5:18 AM, Alex [email protected] wrote:

I have the exact same issue as FlopTheNuts. I don't fully understand how
all the parts combine together, but based on what I've read the settings
that FlopTheNuts has should work and they don't I get ERR max number of
clients reached constantly, along with this:

The max pool size is currently 5; consider increasing it.

I have Redis mini with 50 connections and the CleaDB Drift plan with 30
connections to the Mysql database.

Redis server = 4
Redis client = 1
Sidekiq pool = 5
dynos running in a single job = 3
concurrency (bundle exec sidekiq -c 10) = 10

Mysql connections should be ok 3 * 5 = 15 but I don't know about the rest.

I am running Rails 3.2 with Thin so no mutlithreads.

I've been trying all sorts of combinations that _should work_ according
to what I've read but none of them do.

Any help would be greatly appreciated I'm going crazy with this.

thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com/mperham/sidekiq/issues/117#issuecomment-18964228
.

FlopTheNuts picture FlopTheNuts on 5 Jul 2013

Hello

I've read few issues and wiki pages. I still don't get it. We had the error yesterday (Redis::CommandError: ERR max number of clients reached) using Heroku Redis on Heroku with a maximum connection of 80.

What we have :
sidekiq.yml

production:
  :concurrency: 10

sidekiq.rb

Sidekiq.configure_client do |config|
  config.redis = { size: 3 }
end

Sidekiq.configure_server do |config|
  #no redis settings
end

We have 5 dynos

  • bundle exec puma -C config/puma.rb
  • worker bundle exec sidekiq -q default -q primary -q mailers -i ${DYNO:-1}
  • worker_primary bundle exec sidekiq -q primary -i ${DYNO:-1}
  • worker_stats bundle exec sidekiq -q aggregates_update -i ${DYNO:-1}
  • worker_visibility_and_notifications bundle exec sidekiq -q notifications -q visibility -i ${DYNO:-1}

We use Hirefire to scale and yesterday it fails with

  • Web at 4
  • worker at 4
  • worker_primary at 1
  • worker_stats at 0
  • worker_visibility_and_notifications at 1

What is the way to calculate how to stay under 80 connections?

Thanks in advance

benoittgt picture benoittgt on 5 Jan 2016

Heroku support told us to check also the timeout :

Redis' default behaviour is to never expire idle connections so you might want to close connections properly, or try setting a timeout value on your instance.

Don't know if it helps for me my issue. Any advices ?

benoittgt picture benoittgt on 5 Jan 2016

Sidekiq 4 now takes up to concurrency + 5 connections.

Web = 4 * 3 = 12
Worker = 15 * (4 + 1 + 1) = 90

Sidekiq never closes connections. It pools them so they can be reused when needed. It only opens a connection if needed so a worker process will only use 15 connections if it is really busy.

mperham picture mperham on 5 Jan 2016

Thanks a lot @mperham for the help.

When you write Web = 4 * 3 = 12 I also need to add the puma connections ? On my puma.rb I have MAX_THREADS = 20 and WEB_CONCURRENCY = 2

A good formula for determining the number of connections each application will require is to multiply the MAX_THREADS by the WEB_CONCURRENCY. This will determine the number of connections each dyno will consume.[[1](https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#database-connections)]

So It means I have to add this 40 to my web count ? `Web = 4 * 3 + 40 = 52 ?

Sorry quite out of thread.

benoittgt picture benoittgt on 6 Jan 2016

No, the 20 puma threads share the 3 client connections per process. That's the point of a connection pool.

mperham picture mperham on 6 Jan 2016

Great. Thanks @mperham

benoittgt picture benoittgt on 6 Jan 2016

someone, please add this math to the docs on concurrency so it's easier to calculate the concurrencies required. 

var web_connections   = (web_dynos * (client_conn * web_threads));
var concurrency  = (max_connections - web_connections - (redis_reserved * worker_dynos)) / worker_dynos;
var server_connections  = concurrency + redis_reserved;

source

adiakritos picture adiakritos on 16 Nov 2018

@adiakritos Why don't you add it?

mperham picture mperham on 16 Nov 2018
👍2

I added this info here https://github.com/mperham/sidekiq/wiki/FAQ#how-to-calcultate-the-number-or-redis-connection-used-by-sidekiq

Feel free to remove the link to the blog post if you think it should not be present on the wiki.

benoittgt picture benoittgt on 17 Nov 2018

@mperham I would but I'm not entirely sure of what I'm talking about yet.

Here's a possible draft to add into the wiki, otherwise feel free to leave it here for discussion.

Basically, this is my understanding of the whole setup and how to calculate concurrency in the simplest possible setup.

Use this calculator to figure out how to tune concurrency for your specific configuration.

The different components to understand when setting up your config are the following:

  1. The web process running your app's code
  2. The Sidekiq process
  3. The Redis storage

The first thing to get is that your web process is pushing things to the Redis storage, which is where your queue is physically located.

Once that's clear, then it's easier to see that your Sidekiq process will also connect to Redis to pull things "off" the queue.

So it's:

[Web Process] ---> [REDIS] <---- [Sidekiq]

Ok, now lets assume that you're running one Dyno in Heroku for your web process with 5 threads. Each of those threads counts as a single connection to redis. So you'd have 1 connection per thread to the redis servier, just from the 1 Dyno / Web process.

If you have 2 Dynos running your code, each process with 5 threads, now you have 10 threads, which is 10 connections to Redis.

So,

Web Process Connections To Redis = Web Dynos x Threads x Redis Connections (1)
Web Process Connections To Redis = 1 X 5 X 1
Web Process Connections To Redis = 5

Boom. So now we're telling Redis, listen we're gonna need you to handle at least 5 connections total from me at any given time.

Next, let's imagine the Redis service we're using only allows 20 total simultaneous connections.

At this point we know we're using up 5 of that Redis plan's 20 total possible connections. So we only have 15 left to connect to Redis with.

Ok, so now what about the Sidekiq process?

Well, by default, the Sidekiq process will command 2 connections to Redis. That means that our Redis service with 20 total connections is giving 5 connections to Web Process, and 2 to Sidekiq process. That's 7 total connections used up on our 20 connection plan. So we have a remaining 13 available connections to Redis.

This means you can set your Sidekiq Concurrencies to 13 in this case.

A formula for this situation is as follows: 

   max_connection  = 20 # this is our Redis plan's total allowed connections.
   web_dynos       = 1  # Running a single web dyno and web process for the app.
   web_threads     = 5  # I'm using puma with 5 threads so this is how many threads my Web process is using.
   client_conn     = 1  # The number of connections each web process thread makes to the Redis server.
   worker_dynos    = 1  # Processes running the sidekiq process.
   redis_reserved  = 2; # idk, just roll with it.

   web_connections     = (web_dynos * (client_conn * web_threads));
   concurrency         = (max_connections - web_connections - (redis_reserved * worker_dynos)) / worker_dynos;
adiakritos picture adiakritos on 18 Nov 2018
❤5 🎉1 👍1

@adiakritos That's all correct. Sidekiq requires concurrency + 2 connections per process. Client processes can use less connections, as low as 1. If your web process has 10-20 threads in it, I would recommend leaving the connection pool unsized since the default size is 5 -- the 10-20 client web threads will share 5 redis connections. That should be fine for 99% of apps.

Since @benoittgt already added content to the FAQ, I've leave it as is.

mperham picture mperham on 18 Nov 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

aglushkov picture aglushkov  Â·  3Comments
homanchou picture homanchou  Â·  3Comments
andrewhavens picture andrewhavens  Â·  4Comments
mperham picture mperham  Â·  3Comments
HenleyChiu picture HenleyChiu  Â·  4Comments