Sidekiq: Forbidden when clicking on retry

[Micka33]

The use case

My sidekiq web interface, and sidekiq process, are 2 different rails projects (both connecting to the same redis instance obviously).

When in production mode a weird behavior happens.
If one of the jobs fails, and I want to manually retry it, using any of these buttons , I am redirect on a white page with only Forbidden on it, and my devise session is destroyed.
This does not happen in development mode.

URL when clicking retry: http://localhost:3000/sidekiq/retries/1439457271.0547059-2294ccdf2a6c8594801f1dfc
page content: Forbidden

Environment

Ubuntu 14.04.2 LTS
ruby 2.2.1p85
rails (4.2.3)
sidekiq (3.4.2)
  celluloid (~> 0.16.0)
  connection_pool (~> 2.2, >= 2.2.0)
  json (~> 1.0)
  redis (~> 3.2, >= 3.2.1)
  redis-namespace (~> 1.5, >= 1.5.2)
devise (3.5.2)
   bcrypt (~> 3.0)
   orm_adapter (~> 0.1)
   railties (>= 3.2.6, < 5)
   responders
   thread_safe (~> 0.1)
   warden (~> 1.2.3)

Initializer

template = ERB.new File.new(File.join(Rails.root, 'config', 'sidekiq.yml')).read
raw_yaml = YAML.load template.result(binding)
SIDEKIQ_CONFIG = raw_yaml.merge(raw_yaml[Rails.env])

Sidekiq.default_worker_options = { 'backtrace' => true }

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
  config.redis = {
      :url       => SIDEKIQ_CONFIG[:url],
      :namespace => SIDEKIQ_CONFIG[:namespace],
      :size      => SIDEKIQ_CONFIG[:client_connections]
  }
end

require 'sidekiq/web'
require 'sidetiq/web'
require 'sidekiq-status/web'

sidekiq.yml

default: &default
  :verbose: true
  :pidfile: tmp/pids/sidekiq.pid
  :url: redis://localhost:6379/0
  :client_connections: 10
  :concurrency: 10
  :timeout: 5
  :queues:
    - default

<<: *default


development:
  <<: *default
  :pidfile: tmp/pids/sidekiq_dev.pid
  :namespace: dev

production:
  <<: *default
  :pidfile: tmp/pids/sidekiq_prod.pid
  :logfile: ./log/sidekiq_prod.log
  :namespace: prod
  :verbose: false
  :url: <%= "redis://#{ENV['REDIS_IP'] || '127.0.0.1'}:6379/0" %>
  :concurrency: 17
  :client_connections: 17

Routes

  authenticate :user do
    mount Sidekiq::Web => '/sidekiq'
  end

[mperham]

Did you see the 3.4.2 release notes?

On Aug 13, 2015, at 02:34, Mickaël CASSY [email protected] wrote:

The use case

My sidekiq web interface, and sidekiq process, are 2 different rails projects (both connecting to the same redis instance obviously).

When in production mode a weird behavior happens.
If one of the jobs fails, and I want to manually retry it, using any of these buttons , I am redirect on a white page with only Forbidden on it, and my devise session is destroyed.
This does not happen in development mode.

URL when clicking retry: http://localhost:3000/sidekiq/retries/1439457271.0547059-2294ccdf2a6c8594801f1dfc
page content: Forbidden
Environment

Ubuntu 14.04.2 LTS
ruby 2.2.1p85
rails (4.2.3)
sidekiq (3.4.2)
celluloid (~> 0.16.0)
connection_pool (~> 2.2, >= 2.2.0)
json (~> 1.0)
redis (~> 3.2, >= 3.2.1)
redis-namespace (~> 1.5, >= 1.5.2)
devise (3.5.2)
bcrypt (~> 3.0)
orm_adapter (~> 0.1)
railties (>= 3.2.6, < 5)
responders
thread_safe (~> 0.1)
warden (~> 1.2.3)
Initializer

template = ERB.new File.new(File.join(Rails.root, 'config', 'sidekiq.yml')).read
raw_yaml = YAML.load template.result(binding)
SIDEKIQ_CONFIG = raw_yaml.merge(raw_yaml[Rails.env])

Sidekiq.default_worker_options = { 'backtrace' => true }

Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
config.redis = {
:url => SIDEKIQ_CONFIG[:url],
:namespace => SIDEKIQ_CONFIG[:namespace],
:size => SIDEKIQ_CONFIG[:client_connections]
}
end

require 'sidekiq/web'
require 'sidetiq/web'
require 'sidekiq-status/web'
sidekiq.yml

default: &default
:verbose: true
:pidfile: tmp/pids/sidekiq.pid
:url: redis://localhost:6379/0
:client_connections: 10
:concurrency: 10
:timeout: 5
:queues:
- default

<<: *default

development:
<<: *default
:pidfile: tmp/pids/sidekiq_dev.pid
:namespace: dev

production:
<<: *default
:pidfile: tmp/pids/sidekiq_prod.pid
:logfile: ./log/sidekiq_prod.log
:namespace: prod
:verbose: false
:url: <%= "redis://#{ENV['REDIS_IP'] || '127.0.0.1'}:6379/0" %>
:concurrency: 17
:client_connections: 17
Routes

authenticate :user do
mount Sidekiq::Web => '/sidekiq'
end
—
Reply to this email directly or view it on GitHub.

[Micka33]

3.4.2

• Don't allow Sidekiq::Worker in ActiveJob::Base classes. [#2424]
• Safer display of job data in Web UI [#2405]
• Fix CSRF vulnerability in Web UI, thanks to Egor Homakov for
  reporting. [#2422] If you are running the Web UI as a standalone Rack app,
  ensure you have a session middleware
  configured:

use Rack::Session::Cookie, :secret => "some unique secret string here"

---

In #2460, I see a reference to https://github.com/rails/rails/issues/15843 where you posted the following :

Warning: use at your own risk. Monkeypatching is never a great idea, especially a private method. Put this in config/initializers/session_store.rb:

# Monkeypatch necessary due to https://github.com/rails/rails/issues/15843
require 'rack/session/abstract/id'
class Rack::Session::Abstract::SessionHash
  private
  def stringify_keys(other)
    hash = {}
    other = other.to_hash unless other.is_a?(Hash) # hack hack hack
    other.each do |key, value|
      hash[key.to_s] = value
    end
    hash
  end
end

---

Even though I'm using the latest versions of all my dependencies, is this the workaround for now ?

[mperham]

Sinatra must have a working session store. I don't know how to enable that for your specific environment.

[dalibor]

Had the same issue when using Sidekiq::Web mount.
Fixed it by upgrading rack-protection to > 1.5.1.

[mrnugget]

I just wanted to add that we ran into the same issue. Key components:

rails 4.2.5
sidekiq 4.0.1
sidekiq-pro 3.0.0
devise 3.5.2

Sidekiq was mounted via mount Sidekiq::Web => 'sidekiq' in our Rails application. Submitting forms in the Sidekiq UI failed with the "Forbidden" message and no backtrace.

And the solution by @dalibor fixed it. We were on rack-protection 1.5.0 and updating to rack-protection 1.5.3 solved the issue for us.

Should I add this to the relevant wiki entry, @mperham?

[mrnugget]

Update: the solution above wasn't a "solution". Our logout broke and we still had two session cookies.
But what finally fixed it completely was reusing the Rails session in the mounted sinatra app.

So:

1. Update to rack-protection 1.5.3
2. Setup Sidekiq::Web so it uses the same session as the hosting Rails application:

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

The sessions_options line was especially important to us, since we have a domain option in there.

[syedhassan]

@mrnugget where do I set

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

[mrnugget]

@syedhassan Wherever you want. I set these in config/initializers/sidekiq.rb where I also configure Sidekiq.

[emilkarl]

Doing a request to /sidekiq workouts and I get the sidekiq panel but it logs out my user if do it in the same browser. I am using activerecord-session-store gem.

# config/initializers/sidekiq.rb
require 'sidekiq/web'
Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

# config/initializers/session_store.rb
Rails.application.config.session_store :active_record_store, key: '_app_session'

[DrMavenRebe]

I'm getting a similar error @emilkarl. Did you ever find a solution?

[emilkarl]

@DrMavenRebe: Yes, it is the session store. Seems like the session key is the same for my login system and sidekiq. Each sidekiq request resets or changes this and makes the "auth session" for the page invalid.

My solution for now is to use different browsers for it. Im using Chrome to check the system and Safari to check Sidekiq status. That way I can use both at the same time.

[nitzanav]

This worked for me:

Sidekiq::Web.set :session_secret, ENV['SESSION_SECRET']
run Sidekiq::Web

[gabeodess]

Note that in Rails 4.2+ the token has changed to secret_key_base and can be accessed by Rails.application.secrets.secret_key_base.

That being said, @mrnugget's solution doesn't seem to be working for me.

[sandstrom]

How can one disable/remove Rack::Protection in Sidekiq?

[mperham]

You cannot. You are welcome to open a new issue to explain your problem.

[peterwillcn]

Add Sidekiq::Web.set :session_secret, Rails.application.secrets.secret_key_base to config/initializers/sidekiq.rb

[sandstrom]

Thanks @peterwillcn! Our issue is that we've disabled sessions in Rails entirely, so what you suggest won't solve our particular issue (but it would work with some other types of problems around Rack::Protect).

# we've disabled session store in Rails (it's an API-only system)
Rails.application.config.session_store(:disabled)

[AndyMental]

We are accessing sidekiq web ui from ip where its actually running..

http://139.xx.xx.xx/admin/sidekiq/..
in case you are using ssl.. you can configure a subdomain to point to this link..

Our setup is multiple rails applications with separate database instance.. we have redis enabled on all the servers however we faced the same issue and about work around works for us.