Sidekiq: Getting the dreaded `Forbidden` message

[nwhobart]

ruby 2.1.8p440 (2015-12-16 revision 53160) [x86_64-linux]
sidekiq 4.1.1
Phusion Passenger 5.0.26 (Standalone)
Rails 4.1.1

require 'sidekiq/web'
Sidekiq::Web.use Rack::Session::Cookie, secret: ENV['RACK_SESSION_COOKIE']
map '/sidekiq' do
  use Rack::Auth::Basic, 'Protected Area' do |username, password|
    md5_of_password = Digest::MD5.hexdigest(password)
    username == 'user' && md5_of_password == ENV['SIDEKIQ_HASH']
  end

  run Sidekiq::Web
end

Since the update to 4.1.1 I'm getting a Forbidden message from what looks like nginx when trying to retry jobs, delete jobs, kill jobs, etc. Login and everything else seems to work fine. Not sure what else to do. I've tried my best to follow the suggestions from people having similar issues but nothing has worked so far.

Of note we're using AWS Elasticbeanstalk.

[nwhobart]

I've removed the
Sidekiq::Web.use Rack::Session::Cookie, secret: ENV['RACK_SESSION_COOKIE']
line and it's still giving me Forbidden

[mperham]

There's several closed issues that show how to share the session between Rails and Sinatra.

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

Let us know if that works.

[nwhobart]

@mperham thanks for the consolidation. i really thought i put in a rather deep-dive on the closed issues. i didn't come up with such a concise answer. this seems to be working. again thank you for your help.

[epintos]

I'm having the same issue with Elastic Beanstalk 64bit Amazon Linux 2016.03 v2.1.0 running Ruby 2.2.4 (Passenger Standalone) and Sidekiq 4.1.1.

[PFStein]

@epintos I am a dev that works with @hobakill. I was investigating this issue after he opened a pull request into our application to add the new sidekiq configuration for 4.1. I noticed that the monitoring documentation and the code provided by @mperham uses Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token] which is not actually standard for a rails 4 application.

Rails 3 used secret_token as a base to encrypt and verify session cookies but Rails 4 uses secret_key_base to encrypt session cookies. Essentially, the code will most likely be, in your case if you are running Rails 4.*, Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]

Interesting enough, using the above code for one of our applications actually solved our problem. It allowed us to stop receiving 403 errors when trying to cancel Sidekiq jobs from the monitoring client even though the session_secret was set to nil in Sidekiq::Web. I am interested to know if anyone else has encountered such an issue when upgrading.

[epintos]

@PFStein Adding the following to config.ru doesn't solve the problem:

require 'sidekiq/web'
Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]
Sidekiq::Web.use Rack::Auth::Basic do |username, password|
  username == Rails.application.secrets.sidekiq_username &&
  password == Rails.application.secrets.sidekiq_password
end

Do you have something different?

[PFStein]

@epintos Sorry about the confusion. Try this:

require 'sidekiq/web'
Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]
Sidekiq::Web.set :sessions, Rails.application.config.session_options
Sidekiq::Web.use Rack::Auth::Basic do |username, password|
  username == Rails.application.secrets.sidekiq_username &&
  password == Rails.application.secrets.sidekiq_password
end

[epintos]

@PFStein Still not working :/. I've tried it in my server and locally. Still having the Forbidden error

[PFStein]

@epintos I am sorry to hear you are still having trouble! If you open up a rails console and type the following command: Rails.application.secrets.include?(:secret_key_base) Does it return true?

[epintos]

@PFStein Yeah.. it returns true, and the session_options returns {:key=>"_my-domain_session", :cookie_only=>true}

[JanStevens]

I'm having the same issue:

ruby: 2.3.0
rails: 4.6.0
Sidekiq v4.1.1 / Sidekiq Pro v3.2.1
devise: 3.5.7

We tried the configuration from @PFStein and this doesn't work. I even try to set everything manually (instead of using Rails.application.config.session_options) including the extra options specified in https://github.com/rack/rack/blob/master/lib/rack/session/cookie.rb#L23

Conclusion Forbidden error still pops up, browsing around works as expected but doing one action gives you a forbidden. With those "session sharing" configured I always get logged out after a forbidden error (which is very .... helpful).

I feel that this issue is never really well addressed and the current mechanism for detecting "authenticated users" is really broken, a fix would be helpful. I rather configure a lambda/proc with the request object and do my own authentication then this, excuse my language, clusterfuck between sinatra / rails / sessions / rack. Or insert my own middleware that handles authentication.

Also beware when using subdomains and a top domain cookie (that is shared between the subdomains). In Rails you have to specify domain: :all, sintra will blow up because it expect domain to be a string

[mike-ball]

We have the same issue. I'm not sure why this is marked as Closed, since it still a problem.

Sidekiq web Posts work fine in development, but when we deploy to staging or production on AWS Elastic Beanstalk, every Sidekiq Post request results in Forbidden.

I have tried all the ideas in this thread.
ruby 2.2.3
Rails 4.2.5.1
sidekiq 4.1.2
puma 2.16.0
nginx 1.8.0

We've actually had this problem since we switched from delayed_job about 3 months ago. I can't figure out why it won't work in staging or production. Any ideas would be greatly appreciated.

[mperham]

@mike-ball It's Closed because it's not a Sidekiq issue that I can fix. It's a configuration issue between Rack, Sinatra and your app. You must provide Sidekiq::Web with a valid session. Since everyone's app and session config is slightly different, there's no one way to solve this problem.

[mike-ball]

@mperham That's makes sense.

I'm still hopeful that someone on here can point me in a helpful direction. I thought my app was pretty standard, but maybe something in my middleware is a bit off. The app has been upgraded from Rails 2- > 3 -> 4.0- > 4.2.

Here are the values for my secret_key_base and session_options. I have obviously replaced the strings with A, B, or C. You can see that all the values are different. Is that normal?

Rails.application.secrets[:secret_token]
=> "A
Rails.application.secrets[:secret_key_base]
=> "B"

Rails.application.config.session_options
 => {:key=>"_hal_session", :secret=>"C", :cookie_only=>true} 

(I have tried using both secret_token and secret_key_base, but I don't see any change.)

The values are the same for my development, staging, and production. I can't figure out why it's working on development (using Pow), but not in staging or production on AWS Elastic Beanstalk.

[mperham]

Fire up pry and start debugging. This is the middleware in question.

[iMacTia]

@mike-ball http://www.sourcediver.org/blog/2015/07/01/rack-protection-and-nginx/
Had your same problem, this fixed it!
@mperham Could be worth including it in the troubleshooting? I understand it's probably an edge-case, but I spent days before finding it.

In case the link stops working, just add the following to your nginx config:

proxy_set_header X-Forwarded-Proto $scheme;

[mperham]

@iMacTia It's covered in #2560. A search for "Forbidden" will bring up these issues but I encourage you to update the wiki to point to these issues so others can find them. I don't want to turn the wiki into "Here's how to configure a dozen different webservers" to properly proxy.

[TheJefe]

I was still getting this issue after making all of these suggested changes. The fix for me was to upgrade our sinatra gem from 1.3.6 to 1.4.7.

[dorongutman]

We've also been getting that forbidden error and couldn't interact with anything.
Nothing of the suggested fixes have worked.
The only thing that solved it was updating sinatra from 1.4.3 to 1.4.7 (which updated rack-protection from 1.5.0 to 1.5.3).

Hope this helps anyone!

[AdrianTeh]

Thanks @dorongutman

[zeitnot]

@iMacTia 's solution, that is, setting proxy_set_header X-Forwarded-Proto $scheme; fixes the problem.

[sgeef]

Maybe a very unrelated comment for people using sidekiq in Rails, but if your running Sinatra Standalone, and you use Rack basic auth. assure to set the above mentioned session secert:
Sidekiq::Web.set :session_secret, "MyAwsomeSecret"

I looked over it as i saw "Rails", but in fact it's a sidekiq setting and solved the issue for me. Thumbs up fro the comment above.

[kirqe]

There's several closed issues that show how to share the session between Rails and Sinatra.

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

Let us know if that works.

This should be added to the wiki

[mperham]

So add it.

On Aug 19, 2020, at 08:59, Kirill B. [e-mail] wrote:


There's several closed issues that show how to share the session between Rails and Sinatra.

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options
Let us know if that works.

This should be added to the wiki

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.