Sidekiq: [SOLUTION] Getting 403 Forbidden in Production - Rack-protection & Nginx

[codebender]

I was continuing to get a 403 Forbidden in Production with sidekiq version 3.5.0, but I was unable to reproduce the problem locally with the my app nor the sidekiq sample rails app. All signs pointed to SSL/https. Luckily I found this blog post: Rack-protection and Nginx by @mguentner.

After adding proxy_set_header X-Forwarded-Proto $scheme; to my nginx configuration. :boom:!! It worked!

@mperham Perhaps you should add a link in the Wiki that points to the blog post or the nginx config for those that use nginx and https

Many thanks to @mguentner

[mperham]

Nice! The wiki is publicly editable.

[mguentner]

I am glad that my post helped you! You are welcome :smirk: :muscle:

[rikas]

@codebender it's not working even after I add that line.

Here's my full configuration for the Rails app:

        location @app {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Host $http_host;
                proxy_redirect off;
                proxy_pass http://app;
        }

Any clues?

[mguentner]

Have you tried proxy_set_header X-Real-IP $remote_addr;?

[rikas]

@mguentner with that line I get "Forbidden" when trying to access the sidekick web interface. Before I was getting it only when trying to click action buttons inside "Retry" tab.

[juanibiapina]

We also get Forbidden, but we don't have nginx or anything.

[juanibiapina]

We don't have https either.

[codebender]

@juanibiapina Have you tried cloning the sidekiq repo, and running myapp? Can you reproduce the forbidden response there?

What kind of auth are you using? Can you paste your sidekiq config?

[juanibiapina]

No auth, no sidekiq config. I'll try running the app on our production servers.

[juanibiapina]

Ran the app on our production servers. No https, no auth. Same bug happens.

[juanibiapina]

We are behind a hipache proxy

[hendricius]

@codebender works. Thank you very much!

[rikas]

I'm also behind a haproxy, I guess that should be the problem @juanibiapina

[codebender]

I ran into the dreaded Forbidden when using _Chrome_ in the disable web security mode --disable-web-security. So another possible "fix" would be to make sure you are not using this mode.

Happy kiq'in!

[gotttlieb]

@codebender , the problem seems to be Chrome then, could you confirm this? ( i have checked opera and Firefox and they work as expected.)
And, if this issue is generated by chrome, you think that is a good way to disable sidekiq web security?
Anyway, i send feedback to chrome with no expectations of any response.
Thanks for your time.

[codebender]

@gotttlieb I was able to reproduce the problem when Chrome was in --disable-web-security mode. If that mode is turned off (the default) in Chrome, sidekiq web works just fine for me.

[907th]

@codebender @mguentner Thanks! Setting X-Forwarded-Proto in Nginx helped me!
My setup: Nginx (on 443 port/HTTPS) -> Unicorn (via socket) -> Rails -> Sidekiq::Web (mounted in routes.rb)
I think 403 error was due to the Rack::Protection::HttpOrigin activity.

[mguentner]

@907th thanks for the hint - I've updated the article which now links to the module :+1:

[907th]

@codebender @mguentner @mperham I've updated the wiki and added a note about Rack::Protection and links to possible solution for 403 Forbidden error (link to this issue and the article).

[SaiVardhan]

HI I am facing the issue when retrying to run a dead job in sidekiq for production env but it works fine in staging without preferred nginx settings .
I added the proxy settings as specified but still no luck

[ysynesis]

We using sidekiq gem on production server with nginx, haproxy, thin.
After authentication we try to open /sidekiq. Instead of seeing sidekiq ui we receive only "forbidden". Furthermore, we lost devise session. So if we try to open /admin, activeadmin ask for authentication again. We tried all tweaks for nginx mentioned before but still get signout and forbidden.

[jfgrissom]

Hi,

I just resolve this for a load balancer configuration where the infrastructure stack looks like this.

• F5 LTM Virtual Server (SSL terminates here on port 443).
  
  
  
  • Nginx listening on port 81 (No SSL proxying to unicorn via unix socket).
  
  
  • Unicorn (mounting sidekiq/web in the routes file)
  
  

To get this configuration to work the nginx proxy config for

proxy_set_header X-Forwarded-Proto $scheme;

needed to be set to

proxy_set_header X-Forwarded-Proto https;

Also, I put this at the top of the config/initializers/sidekiq.rb

require 'sidekiq/web'
Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]
Sidekiq::Web.set :sessions,       Rails.application.config.session_options

I've spent about 32 hours or actual trial and error brute forcing my way through this. This is what finally solved it for me.

I hope it helps someone else!

Take it easy,
Jay

[nzajt]

Thanks jfgrissom your fix worked for me as well, and I use puma with nginx, with an AWS ELB.

[ryansch]

@jfgrissom That has the potential to open up your sidekiq interface over http. It's usually better to set the X-Forwarded-Proto with the same component that's doing your ssl termination (F5 LTM Virtual Server in your case).

Example from our haproxy config:

  acl ssl_used dst_port 443
  acl ssl_used dst_port 8443

  http-request set-header X-Forwarded-Proto https if ssl_used
  http-request del-header X-Forwarded-Proto if !ssl_used

(We accept PROXY on 443 and standard TLS on 8443.)

[holyketzer]

I have proxy_set_header X-Forwarded-Protocol $scheme; in my Nginx config and thought that it was proxy_set_header X-Forwarded-Proto $scheme;, a lot of time was wasted to figure out why it wasn't worked. OMG I win anyway!

I hope this note can help somebody to fix the same issue in his Nginx config :)

[bhox]

This fixed it for me too!
After adding proxy_set_header X-Forwarded-Proto $scheme and restarting nginx, it immediately worked.
Thanks for reporting it. For the seo future: I was getting the "Forbidden" text in sidekiq web UI during retry/delete operations.

[pioz]

If you are using Apache add

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Host "example.com"

in your vhost conf file. Worked for me!

[ciddennis]

So I found I had this same kind of issue with using ELB in proxy protocol mode to a docker instance. This also allow shows you how to use action cable with ELB.

What I did was:

In session_store.rb

Rails.application.config.session_store :cookie_store, key: '_app-rails_session', domain: ".mydomain.com", tld_length: 2

In sidekiq.rb

require 'sidekiq'
require 'sidekiq/web'

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

In my nginx server config.

map $http_upgrade $connection_upgrade {
            default        "upgrade";
            ""            "";
        }

        server {
            listen 80 ;

            # Health Checkers do not need https so no redirect
            location  /health_checkers {
                proxy_pass            http://docker;
                proxy_http_version    1.1;

                proxy_set_header    Connection            $connection_upgrade;
                proxy_set_header    Upgrade                $http_upgrade;
                proxy_set_header    Host                $host;
                proxy_set_header    X-Real-IP            $proxy_protocol_addr;
                proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
            }
            # For redirect to HTTPS
            location / {
              return 301 https://$host$request_uri;
            }
        }

        # This is where all https will end up and we use the proxy protocol so websockets will work
        #
        # REMEMBER:
        #   This port must be open outbound of the ELB and inbound on the instance via the security groups.
        #   Run this commands on the ELB to get proxy protocal to work.
        #    aws elb create-load-balancer-policy --load-balancer-name awseb-e-3-AWSEBLoa-1GGGC \
        #          --policy-name EnableProxyProtocol --policy-type-name ProxyProtocolPolicyType \
        #          --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
        #    aws elb set-load-balancer-policies-for-backend-server --load-balancer-name awseb-e-3-AWSEBLoa-1GGGC \
        #          --instance-port 81 --policy-names EnableProxyProtocol
        #


        server {
            listen 9000 proxy_protocol;

            gzip on;
                gzip_comp_level 4;
                gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

            if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
                set $year $1;
                set $month $2;
                set $day $3;
                set $hour $4;
            }
            access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;

            access_log    /var/log/nginx/access.log;

           location /cable {
                proxy_pass            http://docker;
                proxy_http_version    1.1;

                proxy_set_header    Connection            Upgrade;
                proxy_set_header    Upgrade               websocket;
                proxy_set_header    Host                  $http_host;
                proxy_set_header    X-Real-IP             $proxy_protocol_addr;
                # proxy_set_header    X-Forwarded-For       $proxy_add_x_forwarded_for;
                proxy_set_header    X-Forwarded-For       $proxy_protocol_addr;
                proxy_set_header    X-Forwarded-Proto     https;
                proxy_redirect off;
            }

            location / {
                proxy_http_version    1.1;

                proxy_set_header    Connection            $connection_upgrade;
                proxy_set_header    Upgrade               $http_upgrade;
                proxy_set_header    Host                  $http_host;
                proxy_set_header    X-Real-IP             $proxy_protocol_addr;
                # proxy_set_header    X-Forwarded-For       $proxy_add_x_forwarded_for;
                proxy_set_header    X-Forwarded-For       $proxy_protocol_addr;
                proxy_set_header    X-Forwarded-Proto     https;

                proxy_pass          http://docker;
            }
         }

[pnomolos]

To those fighting with this, I've added some notes on the wiki to hopefully aid debugging: https://github.com/mperham/sidekiq/wiki/Monitoring#forbidden

It'd be nice to have a larger list of common pitfalls and how to solve them.

[k0a1a]

For those with Apache2 the fix is:

RequestHeader set X-Forwarded-Proto "https"

inside your block

[petr-duda]

How to solve it on Heroku? This does not work

[Mexxerio]

I'll just leave this here in case someone stumbles on here while trying to find a solution and maybe it can save you hours of trial and error with different nginx headers:

In addition to setting these nginx headers:

proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

You also have to add the following to either routes.rb or config.ru if you're getting a HttpOrigin Error from Rack Protection

require 'sidekiq/web'

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]
Sidekiq::Web.set :sessions, Rails.application.config.session_options
Sidekiq::Web.class_eval do
  use Rack::Protection, origin_whitelist: ['https://your.domain'] # resolve Rack Protection HttpOrigin
end

That did the trick for me.

[jpawlyn]

In addition to proxy_set_header X-Forwarded-Proto https; we also needed proxy_set_header X-Forwarded-Port 443; to resolve the "forbidden" problem.

[ismailakbudak]

How to solve it on Heroku? This does not work

I've faced same problem on heroku(free dynos) with custom domain. It is not solving real problem but it will save your time
https://stackoverflow.com/questions/47427511/heroku-sidekiq-error-when-kill-retry-job-403-forbidden-attack-prevented/53445684#53445684

[jonduarte]

I was getting forbidden error when trying to delete or retry a job. I was using nginx + puma.
I tried to replicate it on my local machine using docker and found out that it was failing because of the gem: rack-protection - precisely the module http_origin

Here is where the requested was halted: https://github.com/sinatra/sinatra/blob/master/rack-protection/lib/rack/protection/http_origin.rb#L30-L36:

      def accepts?(env)
        return true if safe? env
        return true unless origin = env['HTTP_ORIGIN']
        return true if base_url(env) == origin
        return true if options[:allow_if] && options[:allow_if].call(env)
        Array(options[:origin_whitelist]).include? origin
      end

it turns out, the comparison of same origin was failing. Basically my nginx config didn't pass down the proper HTTP_ORIGIN header.

Ex: my app is running on localhost:9292 - and my request come from example.com/sidekiq
this condition: base_url(env) == origin fails and halts the request.

Adding only the following config to nginx config solved the issue:

proxy_set_header    Host                  $http_host;

[aliahmed922]

I was having the same issue and literally I tried every workaround suggested on this issue but nothing ever worked on any of the browsers. The problem was simple, I wanted a basic rails authenticaton on sidekiq web and I had it by copying code from this comment but I forgot to read the serious gotcha which is mentioned in the next comment. This caused me this Forbidden issue and I am glad that everything is working ok now on production.