Gunicorn: "Contradictory scheme headers" error when X-Forwarded-Protocol set to 'https'

[tisdall]

see #1766 for some people having the same issue

I have the following in my nginx config:

        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Protocol $scheme;

This causes an error because the default setting for secure_scheme_headers is {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}. When it compares the ssl to the https the error is thrown in https://github.com/benoitc/gunicorn/blob/19.9.0/gunicorn/http/message.py#L118

[tilgovi]

You're correct about the cause. The value for X-Forwarded-Protocol should be ssl but $scheme will be http or https. You can either change your nginx configuration to set X-Forwarded-Protocol to ssl conditionally, based on the value of $scheme; change secure_scheme_headers; or remove proxy_set_header X-Forwarded-Protocol altogether.

I'm not familiar with any software that sets X-Forwarded-Protocol (only X-Forwarded-Proto), so I can't say whether the default secure_scheme_headers are correct.

I don't see any bug in Gunicorn. Thankfully, this seems like it could be solved with configuration changes.

[tisdall]

From the reading I did, it seems that X-Forwarded-Protocol was from before X-Forwarded-Proto being the agreed standard. That would probably explain why some examples have "https" as a value and others have "ssl", there's no standard.

[tilgovi]

Makes sense!

[tisdall]

I guess there's not much to do here except maybe keep the issue open for a while for people who are upgrading from 19.7.1 and who happen to have a similar setup to mine. It seems Django used to state setting X-Forwarded-Protocol to https before Django 1.6 and I think that's why it was in my nginx config.

Would it be possible to expand on error message to say something like "X-Forwarded-Protocol was set to 'https' but that doesn't match the expected value in secure_scheme_headers"? I had no idea what "Contradictory scheme headers" meant when I first saw it (though, it makes sense in retrospect).

[tilgovi]

We could see about rewriting that code to show all the headers and their values. I'm happy to review pull requests for that!

Saying that the value doesn't match the expected value isn't quite enough information. It's okay if a value doesn't match, that is the expected case when a client connects without SSL. What the error message means is that some header matches and some header does not, which is what it's a contradiction. Some header says the connection is secure and some other header says the connection is not secure.

[berkerpeksag]

We could add a note about X-Forwarded-Protocol to our nginx configuration example:

https://github.com/benoitc/gunicorn/blob/master/examples/nginx.conf

We already have a couple notes and hints on how to configure nginx with Gunicorn in the file.

[tisdall]

Yeah, my first version of that comment had the value, but then I was wondering if that might be a security issue leaking that info. I guess it shouldn't be an issue if the value is hard coded in the proxy or otherwise unmodifiable by the end user.

[tisdall]

@berkerpeksag - I think the only issue is people with legacy configurations as no one really should be setting X-Forwarded-Protocol unless they're using old libraries that haven't adopted to the standard.

[tilgovi]

We might be fine to leave things as is. Our example nginx configuration should work. If people search for "Contradictory secure scheme headers" they'll land of one of the issues here and figure out what's going on.

[benoitc]

maybe adding a short note on our configuration file or in the deploying doc would worth it. Not sure about the phrasing. Any idea?

[benoitc]

bump

[tilgovi]

Either way. If someone wants to write it, I would review the PR, but I'm comfortable with the current documentation.

[tisdall]

What about changing the error message from "Contradictory scheme headers" to something more verbose?

[tilgovi]

If you'd like to make a PR, I would be happy to review it.

gunicorn.http.errors has a InvalidSchemeHeaders exception that right now has no arguments. It is created in gunicorn.http.message, where the conflicting values should be available to you.

[llybin]

If you nginx and app not on one host, probably, this solution: https://docs.gunicorn.org/en/stable/settings.html#forwarded-allow-ips

[tilgovi]

Thanks for the discussion everyone! As this has now been open for a while with no action, I'm going to close it. I will still happily review any documentation PRs.

[akondas]

If you are unable to change the nginx configuration, there is one more way to work around this problem.
Create config file (for example: gunicorn.py):

secure_scheme_headers = {'X-FORWARDED-PROTOCOL': 'https', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}

Then run gunicorn with this configuration:

gunicorn --config /path/to/gunicorn.py

Then everything will work fine without changing the proxy.
I hope I helped someone :wink: :beers:

[aisk]

You're correct about the cause. The value for X-Forwarded-Protocol should be ssl but $scheme will be http or https. You can either change your nginx configuration to set X-Forwarded-Protocol to ssl conditionally, based on the value of $scheme; change secure_scheme_headers; or remove proxy_set_header X-Forwarded-Protocol altogether.

I'm not familiar with any software that sets X-Forwarded-Protocol (only X-Forwarded-Proto), so I can't say whether the default secure_scheme_headers are correct.

I don't see any bug in Gunicorn. Thankfully, this seems like it could be solved with configuration changes.

Hi @tilgovi , I saw mozilla / aws / nginx's documents said that, x-forwarded-proto should be https instead of ssl.