Httpx: Header Normalization

[Lokraan]

https://github.com/encode/httpx/blob/e19bd9bc4bf5849650a3847232a4b1353600635e/httpx/utils.py#L34

why, why make it lower case.

The website I'm sending the request to wants a signature of the request under the header "ACCESS-SIGN", specifically, in all caps.

Everything else works except for the headers being normalized to all lower case.

I'm going to clone, remove the issue, and then install it locally to solve this issue: but I think this is something that should be fixed on the public codebase.

I don't understand why you normalize the headers, if coders are making capitalization typos then maybe they should learn not to do that.

[StephenBrown2]

Issue already raised here: https://github.com/encode/httpx/issues/538
Seems like you've got another real-world example of a server not following the spec.

@tomchristie PRs welcome, I'm sure?

[florimondmanca]

Yes, going to close this as a duplicate of #538.

PRs to change this behavior are much welcome, as I think your use case demonstrates that we can’t just expect real world systems to follow standards. HTTP is messy! 😅

[StephenBrown2]

Seems like the reason for the lowercasing is actually to follow the HTTP/2 spec, the HTTP/1.1 spec is just looser on that.

[tomchristie]

HTTP headers are specified as case insensitive, so it’s valid behaviour, but yeah we’d like to change it.
It’ll be more awkward that just changing it in our codebase tho, since h11 (which we use for actually sending the requests) normalises headers to lower case too.

[VeNoMouS]

I just discovered the same thing... when dealing with anti bot tech like cloudflare... this MATTERS...

[tomchristie]

@VeNoMouS I'd like for us to be able to preserve header casing in both directions while exposing a case-insensitive interface onto the headers yes, although we need to figure out if the h11 maintainers are happy to support that, and how.

Doing a somewhat shouty "this MATTERS" isn't really appropriate or neccessary. Talking though your "when dealing with anti bot tech like cloudflare" situation would be more helpful, since it'd give us more context to take to the h11 team when talking to them about this.

[VeNoMouS]

sorry @tomchristie, was just trying to emphasize the context of the issue, didn't mean to rub anyone up the wrong way

I've been waiting to move cloudscraper to httpx for h2 support since last year... but i can't because of small issues like this.

I do appreciate the groups collective work and not trying to step on anyone's toes, if you recall I've been here since you guys started the project.

I've got nothing but ❤️ for you guys.

[tomchristie]

Sure thing, no worries at all. It's also more complex because it's not only down to us.

The h11 library that we're using does a really good job of being correct-to-the-letter of the spec, and is very carefully designed. That happens to also mean that it lowercases HTTP headers in both directions, in order to ensure case-insensitivity. (Case-sensitive handling of headers is a protocol violation, athough it does actually occur plenty in the wild.)

They've had some discussion about relaxing that, but there's not a decision on the specfics of that yet. https://github.com/python-hyper/h11/issues/31

What'd be helpful in moving the discussion forward would be if you're able to be very specific about exactly what you need from header casing, so that they're in a better position to take a call on "do we only need to preserve header casing on the outgoing request headers" (which is something they've considered) vs. "preserve header casing on requests and on responses".

So I'd start by breaking this down...

I just discovered the same thing... when dealing with anti bot tech like cloudflare...

Gotcha, so what specific behavior did you see, and what aspect was problematic?

Thanks! 😃