Sdk: dart:io HttpClient's request force-transforms the header names to lowercase

[isoos]

The toLowerCase() transformation may be useful when processing the response headers, but when sending, the server could be checking the cased version and fail if it doesn't find it that way. (Yeah, bad server.)

[zoechi]

Sounds like #25120

[isoos]

Yeah, very similar problem.

[maguilar2k]

If the RFC-2616 states that header names are case-insensitive... why force the header names transformation to lowercase? shouldn't that be a decision taken by the developer?

[isoos]

I think the reason for the lowercase transform is HTTP/2: it standardizes all header names to be lowercase, and with that it makes somewhat sense to do the same here.

[shaxxx]

I've spent half a day debugging to find out http client forces headers as lowercase.
Guess what, I'm stuck with 10 year old client devices with HTTP server's using case sensitive headers.
Why force anything? Let me decide what I want to send with MY request

[HoLLy-HaCKeR]

Same here, I ended up copying over a bunch of code from the SDK after debugging a broken WebSocket connection for a few days. I get the rationale but this isn't documented anywhere which is really annoying.

[shaxxx]

Since I can't say to thousands of users to update their receivers or fix their HTTP server implementation, and I don't expect Dart team to change this behavior, I had no other choice but to change default HTTP client.

I took current HttpClient from master branch (requires Dart 2.5, Flutter 1.8+), changed it to leave headers alone and packed in the package. Hopefully it will save someone someone from crying and pulling up their hair. Just use it as a drop in replacement for HttpClient. Tested and works with Dio.

You can find package HERE.

[a-siva]

There seems to be a suggestion that we should add an option to not change the case.

[zichangg]

@lrhn Any idea? This would be a breaking change if adding one more flag.

[zichangg]

To be more specific, A flag is probably needed to be passed from httpclient all the way down to httpheader.
https://github.com/dart-lang/sdk/blob/a1fdfb8fc8abb758f17c070cebfea3026ca53973/sdk/lib/_http/http_impl.dart#L2071
https://github.com/dart-lang/sdk/blob/a1fdfb8fc8abb758f17c070cebfea3026ca53973/sdk/lib/_http/http_impl.dart#L770
https://github.com/dart-lang/sdk/blob/a1fdfb8fc8abb758f17c070cebfea3026ca53973/sdk/lib/_http/http_impl.dart#L1078

Inside headers,
https://github.com/dart-lang/sdk/blob/4102e63f378ea585c6c6e02107e60017912aadde/sdk/lib/_http/http_headers.dart#L41

https://github.com/dart-lang/sdk/blob/4102e63f378ea585c6c6e02107e60017912aadde/sdk/lib/_http/http_headers.dart#L44

https://github.com/dart-lang/sdk/blob/4102e63f378ea585c6c6e02107e60017912aadde/sdk/lib/_http/http_headers.dart#L44

[shaxxx]

Thats the one way to look at it.
Why would HttpClient assume anything?
You have bunch of Abstract classes in HTTP client that are used only as internal implementations. Take HttpHeaders for instance. It's abstract, and HttpClient uses one single concrete implementation, which is internal, and has no way to use some other implementation.
Why making abstract class in a first place if you're forcing it to be used in a single way?
Good and probably non breaking start would be to

• move internal implementations to separate public classes/files
• refactor existing calls to new classes
• add a way for user to provide specific implementation of abstract class (not just HttpHeaders)
• if no specific implementation is provided use default one

Last two can be done as optional class constructor with default value params (better) or as public properties (less transparent IMHO).
I'm speaking from top of my head now, but this shouldn't breake anything.
Dart has no reflection, and internal calls can be easily refactored.
And once you're there, in default HttpHeaders expose map of all headers as public property, leave current behaviour, as is, but let user modify the map directly however he/she wants. That's extra, I'd be fine if I could make HttpClient use my HttpHeaders instead default ones.

Just my 2 cents.

[songfei]

any update?

[zichangg]

Still in progress. I expect this cl to be landed within next few weeks.

[shaxxx]

Is there any ETA when this will land in a Dart release?

[zichangg]

The cl is pretty much done. But it has to wait for breaking change request getting approved.

[cdm2012]

Creating an API Client using HTTPClient with the following request headers:

I/flutter (32611): headers: {
I/flutter (32611):   "Content-type": "application/json",
I/flutter (32611):   "Accept": "application/json",
I/flutter (32611):   "Authorization": "Bearer: E7HlYmfiXwhLQT1nYApOAE4wqhvCZZXndrT7kiRv8tUBycj3npVnYwnQnH34"
I/flutter (32611): }
I/flutter (32611): 
I/flutter (32611): 
I/flutter (32611): 
I/flutter (32611): request.headers: 
I/flutter (32611): content-type: application/json
I/flutter (32611): accept: application/json
I/flutter (32611): authorization: Bearer: E7HlYmfiXwhLQT1nYApOAE4wqhvCZZXndrT7kiRv8tUBycj3npVnYwnQnH34

And the server (nginx/1.15.8) is returning a 400 error:

I/flutter (32611): <html>
I/flutter (32611): <head><title>400 Bad Request</title></head>
I/flutter (32611): <body>
I/flutter (32611): <center><h1>400 Bad Request</h1></center>
I/flutter (32611): <hr><center>nginx/1.15.8</center>
I/flutter (32611): </body>
I/flutter (32611): </html>

Using the same headers in android with the following request headers:

E: Called doInBackground w/ headers: [Accept: application/json, Content-type: application/json, Authorization: Bearer E7HlYmfiXwhLQT1nYApOAE4wqhvCZZXndrT7kiRv8tUBycj3npVnYwnQnH34]

And the same server is returning a 200 response, so I'm really looking forward to integrating the change by @ZichangG : https://github.com/dart-lang/sdk/issues/39657#issue-533108500

[accek]

I/flutter (32611):   "Authorization": "Bearer: E7HlYmfiXwhLQT1nYApOAE4wqhvCZZXndrT7kiRv8tUBycj3npVnYwnQnH34"

It looks like the colon : after Bearer should not be there.

[cdm2012]

Thanks @accek, that was definitely an oversight. I just tried it again with these headers:

I/flutter ( 3473): headers: {
I/flutter ( 3473):   "Content-type": "application/json",
I/flutter ( 3473):   "Accept": "application/json",
I/flutter ( 3473):   "Authorization": "Bearer E7HlYmfiXwhLQT1nYApOAE4wqhvCZZXndrT7kiRv8tUBycj3npVnYwnQnH34"
I/flutter ( 3473): }
I/flutter ( 3473): 
I/flutter ( 3473): 
I/flutter ( 3473): 
I/flutter ( 3473): request.headers: 
I/flutter ( 3473): content-type: application/json
I/flutter ( 3473): accept: application/json
I/flutter ( 3473): authorization: Bearer E7HlYmfiXwhLQT1nYApOAE4wqhvCZZXndrT7kiRv8tUBycj3npVnYwnQnH34

But the result was still the same:

I/flutter ( 3473): responseString: <html>
I/flutter ( 3473): <head><title>400 Bad Request</title></head>
I/flutter ( 3473): <body>
I/flutter ( 3473): <center><h1>400 Bad Request</h1></center>
I/flutter ( 3473): <hr><center>nginx/1.15.8</center>
I/flutter ( 3473): </body>
I/flutter ( 3473): </html>

I didn't think to add the nginx logs before, but here are all the nginx logging from last night:

2019/12/29 03:15:05 [error] 3653#3653: *65 open() "/var/www/reely-api/testing/public/api/version" failed (2: No such file or directory), client: 10.0.0.5, server: reely.the-mac.local, request: "POST /api/version HTTP/1.1", host: "reely.the-mac.local"

Currently there are no logs showing, but I think that error was from using the wrong url protocol.

[sortie]

@accek We're working to make the breaking change, but it's a workaround for bad servers. The standard requires the protocol headers are case insensitive and you get the server fixed instead of debugging the issue here. I assume those bearer tokens are for debugging and have no actual access.

[cdm2012]

Yes the token is not usable @accek , it only worked for a private internal server. Could the solution be to use the HTTP/2 standard optionally @isoos , similar to @ZichangG 's solution (but an opt in solution)?

@sortie I looked into the Server side, and according to the creator of Laravel, Laravel literally does not touch request headers in any form or fashion.

[DartMen]

Issues are to be expected when forcing things like these..

Same as dictating the multipart_request boundary to "dart-http-boundary-xxxx" in https://github.com/dart-lang/http/issues/295

[mit-mit]

Landed in https://github.com/dart-lang/sdk/commit/d39cdf03f83a4b76d0bda122b3745109fcd03299

[MiriDevAndro]

anyone has find an solution ?

[isoos]

@MiriDevAndro: If I understand correctly, the fix has been landed, it may be in the 2.8.0-dev branch already, and it will be in 2.8.0 stable.

[MiriDevAndro]

@isoos yes i have see but i dont have this path/file sdk_nnbd/lib/_http/http.dart in my sdk , i use windows andoid studio and i have download sdk from official page

[MiriDevAndro]

Landed in d39cdf0

there is no such dir like sdk_nnbd/lib/_http/http.dart on my sdk

[a-siva]

@MiriDevAndro the fix has landed in sdk/lib/_http/http.dart the other path (sdk_nnbd) that you refer to is an internal branch of the libraries that we are using for the null safety feature.

[MiriDevAndro]

@MiriDevAndro the fix has landed in sdk/lib/_http/http.dart the other path (sdk_nnbd) that you refer to is an internal branch of the libraries that we are using for the null safety feature.

why then is not working , i have modify all like this d39cdf0 comment above

[a-siva]

What is not working, can you send us a test case that still exhibits the problem,
It looks like you are not using the sdk as is and instead have your own version of the build into
which you are applying this patch, it is possible there is some inconsistency there.

[MiriDevAndro]

@a-siva im using flutter in android studio ,how can i fix lowercase headers?

[mit-mit]

@MiriDevAndro you might have an older version of the Dart SDK that doesn't have this change. If you run flutter --version, the Dart SDK needs to be 2.8.0-dev.10.0 or later.

[MiriDevAndro]

@mit-mit thnx for answers , can i use dart sdk 2.8.0-dev.10.0 + in android studio manually or not will work?

[mit-mit]

If you are doing Flutter development, you need to upgrade your Flutter SDK. That contains an embedded Dart SDK inside it (as reported by flutter --version).

[MiriDevAndro]

The flutter channel dev dart sdk 2.8.0-dev.15.0 has to much bug and stable has ver dart sdk 2.7.0 max

[mit-mit]

Sorry, you will have to wait for the dev channel to stabilize then.

[MiriDevAndro]

ok thnx for explanation

[eirikb]

I'm also struggling with a server only accepting cased headers.
How can I use the above solution?

Tried:

final body = '{"hello":"world"}';
final client = new HttpClient();
final req = await client.put('localhost', 8000, '/');
req.headers.add('Content-Type', 'application/json', preserveHeaderCase: true);
req.headers.add('Content-Length', body.length, preserveHeaderCase: true);
req.write(body);
await req.close();

Versions:

> flutter --version
Flutter 1.15.21 • channel master • /home/eirikb/.cache/yay/flutter-git/flutter
Framework • revision e2b4edd286 (3 weeks ago) • 2020-03-13 02:46:02 -0400
Engine • revision d7a00b8b09
Tools • Dart 2.8.0 (build 2.8.0-dev.14.0 eb9c26bd37)

But headers are still lower-cased.
Tested by running Flutter in Android Emulator and dart in terminal:

> /opt/flutter/bin/cache/dart-sdk/bin/dart --version
Dart VM version: 2.8.0-dev.14.0.flutter-eb9c26bd37 (Fri Mar 13 01:41:41 2020 +0000) on "linux_x64"
> /opt/flutter/bin/cache/dart-sdk/bin/dart api_test.dart

Output from nc (also see same result in Wireshark):

> nc -l -p 8000
PUT / HTTP/1.1
user-agent: Dart/2.8 (dart:io)
content-type: application/json
accept-encoding: gzip
content-length: 17
host: localhost:8000

{"hello":"world"}

Printing headers show they were correctly set in HttpHeaders, but not sent correctly:

print(req.headers);

user-agent: Dart/2.8 (dart:io)
Content-Type: application/json
accept-encoding: gzip
Content-Length: 17
host: localhost:8000

[shaxxx]

@mit-mit I'm having exactly the same issue as @eirikb .
preserveHeaderCase as far as I can see is currently not working.
Using

Dart VM version: 2.8.2 (stable) (Mon May 11 15:06:42 2020 +0200) on "windows_x64"

Can someone please explain what's current status on this, is it working, if yes how, if not what's happening with it? Should this be re-opened?

[zichangg]

@shaxxx Sorry for the delayed reply. Code reviews for my fix took some time, I'll land the fix today.

[shaxxx]

@zichangg Thanks for the update, as far I can understand preserveHeaderCase should work in the next flutter stable?
It wasn't really clear from this thread if this is currently working or not since no one confirmed this is still an issue and this bug is closed. Is there another bug tracking progress on this?

[a-siva]

reopening issue so that we can track when the bug fix CL lands and update on when it gets rolled into Flutter. You will probably see it in the dev channel first.