Actix-web: Strange behavior with an Accept-Encoding header of 'gzip'

[bschwind]

OS: MacOS 10.14.2
rustc: rustc 1.32.0 (9fda7c223 2019-01-16)

Dependency Declaration:

[dependencies]
actix = "0.7"
actix-web = {version = "0.7", features = ["ssl"]}

With this code, I receive an error:

let req = client::post(url)
    // .header("Accept-Encoding", "br, deflate") // This is okay
    .header("Accept-Encoding", "br, gzip, deflate") // This fails
    .form(&body)
    .unwrap();

For context, I'm doing an OAuth2.0 flow with google.

The error:

Payload(Io(Custom { kind: Other, error: StringError("Timeout while waiting for response") }))

Headers on the response for a failed request:

ClientResponse HTTP/1.1 200 OK
  headers:
    "content-type": "application/json; charset=utf-8"
    "vary": "Origin"
    "vary": "X-Origin"
    "vary": "Referer"
    "content-encoding": "gzip"
    "date": "Tue, 22 Jan 2019 13:57:18 GMT"
    "server": "ESF"
    "cache-control": "private"
    "x-xss-protection": "1; mode=block"
    "x-frame-options": "SAMEORIGIN"
    "x-content-type-options": "nosniff"
    "alt-svc": "quic=\":443\"; ma=2592000; v=\"44,43,39\""
    "transfer-encoding": "chunked"

Headers on the response for a successful request:

ClientResponse HTTP/1.1 200 OK
  headers:
    "content-type": "application/json; charset=utf-8"
    "vary": "X-Origin"
    "vary": "Referer"
    "vary": "Origin,Accept-Encoding"
    "date": "Tue, 22 Jan 2019 13:56:54 GMT"
    "server": "ESF"
    "cache-control": "private"
    "x-xss-protection": "1; mode=block"
    "x-frame-options": "SAMEORIGIN"
    "x-content-type-options": "nosniff"
    "alt-svc": "quic=\":443\"; ma=2592000; v=\"44,43,39\""
    "accept-ranges": "none"
    "transfer-encoding": "chunked"

Sample repo

https://github.com/bschwind/server-error-repro/blob/master/src/main.rs

Try swapping out the Accept-Encoding header lines to see the error. I haven't dug too far into the Actix source yet, I wanted to post this issue in case I'm doing something obviously wrong.

Thanks!

[DoumanAsh]

It seems the server is only capable of gzip compression.
As I understand you receive response headers, but while reading body you get timeout right?

[bschwind]

As I understand you receive response headers, but while reading body you get timeout right?

Correct! It ends up in this block:

https://github.com/bschwind/server-error-repro/blob/c4e9631c5ec16655b20ca5d6465f84408539ad43/src/main.rs#L121-L124

Error while fetching oauth token: Payload(Io(Custom { kind: Other, error: StringError("Timeout while waiting for response") }))

[bschwind]

Oh interesting, I bumped up the timeout and it actually finishes with the correct response body after 30s.

Edit: Ran it again in release mode, this time it took 55s

[DoumanAsh]

@bschwind can you try curl or other tools to see if it is the same? I wonder if server is just slow on returning chunked body

[bschwind]

@DoumanAsh I tried with the Insomnia REST client with the same headers and POST params as my code, and it returned in a reasonable amount of time.

[bschwind]

I enabled trace logging while reproducing the issue:

https://gist.github.com/bschwind/cd81a70f9fca76a90cd45630e34f7731

The issue seems to be with the chunked reading logic, need to investigate more.

[DoumanAsh]

Thanks, I'll try to take a look as soon as I'll have time

[bschwind]

No rush, I can omit gzip from the header for the time being

[DoumanAsh]

@bschwind I was trying to reproduce the behavior with your example repo, but it seems the request to google API fails, just wondering if you could let me know how to quickly setup client for this oauth service?

[bschwind]

Hey @DoumanAsh , sorry I forgot to mention, you'll need your own google API keys to properly reproduce the issue.

Follow the steps outlined here to get a CLIENT_ID and CLIENT_SECRET, then fill them in here in the code.

If you're running the test server at localhost:8080, you'll need to set that as an allowed redirect url http://localhost:8080/auth/google/callback (make sure to include the http:// protocol scheme)

If it's set up correctly, you should be able to visit http://localhost:8080/auth/google/redirect, which will redirect you to google. You can confirm you want to continue, and then google will redirect you back to your localhost:8080/auth/google/callback endpoint with a token.

Let me know if any of these steps are unclear. The trickiest part is properly setting up the google project, as they have quite a few options and are very picky about how the URLs and such are entered.

[DoumanAsh]

@bschwind Thanks, I'll try to reproduce it on my laptop to see what can go wrong

[BrandonArp]

I just ran into this problem as well. As a workaround, I had to specify an accept-encoding header of "identity". (Actually, anything but gzip works, but the spec to have no special encoding is identity).

I have created a git repo that demonstrates a simple repro of the problem: https://github.com/brandonarp/rust-repro

Uncommenting the line to set the accept-encoding header will cause a faster response that can be read.

[DoumanAsh]

Thanks, sorry I still haven't time to properly take a look at it.

[colelawrence]

Interesting. I encountered all of these exact issues in my Google OAuth application repository as well!
I'm glad to hear the suggestion to disable gzip!

[colelawrence]

I encounter this issue again while hitting the unofficial Google Play Music APIs. The same change (disabling gzip) solved this for me again.

[DGolubets]

I was playing with Google OAuth as well and had the same issue.
After some debugging I think I found the problem. Indeed it's not Gzip but rather Chunked-encoding bug.

Google replies with strange small chunks of size 1:

00000001\r\n\x1f\r\n00000001\r\n\x8b\r\n00000001\r\n\x08\r\n00000001\r\n\0\r\n00000001\r\n\0\r\n00000001\r\n\0\r\n00000001\r\n\0\r\n00000001\r\n\0\r\n00000001\r\n\x02\r\n00000001\r\n\xff\r\n00000001\r\n\xab\r\n00000001\r\n\xe6\r\n00000001\r\nR\r\n00000001\r\nP\r\n00000001\r\nP\r\n00000001\r\nJ\r\n00000001\r\n-\r\n00000001\r\n*\r\n00000001\r\n\xca\r\n01\r\n/\r\n41\r\nR\xb2RP\xca\xcc+K\xcc\xc9L\x89/J-,M-.Q\xd2\x81\xcb\xc6\xa7\xa4\x16'\x17e\x16\x94d\xe6\xe7\x81TzBT*8\x17\xa5\xa6\xa4\xe6\x95d&\xe6\x14+q\xd5\x02\0_\xea\x9f\x8eN\0\0\0\r\n

First, the decoder returns chunk by chunk asynchronously, even though the buffer has many chunks available (even all of them in case of a small response). Then the decompressor returns None cos it doesn't have enough data. Finally Pipeline returns NotReady and starts waiting for more data that will never come cos the whole body was read into the buffer at this stage.

I'll try to make a PR with a fix on the following week. Unless somebody is quicker :)

[fafhrd91]

@DGolubets could you test with master as well

[DGolubets]

@fafhrd91 I somehow missed 1.0 merge coming! I'll see later today if it reproduces any longer.

[DoumanAsh]

@DGolubets It would still be good to have it fixed on 0.7 so that we could release it before 1.0

[DGolubets]

I tested on the most recent master and got a PROTOCOL_ERROR.
I used same code as in https://github.com/actix/actix-http/blob/master/examples/client.rs (which works) but with Google Oauth2 URL (https://www.googleapis.com/oauth2/v3/userinfo?access_token=..).

[fafhrd91]

@DGolubets could you test again with awc https://docs.rs/awc/0.1.0-alpha.1/awc/

[fafhrd91]

fixed in 0.7 branch

[bschwind]

I've confirmed this is working with commit cc6e0c6d04e868e284c45b9ec38f1cc1fe98ac39, thanks so much for the fix!