Okhttp: Recover from REFUSED_STREAM in HTTP/2

[swankjesse]

OkHttp promises to persevere when there’s trouble, but it doesn’t recover from REFUSED_STREAM in HTTP/2. We should transparently retry when an HTTP/2 server returns a REFUSED_STREAM error.

We should set noNewStreams = true on the connection that refused the stream. That is a good enough policy that’s in spirit of the HTTP/2 spec: it’ll prevent the connection from being used for any future streams. Then when we retry we’ll get a different connection and with any luck that one will accept the stream.

We need a special case if the refused stream’s streamId is 1. That will let us recover when Nginx refuses request bodies sent before the settings ack. In that case we should retry on the same physical connection, which we can assume has since ACK’d the settings. It’s still a little bit racy, but unlikely to be a problem in practice. More discussion on this workaround on issue 2506 and on the http-wg list.

[zyfmix]

I'm okhttp3.2 with Nginx1.10.0, can you tell me how to set noNewStreams = true in my code to make it work:

public class CoamAuthInterceptor implements Interceptor {
    @Override
    public Response intercept(Chain chain) throws IOException {
        Request originalRequest = chain.request();
        Response originalResponse = chain.proceed(originalRequest);

        if (!originalResponse.isSuccessful()) {
        }

        return originalResponse;
    }
}

Thank you very much!

[swankjesse]

After getting a response from the http-wg list, our recovery policy can be simpler: attempt one retry on the same connection, and then permanently fail the request.

[swankjesse]

(And do that one retry on a different connection if the current connection is shutting down.)

[dave-r12]

@swankjesse if you didn't already discover this, one other thing I noted when running experiments with nginx is that even if the timing works out, it's possible OkHttp will never send the request. I traced it to this line, https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/framed/FramedConnection.java#L278

I think the headers will need to be flushed prior to writing the data frames.

[rfc2822]

Is this an okhttp or an nginx problem? People are starting to become nervous because clients which use okhttp (like DAVdroid) don't work with their (up-to-date) nginx servers anymore… how do you recommend to handle this? Disable HTTP2 in okhttp?

[swankjesse]

Disabling HTTP/2 in OkHttp will work. It's sad, but it'll work. Both OkHttp and Nginx are going to make code changes for this issue.

[HLFH]

@rfc2822 I love HTTP/2. I temporarily switched to dmfs.org.

[dave-r12]

A quick update: nginx attached a proposed patch on their issue tracker. I tested patch against OkHttp and it worked perfectly.

[dangfan]

@dave-r12 Would you please give a link of the changeset? I cannot find it in trac.nginx.org. Thanks!

[raoulbhatia]

@terro https://trac.nginx.org/nginx/ticket/959#comment:14

[rfc2822]

A quick update: nginx attached a proposed patch on their issue tracker. I tested patch against OkHttp and it worked perfectly.

So, should the problem go away with nginx 1.10.1 or will we still need to upgrade okhttp to get it working?

[dave-r12]

@rfc2822 it seems nginx will provide a configuration option for the preread buffer. From the commit message:

If the directive's value is lower than the default initial window (65535),
as previously, all streams with data will be rejected until the new window
is acknowledged. Otherwise, no special processing is used and all requests
with data are welcome right from the connection start.

If you change it to lower than default initial window, you'll need to upgrade to 3.3. If you leave it default, everything should just work (this is the behavior most clients expect.)

I'm not familiar with their development process, so it's unclear when that change will get merged in.

[dave-r12]

Also, OkHttp 3.3 will be compatible with nginx 1.9.15, if you must use that version.