Okhttp: java.io.IOException: stream was reset: CANCEL and PROTOCOL_ERROR

[sreekumarr84]

I am using okhttp to connect to google OAuth server (https://accounts.google.com/o/oauth2/token). I am occasionally getting "stream was reset" error. I was using org.apache library earlier and never faced this issue before. First I am getting "stream was reset: CANCEL" and on retry "stream was reset: PROTOCOL_ERROR". On retrying after some time the connection works fine.

java.io.IOException: stream was reset: CANCEL
at com.squareup.okhttp.internal.framed.FramedStream.getResponseHeaders(FramedStream.java:146)
at com.squareup.okhttp.internal.http.FramedTransport.readResponseHeaders(FramedTransport.java:141)
at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909)
at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:93)
at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:894)
at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:748)
at com.squareup.okhttp.Call.getResponse(Call.java:277)
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:234)
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:196)
at com.squareup.okhttp.Call.execute(Call.java:79)

and later-on on retry

java.io.IOException: stream was reset: PROTOCOL_ERROR
at com.squareup.okhttp.internal.framed.FramedStream.getResponseHeaders(FramedStream.java:146)
at com.squareup.okhttp.internal.http.FramedTransport.readResponseHeaders(FramedTransport.java:141)
at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909)
at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:93)
at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:894)
at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:748)
at com.squareup.okhttp.Call.getResponse(Call.java:277)
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:234)
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:196)
at com.squareup.okhttp.Call.execute(Call.java:79)

[swankjesse]

Looks like an HTTP/2 or SPDY issue. You can workaround by disabling HTTP/2 like this:

    client = client.newBuilder()
        .protocols(Arrays.asList(Protocol.HTTP_1_1))
        .build();

We should investigate to see why Google doesn’t like our frames.

[dave-r12]

@sreekumarr84 could you elaborate more on how you are using google Oauth server? I put together a simple project here, https://github.com/dave-r12/google-oauth2-example. Could you take a look and see if I'm roughly on the right path? It doesn't seem to be using Http2/SPDY for me, which I'm still investigating. I'm personally in uncharted waters, so apologies if I'm missing something obvious

[swankjesse]

@dave-r12 my guess is that it requires Jetty’s ALPN .jar to be enabled to be reproduced. For me that means adding the following when I launch a Java 7 JVM:

 -Xbootclasspath/p:/Users/jwilson/.m2/repository/org/mortbay/jetty/alpn/alpn-boot/7.1.2.v20141202/alpn-boot-7.1.2.v20141202.jar

See the Building doc for other exotic stuff like this!

[dave-r12]

Gotcha - thanks

[dave-r12]

No luck so far. One thing I just realized is the use of FramedTransport. I think this means either OkHttp 2.5.0 or 2.6.0

[swankjesse]

Yeah. My guess is there’s a bug in our handling of HTTP/2 error handling. Or perhaps in Google’s!

[vaurdan]

I'm having the same problem connecting with my own API using Volley and OkHttp (2.4.0) Stack. My server is running HTTP/2/SPDY, so that may be the problem.

[swankjesse]

@vaurdan what’s your webserver running? Do you see the same problem on OkHttp 2.7.2 or 3.0.1 ?

[vaurdan]

@swankjesse I'm using nginx with a Let's Encrypt certificate. I haven't tested 2.7.2 yet, I will try it later and I will let you know if it worked.
OkHttp 3 will not work for me, as I'm dependent on the OkHttp2 API

[swankjesse]

Which nginx ?
https://publicobject.com/2015/10/19/okhttp-http2-nginx-1-9-5/

[vaurdan]

It's happening on nginx/1.9.9. So it can be the same problem as reported in
that blog post.

On Mon, Jan 18, 2016 at 4:13 PM, Jesse Wilson [email protected]
wrote:

Which nginx ?
https://publicobject.com/2015/10/19/okhttp-http2-nginx-1-9-5/

—
Reply to this email directly or view it on GitHub
https://github.com/square/okhttp/issues/2204#issuecomment-172573533.

[RyanSpeaking]

In version 2.3, 2.4, it might have the same problem.

This issue could be across 3 feature/bug fix.
I found this on Android L and M version.

And this could be also related to many similar issues discussed here before.

Provide to you for reference.

1.
commit a8327d121b9eeb3b61ae63f06095d423936a5da2
Author: Neil Fuller [email protected] Sat Nov 29 01:38:53 2014
Committer: Neil Fuller [email protected] Fri Dec 19 00:19:06 2014
Fix for a socket leak in OkHttp on Android

In this commit, it added ConnectionPool.performCleanup() to schedule a periodic idle connection cleanup.

2.
commit 6f6d959fdae64065521990413f86bc7ab038d7c1
Author: Marcelo Cortes [email protected] Fri Feb 1 04:14:15 2013
Committer: Marcelo Cortes [email protected] Fri Feb 1 06:23:06 2013
Improvements to connection pool

In this commit, when SpdyConnection.removeStream(), it will setIdle(true) to the connection.

  synchronized SpdyStream removeStream(int streamId) {
    SpdyStream stream = streams.remove(streamId);
    if (stream != null && streams.isEmpty()) {
      setIdle(true);
    }
    notifyAll(); // The removed stream may be blocked on a connection-wide window update.
    return stream;
  }

3.
commit 6e382aa9b807c61adc88eb0a2cddc047382e36d9
Author: jwilson [email protected] Sun Aug 18 08:01:33 2013
Committer: jwilson [email protected] Tue Aug 20 02:09:34 2013
Original File: okhttp-protocols/src/main/java/com/squareup/okhttp/internal/spdy/SpdyStream.java
Read HTTP/2.0 frames and error codes.

In this commit, when SpdyStream.closeLater(), it still need to
connection.writeSynResetLater(id, errorCode) to send data
after closeInternal().

But the connection is set to idle (connection.removeStream()) and might be cleaned by performCleanup().

  public void closeLater(ErrorCode errorCode) {
    if (!closeInternal(errorCode)) {
      return; // Already closed.
    }
    connection.writeSynResetLater(id, errorCode);
  }

  private boolean closeInternal(ErrorCode errorCode) {
    assert (!Thread.holdsLock(this));
    synchronized (this) {
      if (this.errorCode != null) {
        return false;
      }
      if (source.finished && sink.finished) {
        return false;
      }
      this.errorCode = errorCode;
      notifyAll();
    }
    connection.removeStream(id);
    return true;
  }

Such issue will cause exceptions:
1.socket is closed
2.stream was reset: PROTOCOL_ERROR/CANCEL

Possible solution could be:

1. Enhance connection pool cleanup, such as extending cleanup period.
2. Don't setIdle() to a connection when it will be needed later.

[swankjesse]

I think this is obsoleted by our new stream allocations model.

[RyanSpeaking]

@swankjesse , could you be more specific what the commit id or link that the new stream allocation model lies in ?

[swankjesse]

It’s here: https://github.com/square/okhttp/commits/okhttp_27/okhttp/src/main/java/com/squareup/okhttp/internal/http/StreamAllocation.java

[MegatronKing]

The same problem happened in Http2, the okhttp version is 3.8.0, does it have a solution?

okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL
at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:153)
at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:135)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:818)

[omerlh]

I have the same issue, but I am running a bit outdated version - 3.2.0. I cannot update the version above 3.6.0 because of React Native - Facebook/react-native#13996 . My server is IIS 10, not nginx so it not the same issue I guess. This is the callstack:

okhttp3.internal.framed.StreamResetException: stream was reset: CANCEL
okhttp3.internal.framed.FramedStream.getResponseHeaders(FramedStream.java:145)
okhttp3.internal.http.Http2xStream.readResponseHeaders(Http2xStream.java:145)
okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
com.asurion.solutohome.util.restAdapterWrappers.ApiClientHeadersInterceptor.intercept(ApiClientHeadersInterceptor.java:18)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
okhttp3.RealCall.execute(RealCall.java:60)

[jaredsburrows]

@swankjesse @MegatronKing How did you get around this issue?

[shubham1g5]

@swankjesse Repinging on this as we are having the same issue after enabling HTTP2 (via ALPN) on server side. We are using nginx version 1.12.2 and okHTTP 3.8.0. Do you have any other thoughts on why stream reset and protocol errors can occur or is there anything I can do to further debug this issue. Thanks!

[prasannakumark]

+1

[leokelly]

We can resolve this by remove duplicated http header. When you enable HTTP2 prior okhttp 2.6.0, To send two identically named headers, Okhttp will send a header with two values, where the values are separated by a single NUL (0) byte. In SPDY this is right,but in HTTP2 shouldn't do this.
In okhttp 2.6.0 http://androidxref.com/7.1.1_r6/xref/external/okhttp/okhttp/src/main/java/com/squareup/okhttp/internal/http/FramedTransport.java
In okhttp 2.7.5 http://androidxref.com/8.0.0_r4/xref/external/okhttp/okhttp/src/main/java/com/squareup/okhttp/internal/http/Http2xStream.java

In okhttp 2.7.5, the HTTP2.0 header and SPDY header was dealed differently.

[andre-ss6]

Has anyone found a solution to this? Is there a workaround?

[DennisVanAcker1]

Still occuring. I'm encountering it using the Fetch library for Android to download files.

[skrzepto]

Also experiencing this issue with 3.14.2 if you force http/1.1 I have't seen the connection reset error again.

[samG4]

@DennisVanAcker1 Did you find any solution to this? I am facing the same issue as well.

[dingshaoran]

same issue version 3.10.0