Okhttp: Http2Stream to have independent errors for request and response streams

[sinhaashish]

I am using minio for Object storage. Minio is a golang project using go1.12 as the base go version.
The server is HTTP2 enabled. I use using minio java client which uses okhttp3 . when i try to store the object in a bucket in multi threaded mode i get okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR.
Refer https://github.com/minio/minio/issues/7501 .

okhttp3.OkHttpClient uses http2.0 by default which causes such issue.
If i forcefuly set the protocol to 1.1 ,it works fine.
Refer PR https://github.com/minio/minio-java/pull/766.

I find there is some issue with Go1.12 or okhttp3 as i found this issue in the community.
Refer https://github.com/square/okhttp/issues/3955
The only solution provided there was to revert to HTTP1_1 .

Setting the protocol to 1.1 is not the solution as i am not using the functionality of HTTP2.0 enabled server.
Below is the code which replicates the problem

```
import io.minio.MinioClient;
class PutObjectRunnable implements Runnable {
MinioClient client;
String bucketName;
String filename;

public PutObjectRunnable(MinioClient client, String bucketName, String filename) {
    this.client = client;
    this.bucketName = bucketName;
    this.filename = filename;
}

public void run() {
    StringBuffer traceBuffer = new StringBuffer();

    try {

        client.putObject(bucketName, filename, filename);

    } catch (Exception e) {
        System.err.print(traceBuffer.toString());
        e.printStackTrace();
    }
}

}

```

```import io.minio.*;

public class ThreadedPutObject {
public static void main(String args[]) throws Exception {
try{
MinioClient client = new MinioClient("https://play.min.io:9000", "Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG", true);
client.traceOn(System.out);
long startTime = System.currentTimeMillis();
boolean found = minioClient.bucketExists("my-bucketname");
if (found) {
System.out.println("my-bucketname already exists");
} else {
// Create bucket 'my-bucketname'.
minioClient.makeBucket("my-bucketname");
System.out.println("my-bucketname is created successfully");
}
Thread[] threads = new Thread[7];
String[] location = new String[]{"<>",
"<>",
"<>",
"<>",
"<>",
"<>",
"<>" };

  for (int i = 0; i < 7; i++) {
        PutObjectRunnable pr = new PutObjectRunnable(client, "my-bucketname",location[i] );
      threads[i] = new Thread(pr);
  }
  for (int i = 0; i < 7; i++) {
    threads[i].start();
  }
        // Waiting for threads to complete.
  for (int i = 0; i < 7; i++) {
     threads[i].join();
      System.out.println(i);
    }
    // All threads are completed.
    } catch (Exception e) {
        throw e;
    }
    System.out.println("uploaded");
}

}

[yschimke]

@sinhaashish Is there a sample project we can check out and run in Intellij? Or a minimial test reproducing the problem in a single page? e.g. a curl command against https://play.min.io:9000

[sinhaashish]

@yschimke We can not use the curl command to replicate the issue because using curl will directly hit the server and we have the issue in Minio-java sdk which uses okhttp3. I have reported the same bug to Go Language and they have pointed some issue with the client.
Refer

[sinhaashish]

Note: While trying to replicate the issue please use minio-java version: '6.0.5' as the later versions has Protocol forcefully set to 1.1 which wont replicate the issue.
Maven Repo https://mvnrepository.com/artifact/io.minio/minio/6.0.5

[Drkstr]

Hi, is that an update on this? I have a similar issue.
Using http1.1 gives me a "javax.net.ssl.SSLException" error.

[swankjesse]

@Drkstr - executable test case?

[Drkstr]

Sorry, not sure how to provide you with an executable test case.
I am building a mobile app and I am using an API built by another team.
Is there any info I can provide you with that will help with this issue?

[swankjesse]

Steps to reproduce!

[Drkstr]

This is the API:
```
@Multipart
@POST("v1/patients/{uid_slug}/images/")
fun sendScreening(@Path(value = "uid_slug") patient_uid: String,
@Part imagePart: MultipartBody.Part): Call>

Here is how I am building the Retrofit client

private val okHTTPClient = OkHttpClient() .newBuilder()
.addInterceptor(authInterceptor)
.addInterceptor(logging)
.build()

 ```
val retrofit = Retrofit.Builder()
                .baseUrl(BASE_URL)
                .addConverterFactory(MoshiConverterFactory.create())
                .client(okHTTPClient)
                .build()

I am trying to upload an image:

val file: File = File(patientImagePhotoPath)
val filePart = MultipartBody.Part.createFormData("file", file.name, RequestBody.create(MediaType.parse("image/*"), file))

RetrofitClient.getInstance(null).sendScreening(patinetUid, filePart) .enqueue(...)

This is what I see in the log file:

D/OkHttp: <-- HTTP FAILED: okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR

[harshavardhana]

@swankjesse @yschimke were you able to reproduce this? let us know if you need any help in reproducing this issue.

[yschimke]

I'll see if I can at least reproduce it. But don't consider me actively working on this at the moment. It doesn't seem too surprising we might see some issues in this area. The investigation on the server shows it's a fairly atypical use case compared to hitting a known public webserver e.g. Apache or Nginx with web traffic.

https://github.com/golang/go/issues/31534#issuecomment-485104051

I see cases where the client side is reusing a closed stream and the server eventually says to GOAWAY.
After that point, it becomes a constant barage of PROTOCOL_ERROR which then subsides. New connections are created until we hit another issue which is exceeded the number of concurrent streams.
This is when the server side goes into a hard loop.

[swankjesse]

You're using the latest OkHttp version?

[Drkstr]

I am using OkHttp version 3.12.0.

I've noticed that the issue only occurs when the device in on a bad network connection.

[harishrpatel]

We are also having to fix it in our app. We are testing Okhttp's retry mechanism under poor connection. Here is how we reproduce it using Charles Proxy tool on Mac. Basically, we took the URL and set a breakpoint in Charles. When breakpoint hit, Charles will prompt you to either execute it as is or ABORT it. We ABORT it to cause this exception "okhttp3.internal.http2.StreamResetException: stream was reset: INTERNAL_ERROR". We are using retrofit. I wish either one of them would catch this exception and return errorBody so that we can handle it gracefully. So, now we are just going to have to try/catch Retrofit's Call.execute().

[swankjesse]

@harishrpatel if the stream was reset there is no error body.

[birenroy]

It sounds like the okhttp3 client is throwing an exception when it receives a RST_STREAM frame with error code NO_ERROR. Note that the HTTP/2 spec specifically says clients should not discard responses as a result.

https://httpwg.org/specs/rfc7540.html#HttpSequence
"When this is true, a server MAY request that the client abort transmission of a request without error by sending a RST_STREAM with an error code of NO_ERROR after sending a complete response (i.e., a frame with the END_STREAM flag). Clients MUST NOT discard responses as a result of receiving such a RST_STREAM, though clients can always discard responses at their discretion for other reasons."

[PhilippC]

@swankjesse I am running into this issue as well (https://github.com/PhilippC/keepass2android/issues/747). I can reproduce it with the following code:

public class ConnectionInfo {
    public String URL;
    public String username;
    public String password;
}


private OkHttpClient getClient(ConnectionInfo ci) throws NoSuchAlgorithmException, KeyManagementException, KeyStoreException {

    Log.d("KP2AJ","building okhttp client");
    OkHttpClient.Builder builder = new OkHttpClient.Builder();
    final Map<String, CachingAuthenticator> authCache = new ConcurrentHashMap<>();

    com.burgstaller.okhttp.digest.Credentials credentials = new com.burgstaller.okhttp.digest.Credentials(ci.username, ci.password);
    final BasicAuthenticator basicAuthenticator = new BasicAuthenticator(credentials);
    final DigestAuthenticator digestAuthenticator = new DigestAuthenticator(credentials);

    DispatchingAuthenticator authenticator = new DispatchingAuthenticator.Builder()
            .with("digest", digestAuthenticator)
            .with("basic", basicAuthenticator)
            .build();

    builder = builder.authenticator(new CachingAuthenticatorDecorator(authenticator, authCache))
            .pingInterval(1, TimeUnit.SECONDS)
            .writeTimeout(100, TimeUnit.SECONDS)
            .connectTimeout(100, TimeUnit.SECONDS)
            .readTimeout(100, TimeUnit.SECONDS)
            .addInterceptor(new AuthenticationCacheInterceptor(authCache));

    OkHttpClient client =  builder.build();


    return client;
}

class PerformOkHttpTestTask extends AsyncTask<Object, Void, Void> {


    @Override
    protected Void doInBackground(Object... params) {

        android.util.Log.d("KP2AJ", "upload starting");

        ConnectionInfo ci = new ConnectionInfo();
        ci.password = "xxx";
        ci.URL = "https://...";
        ci.username = "userX";

        byte[] data = new byte[3000000];

        Request request = null;
        try {
            request = new Request.Builder()
                    .url(new URL(ci.URL))
                    .put(RequestBody.create(MediaType.parse("application/binary"), data))
                    .build();


            Response response = null;

            response = getClient(ci).newCall(request).execute();
            if((response.code() < 200)
                    || (response.code() >= 300))
            {
                if (response.code() == 404)
                    throw new FileNotFoundException();
                throw new Exception("Received unexpected response: " + response.toString());
            }
            android.util.Log.d("KP2AJ", "upload ok");


        } catch (Exception e) {
            e.printStackTrace();
        }

        return null;
    }
}

 @Override 
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    findViewById(R.id.button1).setOnClickListener(new OnClickListener() {
        public void onClick(View v) {
            new PerformOkHttpTestTask().execute();
        }
    });

Note that this issue also occurs without the pingInterval and read/write/connectTimeout calls on the builder. Note that it does not occur if I upload less data (e.g. 3000 bytes instead of 3MB).

The server I am testing this with is an
Server version: Apache/2.4.25 (Raspbian)
Server built: 2019-10-13T15:43:54
Nextcloud Version 16.0.3

My build.gradle has

 dependencies {
     compile 'com.squareup.okhttp3:okhttp:4.2.2'
     compile 'com.burgstaller:okhttp-digest:1.15'
 }

As many of my users (of Keepass2Android) are waiting for a fix to this: Is there anything I can change in my code or a workaround you can suggest?

Is the sample code enough or do you need a full Android Studio project?

Thanks a lot!

[swankjesse]

Can fix.

[kingsleyfrancis]

Hi @swankjesse, is there any fix yet for this? I am having Protocol_Error version of this error HTTP FAILED: okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR. My okttp version is 3.12.1 and asp.net core 2 backend.

[swankjesse]

@kingsleyfrancis PROTOCOL_ERROR indicates that either OkHttp or the server are doing HTTP/2 incorrectly. Please upgrade to the latest (OkHttp 3.12.8), reproduce the bug, then open a new issue with steps to reproduce. We need to know exactly which server you're talking to, including its version. Providing a URL also works.

[Luensche]

@swankjesse Does your PR fix this issue?

[PhilippC]

@swankjesse I would like to point out that I have lots of users who are waiting for a fix on this (https://github.com/PhilippC/keepass2android/issues/747), so if it is possible in any way to keep the milestone fixed to 4.5 I'd be really happy, as I have seen this being moved from 4.3 to 4.4 and now 4.5.
Anyway: Thanks for your hard work! I really appreciate it and I totally understand that features are shifted to later milestones, so please don't see this as criticism. I just want to make sure you are aware that this is a real issue for real people :-)

[sasisachin12]

Still am facing this issues( stream was reset: NO_ERROR) on my POST api call. Any one have solution for that?

i tried all this

implementation 'com.google.code.gson:gson:2.8.5'
implementation 'com.squareup.retrofit2:retrofit:2.8.1'
implementation 'com.squareup.retrofit2:converter-gson:2.8.1'

implementation 'com.squareup.okhttp3:okhttp:4.5.0'
implementation 'com.burgstaller:okhttp-digest:2.3'

but no results.

[anjalsaneen]

We are also facing the same issue for long on a Network Request to our server. it's happening across All OS Version (Version 4.2.2)

okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
    at okhttp3.internal.http2.Http2Stream.takeHeaders(SourceFile:7)
    at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(SourceFile:1)
    at okhttp3.internal.connection.

[yschimke]

I've tried to reproduce this running minio, and @PhilippC test code. Thanks for this.

https://github.com/yschimke/okhttp/tree/repro

But it's not negotiating HTTP/2, only HTTP/1. I also get a timeout error when I try uploading 300MB. Less than that and I'm getting a 404, so I'm also doing something wrong.

So I do wonder whether it's just a problem with minio's support for http/2? Until you can confirm that minio supports HTTP/2 properly, can you disable this in your OkHttpClient instance?

ANDROID_SERIAL=emulator-5554 ANDROID_SDK_ROOT=/Users/yschimke/Library/Android/sdk ./gradlew :android-test:connectedCheck

[yschimke]

Just saw "Failing test for NO_ERROR stream reset", I missed that context. I guess I'm just getting up to speed.

So ideally we would support the cancellation of the large request upload and return a useful 4XX error. But our current implementation just fails, with a vague error NO_ERROR.

[swankjesse]

If the request stream is canceled we may still successfully read the response stream. That could even be a 200 if the server didn't need a request body to generate the response.

To implement this we need to change Http2Stream to have independent errors for request and response streams.

[yschimke]

What would the correct title of the bug be then?

[yschimke]

n.b. minio server has disabled http/2 for now, anyway. So a repro, isn't as simple as running minio locally.

https://github.com/minio/minio/blob/master/cmd/http/server.go#L201

Can repro with a large PUT upload to https://nghttp2.org/httpbin/put

[yschimke]

@swankjesse Do you plan to do this sooner? or later? 4.7 seems like it's a near term target.

[genalex2]

I found the solution!!!

MultipartBody.Builder builder = new MultipartBody.Builder();
builder.setType(MultipartBody.FORM);

Only set the type on the builder multipart body.

Regards.

[swankjesse]

Closed with https://github.com/square/okhttp/pull/6295

[chanjungkim]

For okhttp3.internal.http2.StreamResetException: stream was reset: INTERNAL_ERROR
solution: Checkout here: https://github.com/square/okhttp/issues/3936#issuecomment-738542336