Okhttp: [Question] Proxy connection always fail with unexpected end of stream.

[smatei]

Hi,

On some proxies (squid proxy) OKHttpClient works well. On others (also squid proxies), for an unknown reason, I always get the "unexpected end of stream" error (Caused by: java.io.EOFException: \n not found: size=0 content). For all the browsers and other http clients (java, apache, jetty), the proxy works.

Exception in thread "main" java.io.IOException: unexpected end of stream at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:215) at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) at okhttp3.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:732) at okhttp3.internal.http.HttpEngine.access$2(HttpEngine.java:726) at okhttp3.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:714) at okhttp3.internal.http.HttpEngine.readResponse(HttpEngine.java:566) at okhttp3.RealCall.getResponse(RealCall.java:245) at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:200) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:162) at okhttp3.RealCall.execute(RealCall.java:59) at testok.run(testok.java:54) at testok.main(testok.java:61) Caused by: java.io.EOFException: \n not found: size=0 content=... at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:191) ... 11 more

I have tried to debug it using the OKHttp and okio sources, by sniffing with Wireshark, but no luck. Somehow the server response is missing. How can I debug this?

The code that I use is quite simple.

` Authenticator proxyAuthenticator = new Authenticator() {
public Request authenticate(Route route, Response response) throws IOException {
String credential = Credentials.basic(user, password);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};

OkHttpClient client = new OkHttpClient.Builder()
    .protocols(Arrays.asList(Protocol.HTTP_2, Protocol.HTTP_1_1, Protocol.SPDY_3))
    .connectTimeout(60, TimeUnit.SECONDS)
    .writeTimeout(60, TimeUnit.SECONDS)
    .readTimeout(60, TimeUnit.SECONDS)
    .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxy, port)))
    .proxyAuthenticator(proxyAuthenticator)
    .authenticator(proxyAuthenticator)
    .build();

Request request = new Request.Builder()
    .url(url)
    .header("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0")
    .build();

Response response = client.newCall(request).execute();
System.out.println("response " + response.code());
return response.body().string();`

Best Regards,
Stefan Matei

[dave-r12]

@smatei one scenario that might trigger this can occur when the server closes the underlying TCP connection. Can you check with wireshark to see if the server sends a FIN packet to close the connection?

[smatei]

Hi,

I have attached a screenshot. On the left side is OKHttp request, on the other apache http client (with 200 response). I can see a FIN packet, but I am sending it using okhttp (11.0.0.71 is my address). This is strange. I send the request, and instead of receiving an answer, I close the connection. Or maybe I close the connection because of the eof exception.

[dave-r12]

Cool, thanks! It looks like on the left, the initial HTTP request is sent (No. 4), the server responds with a 407 (No. 6) and then closes the connection (No. 8). Can you paste in the response headers of the 407 response?

On the right, there is no 407 response, so my guess is the authentication info is included with the original request?

[smatei]

Hi,

On the right side there is apache http client, with preemptive authentication. No 407, direct authentication. As I wrote in https://github.com/square/okhttp/issues/2435, I would like that implemented in OKHttp.

You are right, the answer is in the 407 response. I have figured out the problem. Please take a look at the attached picture

Now both Wireshark sessions use OKHttp. On the left I have a not-working proxy, on the right, I have a working proxy. The response for the not-working proxy is Proxy-Connection: close. The working proxy however keeps the connection alive and the next request works (response 200).

In the class RealCall.getResponse, if I comment *_if (!engine.sameConnection(followUp.url())) *_, the stream is closed, and the next request is successful, as a new connection is done.

` //if (!engine.sameConnection(followUp.url()))

  {

    streamAllocation.release();

    streamAllocation = null;

  }

`

I will think about a solution to this, as I have to use the library in a scraping software, and the users may have all kinds of proxy servers, that may have this behavior.

I am thinking of creating a branch of OKHttp and implement the direct authentication myself. This would save an extra request and avoid the followup that fails due to proxy closing the connection.

Best Regards,
Stefan Matei

[swankjesse]

Duplicate of https://github.com/square/okhttp/pull/2458

[DmitriyYakovlev]

Hi. Is there any solution for described problem in any version? I have the same problem