After upgrading our android app from okhttp 3.12.2 to 3.14.0, we started seeing this error (stacktrace below) coming through Crashlytics in production at a very high rate. Because we upgraded directly from 3.12 to 3.14, I'm not sure if this error was introduced in 3.13 or 3.14.
With an admittedly small sample size, the crash was affecting over 10% of sessions and over 20% of users, so we were forced to roll back. It appeared across Android versions 7, 8, 9, and Q.
Unfortunately, I wasn't able to reproduce it locally. However, it seems very similar to this issue from 2 years ago, which appears to have been re-introduced to the library: https://github.com/square/okhttp/issues/3308
I am not sure if this is related but we are compiling our app with R8 version 1.4.72
Our app uses Java 8, min SDK 21, and the latest version of retrofit (2.5.0).
Hopefully this information and the stack trace below is enough to be helpful, apologies for not being able to include a reproducible case outside of production.
Thank you for the amazing library and everything you do for android open source!
Fatal Exception: java.util.NoSuchElementException
at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:75)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:107)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:87)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:162)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at com.redacted.myapp.MyNetworkingModule.doRequestWithExceptionCatching(MyNetworkingModule.java:150)
at com.redacted.myapp.MyNetworkingModule.lambda$provideOkhttpClient$0(MyNetworkingModule.java:99)
at com.redacted.myapp.-$$Lambda$MyNetworkingModule$f9YhFuxMEYBUFTv_jyu3YSQWZX8.intercept(-.java:2)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)
Yikes. Can fix.
Just because it might help to isolate the problem – does your interceptor retry requests? The stacktrace suggests something like that might be the underlying cause.
Are we missing a okhttp_3.14.x branch?
Ignore: misunderstood the point of the check
The hasNext() and next() are on different objects, which looks weird.
boolean newRouteSelection = false;
if (selectedRoute == null && (routeSelection == null || !routeSelection.hasNext())) {
newRouteSelection = true;
routeSelection = routeSelector.next();
}
FWIW - I'm not actively fixing this, so just flagging for swankjesse or whoever picks it up
@swankjesse yes, our interceptor conditionally retries requests based on a Retry header that we send, and this stack trace is indeed coming from inside the retry code. This is what that looks like:
// Only if the request has a custom header for retrying, retry on timeout
if (request.headers().get(RETRY_HEADER) != null) {
response = doRequestWithExceptionCatching(chain, request);
int tries = 0;
while (response == null && tries++ < TOTAL_RETRIES) {
Timber.v("OkHttp: Retrying %s: %d", request.url(), tries);
response = doRequestWithExceptionCatching(chain, request); // CRASH OCCURS HERE
}
if (response == null) {
throw new IOException(String.format(
"OkHttp: Request to %s failed after multiple retries", request.url()));
}
} else {
response = chain.proceed(request);
}
The implementation of doRequestWithExceptionCatching is pretty straightforward:
private Response doRequestWithExceptionCatching(Interceptor.Chain chain, Request request) {
Response response;
try {
response = chain.proceed(request); // CRASH OCCURS HERE
} catch (IOException e) {
return null;
}
return response;
}
Thanks for looking into it!
Yep, I see how we made this mistake.
@yschimke good idea. This plus the other Exchange regression motivates us to make an okhttp_3.14.1 release. Folks who are (unreasonably!) grumpy about Kotlin should still have a rock-solid 3.x branch to ship.
We are still having the same issue with 3.14.1, retrofit 2.5.0, minSdk 21
Fatal Exception: java.util.NoSuchElementException
at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:75)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:107)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:87)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at it.subito.networking.utils.TimingLoggerInterceptor.intercept(TimingLoggerInterceptor.java:26)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at it.subito.networking.okhttpclient.interceptors.HeaderInterceptor.intercept(HeaderInterceptor.java:49)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:818)
@danibart93 yikes. Will take a look.
@danielebart I was trying to reproduce this tonight. Any idea what happens before this exception? Maybe a redirect, connection failure, or interceptor retry?
Sorry, i have no idea. All i can say is that this is the top crash in our latest build and we didn't have this crash before updating to the 3.14.1 (from 3.12). I'll keep you posted if something comes up, thank you!
A possible scenario may be something along these lines:
ExchangeFinder.findConnection() sets up a separate connection for each call.RealConnection.isHealthy() is called on the reused connection, it becomes unhealthy and ExchangeFinder.findConnection() is executed again.ExchangeFinder has already called RouteSelector.next() once to set up the original connection, which was closed. It now calls RouteSelector.next() again, but there is no guarantee that there is another Selection to return.I'll try to write a test that reproduces this.
I reproduced the exception in an improvised test where a connection is shutdown prematurely:
SocketException is thrown, RetryAndFollowUpInterceptor decides to retry because ExchangeFinder.retryCurrentRoute() and ExchangeFinder.hasRouteToTry() return true:ExchangeFinder.findConnection() is reached, a ConnectionShutdownException is tracked on the Connection and increments Connection.routeFailureCount.ExchangeFinder.findConnection() decides to try a new route because retryCurrentRoute() now returns false due to Connection.routeFailureCount > 0, but there is no other route:Not sure if it will help, but we also got the crash with 3.14.1 with a slight different trace:
Fatal Exception: java.util.NoSuchElementException
okhttp3.internal.connection.RouteSelector.next (RouteSelector.java:27)
okhttp3.internal.connection.ExchangeFinder.findConnection (ExchangeFinder.java:27)
okhttp3.internal.http.RealInterceptorChain.connectTimeoutMillis (RealInterceptorChain.java)
okhttp3.internal.connection.ExchangeFinder.find (ExchangeFinder.java:7)
okhttp3.internal.connection.Transmitter.acquireConnectionNoEvents (Transmitter.java:16)
okhttp3.internal.connection.ConnectInterceptor.intercept (ConnectInterceptor.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:19)
okhttp3.internal.cache.CacheInterceptor.intercept (CacheInterceptor.java:19)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:34)
okhttp3.internal.http.BridgeInterceptor.intercept (BridgeInterceptor.java:34)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept (RetryAndFollowUpInterceptor.java:25)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:2)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.RealCall.getResponseWithInterceptorChain (RealCall.java:10)
okhttp3.RealCall$AsyncCall.execute (RealCall.java:2)
okhttp3.internal.NamedRunnable.run (NamedRunnable.java:17)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1162)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:636)
java.lang.Thread.run (Thread.java:784)
https://github.com/square/okhttp/pull/5013 adds a test that reproduces the stack trace in the description.
3.14.2 is out and has this fix. Go get it!
sorry, just wondering if this issue has been implemented on 3.12.X ?
It was a regression introduced in 3.14.0, so 3.12.x has neither the bug nor it's fix.
Just saw this in 3.14.2
java.util.NoSuchElementException:
at okhttp3.internal.connection.RouteSelector.next()(RouteSelector.java:15)
at okhttp3.internal.connection.ExchangeFinder.findConnection()(ExchangeFinder.java:21)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection()(ExchangeFinder.java:1)
at okhttp3.internal.connection.ExchangeFinder.find()(ExchangeFinder.java:6)
at okhttp3.internal.connection.Transmitter.newExchange()(Transmitter.java:5)
at okhttp3.internal.connection.ConnectInterceptor.intercept()(ConnectInterceptor.java:5)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at okhttp3.internal.cache.CacheInterceptor.intercept()(CacheInterceptor.java:22)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at okhttp3.internal.http.BridgeInterceptor.intercept()(BridgeInterceptor.java:22)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept()(RetryAndFollowUpInterceptor.java:6)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at com.redacted.myapp.networking.CloudinaryWebPInterceptor.intercept()(CloudinaryWebPInterceptor.java:21)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at com.redacted.myapp.interceptors.DaliRedirectInterceptor.intercept()(DaliRedirectInterceptor.java:26)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at com.redacted.myapp.shared.features.common.utils.image.DaliCompatInterceptor.intercept()(DaliCompatInterceptor.java:1)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at com.redacted.myapp.core.interceptors.UserAgentInterceptor.intercept()(UserAgentInterceptor.java:22)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:10)
at okhttp3.internal.http.RealInterceptorChain.proceed()(RealInterceptorChain.java:1)
at okhttp3.RealCall.getResponseWithInterceptorChain()(RealCall.java:13)
at okhttp3.RealCall$AsyncCall.execute()(RealCall.java:2)
at okhttp3.internal.NamedRunnable.run()(NamedRunnable.java:3)
at java.util.concurrent.ThreadPoolExecutor.runWorker()(ThreadPoolExecutor.java:1133)
at java.util.concurrent.ThreadPoolExecutor$Worker.run()(ThreadPoolExecutor.java:607)
at java.lang.Thread.run()(Thread.java:761)
@AndrewWestberg yikes! Mind opening a new issue? Also, what’s up with the line numbers in your stacktrace? Those aren’t right.
Seconding @AndrewWestberg's comment. We've been following the thread here and were optimistic that 3.14.2 would resolve this. However, I am still seeing errors on 3.14.2:
okhttp3.internal.connection.RouteSelector.next (RouteSelector.java:95)
okhttp3.internal.connection.ExchangeFinder.findConnection (ExchangeFinder.java:138)
okhttp3.internal.connection.ExchangeFinder.findHealthyConnection (ExchangeFinder.java)
okhttp3.internal.connection.ExchangeFinder.find (ExchangeFinder.java:22)
okhttp3.internal.connection.Transmitter.newExchange (Transmitter.java:16)
okhttp3.internal.connection.ConnectInterceptor.intercept (ConnectInterceptor.java:23)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.cache.CacheInterceptor.intercept (CacheInterceptor.java:124)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.BridgeInterceptor.intercept (BridgeInterceptor.java:161)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept (RetryAndFollowUpInterceptor.java:23)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:153)
okhttp3.internal.http.RealInterceptorChain.proceed (RealInterceptorChain.java:4)
okhttp3.RealCall.getResponseWithInterceptorChain (RealCall.java:120)
okhttp3.RealCall$AsyncCall.execute (RealCall.java:12)
okhttp3.internal.NamedRunnable.run (NamedRunnable.java:17)
I'm happy to open that new issue if it's still needed.
@swankjesse You can ignore my line numbers. We're forced to use NewRelic for our crash logging and they don't always properly de-obfuscate crashes since R8.
How can I solve this problem now?
Still present in 4.0.1, same stack trace.
So is this issue resolved or not, does anyone know? Want to update to 4.1.0 but don't want to run into this again (we tried updating a couple months ago)
@leofirespotter we were able to nail down an instance with an executable test case and made a fix in 4.1.0. Please give it a shot if you can.
@dave-r12 we're still seeing this issue in 4.1.0. The request seems to succeed, but the result is that the callback returns with a failure. FYI, we're using this with Retrofit 2.5.0, if that matters.
This is how we're constructing our OkHttpClient:
fun initializeOkHttpClient(
timeoutSeconds: Long = 60,
vararg networkInterceptors: Interceptor
): OkHttpClient {
val okHttpBuilder = OkHttpClient.Builder()
.connectTimeout(60, TimeUnit.SECONDS)
.readTimeout(60, TimeUnit.SECONDS)
.writeTimeout(60, TimeUnit.SECONDS)
okHttpBuilder.addInterceptor(HttpLoggingInterceptor(object : HttpLoggingInterceptor.Logger {
override fun log(message: String) {
Log.e("HttpLoggingInterceptor", message)
}
}).apply {
level = HttpLoggingInterceptor.Level.BASIC
})
networkInterceptors.forEach { okHttpBuilder.addNetworkInterceptor(it) }
return okHttpBuilder.build()
}
Here's the stack trace:
HttpLoggingInterceptor --> POST https://<removed> (65-byte body)
HttpLoggingInterceptor <-- 200 https://<removed> (1137ms, unknown-length body)
java.net.SocketException: socket is closed
at com.android.org.conscrypt.ConscryptFileDescriptorSocket$SSLInputStream.read(ConscryptFileDescriptorSocket.java:554)
at okio.InputStreamSource.read(Okio.kt:102)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:159)
at okio.RealBufferedSource.request(RealBufferedSource.kt:62)
at okio.RealBufferedSource.require(RealBufferedSource.kt:55)
at okhttp3.internal.http2.Http2Reader.nextFrame(Http2Reader.kt:88)
at okhttp3.internal.http2.Http2Connection$ReaderRunnable.run(Http2Connection.kt:564)
at java.lang.Thread.run(Thread.java:919)
@mattswitch I believe that's a new problem. And I'm happy that it's a nice IOException rather than an unexpected NoSuchElementException. If you're still seeing it on OkHttp 4.2.0 please open a new issue.
I'm closing this one. Thanks everyone for your patience.
Hi @swankjesse,
We are still seeing this issue with latest okhttp . I am using 4.4.0 of it. Retrofit version is 2.7.2 and okio is 2.4.3.
2020-02-28 16:43:50.282 7349-9218/co.switchapp W/System.err: java.net.SocketException: socket is closed
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at com.android.org.conscrypt.ConscryptFileDescriptorSocket$SSLInputStream.read(ConscryptFileDescriptorSocket.java:588)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at okio.InputStreamSource.read(Okio.kt:93)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at okio.RealBufferedSource.request(RealBufferedSource.kt:207)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at okio.RealBufferedSource.require(RealBufferedSource.kt:201)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at okhttp3.internal.http2.Http2Reader.nextFrame(Http2Reader.kt:88)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at okhttp3.internal.http2.Http2Connection$ReaderRunnable.run(Http2Connection.kt:615)
2020-02-28 16:43:50.283 7349-9218/co.switchapp W/System.err: at java.lang.Thread.run(Thread.java:919)
I am not sure where does this issue actually belongs to, so I have raised bug in OKIO as well. Her eis the link to it https://github.com/square/okio/issues/702#issue-572699511
I updated okhttp from 3.14.2 to 3.14.7 and that "NoSuchElementException" error started to appear ( very small chance, but it is )
Has anyone encountered this on 4.5.0 or newer? I think the fix was here:
https://github.com/square/okhttp/pull/5888
Believe fixed, will reopen with evidence from 4.5.0+
Most helpful comment
3.14.2 is out and has this fix. Go get it!