Okhttp: Recover gracefully from corrupted cache

[paour]

In an application using Retrofit 2.2.0 and OkHttp 3.6.0, I'm getting several crashes per day with a strange error inside OkHttp classes, which I can't catch because I'm using async calls.

Fatal Exception: java.lang.IllegalArgumentException: unexpected url: ://example.com/api/recommend/Ans%20
       at okhttp3.Request$Builder.url(Request.java:141)
       at okhttp3.Cache$Entry.response(Cache.java:705)
       at okhttp3.Cache.get(Cache.java:210)
       at okhttp3.Cache$1.get(Cache.java:144)
       at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:70)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
       at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
       at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
       at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
       at okhttp3.RealCall.access$100(RealCall.java:33)
       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:120)
       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
       at java.lang.Thread.run(Thread.java:833)

I guess what's happening is the cache is somehow corrupted (the URL scheme is missing), and when the CacheInterceptor tries to create an HttpUrl, this exception gets thrown, and since it's not an IOException, it's not caught and given to my failure handler.

Shouldn't such RuntimeExceptions be caught by RealCall and wrapped in an IOException for delivery to the failure handler?

[swankjesse]

Yeah, we should definitely recover better when the cache breaks badly.

[mkrussel77]

Is there any knowledge on what causes this to happen? I've seen it in a couple of Android apps I support.

My first thought was that they were creating two instances of OkHttpClient both pointing to the same cache directory, but that doesn't seem to be the case.

Other thought is we use context.getCacheDir() as the base directory for the cache, and maybe the OS is deleting files from the cache directory, but it seems strange that just the scheme winds up missing.

Also would the solution that goes in, not just make the request fail with an IOException, but somehow go and make the real request and then update the cache again?

[jinatonic]

We also have several thousands of these crashes a week, any ways to mitigate the issue?

[swankjesse]

@jinatonic executable test case to reproduce would be a great start.

[danh32]

We're still investigating, hoping to get a reproducible test-case. One interesting thing we found is that when we get this exception, the actual cache entry's url on the first line doesn't appear corrupted. We added an interceptor that catches this specific crash and deletes the offending request's cache entry. We're also printing the contents of the entry and report a non-fatal to help us debug. So we're seeing:

java.lang.IllegalArgumentException: unexpected url: ://midlands.robinhood.com/news/CWCO/

get thrown, and then then:

*** CACHE ENTRY START ***
https://midlands.robinhood.com/news/CWCO/
...
*** CACHE ENTRY END ***

So it seems like the missing scheme is happening somewhere after being read from the disk.

[awenger]

We also started to get this crash with the latest update of our app (before the update we got it very infrequently). With this version we also observed #2281 for the first time. So I think they have the same cause (corrupted cash)?
I went through the diff, and this was the biggest change we had regarding retrofit/okhttp:

-            .addCallAdapterFactory(RxJavaCallAdapterFactory.createWithScheduler(Schedulers.io()))
+           .addCallAdapterFactory(RxJavaCallAdapterFactory.createAsync())
+           .addCallAdapterFactory(RxJava2CallAdapterFactory.createAsync())

Retrofit 2.3.0
OkHttp 3.9.0

[hugosvent]

any workaround on handling this issue?

got so many crash because of
Fatal Exception: java.lang.IllegalArgumentException
Expected URL scheme 'http' or 'https' but no colon was found

should I just add an Interceptor which try the builder and catch the IllegalArgumentException then force it into network instead of cache by using cacheControl(FORCE_NETWORK)?

it's so hard to reproduce it internally