Okhttp: IllegalArgumentException for the url in the Cache Entry

[brunoescalona]

The following error is one of our main crashes right now, around 25% of our crashes.

Fatal Exception: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found
       at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.java:1260)
       at okhttp3.HttpUrl$Companion.get(HttpUrl.java:1632)
       at okhttp3.Request$Builder.url(Request.java:184)
       at okhttp3.Cache$Entry.response(Cache.java:641)
       at okhttp3.Cache.get$okhttp(Cache.java:183)
       at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:47)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:109)
       at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:83)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:109)
       at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:76)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:109)
       at de.stocard.dagger.ProvidedDependenciesModule$provideHttpClient$$inlined$-addInterceptor$1.intercept(ProvidedDependenciesModule.java:1083)
       at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:109)
       at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.java:201)
       at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.java:517)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
       at java.lang.Thread.run(Thread.java:919)

The same error was reported https://github.com/square/okhttp/issues/4322 and https://github.com/square/okhttp/issues/4918

I would like to reopen the discussion and maybe together we can find the root cause and a solution.

During the last week we were building a custom version based on the 4.9.0 to report the url contained in the Cache Entry. We modified the following line with the code below and we also modified the cache path directory to avoid possible old corrupted files.

throw IllegalArgumentException("Expected URL scheme 'http' or 'https' but no colon was found for the url input $input")

We are getting in that case the corrupted url stored in the cache file.

For example, with the following retrofit configuration:

val clientBuilder = OkHttpClient.Builder()

clientBuilder.protocols(listOf(Protocol.HTTP_1_1))

clientBuilder.addInterceptor { chain ->
    val originalRequest = chain.request()
    val requestWithUserAgent = originalRequest.newBuilder()
            .header("User-Agent", "Example/1.0.0 HTTPClient Android").build()
    chain.proceed(requestWithUserAgent)
}

val certificatePinner = CertificatePinner.Builder()
certificatePinner.add("*.example.com", "sha256/public-key-1")
clientBuilder.certificatePinner(certificatePinner.build())

val cacheDir = File(context.cacheDir, "new_caching_directory")
val cache = Cache(cacheDir, (16 * 1024 * 1024).toLong())
clientBuilder.cache(cache)

val okhttpClient = clientBuilder.build()

val retrofit = Retrofit.Builder()
                .baseUrl("https://www.example.com/")
                .client(okHttpClient)
                .build()

val exampleApi = retrofit.create(ExampleApi::class.java)

interface ExampleApi {
    @GET("users")
    fun getUsers(): Single<List<User>>
}

We are getting in the error stack traces the following urls:

Fatal Exception: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found for the url input 

5����e?��뷘b���@&-����1�W�ہ�n��?�!y6�j٬�6N�S��https://www.example.com/users

Fatal Exception: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found for the url input 

5T�k;=�fEr��,��k�H��2�C�A*�����s�������.�^https://www.example.com/users

Fatal Exception: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found for the url input ://www.example.com/users

The url stored in the cache file, as you can see, can contain some corrupted text or the https is removed from the url.

Could you have a look? I can also investigate further or give you more data if needed.

Regards

[yschimke]

@brunoescalona thanks for the additional debug information.

[dave-r12]

Yes I'd like to get to root cause on this one too!

@brunoescalona I don't see a cache configured in your example. Is that happening elsewhere? Is it possible to provide a minimal test or code sample that you can reliably reproduce with?

[brunoescalona]

@dave-r12 I have updated the description with a more detailed and similar configuration. But I also can include it here below.

We can not reproduce the issue, we tried different things but it was impossible so far. After few ideas I could only simulate the behaviour of the issue "corrupting" manually the cache files and that way I was able to get the same stack trace and error, but I couldn't reproduce the root cause and understand why the cache file was corrupted.

We are also happy to provide more detailed information with few production endpoints causing the issue in a private channel if needed.

We are currently using two Okhttp instances configured as follow:

val clientBuilder = OkHttpClient.Builder()

clientBuilder.protocols(listOf(Protocol.HTTP_1_1))

clientBuilder.addInterceptor { chain ->
    val originalRequest = chain.request()
    val requestWithUserAgent = originalRequest.newBuilder()
            .header("User-Agent", "Example/1.0.0 HTTPClient Android").build()
    chain.proceed(requestWithUserAgent)
}

val certificatePinner = CertificatePinner.Builder()
certificatePinner.add("*.example.com", "sha256/public-key-1")
clientBuilder.certificatePinner(certificatePinner.build())

val cacheDir = File(context.cacheDir, "new_caching_directory")
val cache = Cache(cacheDir, (16 * 1024 * 1024).toLong())
clientBuilder.cache(cache)

val okhttpClient = clientBuilder.build()

val loggingInterceptor = HttpLoggingInterceptor()
loggingInterceptor.level = HttpLoggingInterceptor.Level.NONE

val clientBuilder = OkHttpClient.Builder()
        .protocols(listOf(Protocol.HTTP_2, Protocol.HTTP_1_1))
        .addInterceptor(loggingInterceptor)

val certificatePinner = CertificatePinner.Builder()
certificatePinner.add("*.example2.com", "sha256/public-key-2")
clientBuilder.certificatePinner(certificatePinner.build())

val httpClient = clientBuilder.build()
httpClient.dispatcher.maxRequestsPerHost = 32
httpClient.dispatcher.maxRequests = 32

val cacheDir = File(context.cacheDir, "new_caching_directory_2")
val cache = Cache(cacheDir, (8 * 1024 * 1024).toLong())
httpClient.newBuilder().cache(cache).build()

[gildor]

I can confirm that we have the same issue, see it on clients with cache. And it's not an issue of request itself, we have a debug and request URL is correct, so after digging into sources, it may happen only if a cache entry has invalid value for URL

Never be able to reproduce it, only on prod. We have not so many such cases, 10-30 per day maximum, but still it in top 10 our crashes

[im-lakshya]

We've been also getting the exact same crash, and the count is now getting higher and higher. Is there any way to suppress it or to solve it?

[yschimke]

@im-lakshya is there any additional context here? Is it all Android versions? Any chance you have multiple clients sharing a cache?

[im-lakshya]

We're one of the libraries in which we use Picasso for image caching. And our users suspect that the crash is happening because of us and I'm not able to find out the root cause behind it, and also verified that Picasso is also internally catching the exceptions when it makes connections via OkHttp.

And the crash is happening across all the major versions (9, 10, 11) but I can see that 90% of crashes are happening in Android 10 devices. We also checked that there are multiple libraries present in the client's app which internally uses OkHttp. So we're not sure how do we ensure which library is causing this crash, or is it OkHttp doing something internally due to which it is crashing.

Let me know if you require any more information, I'll try to get them from the client.

[dave-r12]

Thanks for the info. So far, I haven't been able to spot how this could happen in practice. It seems .... impossible 😄. But there's gotta be an explanation!

This person had mentioned that when they read the cache file directly upon seeing this exception the first line was actually fine https://github.com/square/okhttp/issues/3251#issuecomment-405322270. Could anyone else confirm if reading the cache file directly after seeing this exception shows a valid URL as the first line?

[yschimke]

Anything we could put in the next release to confirm a theory? Or recover at least enough to make this non-fatal?

} catch (re: RuntimeException) {
  throw new IOException("please report to https://github.com/square/okhttp/issues/6453 " + firstLineOfCache, re)
}

[brunoescalona]

@dave-r12 we were reporting the URL for each crash from here:

if (schemeDelimiterOffset != -1) {
  when {
    input.startsWith("https:", ignoreCase = true, startIndex = pos) -> {
      this.scheme = "https"
      pos += "https:".length
    }
    input.startsWith("http:", ignoreCase = true, startIndex = pos) -> {
      this.scheme = "http"
      pos += "http:".length
    }
    else -> throw IllegalArgumentException("Expected URL scheme 'http' or 'https' but was '" +
        input.substring(0, schemeDelimiterOffset) + "'")
  }
} else if (base != null) {
  this.scheme = base.scheme
} else {
  throw IllegalArgumentException("Expected URL scheme 'http' or 'https' but no colon was found for the url input $input")
}

And the urls are corrupted. I didn't go through all the code, but I was expecting to have the text url obtained from the cache file directly.

Should we also try to read the file to compare the url stored there?

@yschimke where are you planning to include that check?

[yschimke]

@brunoescalona not sure, I'm brainstorming what we can do in Version+1, to get information to resolve by Version+2. And wondering if we should ask for reports and also make it not fatal until then.

[gildor]

We see this crash on all versions of Android, but ~70-80% are from Android 10.

And we indeed reuse cache between multiple okhttp clients. I will try to split http cache for every client (it's not really reused, it probably never request the same url, only reuses cache dir) and let you know

[swankjesse]

Are you creating multiple Cache instances that all point at the same directory? That might be your problem. You want a single Cache instance that the different OkHttpClient instances all share.

[gildor]

@swankjesse No, we create a single Cache instance which used by multiple clients.

[yschimke]

@brunoescalona an example we could check, print a noisy warning if we create two cache instances with same directory.

[hoppham2332]

I'm stuck in the same issue and I find that it occurs if you try to make a URL with more than 1 period in the domain.
Example: It's ok with http://abc.xyz/foo/ba/
But it will throw the error if domain like: http://abc.xyz.com/foo/ba/

[brunoescalona]

@yschimke we included some logs when the cache object is instantiated. After checking few logs of the affected users I could see that the cache is instantiated only once.

Can we do something else to help?

Edit: by the way @yschimke we are using dagger to instantiate the cache and we are using the @Singleton annotation.