Okhttp: IOException in DiskLruCache in 1.5.0.

[chrisbanes]

Upgraded to v1.5.0 and I'm seeing a lot of these exceptions thrown.

java.lang.RuntimeException: java.io.IOException: failed to delete /data/data/app.philm.in/cache/picasso-cache/452f1b31c49eebbd77673818036d6560.0
       at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:175)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:841)
Caused by: java.io.IOException: failed to delete /data/data/app.philm.in/cache/picasso-cache/452f1b31c49eebbd77673818036d6560.0
       at com.squareup.okhttp.internal.DiskLruCache.remove(DiskLruCache.java:584)
       at com.squareup.okhttp.internal.DiskLruCache.trimToSize(DiskLruCache.java:639)
       at com.squareup.okhttp.internal.DiskLruCache.access$100(DiskLruCache.java:86)
       at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:169)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:841)

[BlimBlim]

Same Problem here, also with 1.5.1.
Also tried to set up a new Cache Directory cacheDir.getAbsolutePath() + "/okhttpCache"
Also using PicassoLibrary in my project.

03-13 05:45:07.065: E/AndroidRuntime(3329): java.lang.RuntimeException: java.io.IOException: failed to delete /data/data/xxx.xxxx.xxxx/cache/okhttpCache/dc1c01325a3aed57dde90c3232c1e565.0

[keshwans]

I started getting the same error today after upgrading to OkHttp 1.5.1 and Picasso 2.2.0
ava.lang.RuntimeException: java.io.IOException: failed to delete /data/data/com.mycom.app/cache/picasso-cache/1cc0afe5e0f05e3b1586e04b713d5e5d.0
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
at java.lang.Thread.run(Thread.java:856)
Caused by: java.io.IOException: failed to delete /data/data/com.mycom.app/cache/picasso-cache/1cc0afe5e0f05e3b1586e04b713d5e5d.0
at com.squareup.okhttp.internal.DiskLruCache.remove(DiskLruCache.java:584)
at com.squareup.okhttp.internal.DiskLruCache.trimToSize(DiskLruCache.java:639)
at com.squareup.okhttp.internal.DiskLruCache.access$100(DiskLruCache.java:86)
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:169)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
            at java.lang.Thread.run(Thread.java:856)

[swankjesse]

I believe the problem is that DiskLruCache + fat32 SD cards get grumpy when you delete a file that's open.

And I think there's a regression where in some case we aren't closing the cache.

Will fix ASAP and cut a 1.5.2.

[chrisbanes]

@swankjesse My stack trace was from a N5 with the cache on the data partition, so it's not a FAT32 issue.

[BlimBlim]

@swankjesse I also reproduced this on Nexus5 and Android4.2.2 Emulator on Windows. Can try more devices if neccesary (S2, S3, N4), just tell me!

[narayank]

@nfuller is investigating the same issue. We've had several reports of it from nexus 4s and nexus 5s

[nfuller]

I was talking to Jesse about this yesterday. Given there are people in GMT-like TZs looking at the issues too an update might be useful. Here's some information I sent out based on my investigations last night:

---

I grabbed a copy of the tester's on disk HTTP cache. The file it's complaining about in the exception really isn't there, i.e. rather than being a problem with the file being there and the delete failing, it can't delete because it doesn't exist.

OkHttp in previous versions (apparently) still treated a failure like this as a fatal exception and the exception from the older okhttp-bundled app confirms this. The file name is coming from the journal file that the DiskLruCache keeps so there are a few follow up questions I will be investigating:

1) Is this an indication of the journal file becoming corrupted. There's a chance this is a symptom of a prior failure that is corrupting/truncating the journal, and the log from that failure could be revealing.
2) Should OkHttp be more lenient when the file system doesn't match the contents of the journal file?
3) Has anything changed in Android that would cause some files but not others to be deleted from the cache dir under a running app?

---

I noticed that it's not just one file that's mentioned in the journal file and missing, but many. Missing ones are concentrated mostly at the top of the journal file as far as I can see. There are no REMOVE entries in the journal at all, and there is no backup file.

Best guess is that something external deleted all those files, or somehow all the REMOVE calls from valid deletes failed to make it into the journal.

I notice that there's no journalWriter.flush() around line 591 in DiskLruCache and the implementation changed in the most recent code to be an Okio.buffer(). There wasn't a flush() when it was a BufferedWriter either, but perhaps it was working differently. trimToSize() removes a lot of things and then closes (which should flush). Might what we're seeing be related to that not-closing-the-cache problem you mentioned?

This looks related to me:
https://github.com/square/okhttp/issues/521

[cevcev]

Got same error from different devices.

GENERAL_MOBILE,Galaxy Note3 (ha3g),Galaxy S3 (m0),Galaxy S4 (ja3g),Galaxy Note II (t03g),
LG G2 (g2),Galaxy Grand duos (baffin),HTC One X (endeavoru),Galaxy W (GT-I8150),
Xperia Z (C6603),Xperia L (C2105),Galaxy S3 Mini (golden),ST18i (ST18i),ST25i (ST25i)
Xperia Z1 (C6903),Galaxy S4 (jflte),Galaxy S2 (GT-I9100),HTC One (m7),Galaxy Ace2 (GT-I8160)
Nexus 4 (mako),Galaxy S4 Mini (serrano3g),Galaxy Mega 6.3 (melius3g),Galaxy Win (delos3geur)
Atrix (olympus)

Couldnt reproduce issue on my test devices or emulator.

[swankjesse]

Fix: https://github.com/square/okhttp/pull/650

[swankjesse]

I've posted 1.5.2-SNAPSHOT to Maven central. Please give it a try and see if it fixes your problem!

[swankjesse]

(If it does fix the problem, that's great news and we can do a 1.5.2 final release)

[tmysik]

In my build.gradle, I tried compile 'com.squareup.okhttp:okhttp:1.5.2-SNAPSHOT' but got Could not find com.squareup.okhttp:okhttp:1.5.2-SNAPSHOT. Do I need to do something more?

[swankjesse]

Via JohnPristine on StackOverflow, you need to add this to your ~/.m2/settings.xml::

<profiles>
  <profile>
     <id>allow-snapshots</id>
        <activation><activeByDefault>true</activeByDefault></activation>
     <repositories>
       <repository>
         <id>snapshots-repo</id>
         <url>https://oss.sonatype.org/content/repositories/snapshots</url>
         <releases><enabled>false</enabled></releases>
         <snapshots><enabled>true</enabled></snapshots>
       </repository>
     </repositories>
   </profile>
</profiles>

[cevcev]

I tested this my app with this snapshot and see no problem atm.

[BlimBlim]

Works great now with 1.5.2! Thanks a lot!

[ibmkhd]

We are having the same issue using
okhttp-urlconnection:2.1.0 , okhttp:2.1.0 and picasso:2.4.0.
It happened twice with a 4.3 Note 3.

java.lang.RuntimeException: java.io.IOException: failed to delete /data/data/com.opensooq.OpenSooq/cache/picasso-cache/5243ac9aaec66b953ad21b26712379d6.0
       at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:173)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
       at java.lang.Thread.run(Thread.java:841)
Caused by: java.io.IOException: failed to delete /data/data/com.opensooq.OpenSooq/cache/picasso-cache/5243ac9aaec66b953ad21b26712379d6.0
       at com.squareup.okhttp.internal.DiskLruCache.deleteIfExists(DiskLruCache.java:386)
       at com.squareup.okhttp.internal.DiskLruCache.removeEntry(DiskLruCache.java:600)
       at com.squareup.okhttp.internal.DiskLruCache.trimToSize(DiskLruCache.java:653)
       at com.squareup.okhttp.internal.DiskLruCache.access$100(DiskLruCache.java:85)
       at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:167)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
       at java.lang.Thread.run(Thread.java:841)

[rockerhieu]

Same here, still happen with okhttp-urlconnection:2.1.0 , okhttp:2.1.0 and picasso:2.4.0.

Fatal Exception: java.lang.RuntimeException
java.io.IOException
 raw
com.squareup.okhttp.internal.DiskLruCache$1.run (DiskLruCache.java:173)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1080)
java.lang.Thread.run (Thread.java:838)
Caused by java.io.IOException
com.squareup.okhttp.internal.DiskLruCache.renameTo (DiskLruCache.java:395)
com.squareup.okhttp.internal.DiskLruCache.rebuildJournal (DiskLruCache.java:375)
com.squareup.okhttp.internal.DiskLruCache.access$300 (DiskLruCache.java:85)
com.squareup.okhttp.internal.DiskLruCache$1.run (DiskLruCache.java:169)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1080)
java.lang.Thread.run (Thread.java:838)

@swankjesse should we reopen this issue?

[swankjesse]

@rockerhieu try upgrading your OkHttp to the release that has the fix first!

[swankjesse]

(We've fixed cache bugs since 2.1.0.)

[rockerhieu]

@swankjesse but as I mentioned, the issue still happen on version 2.1.0.

[parkjaeiek]

i got same probleam
okhttp 2.5.0 picasso 1.9.0

this error

Fatal Exception: java.lang.RuntimeException: java.io.IOException: failed to rename /data/data/com.vitalhint.haemukja/cache/picasso-cache/journal.tmp to /data/data/com.vitalhint.haemukja/cache/picasso-cache/journal
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
at java.lang.Thread.run(Thread.java)
Caused by java.io.IOException: failed to rename /data/data/com.vitalhint.haemukja/cache/picasso-cache/journal.tmp to /data/data/com.vitalhint.haemukja/cache/picasso-cache/journal
at com.squareup.okhttp.internal.io.FileSystem$1.rename(FileSystem.java)
at com.squareup.okhttp.internal.DiskLruCache.rebuildJournal(DiskLruCache.java)
at com.squareup.okhttp.internal.DiskLruCache.access$400(DiskLruCache.java)
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
at java.lang.Thread.run(Thread.java)

[kaeawc]

I'm seeing a similar error as what @parkjaeiek found, same versions of Picasso 1.9.0 and OkHttp 2.5.0:

0 java.lang.RuntimeException: java.io.IOException: failed to delete /data/data/co.hinge.mobile.android/cache/picasso-cache/cc45f6004dd1f99955745fd84140bb80.1
1   at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:181)
2   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
3   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
4   at java.lang.Thread.run(Thread.java:818)
5 Caused by: java.io.IOException: failed to delete /data/data/co.hinge.mobile.android/cache/picasso-cache/cc45f6004dd1f99955745fd84140bb80.1
6   at com.squareup.okhttp.internal.io.FileSystem$1.delete(FileSystem.java:70)
7   at com.squareup.okhttp.internal.DiskLruCache.removeEntry(DiskLruCache.java:607)
8   at com.squareup.okhttp.internal.DiskLruCache.trimToSize(DiskLruCache.java:664)
9   at com.squareup.okhttp.internal.DiskLruCache.access$200(DiskLruCache.java:88)
10  at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:175)
11  ... 3 more

[metcorne]

Any solution? I have the same crash as @parkjaeiek a couple of times a day:

Fatal Exception: java.lang.RuntimeException: java.io.IOException: failed to rename /data/data/<app name>/cache/journal.tmp to /data/data/<app name>/cache/journal
       at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:181)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:841)
Caused by java.io.IOException: failed to rename /data/data/<app name>/cache/journal.tmp to /data/data/<app name>/cache/journal
       at com.squareup.okhttp.internal.io.FileSystem$1.rename(FileSystem.java:85)
       at com.squareup.okhttp.internal.DiskLruCache.rebuildJournal(DiskLruCache.java:409)
       at com.squareup.okhttp.internal.DiskLruCache.access$400(DiskLruCache.java:88)
       at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:177)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:841)

Using okhttp 2.5.0 and picasso 2.5.2

[swankjesse]

@metcorne can you reproduce this reliably? What file system + OS?

[metcorne]

I cannot reproduce it 100%, it just happens when i'm testing and developing my app. Sometime a couple of times a day, some days none.
Today i've seen it on these devices used by our testers;
Sony Xperia Z5 Android 5.1.1
Samsung GT P5210 Android 4.3
Samsung Galaxy Android 4.4.1

All developer devices so not many apps or no free storage.
We use Picasso and OkHttp, but i'm not sure it has anything to do with Picasso, the crash log is not very descriptive.

[swankjesse]

@metcorne cool. Will attempt to reproduce.

[grschroe]

I was experiencing this using okhttp 2.5.0 with picasso 2.3.4 and retrofit 1.9.0 I have since removed picasso completely however I see this error pop up quite a bit in my crashalytics data. This is by far the most frequent crash inside my app so im hoping this issue will be tracked down soon.

[swankjesse]

@grschroe if you can provide a reliable way to reproduce this, that’ll make it significantly easier for me to fix it.

[pristalovpavel]

Does anybody have a stable workaround? This is one of the most often crashes in my app!

[xinthink]

Experiencing this with okhttp 2.6.0, on Android 5.x
shared stacktrace: http://crashes.to/s/e7df164324f

[afchamorro]

I have the same problem in my app with okhttp 2.7.0 and picasso 2.5.2, on Android 2.x, 4.x and 5.x

Fatal Exception: java.lang.RuntimeException: java.io.IOException: failed to rename /data/data/PACKAGE/cache/journal.tmp to /data/data/PACKAGE/cache/journal
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:181)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:818)
Caused by java.io.IOException: failed to rename /data/data/PACKAGE/cache/journal.tmp to /data/data/PACKAGE/cache/journal
at com.squareup.okhttp.internal.io.FileSystem$1.rename(FileSystem.java:85)
at com.squareup.okhttp.internal.DiskLruCache.rebuildJournal(DiskLruCache.java:409)
at com.squareup.okhttp.internal.DiskLruCache.access$400(DiskLruCache.java:88)
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:818)

Any solution?

Thank you.

[swankjesse]

@afchamorro full disk?

[afchamorro]

@swankjesse According Crashlytics reports does not seem that's the problem.

[swankjesse]

@afchamorro big plans to improve fault tolerance in the disk cache in OkHttp 3.1.

[swankjesse]

(Late January at the earliest)

[parahall]

Having the same issue. Crash number 3 among all the crashes
OkHttp 2.7.0, 12k CRASHES from 6746 USERS

Fatal Exception: java.lang.RuntimeException
java.io.IOException: failed to rename /data/user/0/com.gettaxi.dbx.android/cache/http-cache/journal.tmp to /data/user/0/com.gettaxi.dbx.android/cache/http-cache/journal
raw

Fatal Exception: java.lang.RuntimeException: java.io.IOException: failed to rename /data/user/0/PACKAGE/cache/http-cache/journal.tmp to /data/user/0/PACKAGE/cache/http-cache/journal
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:181)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
Caused by java.io.IOException: failed to rename /data/user/0/PACKAGE/cache/http-cache/journal.tmp to /data/user/0/PACKAGE/cache/http-cache/journal
at com.squareup.okhttp.internal.io.FileSystem$1.rename(FileSystem.java:85)
at com.squareup.okhttp.internal.DiskLruCache.rebuildJournal(DiskLruCache.java:410)
at com.squareup.okhttp.internal.DiskLruCache.access$400(DiskLruCache.java:88)
at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)

[crossle]

Many this error

[afchamorro]

Having the same issue with OkHttp 3.1.1

Fatal Exception: java.lang.RuntimeException: java.io.IOException: failed to rename /data/data/PACKAGE/app_service_api_cache/journal.tmp to /data/data/PACKAGE/app_service_api_cache/journal
at okhttp3.internal.DiskLruCache$1.run(DiskLruCache.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:841)
Caused by java.io.IOException: failed to rename /data/data/PACKAGE/app_service_api_cache/journal.tmp to /data/data/PACKAGE/app_service_api_cache/journal
at okhttp3.internal.io.FileSystem$1.rename(FileSystem.java:85)
at okhttp3.internal.DiskLruCache.rebuildJournal(DiskLruCache.java:405)
at okhttp3.internal.DiskLruCache.access$400(DiskLruCache.java:84)
at okhttp3.internal.DiskLruCache$1.run(DiskLruCache.java:173)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:841)

[matthewrice345]

We still see this error on 3.1.2 as well.

Fatal Exception: java.lang.RuntimeException: java.io.IOException: failed to rename /data/data/com.instructure.candroid/cache/responses/journal to /data/data/com.instructure.candroid/cache/responses/journal.bkp at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:181) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587) at java.lang.Thread.run(Thread.java:818) Caused by java.io.IOException: failed to rename /data/data/com.instructure.candroid/cache/responses/journal to /data/data/com.instructure.candroid/cache/responses/journal.bkp at com.squareup.okhttp.internal.io.FileSystem$1.rename(FileSystem.java:85) at com.squareup.okhttp.internal.DiskLruCache.rebuildJournal(DiskLruCache.java:407) at com.squareup.okhttp.internal.DiskLruCache.access$400(DiskLruCache.java:88) at com.squareup.okhttp.internal.DiskLruCache$1.run(DiskLruCache.java:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587) at java.lang.Thread.run(Thread.java:818)

Our implementation is open source if that helps. https://github.com/instructure/CanvasAPI

[duanbo1983]

Seeing this on okhttp 2.7.2

[PaulProject]

error continues to 3.2.0

[dave-r12]

FWIW, I've opened up a pull request #2388 to allow the DiskLruCache to recover from this scenario. It may take some time to iron out all the details though.

[maxsdev]

Seeing this on okhttp 3.11.0 :(

[TapanM]

Same is happening.

[ningjiaqi]

how to fix, I use retrofit, same problem, update to 2.5.0, no effect

[dave-r12]

@maxsdev @TapanM @ningjiaqi could you post your stack trace?

[kevinROctavian]

still get this error from okhttp:3.13.1.

Caused by java.io.IOException
failed to rename /data/user/0/com.goodboystudios.petparade/cache/picasso-cache/journal.tmp to /data/user/0/com.goodboystudios.petparade/cache/picasso-cache/journal

com.squareup.okhttp.internal.io.FileSystem$1.rename (FileSystem.java:85)

[Shimingli]

help me