Kiwix-android: Storage Not Accessible in API 28

[mhutti1]

I am unable to access storage on my API 28 emulator.

[mhutti1]

Bug

2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err: java.io.IOException: Permission denied
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.io.UnixFileSystem.createFileExclusively0(Native Method)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.io.UnixFileSystem.createFileExclusively(UnixFileSystem.java:281)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.io.File.createNewFile(File.java:1008)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at eu.mhutti1.utils.storage.StorageDevice.createLocationCode(StorageDevice.kt:77)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at eu.mhutti1.utils.storage.StorageDevice.<init>(StorageDevice.kt:40)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at eu.mhutti1.utils.storage.StorageDevice.<init>(StorageDevice.kt:36)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at eu.mhutti1.utils.storage.StorageDeviceUtils.externalFilesDirsDevices(StorageDeviceUtils.kt:47)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at eu.mhutti1.utils.storage.StorageDeviceUtils.getStorageDevices(StorageDeviceUtils.kt:37)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at org.kiwix.kiwixmobile.utils.files.FileSearch.directoryRoots(FileSearch.kt:75)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at org.kiwix.kiwixmobile.utils.files.FileSearch.scanFileSystem(FileSearch.kt:66)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at org.kiwix.kiwixmobile.utils.files.FileSearch.access$scanFileSystem(FileSearch.kt:35)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at org.kiwix.kiwixmobile.utils.files.FileSearch$scan$1.call(FileSearch.kt:41)
2019-08-16 16:43:26.011 5413-5570/org.kiwix.kiwixmobile W/System.err:     at org.kiwix.kiwixmobile.utils.files.FileSearch$scan$1.call(FileSearch.kt:35)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at io.reactivex.internal.operators.flowable.FlowableFromCallable.subscribeActual(FlowableFromCallable.java:39)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at io.reactivex.Flowable.subscribe(Flowable.java:14827)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at io.reactivex.Flowable.subscribe(Flowable.java:14774)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
2019-08-16 16:43:26.012 5413-5570/org.kiwix.kiwixmobile W/System.err:     at java.lang.Thread.run(Thread.java:764)

[Aditya-Sood]

IIRC starting from Pie there was a change in how files could be accessed. With the new scoped storage, apps can only access those directories which they've created themselves. For other directories, the user needs to explicitly grant access for each of them. So just having WRITE_EXTERNAL_STORAGE permission won't suffice.

[mhutti1]

We will likely need to work out a workaround for this then.

[macgills]

This is thorny but it looks like we can kick the problem down the line to a later release by doing this

<!-- This attribute is "false" by default on apps targeting Android Q. -->
  <application android:requestLegacyExternalStorage="true" ... >

[macgills]

Tch, ScopedStorage is 29>=, this is a problem on 28. The end solution might be similar to supporting Scoped Storage

[macgills]

After reading all of The Death of External Storage articles I have concerns. Particularly in the performance department

[Aditya-Sood]

@mhutti1 @macgills is it a problem if we shift to using the new app-specific directory /data/media/{user}/Android/... instead? We won't have to use the 'Storage Access Framework' then (& avoid the performance issues I guess)

As long as zims are downloaded through app (internet or local file transfer), we can restrict the zims to that directory. Only downside is that it will become non-trivial for a user to load zims onto their mobile through a computer/removable storage

[macgills]

I tried to set the download location in downloadManager to the internal directory on removable storage and it still errored, possibly because downloadManager does not have the permissions to write to our internal removable directory? I need to do a lot more investigation

[macgills]

Api 28 P 9.0: Can only write to sd card location that are private to your app, can read from sd card, downloadmanager can not save files directly to sd card
Api 27 O 8.1: can't find sd card??? emulator bug?
Api 26 O 8.0: can't find sd card??? emulator bug? It shows one storage but changes its mind on wether it is internal or external
Api 25 N 7.1.1: works fine
Api 24 N 7.0: works fine

This is very complex.

Options

1. Replace downloadmanger (maybe only on 28>?) with fetch and this would allow us to download to our private directory on the sd card. Alternatively we could continue using downloadmanager and download to internal and then copy to external when a download finishes.

2. Migrate to Storage Access Framework which looks hideous but will be necessary for the future of android (Q/R) but maybe it won't be mandatory for a year or two? It is very weird right now and devs hate it. Only works 4.4>=

3. Do Nothing We can say only support sd cards that are formatted as internal storage for downloading files. We will still be able to read zims off of removable sd cards. We should probably also verify that a storage is writable before using it as a download location

[macgills]

@kelson42 I will keep working through emulators and if anyone else wants to verify my findings that could be helpful

[macgills]

There is a 4th unsavoury option
Give us root so we can do whatever we like when we like it. Terrible option for all but the superuser

[MidUK]

May I suggest an alternative option?

Allow users to download ZIMs in 4GB chunks so we can manage file storage ourselves.

[macgills]

@MidUK that was the solution we were using prior to 2.4 and found it to be slow and error prone. I don't think this solution is viable for the 3.0 milestone but I would say we are open to reimplementing the feature in the future either as an experimental feature to be enabled in settings or as a full scale download replacement.

[MidUK]

I hadn't realised there was a location to be able to download ZIM chunks. Secondly, I fear that there isn't the information to get those chunks working once copied onto the SD card (does it need to be in a specifically labelled folder or do I just dump them in the Kiwix folder and Kiwix auto detects them).

Just out of curiosity, for any other users reading this, what reason do you not download the files manually, split them up and then put them on your external storage (SD card)?

Is it because you're lazy, not technical enough to do so, fear that something will go wrong, don't know how to get your split files working on Kiwix, etc?

[abdulwd]

@macgills IMO we should replace the download manager with Fetch for all APIs if there's no con apart from rewriting the code. I also found this library some time ago https://github.com/lingochamp/okdownload

[abdulwd]

I think Fetch is better we can also use it to share zims between devices.

[macgills]

@MidUK It is a dense topic. SD cards were not officially supported in android until 4.4 (surprising I know) and we support 4.0.3-9.0. Android's ability to report an SD card even being present varies wildly by manufacturer and version and even then the permission to write to these sd cards changes.
We choose the kiwix folder because it is very public and easily accessible to users and we want those files to exist on your device even after an uninstall.
Split files on kiwix should work currently but the app does not produce them from 2.5 onwards as I said

we are open to reimplementing the feature in the future either as an experimental feature to be enabled in settings or as a full scale download replacement.

but 3.0 takes priority and Kiwix is shifting to faster releases so I am afraid I must ask for patience. Split files (nor zim files) do not need to be anywhere specific as we scan every point of your device we can find for zim files.
The issues we face are resource allocation, error prone android apis, inconsistent behaviour across android versions and indeed the complexity of the task.
DownloadManager was chosen as it is an android component that lives outside of our lifecycle - you can force close the app while a download is ongoing and it will still continue - is available on all android versions, handles retries and connectivity outages, allows choosing of network to download on -wifi or mobile - was relatively easy to use and is the recommended way in the android documentation to perform long running downloads.

If at all possible I recommend formatting an sd card as internal as this will get rid of many problems. It is not desirable to all users but kiwix functions better that way. As kiwix evolves this functionality will improve.

If you are an android/java/kotlin developer yourself we welcome contributions.

[macgills]

@abdulwd replacing with fetch seems like the most frictionless way forward. We do lose the functionality of the downloads continuing outside of our app and we would have to move zim files to our private storage (only on sd cards?) so they will now be deleted on uninstall.
We currently have zim file sharing thanks to Aditya so it may not be necessary to replace that with fetch

[macgills]

I will pursue a fetch implementation, all files will be saved to the private app storage whether on internal or removable storage so they will be deleted on unistall

[DoS007]

I'm a user without programming know how. just want to say: Downloading on internal and then copying on external is often on big things like wiki with pictures not possible.

[abdulwd]

@macgills Files can be saved to the Kiwix folder using Fetch, if we encounter some error then we can try the private storage.

[macgills]

On 28 you cannot save files onto an sd card anywhere but private, for uniformity of code and user experience I will be exclusively using private storage to save files.
In the future we may start using Storage Access Framework (in tandem with Fetch) but for now that is not supported on all apis we support and is not backported.