Kiwix-android: Unable to open ZIM files that Kiwix for Android did not download.

[virtadpt]

Describe the bug
BLUF: It seems like it should be possible for Kiwix for Android to open .zim files that it did not download. It can't.

Background: I live in a very bandwidth contstrained location; it is feasible for me to use Kiwix to download smaller .zim files (e.g., InstallGentoo Wiki - 2019-09-18 - 9MB) but not to download larger ones (wikipedia_en_all_maxi_2020-12.zim - 73.9 GB). So, my workaround is to take a flash drive to work, plug it into my machine there, and have it download .zim files with wget. Then the next time I'm in the area I pick up the flash drive, plug it into my laptop, plug my phone or tablet's microSD card into my laptop, and copy the .zim file over.

The thing is, this doesn't actually work. Only files that Kiwix for Android has downloaded are in the Kiwix Library.

If I use my file manager to open the .zim file (Open With -> Kiwix) the Kiwix app open and very briefly the "loading" indicator shows at the top, but the .zim file is not added to the Library.

On all of my devices I have Kiwix configured to use my phone's microSD card as its source of truth (sandwich menu -> Settings -> Storage -> External).

Expected behavior
If a new .zim file is in the storage location Kiwix is configured for: By opening the Kiwix app's Library screen the new .zim file should be present and I should be able to open and read through or search it just like .zim files that Kiwix downloaded on its own.

If a new .zim file has been copied onto the storage location Kiwix is configured for: I should be able to open the file's "do stuf" menu, tap on "Open with", select Kiwix, and Kiwix should add that file to its Library screen and let me read through or search it.

Steps to reproduce the behavior:

1. Download a .zim file, but not with Kiwix. Use a web browser, or copy a .zim file onto the Android device's storage somehow (say, by copying the file with a file manager)
2. Open Kiwix
3. Tap the Library icon
4. The .zim file is not present

Alternatively, trying to open the file with "Open with"

1. Download a .zim file, but not with Kiwix. Use a web browser, or copy a .zim file onto the Android device's storage somehow (say, by copying the file with a file manager)
2. Open the Android device's file manager.
3. Navigate to where the new .zim file is stored (on my devices, that is my phone's microSD card -> Android -> Data -> org.kiwix.kiwixmobile -> Files -> Kiwix)
4. Tap on the "do stuff with this file" menu icon for the file (for me it's a kebab menu icon)
5. "Open with"
6. Select Kiwix
7. The .zim fie does not open. The .zim file is also not in the Library.

Screenshots
Can supply screenshots upon request.

Environment

• Version of Kiwix Android : v3.4.3 build 623403 from the Google Play store on both devices
• Device :
  
  
  
  • Samsung SM-G965U phone
  
  
  • Samsung SM-T290 tablet
  
  
• OS version : Both are running Android 10, unmodified

Logs

• Phone: https://drive.google.com/file/d/1umINHzLqnYJwWGFbjT-P2DH1BmehB48J/view?usp=sharing
• Tablet: https://drive.google.com/file/d/1YaJRE4gjD64iSNtzqSflIk_TVdSR8ciC/view?usp=sharing

[kelson42]

@virtadpt Thank you very much for your quality bug report. This should indeed just work. We have a detection system which is definitly buggy and fails to detect your SD card I guess. @mhutti1 You had worked on this and @BBT-SeanMacGillicuddy had afterward incorporated/modified your code. Do you see here what is going wrong?

[mhutti1]

I think the not opening with the library is an issue I have seen on my Samsung device, the way the file is passed into Kiwix is not being handled correctly. It's tricky becuase each device maker often has their own file browser with its own way of passing files. I havn't looked at Android file search in a while but when I did the main issue was again it is up to device manufactures and often there is no way to know where sd cardsare mounted.

[kelson42]

@mhutti1 What I don’t understand is that this is the same kind of explanation that I heard from you 4-5 years ago. Many tickets have been closed meanwhile, just an example about a recent one https://github.com/kiwix/kiwix-android/issues/2354. This gives me the feeling that the problem is not understood or tackle properly.

[mhutti1]

My understanding is this is another example of there being no unified way to do this in Android, instead things change. This did used to work with the samsung brower I think. I imagine they have simply changed how it works and we need to change our code to account for this unfortunately.

[kelson42]

@mhutti1 I have a few questions which I believe their answers would help to bring clarity and fix the bug:

• This tickets reports two different bugs (1) the new ZIM file is not automatically detected in the Kiwix local library (2) The user can not open from the "file browser". The real problem I want to treat is that the (1), one time fixed we could look like to (2).
• What is the current approach to detect automatically new content in the local library? I believe it would be good to create a wiki page just for this and briefly explain the approach and its weaknesses.
• Does the diagnostic - uploaded by @virtadpt and created via Kiwix - allows us to know the mouting point of the SD cards and why Kiwix fails to identify the new content?
• Since a few versions, we have revamped the SD card detection algorithm which is used in the settings to populate the Settings Storage list of devices. My experience is that this feature works fine now. Do we know if the SD card detection has worked fine there (from the diagnostic)? Are somehow two things related? If the SD card detection works fine, why not just ask to scan the SD card if there is one?

[virtadpt]

@kelson42

• Yeah, I was wondering about that... what does having no File -> Open menu have to do with an Android misfeature?
• How about adding a File -> Open menu so we can select ones that Kiwix doesn't already know about? Kiwix is great for making wikis available offline, but it's missing the "download files so they can be added offline" use case.
• When reading through the diagnostic dump I sent, I saw some stuff about "zim files in the database" but nothing about "zim files not in the database." I spent some time poking around looking for said SQLite database so I could add the file manually, but didn't find it.
• It worked fine, because it located the .zim archives that I was able to download through Kiwix (by browsing the Kiwix library) but not the ones that I downloaded some other way. It doesn't, for example, detect any .zim files that I download with Firefox on my phone or tablet (that I move into the Kiwix .zip library directory) that are small enough for me to grab in an hour or less at home.

[mhutti1]

• We do have a file open menu for some apps, its not an easy thing to do for everything as you have to tell apps that you support opening certain file types but different apps want you to tell them in different ways. It is a constant battle to update what we tell apps we can open as they for no apparent reason change how they want us to tell them.
• I can have a look into creating a wiki on what we currently do.
• We can try scan the SD card but as I said we might not find the sd cards root directory correctly. If we can't find that we have issues. This also is device specific.

[shapeshifter910]

So what's the point of having the download from our repository link in your FAQ. Because anybody that downloads files that way will not be able to use them in Kiwix.
The whole point of this app is to have wiki files and being able to open them on various devices OFFLINE.
Your app has it's core functionality BROKEN!
It's relatively easy to use a tablet or phone when being offgrid and recharging them using solar panels or power banks unlike using laptops which need Inverter to recharge.
And trying to get me to download 80GB over wifi to my tablet is not a good idea.
So if you truly want to provide an offline wikipedia for people, the android and ios apps are far more important than the computer applications.
You should find some qualified android developers. There are some out there for sure because I have some quality apps on my android tablet that have no problems with anything at all - external storage or internal, doesn't matter, open anything from anywhere.

[virtadpt]

@shapeshifter910 This. So much this. I'm pretty much at the point of building a PiPhone to run Kiwix Desktop because I can open open .zim files that I download at home (the tiny ones).

[mhutti1]

I understand the frustration. We are 100% volunteers working on the Android app at the moment. I assure you it does work on some devices but device manufacturers keep changing their file browser specs underneath us so we have to be constantly playing catchup.

The underlying issue is that ZIM files don't have a mimetype we can query for. This is what allows other apps to open word documents etc and it is trivial for apps to open these types of file as you elude. Unfortunately without an identifiable mimetype it is much much harder.

[mhutti1]

The issue is the code here: https://github.com/kiwix/kiwix-android/blob/develop/app/src/main/AndroidManifest.xml#L46 I have spent many hours trying various additional path configurations but none work with for example the samasung file browser. We are telling the browser we accept the file. It is not listening though...

[mhutti1]

https://github.com/kiwix/kiwix-android/wiki/Finding-Files-In-Android