Kiwix-android: German Wikipedia (using ZIM chunks) articles are not all rendered properly with 2.4

[uwemock]

I am running Kiwix 2.2 Build 53 from the Google Play Store.

I have a couple of ZIM files on my SD card, such as german Wikibookx, german Wikiquote, german and english Wikivoyage... They all work fine. However, I could not get the german Wikipedia up and running.

I tried to use the -nopic and the-novid files, but they just won't show up in the Kiwix content list.

I tried to split the -novid into 2GB chunks, but it still doesn't show up in Kiwix. Yes, I checked the naming, and all chunks are in the same directory.

The german Wikipedia is also not in the remote files list, so I could not test if it works to have Kiwix download the files.

Any hints?

[Martin5372]

Hi,
I just have the same problem and found your entry :-)
I tried the following files:

MD5 hashes: created from my downloaded files:
f8af88cc286c0b4e8d311e5c89d9af5f *wikipedia_de_all_nopic_2018-05.zim
f7119ed39fcf12f34afa9968d4aa5fc4 *wikipedia_de_all_novid_2018-05.zim
a45623bd6993b3f7c374082e4c2b5408 *wikipedia_de_all_novid_2018-06.zim

The MD5 hash for wikipedia_de_all_novid_2018-06.zim.md5 is OK:

http://download.kiwix.org/zim/wikipedia/wikipedia_de_all_novid_2018-06.zim.md5
a45623bd6993b3f7c374082e4c2b5408 wikipedia_de_all_novid_2018-06.zim
For the others I found no reference hash file.

I want to use the latest wikipedia_de_all_novid_2018-06.zim on my Android smartphone (.zimaa, ...).
It failed so I tried to open the ZIM-file in Kiwix for windows (V0.9).
But all of the above files reported the same error message:
Message in German: " kann nicht geladen werden. Bist du sicher, dass dies eine ZIM-Datei ist?"
Message in English: "Unable to load *. Are you sure this is a ZIM file?"

A test with the following file
d57cc00bb3f10c057ed0889f4a39d70d *wikipedia_en_ray_charles_2015-06.zim
in Kiwix V0.9 for Windows was successfull!
I guess that the german ZIM files are corrupt.

[FTno]

Had the same problem with the english Wikipedia file 'wikipedia_en_all_novid_2018-05.zim'. Other zim files like Wikivoyage, Wikiquote and some more worked fine.

Then I updated to v2.3 (kiwix-2.3_arm64-v8a.apk - https://sourceforge.net/projects/kiwix/files/android/2.3/) and the Wikipedia file worked.

But the search feature does not work for the Wikipedia file, just a blank page whatever I search for.
Hope this will be fixed for the next version.

[Martin5372]

I just switched from Kiwix 2.2 (latest version in the Play Store) to 2.3 from
https://sourceforge.net/projects/kiwix/files/android/2.3
thanks to FTno :+1:

With this new Kiwix version for Android the described access problem for my ZIM file
a45623bd6993b3f7c374082e4c2b5408 *wikipedia_de_all_novid_2018-06.zim
was fixed.

But I have the same problem like FTno:
The search function is working, but for most of the clicked articles I get a blank page.
The blank page keeps active and the CPU utilization is low = no background activity and no progress.

[mhutti1]

Could you try the nighties here: http://download.kiwix.org/nightly/2018-06-08/ and see if this fixes it?

[Martin5372]

Hi,
I installed kiwix-android_arm-debug.apk from
http://download.kiwix.org/nightly/2018-06-08/
but the problem is still the same:
The search function is working, but for most of the clicked articles I get a blank page.
The blank page keeps active and the CPU utilization is low = no background activity and no progress.

Here are the splitted ZIM files I use on Android 7.1.2 (LineageOS 14.1-20180628):
4.194.304.000 wikipedia_de_all_novid_2018-06.zimaa
4.194.304.000 wikipedia_de_all_novid_2018-06.zimab
4.194.304.000 wikipedia_de_all_novid_2018-06.zimac
4.194.304.000 wikipedia_de_all_novid_2018-06.zimad
4.194.304.000 wikipedia_de_all_novid_2018-06.zimae
4.194.304.000 wikipedia_de_all_novid_2018-06.zimaf
4.194.304.000 wikipedia_de_all_novid_2018-06.zimag
4.194.304.000 wikipedia_de_all_novid_2018-06.zimah
989.859.315 wikipedia_de_all_novid_2018-06.zimai

[mhutti1]

How about https://drive.google.com/open?id=1nI8vzMB-cXyrDLL9iPWXa7RCXekciCgo ?

[Martin5372]

Hello mhutti1,
here are the results for my smartphone with Android 7.1.2 (LineageOS 14.1-20180628):

• Kiwix 2.4 + wikipedia_de_all_novid_2018-06.zim (ZIM splitted in 4000 MB(!) pieces):
  This combination did not work. Still the same problems:
  > The search function is working, but for most of the clicked articles I get a blank page.
  > The blank page keeps active and the CPU utilization is low = no background activity and no progress.

Then I tried the following combination on my smartphone:

• Kiwix 2.4 + wikipedia_de_all_novid_2018-06.zim (ZIM splitted in 1500 MB(!) pieces):
  Result: This combination works!! :-)
  Very fast access to all articles and pictures and no blank screen. :-)

I already checked the SDCARD with H2testw: no errors.
Afterwards I make the same test again 4000 MB pieces vs. 1500 MB pieces.
The result is still the same.

Thank you very much for your help!
Please let me know, if you need more information to identify the size problem.

---

Does your Kiwix version 2.4 requires a newer android version?
On my old tablet with android 4.2.2 this version is not stable.
black screen, App freezes, ... it seems, that the behaviour is independent from the ZIM files.
Solution: I switched from Kiwix 2.4 to 2.3 and the following combination works for me:
Kiwix 2.3 + wikipedia_de_all_novid_2018-06.zim (ZIM splitted in 1500 MB(!) pieces).

have a nice day!

[Martin5372]

Hi,
the workaround (to use 1500 MB pieces) is still working for wikipedia_de_all_novid_2018-06.zima. :-)

What can I do for
http://download.kiwix.org/zim/wikipedia/wikipedia_en_all_novid_2018-05.zim with 77 GB?
There are not enough letters (*.zimaa to *.zimaz) to use 1500 MB pieces.

I tried 3500 MB pieces but the problem occured again:

• Some artices work
• some not
• some problems with missing pictures / only an empty frame appears
• and the problem with the blank page, again.

Any hints or workarounds?
thank you

[uwemock]

While Kiwix 2.3 (from the above link) was able to read the german Wikipedia, the newer 2.4 that came through the Play Store can't read my files. It even refused to read files that were fine with 2.2. So I went back to 2.3.

Sorry, guys - this is a mess. Release versions should always be able to read the format of files that have been published to deliver the application workload.

[mhutti1]

@Martin5372 you should be able to go .zimaa -> .zimaz ->.zimba -> .zimbz -> .zimzz

[mhutti1]

Does this problem occur with 2000MB chucks (the size we use automatically for our downloads)?

[uwemock]

I tried with Kochwiki which is only 60MB large. Kiwix 2.2 and 2.3 will read the file with no problems, Kiwix 2.4 just quits after a few seconds of white screen.

[kelson42]

@uwemock I just have tried 2.4 with Kochwiki and it works fine for me. That said, if you can achieve to get the log of your Android device corresponding to that crash scenario that would be great to open a dedicated ticket.

[uwemock]

@kelson42 Where will I find that log?

[kelson42]

@mhutti1 What would be the best approach you would recommend to @uwemock to share with us his Kiwix Android crash log?

[Martin5372]

@mhutti1

@Martin5372 you should be able to go .zimaa -> .zimaz ->.zimba -> .zimbz -> .zimzz

for wikipedia_en_all_novid_2018-05.zim I tried it with 1500 MB chunks with Kiwix 2.3 on my old tablet.
It's working :-)

Does this problem occur with 2000MB chucks (the size we use automatically for our downloads)?

for wikipedia_en_all_novid_2018-05.zim I tested afterwards 2000 MB chunks with Kiwix 2.3 on my old tablet.
It's working :-)

thank you very much!

[kelson42]

@mhutti1 I can confirm that we have here something really buggy. Depending of the content and chunk size, Kiwix for Android seems not always able to deal properly with it. I have splitted wikipedia_de_all_novid_2018-06 in 3500 MB chunks and then I can confirm that not all contents (images, pages) can be loaded properly... but if I try with kiwix-serve, with the same files, everything works fine.

[mhutti1]

@kelson42 hmm interesting. If standard sized chunks work then this isn't a major issue at the moment but we should certainly look into the root cause behind instability with larger chunk sizes. Possibly the use of Java int's overflowing somewhere I am not sure.

[Wikinaut]

I run wikipedia_de_all_novid_2018-06 (2000 MB chunks) but a search for "PDP" does not give any result (but pages like "PDP-11" are present, when I follow a link on the page "VAX" to it.

• Kiwix 2.4 Build: 1181980
• wikipedia_de_all_novid_2018-06 with 2000 MB ( 1,95 GB) chunks

@kelson42 : pls. can you confirm, that this not-fully working search or index again breaks the Kiwix functionality (it's a show-stopper again, in my view).

updated and including my other post:
@Martin5372 can you help me, again, I found Kiwix not fully working. I downloaded and split wikipedia_de_novid_2018-06 to 2000MB chunks, like in the tutorial, but there are no results for a search for "PDP". Can you confirm that a split into smaller chunks really helps?

[mhutti1]

Hi @Wikinaut this is a known bug with our C++ backend. In larger files with a full text index the index must fit in a chunk for it to work unfortunately. We should be falling back to title search in this case, thanks for providing a specific reproduction case. I will look into this and see what is happening.

[Wikinaut]

@mhutti1 and @kelson42 Yesterday, I made a larger check, here are my findings, and I am willing to help and to fix this issue very soon [ I want to have the Kiwix running since years, but not a single German (or other version) worked for me in the last three years. This leaves me quite puzzled, because you are promoting Kiwix so much and loud, but as it never works here, I really doubt what is going on... well, let's try to fix this. ]

• HUAWEI P8 LITE 2017
• Android 7.0
• SD CARD SAMSUNG PRO 128 GB
• KIWIX Android App 2.4 Build 1181980

I tried these files yesterday:

• Wikipedia_de_all_novid_2018_06 chunks of 1000 MB (almost no entries at all, only a very few)
• Wikipedia_de_all_novid_2018_06 chunks of 2000 MB (no entries for "pdp")
• Wikipedia_de_all_novid_2018_06 chunks of 1000 MB (no entries for "pdp")

Let me know, if you want me to test something else, usually, I will responded very quickly.

P.S. @kelson explained me yesterday in the chat, that only KIWIX with SD card with a single big ZIM file will work reliably, at the moment. Perhaps you can confirm this. I don't know: when you prepare a SD card with the Android 7.0 setup/formatter, I think, this (formatting) cannot be done on the device. Am I correct?

[IrgendwerA8]

I think this is a very serious issue too. On my Android 4.1.2 device, I'm not able to open a more current version of the German Wikipedia, even when spitted to the standard 2000MB chunk size:

Official 2.2: Fehler: Die ausgewählte Datei ist keine gültige ZIM-Datei.
2.3 from above: Crashes during start-up
2.4 from above: Starts and "seems" to open the ZIM after selecting, but after a second or so switches to black screen and hangs.

No problems at all with the two year old German Wikipedia and 2.2.

[kelson42]

@IrgendwerA8 What happens if you do not split it?

[IrgendwerA8]

I cannot put it on my FAT32 formatted SD-Card? ;)

[kelson42]

@IrgendwerA8 No way to get a card using exFAT? What you describe is a bug, but - up to now - to get full featured new ZIM, users will have to keep them in one (so, stop to split them)... so stop use FAT32 as a filesystem.

[uwemock]

Things are getting weird...
While Kiwix 2.3 is fully functional on my phone, the same version behaves differently on my tablet. I can see the start page of each ZIM, and I can also follow links to single articles. But I cannot use the search function. The results list remains empty.
This occurs on the -nopic Wikipedia split into 2000M chunks as well as on the 60MB Kochwiki.

[IrgendwerA8]

@kelson42 Just found my assumption confirmed by testing it: My device doesn't support exFAT and the microSD isn't recognized then. So splitting the data seems to be the only solution. I wonder what happened to the zim files that older ones work flawlessly and newer ones aren't valid anymore?

[FTno]

As far as I've understood it, there are advices here to use standard sized chunks (2000MB) and to stop to split the zim files. Rather confusing!

When I wrote my first post in this thread (near the top), I tried to use the full english zim file (no split) and search did not work.
Later I have tried to split in those 2000MB chunks and the search feature works. (I haven't done more than about a dozen searches, but at least those worked.)

NB! I think the title of this thread should be changed because the issue here is not specific for the german zim file.

[Wikinaut]

Changing the title is up to the original poster: @uwemock pls. action

[Wikinaut]

@kelson42 pls. action! I mean, the Kiwix pages are full of instructions how to split the files, which, respectfully stated, appears to be bullshit in the view of the recent information in this issue. Split or not to split, that is here the question.

[uwemock]

Actually, the title holds for my original problem: The german Wikipedia did not work in Kiwix 2.2. This problem has been resolved by Kiwix 2.3.

Unfortunately, two things happened:
1) Kiwix 2.4 messed it all up again.
2) People having problems with splitting ZIM files up took over my thread.

I should, again, note that my problems with search not working in Kiwix 2.3 on my tablet apply to the Kochwiki ZIM file which is 60 MB in size (this is megabytes, not gigabytes) - and, of course, I didn't split this file up in smaller parts.

So I recommend that someone should file an issue for the splitting topic.

[mhutti1]

I have done some testing. My results are:

Full 30GB German file -> Searching "PDP" works (is a bit slow, be sure you don't mistake this for no results)
Split into ~2000MB chunks -> searching pd searches using the fallback searchsuggestions method. Searching pdp tries to do this but finds nothing. Searching pdp-11 gives nothing as well. If we however use developer tools to search PDP-11 and don't reset the capitalisation then it returns the singular correct result. @kelson42 this is definitively an issue with seachsuggestions. Likely a regression if this didn't happen before.
Haven't tried other sizes because I think I have found the issue.

[mhutti1]

Some more details. It isn't the length of a string that causes this issue. "hello" works fine. It might be the "-" or possibly the capitalisation.

[Wikinaut]

@mhutti1 V i e l e n D a n k

[kelson42]

@mhutti1 The original problem here is not the suggestions system (@Wikinaut should open an other ticket instead of hijacking this one), but that articles are not rendered properly when using splitted ZIM files. Are you able to reproduce it with my concrete example?

[mhutti1]

@uwemock Could you try wiping the app data (not deleting the ZIM files) from settings. We really need a log to be able to solve this as we can't reproduce.

[kelson42]

This bug (big files not open or all articles not rendered properly) has been fixed and is available in 2.4 (latest build) and in nightlies.

[IrgendwerA8]

@kelson42 Sorry to say, but no, on my device the local selection of zim files is still empty...

[kelson42]

@IrgendwerA8 This ticket has gathered many kind of different use cases. We think to have fixed the most important one, if you have still a problem, please open a new ticket and share again/copy your details after a new test with 2.4.

[Wikinaut]

@kelson42 @mhutti1 I installed (beta tester) the

• Kiwix 2.4 Build 1182160 from Google Play Store and
• wikipedia_de_all_novid_2018-06 .zim with 2000 MB ( 1,95 GB) chunks

and I confirm that this works now on

• HUAWEI P8 LITE 2017
• Android 8.0.0
• SD CARD SAMSUNG PRO 128 GB

[Wikinaut]

@kelson42 Did I say "Thank you" ? Thank you!

[uwemock]

I tried the current release from the Play Store (Kiwix 2.4 Build 1182160), and I deleted the app data after the upgrade.

It works on my Samsung S5 mini (Android 6.0.1).
It doesn't work on my Samsung Galaxy Tab A6, Android 5.1.1.

On the Galaxy Tab, I have the following issues:

• The UI is in English while the device is set to German.
• Article search works on smaller files, e.g. KochWiki, not on Wikipedia.
• Sometimes, accessing an article closes the app.

[kelson42]

@uwemock Please open new tickets, one problem per ticket. "Work" or "Does not work" is not something a developer can start with. Please give a concrete example by confronting what you get with what you expect.

[Wikinaut]

@uwemock wrote

The UI is in English while the device is set to German.

I confirm this issue (since the beginning of Kiwix).