Kiwix-android: Very slow search function

[roydok]

Describe the bug
Slow search function. Takesna minute or 2 and sometimes never for the search results to come out.

Expected behavior
Quick search suggestion and results come out lile in the older 2.5 version

Steps to reproduce the behavior:

1. Upgrade to the latest version 3.0 and above
2. Use search
3. Slow to no results

Environment

• Version of Kiwix Android : 3.3.1
• Device : Samsung Galaxy S20 Ultra with 512 gb microSD card
• OS version : Android 10 and ONE UI 2.1
• 32 gb Wikipedia off-line installed microSD card

[kelson42]

@roydok Thank you for your bug report. Can you give one example of a search which takes two minutes to give results? Same question for a search giving no result at all?

[roydok]

Hi. Thank you very much for your quick reply. I tried moving the 35.8 gb
off-line Wikipedia file from the 512 gb microSD card to my 128 gb internal
storage. The search speed improved but still slower compared to the older
Kiwix version 2.5. The older Kiwix version 2.5 has a fast search and search
suggestion if your file is located in either the microSD card or internal
storage. The current Kiwix version 3.3.1 only does fast search if the file
is in the internal storage but super slow to no search in the microSD card.
I am going to keep my file in the internal storage until this is improved.

Thank you again.

On Tue, May 12, 2020, 1:53 PM Kelson, [e-mail] wrote:

@roydok https://github.com/roydok Thank you for your bug report. Can
you give one example of a search which takes two minutes to give results?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kiwix/kiwix-android/issues/2082#issuecomment-627125755,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APRPGMOHGJVFATI4YLHBU23RRDP3XANCNFSM4M6NAE7A
.

[kelson42]

@roydok This is great if it works better for you. That said, without clear and precise reproduction steps, we won't be able to work on this ticket and will close it.

[macgills]

I could maybe build an apk which is 2.5.0 but with latest kiwixlib and see if the slowdown is a result of kiwixlib or if it is the android app?

Otherwise I don't know how to diagnose this. Could it also be the age of the file? 32 gb Wikipedia off-line, knowing the date of this file wouldn't hurt anyways.

With a 2018-07-01 Wikipedia 77.2GB search is near instantaneous for me on a Galaxy Tab S running 6.0.1.

@roydok could you send a diagnostic report? The functionality can be found in the help screen of the app.

[kelson42]

@macgills I strongly suspect this was at a time and with a ZIM file without a title Xapian index. This is why I ask for a concrete up2date use case. I don't think this make sense to come back to 2.5.

[roydok]

Diagnostic sent as you requested. I transfered back the 35.8 gb off-line
Wikipedia file back to my 512 microSD card and ran Kiwix version 3.3.1
doing the search and generating the diagnostic as you suggested. Search is
still slow.

On Tue, May 12, 2020, 11:16 PM Seán Mac Gillicuddy, <
[email protected]> wrote:

I could maybe build an apk which is 2.5.0 but with latest kiwixlib and see
if the slowdown is a result of kiwixlib or if it is the android app?

Otherwise I don't know how to diagnose this. Could it also be the age of
the file? 32 gb Wikipedia off-line, knowing the date of this file
wouldn't hurt anyways.

With a 2018-07-01 Wikipedia 77.2GB search is near instantaneous for me on
a Galaxy Tab S running 6.0.1.

@roydok https://github.com/roydok could you send a diagnostic report?
The functionality can be found in the help screen of the app.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kiwix/kiwix-android/issues/2082#issuecomment-627409790,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APRPGMJ6FNFRFITR2LCZATTRRFR3VANCNFSM4M6NAE7A
.

[roydok]

Diagnostic sent

On Tue, May 12, 2020, 11:57 PM Roy Concepcion, [e-mail]
wrote:

Diagnostic sent as you requested. I transfered back the 35.8 gb off-line
Wikipedia file back to my 512 microSD card and ran Kiwix version 3.3.1
doing the search and generating the diagnostic as you suggested. Search is
still slow.

On Tue, May 12, 2020, 11:16 PM Seán Mac Gillicuddy, <
[email protected]> wrote:

I could maybe build an apk which is 2.5.0 but with latest kiwixlib and
see if the slowdown is a result of kiwixlib or if it is the android app?

Otherwise I don't know how to diagnose this. Could it also be the age of
the file? 32 gb Wikipedia off-line, knowing the date of this file
wouldn't hurt anyways.

With a 2018-07-01 Wikipedia 77.2GB search is near instantaneous for me on
a Galaxy Tab S running 6.0.1.

@roydok https://github.com/roydok could you send a diagnostic report?
The functionality can be found in the help screen of the app.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kiwix/kiwix-android/issues/2082#issuecomment-627409790,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APRPGMJ6FNFRFITR2LCZATTRRFR3VANCNFSM4M6NAE7A
.

[macgills]

@roydok interesting.... your sdcard reports its filesystem as sdfat which I believe should not support files of this size.

[roydok]

Thank you very much.

On Wed, May 13, 2020, 12:00 AM Roy Concepcion, [e-mail]
wrote:

Diagnostic sent

On Tue, May 12, 2020, 11:57 PM Roy Concepcion, [e-mail]
wrote:

Diagnostic sent as you requested. I transfered back the 35.8 gb off-line
Wikipedia file back to my 512 microSD card and ran Kiwix version 3.3.1
doing the search and generating the diagnostic as you suggested. Search is
still slow.

On Tue, May 12, 2020, 11:16 PM Seán Mac Gillicuddy, <
[email protected]> wrote:

I could maybe build an apk which is 2.5.0 but with latest kiwixlib and
see if the slowdown is a result of kiwixlib or if it is the android app?

Otherwise I don't know how to diagnose this. Could it also be the age of
the file? 32 gb Wikipedia off-line, knowing the date of this file
wouldn't hurt anyways.

With a 2018-07-01 Wikipedia 77.2GB search is near instantaneous for me
on a Galaxy Tab S running 6.0.1.

@roydok https://github.com/roydok could you send a diagnostic report?
The functionality can be found in the help screen of the app.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kiwix/kiwix-android/issues/2082#issuecomment-627409790,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APRPGMJ6FNFRFITR2LCZATTRRFR3VANCNFSM4M6NAE7A
.

[roydok]

My microSD card is a Samsung Evo plus 512 gb. I used it as is and not
reformatted it. I'll transfer the large 35.8 gb file back to the main
internal storage to have a faster search in Kiwix version 3.3.1. Thank you
very much for your time.

On Wed, May 13, 2020, 12:10 AM Seán Mac Gillicuddy, <
[email protected]> wrote:

@roydok https://github.com/roydok interesting.... your sdcard reports
its filesystem as sdfat which I believe should not support files of this
size.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kiwix/kiwix-android/issues/2082#issuecomment-627441564,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APRPGMPFX2XG3TZMU3WSMRLRRFYG5ANCNFSM4M6NAE7A
.

[macgills]

If you can reformat to exfat the sdcard I would recommend that? Honestly it shouldn't have been able to copy the files over at all if the filesystem does not support files of this size. We could have a flaw in our methodology for detecting filesystems, very few pieces of code are perfect.

[roydok]

I'll try to reformat the card tom and report back to you and send you the
diagnostic. Thanks again for your help.

On Wed, May 13, 2020, 12:15 AM Seán Mac Gillicuddy, <
[email protected]> wrote:

If you can reformat to exfat the sdcard I would recommend that? Honestly
it shouldn't have been able to copy the files over at all if the filesystem
does not support files of this size. We could have a flaw in our
methodology for detecting filesystems, very few pieces of code are perfect.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kiwix/kiwix-android/issues/2082#issuecomment-627444579,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APRPGMJDOEUYY4YHLNAK2HDRRFY3BANCNFSM4M6NAE7A
.

[macgills]

Thank you for helping diagnose the problem, hopefully we can improve kiwix.

[kelson42]

@roydok @macgills What is the status here? How does it work with ve rsion 3.3.2 in beta an a recent ZIM file?

[macgills]

We have greatly improved performance with thread safe searches and also helped the memory footprint by disposing unused readers used for searching. I was never able to replicate 2 min searches as the search results appeared almost instantaneously for me

[dbedrenko]

@macgills Those improvements that you mentioned were introduced in which version?

I am running v3.3.3 Build: 5230303 on my tablet and search is unbearably slow with the new wikipedia_en_all_maxi_2020-06.zim. By comparison, v2.5 and the 2016 English Wikipedia maxi, that I upgraded from, searched instantaneously.

After typing in "faceboo" it takes the app 20-25 seconds to return a list of results, which makes the app painful to use. Recently, another user described having the same issue.

What's interesting is that, after this initial delay, deleting characters from the end or appending characters updates the results far quicker (a second or 2 at most). This makes me wonder whether KiwiX is doing some sort of unnecessary, exhaustive searching upfront rather than solely focusing on getting results that start with "faceboo".

[Beater34]

1. Please forgive me if I should not post here, but I have the same slow search problem as roydok with Kiwix 3.3.3 and the 89GB Wikipedia_en_all_maxi_2020-06.zim. Searches for "facebook" and "Canberra" both took 90 seconds.
2. I previously had rapid searches with Kiwix 2.5 and the 78GB Wikipedia_en_all_maxi_2018-06.zim.
3. I have a Motorola G6 Plus phone (XT1926-5) running Android 9 (PPW529.116-11-23 1st May 2020) with a 256GB Samsung Evo Plus microSD card (Grade 3, Class 10) formatted exFAT. I can record 4K video with my phone and SD card (demonstrating good performance).
4. I have sent a diagnostic from Kiwix.
5. @macgills: (a) You get rapid searches with the 77GB 2018-07.zim, just as I got rapid searches with the 78GB 2018-10.zim – have you tried searching the 89GB 2020-06.zim (or a similar recent zim)? (b) Samsung Evo Plus microSD cards are shipped exFAT, and roydok cannot have FAT given that he loaded a 35.8GB file. (c) It seems a good idea to build an apk which is 2.5.0 but with the latest kiwixlib to see if the slowdown is the lib or the app.
6. Hearty thanks to both kelson42 and roydok for helping.

[macgills]

@dbedrenko 3.3.3 includes our latest fixes to search and the latest version of kiwixlib.

I will attempt to reproduce the issue on my tablet but this will take a while as the download is quite large, right now it reports as 4 day left.

I think this issue is related to the zim file but @kelson42 may be able to speak with more clarity there.

[kelson42]

I'm not able to reproduce the problem with my Google Pixel2. I believe the ZIM file to be correct. For the moment, I have actually no clue why on a few devices the search is that slow.

[macgills]

With the Pixel2 I doubt it has a sd card which seems to be a key component in this issue.

[Beater34]

Yes, the problem is that Kiwix is very slow to search a _recent_ ZIM file on an _SD card_ - even when the SD card is high-speed.

[dbedrenko]

Do let me know if I can help debug somehow. I have a rooted device, so you could give an .apk with whatever debug messages would be helpful to you. I could then upload this log file.

If it is not too difficult I could also access the device with a debugger? Via AndroidStudio or whatever (I'm not familiar with this environment). Not sure if this would be helpful, offering just in case.

[macgills]

Thank you very much for the offer but I think raw log inspection may not be very helpful unless I were to deliver you a version with enhanced search logging. I may yet do that and take you up on the offer if my own attempts to recreate the problem are fruitless.

Depending on your code/kotlin literacy I would invite you to look at SearchResultGenerator if you want an indepth understanding of the code.

[kelson42]

@macgills What might be great here to know is if the process get stuck inside or outside the libkiwix.

[macgills]

I have been unable to replicate on my tablet Galaxy Tab S SM-T700 Android 6.0.1 with Wikipedia Maxi 88.5GB 2020-06-26

On the very first search I did experience some delay but I would say more in the region of 5 seconds before results appeared and all subsequent searches were fine. @kelson42 is the some index building process on initial search?

[kelson42]

@macgills My UX is similar and is not coherent with the bug report which is more about 60s. The first search needs a bit of I/O, this is the best explanation I have for the moment. Nothing different otherwise in term of processing.

[macgills]

I can make a version with logs for @dbedrenko to help us identify the bottleneck but it may be tomorrow with the PRs on my plate

[dbedrenko]

I can make a version with logs for @dbedrenko to help us identify the bottleneck but it may be tomorrow with the PRs on my plate

Take your time, I will be on holiday until Tuesday (and thus unavailable).

Thanks a lot for trying to help the users that DO have this issue! :)

[macgills]

Here is some example output of me searching for "a"

And here is a link to the apk:
https://drive.google.com/file/d/1ekCGG8xoR4jXI33Ll35RrWrVK2sM-lWB/view?usp=sharing

Enjoy your holidays and let me know if you encounter any problems

[dbedrenko]

@macgills Attached is the output from the point I begin typing "facebo"

The big "Search Suggestion" output is outputted all at once about 15-20 seconds since I began typing. The search results display on screen about ~8 seconds after that.

search_bug1.txt

top displays the kiwixmobile to be the top process throughout, at ~47% CPU. What's noteworthy is that the sdcard process is not to be seen (but it was top process when the app was scanning for the .zim file in the Library screen.)

Please let me know if I can do any more debugging, would be happy to help.

[macgills]

Thank you for the logs. It would be handy if I could get timestamps in the logs, did you get these from the logcat file in the kiwix directory or run logcat yourself?

I wonder if you would get faster results from typing faster, we only submit a term to the results generator if there hasn't been another key entered within 500ms of another key

E/SEARCH_BUG( 5505): about to call searchSuggestions for fa on RxCachedThreadScheduler-9
E/SEARCH_BUG( 5505): about to call searchSuggestions for f on RxCachedThreadScheduler-14
E/SEARCH_BUG( 5505): about to call searchSuggestions for face on RxCachedThreadScheduler-13
E/SEARCH_BUG( 5505): about to call searchSuggestions for faceb on RxCachedThreadScheduler-12
E/SEARCH_BUG( 5505): about to call searchSuggestions for facebo on RxCachedThreadScheduler-22

and as seen here we are operating 5 searches simultaneously in separate threads

[dbedrenko]

Thank you for the logs. It would be handy if I could get timestamps in the logs, did you get these from the logcat file in the kiwix directory or run logcat yourself?

I ran logcat via adb, didn't know there was a log file being created somewhere. Not sure how to get this with timestamps...

I wonder if you would get faster results from typing faster, we only submit a term to the results generator if there hasn't been another key entered within 500ms of another key

I made it a point to type with good speed (still using one hand as usual though).

and as seen here we are operating 5 searches simultaneously in separate threads

If searches do not depend on preceding searches, what is the benefit of continuing to fetch the results for "f", "fa", "fac", "face" e.g. at the point in time when the user has entered "faceb"? When the user has entered "faceb" I don't think that they want to see the results for "f" or "fa", etc.

Maybe we should cancel those searches?

[kelson42]

and as seen here we are operating 5 searches simultaneously in separate threads

If searches do not depend on preceding searches, what is the benefit of continuing to fetch the results for "f", "fa", "fac", "face" e.g. at the point in time when the user has entered "faceb"? When the user has entered "faceb" I don't think that they want to see the results for "f" or "fa", etc.

This remark seems absolutly pertinent to me. On Kiwix-Desktop for Linux/Windows we stop "old" threads before starting one and I thought we had fixed this a few months ago on Kiwix-Android honestly.

[macgills]

https://drive.google.com/file/d/10LPnhWsW11lwTQGbLTKBeJ4HkZXwSxrn/view?usp=sharing here is a version with task cancellation.

it doesn't have the logging in it though so it is a very much "see if it works", for future reference logcat format is controlled with a -v option link

The task cancellation was much uglier code and didn't seem necessary during my testing of fixing the bug of the search results from the reader not being thread safe. I still feel like we are not addressing a core issue because of not being able to reproduce this consistently across devices but if this fixes it then it fixes it.

[dbedrenko]

Thank you for the second build, but unfortunately it is still very slow (although it's faster).

Attached is the log for the first debug build, now with timestamps:
first-debug-build-with-time.log

And the log from the new debug build that cancels old searches:
debug-build-cancel-searches.log

Do you see anything that hints at in which operation most of the time is being spent?

[kelson42]

@dbedrenko Thank you for keeping testing and helping us to fix the problem. I still believe that somehow there is chance that part of the problem comes from the SD card or the SD card reader. If I understand properly, the Samsung Galaxy S20 Ultra is a really high end device with the lot of internal memory. I would really be interested to know how it works if the ZIM is stored on the internal memory.
@macgills I would recommend to have a look to both ticket/PR of https://github.com/kiwix/kiwix-desktop/issues/265. Similar problem/solution for Kiwix-Desktop and now it works like a charm.

[dbedrenko]

I would really be interested to know how it works if the ZIM is stored on the internal memory.

@kelson42 Unfortunately, my tablet doesn't have such a large internal storage so I cannot try this. As mentioned previously, while the search is happening the sdcard process is not using CPU like it was when scanning the storage card for a .zim file. The CPU was all used up in the kiwix process. This sounds to me like searching is CPU-bound rather than IO-bound.

[macgills]

07-17 21:16:43.914 E/SEARCH_BUG(22157): received search term g
07-17 21:16:44.414 E/SEARCH_BUG(22157): sending g to result generator
//500 ms delay completely expected
07-17 21:16:51.695 E/SEARCH_BUG(22157): about to call searchSuggestions for g on RxCachedThreadScheduler-4
//7 second delay here, note  RxCachedThreadScheduler-4 indicating we absolutely are on a background io thread before calling kiwixlib
07-17 21:17:19.937 E/SEARCH_BUG(22157): result of searchSuggestions for g is true
//28 second delay to get a result from kiwixlib
07-17 21:17:30.453 E/SEARCH_BUG(22157): found suggestion SearchSuggestion(title=Ge, url=/A/Ge)
// results start coming in 10 sec later

so the delay is largely in kiwixlib, it might be beneficial to add logging to the task cancellation apk if you are still willing to test @dbedrenko but this may be tomorrow as I have a good few tickets to get to today.

It is quite technically challenging but in Android Studio there is a Profiler that can be run to collect CPU usage data, I am not too familiar with it and native code but it could offer insight but would require some work, they can be exported as mentioned here and I could do the analysis?

[dbedrenko]

so the delay is largely in kiwixlib, it might be beneficial to add logging to the task cancellation apk if you are still willing to test @dbedrenko but this may be tomorrow as I have a good few tickets to get to today.

Would be happy to test; no urgency, please take your time. I am really grateful that you give attention to this bug :)

It is quite technically challenging but in Android Studio there is a Profiler that can be run to collect CPU usage data, I am not too familiar with it and native code but it could offer insight but would require some work, they can be exported as mentioned here and I could do the analysis?

I will try to profile the app. My next hurdle is that Android Studio doesn't detect my device for some reason.

I see that adb allows you to profile the app also; I will try this first.

[macgills]

@dbedrenko https://drive.google.com/file/d/10LPnhWsW11lwTQGbLTKBeJ4HkZXwSxrn/view?usp=sharing a task cancellation version with logging

[dbedrenko]

I attach a log with the above version.

I am trying to make the profiling work: this should be very helpful.
with-cancelling-and-logging.log

[dbedrenko]

I have a profile trace of me searching for "shakes" with the cancel searches, logging enabled build.

This is how it looks like in Android Device Monitor (to run this tool, go to <YOUR-SDK-DIR>/tools and run ./monitor)

You can also open it in with Android Studio (this is how it looks like) and select the thread to analyse at the top left. It's weird that here you don't see all the columns that were in the Android Device Monitor like "Excl. CPU Time %". I guess they are buried somewhere in the interface...

shakes-search.trace.zip

The trace doesn't make sense to me, how is the "Incl CPU Time %" column not adding up to 100% ? Does it make sense to you?

[macgills]

I attach a log with the above version.

I am trying to make the profiling work: this should be very helpful.
with-cancelling-and-logging.log

Oh how very annoying, the task cancellation isn't cancelling the tasks. I have a few options here 1 which I like but could make things a bit more complex overall and another I hate but keeps the ugly in this file.

1) Introduce coroutines to the project, their ease of cancellability is much touted, they are the future of async programming on Android but RxJava is fulfilling a similar role right now so we would have 2 competing tools.
2) Just start a thread for searching so when I kill it it actually gets killed

[dbedrenko]

My opinion is probably not informed, but if one option is much easier to do than the other, I would do that one first to see if this actually fixes the slow search bug (remember, most people get instantaneous search results even without search cancellation). If it doesn't solve it, you don't have to merge it and we would look for the solution elsewhere.

[macgills]

https://drive.google.com/file/d/10LPnhWsW11lwTQGbLTKBeJ4HkZXwSxrn/view?usp=sharing

If you can try this version, coroutines with logging. Coroutines ended up being the nicest to implement with the finest grain of control. Thanks for your continued patience and assistance

[dbedrenko]

Sorry for the delay, here is the log and the trace for search term "faceboo":

coroutines-with-logging.log
coroutines-with-logging.trace.zip

This was the first search after starting the app. It took around 25 seconds from last letter touched to search results. Does the profiling trace illuminate anything? Thank you for not giving up on this issue :-)

If I do searches after that they are a lot quicker: around 15 seconds for "mango", 15 seconds for "john", 8 seconds for "bloke", 6 seconds for "shakespeare", 15 seconds for "google" (the duration is from the last letter touched to search results appearing).

[macgills]

07-30 20:33:11.511 E/SEARCH_BUG( 8370): before searchSuggestions for f
07-30 20:33:31.621 E/SEARCH_BUG( 8370): after searchSuggestions for f
07-30 20:33:31.621 E/SEARCH_BUG( 8370): before searchSuggestions for fa
07-30 20:33:31.628 E/SEARCH_BUG( 8370): before searchSuggestions for face
07-30 20:33:31.644 E/SEARCH_BUG( 8370): before searchSuggestions for faceboo

This has actually uncovered something a little interesting.

I assume in the 20 seconds between typing "f" and reaching "after searchSuggestions" for "f" you had typed the rest of "faceboo".

In a typical situation I would expect to see multiple "before searchSuggestions" arrive. This leads me to believe only a singular thread is free? Otherwise the other operations would just start and not be blocked by "searchsuggestions f"

Additional threads in this pool are created and are shutdown on demand. The number of threads used by this dispatcher is limited by the value of “kotlinx.coroutines.io.parallelism” (IO_PARALLELISM_PROPERTY_NAME) system property. It defaults to the limit of 64 threads or the number of cores (whichever is larger).

or you have a device with a single core? Doesn't seem likely.

I can try one more thing before I need to take a look at maybe re-implementing search suggestion from kiwixlib in kotlin as a cancellable suspend function? Which may very well be infeasible.

Unfortunately it will be monday before I can come back to this and hopefully dive into the traces

[dbedrenko]

or you have a device with a single core? Doesn't seem likely.

My tablet has a dual-core CPU: "Dual-core 1.0 GHz Cortex-A9" to be exact.

[kelson42]

@macgills Any feedback here? Would like to know if you need anything from libkiwix.

[macgills]

I think to resolve this I would need for the kiwixlib api offer me a way to instead of doing 1 call searchSuggestions(searchTerm, 200) that blocks execution I would need something I can call iteratively so that execution returns to my control and I can call yield or ensureActive which will actually halt the execution entirely so the thread can stop.

I still haven't really analysed the CPU trace, it is hard to fit in to the daily routine of reviews, notification clearings, etc as it is a lengthy enough task. Buying the device in question might be feasible, I also have a budget alcatel device that might have 2 cores? It barely turns on it has so little memory though so that is always an annoying process.

[kelson42]

@macgills Thank you for the feedback. Once you will have completed the CPU trace analysis, let us know here about the next steps and if you need anything. This is a serious problem.

[kelson42]

@mgautierfr Might that be that https://github.com/kiwix/kiwix-lib/pull/374 is somehow related to this problem?

[mgautierfr]

Yes and no :)

The search api exposed by kiwix-lib to android is not thread safe indeed. The searchSuggestions use a internal vector to store the result and if you do several search in the same time you will get wrong (mixed) results. We may even have crash because of two thread modified the vector at the same time and scrambling the memory.

But it will not help to have a async search api nor to speed up the search process.

[kelson42]

@mgautierfr AFAIK, the only way to provide a good experience of suggestion search is to use multithreading? You confirm?

[mgautierfr]

Yes. We may have good surprise on the search code itself, but from what I know, we should try to move this way.
This should not be made on android however. As @macgills suggests, it would be to kiwixlib/libzim to provide a async api. And user code (anrdoid, desktop, serve..) will have to adapt to take advantage of this api (incremental displaying of the results...)

[kelson42]

@mgautierfr Do we have a ticket for that already?

[mgautierfr]

No

[kelson42]

@mgautierfr @macgills OK, so we need one and I don't want to release any new version of Kiwix Android without having this bug fixed and the suggestion working properly. Beside the basic ZIM opening and rendering, I hardly see something more important than the suggestions system. As this problem/strong limitation seems to have been understood earlier I would have appreciate to see a ticket already open. It is not optimal to wait to get full users impact before doing something.

[stesee]

An other thing that would be a enhancemant would be ui feedback that something happens while searching. Even when it takes some time, this would tell the user that there should be search result anytime.... in the first run I thought, that kiwix had no search hit, before I found this issue... would be a quick win too.

[kelson42]

@stesee Indeed, I can not reproduce the problem, but another user impacted by the problem reported that it is written "no results" whereas it shouldl be maybe written "please wait" or "searching..." as long a the search is not over.

[mgautierfr]

OK, so we need one and I don't want to release any new version of Kiwix Android without having this bug fixed and the suggestion working properly.

I don't know what you're calling "suggestion working properly" but if you think about having a async api I hope you are not hurry to make a new release of kiwix-android.
There is nothing async at all in libzim/kiwix-lib for now.

[macgills]

@dbedrenko it has been a long time coming but can you test on this https://drive.google.com/file/d/10LPnhWsW11lwTQGbLTKBeJ4HkZXwSxrn/view?usp=sharing ?

I am about to open a PR after a complete overhaul of the internals of this screen

[kelson42]

@macgills It is a bit unfortunate that we don't have had any transparent exchange here between @mgautierfr and you on the technical approach to solve the problem. I hope at least you made a clear analysis of the problem together and came on a plan about what to do in a private chat.

On my side I discussed the topic in depth with @mgautierfr and clearly came to the conclusion that as long as the Kiwix for Android suggestion system is not able to handle multiple suggestion threads then the usability of the whole suggestion system usability will be challenged. The reason being that you have only one thread and you can not kill it to start a new search, so you have to wait it finishes before starting over a new one (so the user as well... waits).

[macgills]

So this ticket introduces a loading UI and speed increases, it isn't going to "fix" the problem but it is as much as we can do

[kelson42]

@macgills What is missing from your point of view to fix the problem? Do you agree with my conclusion that Kiwix Android needs to open many threads to makes suggestions (if not why)? Have you introduced a delay beetween search requests so a user typing quickly 5 letters does not make 5 suggestions searches?

[macgills]

We are now using coroutines, the concurrency primitive of kotlin, which are way more easily cancellable, when a load is in progress we display a spinner (we weren't previously because I was under the impression loads happened in milliseconds). Right now I believe it to be a good usability improvement. If @dbedrenko can confirm that I would be grateful as I cannot replicate their scenario locally with my devices but they have done enough over the course of this ticket.

To really fix this problem then kiwixlib would have to have an interruptable API but that doesn't seem likely as that would mean a rewrite of an external API. Even then I don't think it would be very effective as the majority of time is spent locating the xapian index which must be done anyways?

We could try empty searches whenever a user opens a book and hope to warm it up ahead of a search?

I can't remember everything we have tried over the past 4 months

[kelson42]

We are now using coroutines, the concurrency primitive of kotlin, which are way more easily cancellable, when a load is in progress we display a spinner (we weren't previously because I was under the impression loads happened in milliseconds). Right now I believe it to be a good usability improvement. If @dbedrenko can confirm that I would be grateful as I cannot replicate their scenario locally with my devices but they have done enough over the course of this ticket.

The spinner display is good, but this is only a communication thing... it does not solve the search speeed problem.

The Kotlin coroutines might solve the problem, I can not judge that as I don't know the technical background of them.

That said, if the user is happy, this is all good to me.

To really fix this problem then kiwixlib would have to have an interruptable API but that doesn't seem likely as that would mean a rewrite of an external API. Even then I don't think it would be very effective as the majority of time is spent locating the xapian index which must be done anyways?

@mgautierfr You agree with that? To me I remember you said, that thread being created at Kotlin level should be interrupted at Kotling level. If you disagree, please do the necessary with @macgills To have a full written agreement of what needs to be done to solve properly this ticket.

We could try empty searches whenever a user opens a book and hope to warm it up ahead of a search?

It seems indeed something has to be done here, this is one of the conclusions of https://github.com/openzim/libzim/issues/418 as well. Pretty sure nothing should be done at Kiwix-Android level (at least not now). This is more something at a lower level IMO.

[mgautierfr]

To really fix this problem then kiwixlib would have to have an interruptable API but that doesn't seem likely as that would mean a rewrite of an external API. Even then I don't think it would be very effective as the majority of time is spent locating the xapian index which must be done anyways?

I agree here. It will not really be possible to interrupt the search usefully. There is only one xapian function taking a lot of time. Libzim don't have the hand on this and cannot interrupt the long step.
The only real thing to do is to do the search in a different thread. This way we can still answer ui event (and tell user a search is going one).
If we don't want the search result (because user enter another query) we have to launch another thread and discard result of the previous search. (Ideally we would kill the previous search and launch another one, but killing a thread could lead to memory leak and inconsistent state).

The question is where we create this thread.
For now it is to the user to create this thread and handle this correctly.
Creating it on the libzim level is pretty complex has we have to adapt the whole api to have async api. (Not speaking about the wrapping). It may come in the future but I don't thing it will come soon.

Coroutine works pretty well with cooperative "tasks" (where the tasks regularly give back control to the "tasks mananger" and allow other tasks to run). This is not specific to kotlin, this is specific to coroutine. Without a async api, it will not be possible to have cooperative search task.

To me I remember you said, that thread being created at Kotlin level should be interrupted at Kotling level

I don't remember I said that but I totally agree. Thread created by one "system" but be handle by this "system".

---

To conclude, I think there will be no real solution coming from libzim/kiwix-lib soon.
The best way to solve this is to use (several) thread to do the search.

[macgills]

The best way to solve this is to use (several) thread to do the search.

Which is what the updated solution does and in fairness what the original solution did but much less cooperatively because of the nature of RxJava's implementation of Reactive Streams. We will always have problems on device with low thread counts that search on an sd card

The idea of the "interuptable" xapian function is quite unpalatable but would work from the app side like so

searchSuggestions(etc) { //this function gets executed at intervals in a task and would cancel my coroutine }

but the xapian function would need to be rewritten and I don't think you can pass lambdas across JNI for starters? It is not what we want to do

[mgautierfr]

Which is what the updated solution does and in fairness what the original solution did but much less cooperatively because of the nature of RxJava's implementation of Reactive Streams

This is something I cannot comment :)

We will always have problems on device with low thread counts that search on an sd card

I agree here. If the device need time to get the data on the sdcard, it will takes time. What we can do is to ensure the application do not freeze. And maybe do a bit of warmup to start loading data before the user needs it.

The idea of the "interuptable" xapian function is quite unpalatable but would work from the app side like so

searchSuggestions(etc) { //this function gets executed at intervals in a task and would cancel my coroutine }

but the xapian function would need to be rewritten and I don't think you can pass lambdas across JNI for starters? It is not what we want to do

I don't really understand that.
But there we(I) will not rewrite xapian methods. This is an external library we are using.
A cooperative task/api means that we give back control when we are waiting for data (io). Here we are waiting for xapian to run. On higher level everything run on the same thread but on lower level it means that we use thread to do the search while we give back control on main thread.
I suppose the low thread counts is low whatever the code using threads. If this is too low for kotlin, it will be too low for cpp also.

[Beater34]

Thank you all for doing so much work to fix the slow search problem. I had slow searches with Kiwix 3.3.3 on a Motorola G6 Plus phone with wikipedia_en_all_maxi_2020-06.zim on an SD card (see my post of Jul 8, above). I now have Kiwix 3.4.0 build 6230400, but searches still take 1-2 minutes, and sometimes selecting one of the search results gets no response and I have to wait another minute or so until the selected article opens (or I restart Kiwix). Please let me know if I can provide any more information that would be helpful.

[macgills]

Please await Kiwix 3.4.1's release and if you still experience these issues then opening a new ticket would be best

[Wikinaut]

I built and installed the current develop branch be91e6ba6 and found that the search time in the big enwiki file (80 GB) went down from 1..2 minutes (beta release 3.3.4 or so) to 10..20 seconds. ("external" SD Card 256GB on HUAWEI P Smart 2019)

[Wikinaut]

So from my side: Go on!

[kelson42]

@Wikinaut It is a good news, but still 10 times too slow. Hopefullz 3.4.1 will solve the problem.

[Wikinaut]

Yes, I was still disappointed, because I thought, the develop has that built in, hasn't it?

[kelson42]

@Wikinaut Yes it does, you are right. We will release 3.4.1 and then probably have to start a new round of investigation/fix to come down to reasonable times even in external SD cards worse scenarios.