Xowa: Full-text search: Initial release

[gnosygnu]

Hi all. This issue will track the upcoming release of full-text search for the Wikipedias.

I have a primitive version working on my machine that does full-text search over all articles in 2017-03 English Wikipedia. I still have a lot to do, but the initial functionality is pretty promising. I'm planning to add it officially to the XOWA release sometime in the next few weeks.

I'm looking for testers who could do the following:

• Test the performance on their machines and let me know details. For example, my machine is a Windows 7 box with a i7-6700K CPU @ 4.00 GHz, 32 GB RAM, and an SSD. A search for something like "third planet sun solar system life terrestrial world tectonic atmosphere" brings back the first 100 results in 1 second.
• Test the search in non-English wikis. I haven't tested French, German, or any others. I'm curious how accents, stop words (le in French), and other possible multi-lingual aspects work.
• Test various search phrases and let me know how useful / useless it is. If you're going to do this, please note that full-text search isn't a search engine. The results are going to be less like Google, and more like https://en.wikipedia.org/w/index.php?title=Special:Search . Still, XOWA uses Lucene, so you can do some fancy stuff. See https://lucene.apache.org/core/2_9_4/queryparsersyntax.html
• Suggest minor improvements / changes. I have my own list I'm working through, but I'm always interested in what others think.

If you want to help out, here's what you'd have to do:

• Download and install v4.4.1 from https://github.com/gnosygnu/xowa/releases/tag/v4.4.1.1703. Note that this is a beta release, so there might be regression bugs. If there are, I'll fix them as soon as possible, but you may have to wait a few days.
• Use a wiki from Download Central. Note that this won't work out-of-box for wikis that are imported from Import Online / Import Offline.
• Go to home/wiki/Special:XowaSearchBuilder
  
  
  
  • Enter in your wiki's domain. For example, "simple.wikipedia.org"
  
  
  • Press the run button and wait. You'll see a progress message every 10,000 pages. For context, Simple Wikipedia takes 2 minutes on my machine, and English Wikipedia takes 3 hours.
  
  
  • Note: if you're building English Wikipedia, you'll need about 37 GB of free space for the search index.
  
  
• Once the build is done, go to Special:XowaSearch for your wiki. For example, simple.wikipedia.org/wiki/Special:XowaSearch
• Enter search phrases. For anyone using the field syntax, there are only two fields: title and body. Again, a Lucene reference would be helpful: https://lucene.apache.org/core/2_9_4/queryparsersyntax.html

Anyway, if you have questions / comments, please add below. Otherwise, I'll keep this issue open until full-text search makes it into the official release.

Thanks.

[Ope30]

I tested the full-text-search function on Windows 10 64 bit Intel Core i5-6600K @ 3.50GHz 16GB RAM.

I was using Simple Wikipedia and I get very fast search results. I searched "oldest", but it actually doesn't matter what I look for. I got Simple Wikipedia off the Download Central, I redownloaded it. The search build took me 2 minutes as far as I remember. There is not much to say about it because it works the way it should, you know. I still have to test it with other wikis. I believe it doesn't make a big difference. I could be wrong. Do I have to do this for the rest of the wikis in order for this to work? Or can I download it?

However, I hope I provided enough information.

I have two requests, can you add it next to "Search using Special:All Pages" or atleast somewhere noticeable? Also, can you make it so that you can see information like https://simple.wikipedia.org/w/index.php?search=castle&title=Special:Search&profile=default&fulltext=1&searchToken=b0krg4zxw24q2udauj6rv66v4, so you can see some text?

-I get random freezes (XOWA not responding). After a few seconds it's restored. Not sure who is the trigger.
-Here's some "error": http://image.prntscr.com/image/067da0b2d7234a00b2e1991ec2264bb1.png
I got it when I clicked on an result after using full-text-search. It's random. Or?

[gnosygnu]

@Ope30 Thanks for testing! I'm running out the door now, but I'll quickly answer one:

Do I have to do this for the rest of the wikis in order for this to work? Or can I download it?

Nope. You shouldn't have to redownload anything. It should work with any Download Central wiki.

Next time, try restarting after rebuilding the search index. If that still doesn't work, let me know, and I'll investigate further.

I'll answer the rest later.

---

For everyone else: I goofed in yesterday's release. It included a version of Lucene that only works for Java 1.8. I want to give other users time to move to Java 1.8, so I can't force it on everyone yet.

I uploaded a new one for Java 1.7 here: https://github.com/gnosygnu/xowa/releases/tag/v4.4.2.1703. If you've built search indexes for your wiki, please download this version and rebuild them.

Sorry about that.

[Ope30]

Nope. You shouldn't have to redownload anything. It should work with any Download Central wiki.

Ah. Okay, but I do have to build all the indexes for all the other wikis?

Next time, try restarting after rebuilding the search index. If that still doesn't work, let me know, and I'll investigate further.

Uhm, I think my comment wasn't as detailed as I expected, sorry. Ehm, I don't have any problems regarding building-indexes. What I meant by "download it" was to download the indexes separately. :P

[gnosygnu]

I tested the full-text-search function on Windows 10 64 bit Intel Core i5-6600K @ 3.50GHz 16GB RAM.

Cool. Thanks for the machine specs!

I was using Simple Wikipedia and I get very fast search results. I searched "oldest", but it actually doesn't matter what I look for.

Yeah, the Lucene search index is pretty fast. It should handle any one word or combination of words you throw at it.

I got Simple Wikipedia off the Download Central, I redownloaded it.

Cool. Any existing Download Central wiki is viable for the index build

The search build took me 2 minutes as far as I remember.

Just as fast as my machine. :)

There is not much to say about it because it works the way it should, you know. I still have to test it with other wikis. I believe it doesn't make a big difference. I could be wrong. Do I have to do this for the rest of the wikis in order for this to work? Or can I download it?

Yeah, for now, you have to build the wikis manually on your side. It's still too early for me to upload anything to Download Central.

Also, please make sure you build the indexes with today's 4.4.3 release, not yesterday's 4.4.2 release. Today's release should also have better support for non-English languages.

I have two requests, can you add it next to "Search using Special:All Pages" or atleast somewhere noticeable?

Well, I'm planning to change the box to go directly to Special:XowaSearch and demote Special:AllPages to a choice off the main menu

Also, can you make it so that you can see information like https://simple.wikipedia.org/w/index.php?search=castle&title=Special:Search&profile=default&fulltext=1&searchToken=b0krg4zxw24q2udauj6rv66v4, so you can see some text?

It's sort of there in today's release already. If you do https://simple.wikipedia.org/wiki/Special:Search?query=castle , you'll get "castle" in the query box. You'll still have to hit enter though. I can make it run the search automatically for the next update.

-I get random freezes (XOWA not responding). After a few seconds it's restored. Not sure who is the trigger.
-Here's some "error": http://image.prntscr.com/image/067da0b2d7234a00b2e1991ec2264bb1.png
I got it when I clicked on an result after using full-text-search. It's random. Or?

Not sure. I haven't seen it yet. A few questions:

• Can you reproduce this error reliably? If so, what words are you using?
• Are you executing the searches right after each other. I need to handle cancelling previous searches, so it's possible that an earlier search is messing up the current one.
• Do you have Search open in two different tabs? I need to put in some changes to handle Search from multiple tabs

Ah. Okay, but I do have to build all the indexes for all the other wikis?
Ehm, I don't have any problems regarding building-indexes. What I meant by "download it" was to download the indexes separately. :P

Yeah, for now, the indexes will need to be built. It's still too early to incorporate them into Download Central as the format is still in flux (as shown by today's release). In about a month or two, the new Download Central wikis will have them as an option to downoad. Until then, anyone who wants to experiment with them can build it on their own. Hope that makes sense.

Thanks!

[Ope30]

Yeah, for now, you have to build the wikis manually on your side. It's still too early for me to upload anything to Download Central.

Okay.

Well, I'm planning to change the box to go directly to Special:XowaSearch and demote Special:AllPages to a choice off the main menu

Okay. Let's see the outcome of that. =/

Also, please make sure you build the indexes with today's 4.4.3 release, not yesterday's 4.4.2 release. Today's release should also have better support for non-English languages.

It's sort of there in today's release already. If you do https://simple.wikipedia.org/wiki/Special:Search?query=castle , you'll get "castle" in the query box. You'll still have to hit enter though. I can make it run the search automatically for the next update.

I still have 4.4.1 1703, where do I get 4.4.3 from? And yeah, I agree, make it automatically!
Just a question, I did not know you can search images as well, would it be possible for you to add that function to the full-text-search function? I don't believe that's even possible.

Can you reproduce this error reliably

So, I found out that it was appearing because, if you search for "car" (or anything else) through 1000 max pages per wiki, and you immediately click on an result, you get the error. I tested and waited for the search to be complete, because it took me like 10 seconds through 1000 pages, and I did not get the error.

Yeah, for now, the indexes will need to be built. It's still too early to incorporate them into Download Central as the format is still in flux (as shown by today's release). In about a month or two, the new Download Central wikis will have them as an option to downoad. Until then, anyone who wants to experiment with them can build it on their own. Hope that makes sense.

Makes sense. I will try to build them in the next coming days, or I could just wait to download them, I don't know. By the way, I as well will try to download 2017-03 Wikipedia. I need the time. Just so you know.

[gnosygnu]

Okay. Let's see the outcome of that. =/

Hmm... I take it you don't agree? Do you use Special:AllPages a lot? Or would you rather have another text box. Personally, I don't want to add a third, because it makes it cluttered. Also, Windows Explorer and Firefox have two boxes, but not necessarily three

I still have 4.4.1 1703, where do I get 4.4.3 from?

Ooops. Type. Meant v4.4.2. See: https://github.com/gnosygnu/xowa/releases/tag/v4.4.2.1703

Just a question, I did not know you can search images as well, would it be possible for you to add that function to the full-text-search function? I don't believe that's even possible.

Unfortunately, not really. You would need commons and even then, the search could be unsatisfying as there may not be enough keywords.

So, I found out that it was appearing because, if you search for "car" (or anything else) through 1000 max pages per wiki

Yeah, 1000 is too much. ;) I'll add paging in a future release, but 20 - 100 will probably be the recommended size

Makes sense. I will try to build them in the next coming days, or I could just wait to download them, I don't know. By the way, I as well will try to download 2017-03 Wikipedia. I need the time. Just so you know.

Sure, up to you! Just make sure you use 4.4.2 if you build any more. Let me know if you run across anything. Thanks!

[Ope30]

I take it you don't agree? Do you use Special:AllPages a lot? Or would you rather have another text box. Personally, I don't want to add a third, because it makes it cluttered. Also, Windows Explorer and Firefox have two boxes, but not necessarily three

Can you let it there for me please?! Or atleast add a possible option to turn all three off/on? :(

Unfortunately, not really. You would need commons and even then, the search could be unsatisfying as there may not be enough keywords.

Damnit. I would have loved to see this. But okay!

Yeah, 1000 is too much. ;) I'll add paging in a future release, but 20 - 100 will probably be the recommended size

Agreed. 100 is the best you can get.

Sure, up to you! Just make sure you use 4.4.2 if you build any more. Let me know if you run across anything. Thanks!

I definitely will let you know.

[gnosygnu]

Can you let it there for me please?! Or atleast add a possible option to turn all three off/on? :(

I'll take a look at it, but it's really easiest for me to leave two boxes (URL and AllPages / Search) then to make it three boxes (URL and AllPages and Search) with corresponding options

[gnosygnu]

Another note. It looks like there will be yet another change to the search . It turns out Lucene 6.4 is more accurate out-of-box than Lucene 5.5. That means I may have to go back to Java 1.8.

I'll upload another build by the end of this week with hopefully a finalized format. In the meantime, if you're generating search indexes, please only generate them for a handful of wikis. The search index format is definitely going to be in flux for another release or two, so please keep in mind that any search index you generate will have to be regenerated again when you get the next version.

Sorry about the trouble. Thanks.

[Ope30]

I'll take a look at it, but it's really easiest for me to leave two boxes (URL and AllPages / Search) then to make it three boxes (URL and AllPages and Search) with corresponding options

Alright.

Another note. It looks like there will be yet another change to the search . It turns out Lucene 6.4 is more accurate out-of-box than Lucene 5.5. That means I may have to go back to Java 1.8.

Sure. Do it!

I'll upload another build by the end of this week with hopefully a finalized format. In the meantime, if you're generating search indexes, please only generate them for a handful of wikis. The search index format is definitely going to be in flux for another release or two, so please keep in mind that any search index you generate will have to be regenerated again when you get the next version.

Roger that. I wanted to generate them for DE and EN only. To be honest, I don't need it for the rest.

[gnosygnu]

Full-text search has been included in tonight's v4.5 release. For more information, see the following:

• http://xowa.org/home/wiki/App/Full-text_search
• http://xowa.org/home/wiki/App/Full-text_search/Lucene/Search_indexes/Getting
• http://xowa.org/home/wiki/App/Full-text_search/Lucene/Search_indexes/Building

There will probably be more changes to full-text search in future versions, but they will get tracked under different issues. In the meantime, I'm going to close this issue out. Thanks.