Xowa: Offline Installation fails with windows_64_v3.11.2.1 due to file naming issue.

[XowaWilliam]

I did offline installation of recent November files I downloaded from en.wikipedia.org and the installation failed.
I then tried the installation with older windows_64_v3.8.1.2 and it worked fine.
After much fiddling, I finally pinpointed the problem to file naming. The current windows_64_v3.11.2.1 does not recognise dated files.

When I download files from en.wikipedia.org, I habitually use the dated files like enwiki-20161120-pages-articles.xml.bz2 and not enwiki-latest-pages-articles.xml.bz2 _( I do this so one does not confuse files from another date )_

After supplementing date with latest, windows_64_v3.11.2.1 crunched right through to the end perfectly.

So I conclude that somewhere between windows_64_v3.8.1.2 and windows_64_v3.11.2.1 a change has taken place in the software.

Sorry, unfortunately in all the fiddling, I never saved a log file. But I am glad I found a workaround :)

[gnosygnu]

Hey. Thanks for writing up the issue, as well as all the detailed info.

Unfortunately, I wasn't able to reproduce it. I list my steps below. Can you take a look at it and let me know what I'm doing differently?

Thanks.

---

• I downloaded the English Wikinews XML dump: https://dumps.wikimedia.org/enwikinews/20161120/enwikinews-20161120-pages-articles.xml.bz2
  
  
  
  • Note that I'm using Wikinews and not Wikipedia because Wikinews is much smaller. I don't think the issue is tied directly to Wikipedia (XOWA has practically no special code that says "if English Wikipedia" do this)
  
  
• I went to Import/Offline at home/wiki/Dashboard/Import/Offline and changed the following:
  
  
  
  • Wiki to "Wikipedia" to "Wikinews"
  
  
  • Language from "Simple English" to "English"
  
  
  • "Where to get the dump" from "download" to "read from file".
    
    
    
    
    
    • I then chose the file at C:\xowa\wiki\en.wikinews.org\enwikinews-20161120-pages-articles.xml.bz2
    
    
    
  
  
• Pressed "Run Script"

The import proceeded. It popped up a message box due to the missing category dumps, but otherwise English Wikinews opened.

[XowaWilliam]

I just tried to reproduce the fault with Simple wiki and it worked perfectly.
So the problem must somehow be unique to en.wikipedia.org
Unfortunately I am pressed for time ( Friday ), but will try again this weekend with en.wikipedia.org and send you the log file.

[gnosygnu]

Cool. Thanks for trying to reproduce it.

I'll give it a go with English Wikipedia. I started downloading the dump from https://dumps.wikimedia.org/enwiki/20161120/enwiki-20161120-pages-articles.xml.bz2 .

I'm going to let it run overnight, do an import tomorrow morning, and report back in this thread. Thanks.

[XowaWilliam]

I must just stress that I am not placing the blame on the primary file enwiki-20161120-pages-articles.xml.bz2
I placed all 4 files in the xowa\wiki\en.wikipedia.org directory, and the blame could be any one or more of the following files.

I failed to mention that when I did the test with Simple wiki this morning, even though I placed the four files as above, I happened to notice it downloading files from the internet, even though those files were in the directory, except my files were dated, and the files it downloaded were labeled latest.
I think this is how I picked up the fault yesterday. I knew it should not be doing any downloading.

Maybe I should not be placing all these 4 files in the same directory, maybe you can enlighten me.

Got to run now.

[gnosygnu]

I failed to mention that when I did the test with Simple wiki this morning, even though I placed the four files as above, I happened to notice it downloading files from the internet

Hmm... This could be the issue. XOWA may look for the latest files and try to downloaded them if not there (ignoring the 20161120 files). If so, you could probably get around that by disabling Web Access (at Options/Security). Of course, as you cleverly discovered, renaming your files to latest also works. ;)

Maybe I should not be placing all these 4 files in the same directory, maybe you can enlighten me.

Nope. You're correct. The easiest thing to do is put them in the xowa\wiki\en.wikipedia.org directory. XOWA looks there first. If XOWA is downloading the latest files, then that's something I'll need to fix.

I'm downloading the other three files now. Will report back sometime tomorrow. Thanks!

[gnosygnu]

I imported them today and they worked -- both with Web Access Enabled and disabled.

Web Access Enabled does have a bug in that it ignored the 20161120 category files and downloaded the "latest" category files again. I'll fix that again for a future release.

With that said, I think this issue is similar to https://github.com/gnosygnu/xowa/issues/98#issuecomment-263153111 . v3.11.2 had a bug that could cause the category import to stall on some older machines. Based on your log, it looks like it froze there.

Can you try reimporting with the version mentioned therein?

Thanks!

[XowaWilliam]

I can confirm that the FIX xowa_app_windows_64_v3.11.2.8 from your Google Drive account solved the lock up problem. https://drive.google.com/open?id=0B9cb52zjL2rIY1ppR3MwbjdoWXc

However the build time had tripled.
Hopefully this can be remedied.
I am willing to run more trials if needed.

I must just add that this offline test was done with my internet literally off to prevent downloading of enwiki-latest-categorylinks.sql

[gnosygnu]

Cool. Again, many thanks for the work and confirmation.

In short:
v3.9.2 through v3.11.2 will stall when importing English Wikipedia on older systems. Tonight's v3.12.1 will fix this issue, though the process will still slow down from about 2 hours to as many as 5 hours.

With more detail:
Wikipedia changed the category system to handle numeric sorting around 2016-09. This involved enabling something called "UCA-collation" ( https://phabricator.wikimedia.org/T136150).

Unfortunately, this UCA-collation broke paging forward / backward in the XOWA Category pages. The only way to fix it was to reproduce Wikipedia's Category system, which involved processing a lot more data. This new data (about 10 GB) is what's responsible for the slowdown.

A newer system with an SSD, or even a stronger processor will make this process faster, but it'll probably still weigh in around 3 hours total. I can try to optimize later, but ultimately, there is just so much data that the import will be slower.

[gnosygnu]

This fix is included in tonight's v3.12.1.1 release. As #108 was related and closed, I'm going to go ahead and close this as well. Feel free to continue the discussion here.

Thanks.