Xowa: Wikis: 2017-03 progress thread

[gnosygnu]

This is a general thread for XOWA wikis produced for 2017-03.

March has gotten to an early start as I released 2017-03 English Wikipedia this weekend. I'm still going to keep the queue light, as I want to focus as much attention as possible on full-text search.

With that said, here's the queue:

• 2017-03 English Wikipedia released 2017-03-13
• 2017-03 Hindi wikis released 2017-03-19
• 2017-03 Tibetan wikis released 2017-03-19
• 2017-03 Russian wikis released 2017-03-21
• 2017-03 Czech wikis
• 2017-03 Haitian wikis

Also, as a general reminder, here's how to get any of these wikis in XOWA or XOWA Android

• Go to Download Central
  
  
  
  • On XOWA: Tools -> Download Central (also home/wiki/Special:XowaDownloadCentral)
  
  
  • ON XOWA Android: Left menu -> XOWA -> Download button
  
  
• Look at the "Available" table
• Click the "blue down arrow" all the way to the right
• You'll see a status message in the bottom right about the recent additions

[Ope30]

Hey.

I just wanted to say that I decided to only download the rest of the list other than 2017-03 EN Wikipedia. I may do it in the future. Is there a reason to get 2017-03?

[gnosygnu]

Is there a reason to get 2017-03?

No reason. I just like keeping English Wikipedia up to date.

You may want to wait until 2017-04 (or 2017-05). By then, I'll have the full-text search databases integrated into Download Central.

Hope that helps.

[Ope30]

No reason. I just like keeping English Wikipedia up to date.

Me too. That's the only meaningful reason to keep redownloading. Just feel like 1 month doesn't make that big of a difference. But yes, I will download 2017-04 or 2017-05.

[gnosygnu]

Just feel like 1 month doesn't make that big of a difference. But yes, I will download 2017-04 or 2017-05.

Yup, agreed. One month doesn't make a big difference. It's nice to have an up-to-date English Wikipedia though. ;)

Btw: I uploaded the 2017-03 Russian wikis tonight. I'll do Czech / Haitian later this week / next week.

Thanks.

[Ope30]

Thank you for uploading the Russian Wiki(s)! Was about time!
Anyways, I only downloaded the big one. As always, I tested it just to be sure and, I believe we have talked about this already before, check out ru.wikipedia.org/wiki/Категория:Скрытые_категории
can you get to the next page? Do you see any articles in the categorie? I don't. I think you mentioned something with the System/Category or so. However.
Sorry, lol, can you check out ru.wikipedia.org/wiki/Маргарян, Андраник Наапетович for me? Just want to know if that's a missing image or not, I believe it's not, if it's one, I don't care. Just curious.

[gnosygnu]

Thank you for uploading the Russian Wiki(s)! Was about time!

No problem, and sorry for the delay.

check out ru.wikipedia.org/wiki/Категория:Скрытые_категории can you get to the next page?

Nice catch. I'm not seeing entries either. I believe the problem is related to the Cyrillic language. I looked into it a little tonight, but I think this is going to be difficult to fix. XOWA uses a library called ICU which is difficult to debug. I'll take a look at it more later this week, but there probably won't be a quick fix

can you check out ru.wikipedia.org/wiki/Маргарян, Андраник Наапетович for me? Just want to know if that's a missing image or not

I checked now, and if you mean the main infobox image, then it should be there. It was added back in 2016: https://ru.wikipedia.org/wiki/File:AndranikMargaryan_(cropped).jpg

When you get a chance, try the following:

• Confirm if you have these two files: ru.wikipedia.org-file-ns.000-db.002.xowa and ru.wikipedia.org-file-ns.000-db.009.xowa
• Go to: home/wiki/Special:XowaDiag?type=file.check&wiki=ru.wikipedia.org&file=AndranikMargaryan_(cropped).jpg . Paste the output here. I list mine below

Thanks.

---

machine.op_sys = windows
app.version = 4.4.2.1703
cfg_file_retrieve = y
fsdb.schema_is_1 = n
fsdb.mnt_file = C:\xowa\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa
fsdb.abc_file = C:\xowa\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa
fsdb.atr_file = C:\xowa\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa
fsdb.orig = gplx_key=sqlite;data source=C:\xowa\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa;version=3;

------------------------------------------------------
fsdb.orig.select: SELECT * FROM orig_reg WHERE orig_ttl = 'AndranikMargaryan_(cropped).jpg';
------------------------------------------------------
AndranikMargaryan_(cropped).jpg|0|1|2|361|986||

------------------------------------------------------
fsdb.fil.select: SELECT * FROM fsdb_fil WHERE fil_name = 'AndranikMargaryan_(cropped).jpg';
------------------------------------------------------
345731|1|1|2|-1|AndranikMargaryan_(cropped).jpg|-1|||

------------------------------------------------------
fsdb.thm.select: SELECT * FROM fsdb_thm WHERE thm_owner_id = 345731;
------------------------------------------------------
345732|345731|146|399|-1|-1|2|13329|||
1543309|345731|100|273|-1|-1|9|7850|||

------------------------------------------------------
fsdb.bin.select: SELECT * FROM fsdb_bin WHERE bin_owner_id = 345732;
------------------------------------------------------
345732|2|-1||
fsdb.bin.export = C:\xowa\wiki\ru.wikipedia.org\tmp\AndranikMargaryan_(cropped)_345732.jpg
fsdb.bin.len = 13329

------------------------------------------------------
fsdb.bin.select: SELECT * FROM fsdb_bin WHERE bin_owner_id = 1543309;
------------------------------------------------------
1543309|2|-1||
fsdb.bin.export = C:\xowa\wiki\ru.wikipedia.org\tmp\AndranikMargaryan_(cropped)_1543309.jpg
fsdb.bin.len = 7850

[Ope30]

Nice catch. I'm not seeing entries either. I believe the problem is related to the Cyrillic language. I looked into it a little tonight, but I think this is going to be difficult to fix. XOWA uses a library called ICU which is difficult to debug. I'll take a look at it more later this week, but there probably won't be a quick fix

I have no idea what you're talking about (library ICU etc) because I have no idea in coding. :p But yeah, take your time.

I checked now, and if you mean the main infobox image, then it should be there. It was added back in 2016:

http://image.prntscr.com/image/16b09897c1614242b3dd296959290de1.png can you see it?

`machine.op_sys = windows
app.version = 4.4.0.1703
cfg_file_retrieve = y
fsdb.schema_is_1 = n
fsdb.mnt_file = E:\XOWA\XOWA\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa
fsdb.abc_file = E:\XOWA\XOWA\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa
fsdb.atr_file = E:\XOWA\XOWA\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa
fsdb.orig = gplx_key=sqlite;data source=E:\XOWA\XOWA\wiki\ru.wikipedia.org\ru.wikipedia.org-file-core.xowa;version=3;

------------------------------------------------------
fsdb.orig.select: SELECT * FROM orig_reg WHERE orig_ttl = 'AndranikMargaryan_(cropped).jpg';
------------------------------------------------------
AndranikMargaryan_(cropped).jpg|0|1|2|361|986||

------------------------------------------------------
fsdb.fil.select: SELECT * FROM fsdb_fil WHERE fil_name = 'AndranikMargaryan_(cropped).jpg';
------------------------------------------------------
345731|1|1|2|-1|AndranikMargaryan_(cropped).jpg|-1|||

------------------------------------------------------
fsdb.thm.select: SELECT * FROM fsdb_thm WHERE thm_owner_id = 345731;
------------------------------------------------------
345732|345731|146|399|-1|-1|2|13329|||
1543309|345731|100|273|-1|-1|9|7850|||

------------------------------------------------------
fsdb.bin.select: SELECT * FROM fsdb_bin WHERE bin_owner_id = 345732;
------------------------------------------------------
345732|2|-1||
fsdb.bin.export = E:\XOWA\XOWA\wiki\ru.wikipedia.org\tmp\AndranikMargaryan_(cropped)_345732.jpg
fsdb.bin.len = 13329

------------------------------------------------------
fsdb.bin.select: SELECT * FROM fsdb_bin WHERE bin_owner_id = 1543309;
------------------------------------------------------
1543309|2|-1||
fsdb.bin.export = E:\XOWA\XOWA\wiki\ru.wikipedia.org\tmp\AndranikMargaryan_(cropped)_1543309.jpg
fsdb.bin.len = 7850`

[gnosygnu]

I have no idea what you're talking about (library ICU etc) because I have no idea in coding. :p But yeah, take your time.

Yeah, I usually provide too much detail. But, it's going to be difficult

can you see it?

Oops. I actually had page sync on before, and that's why I see the image.

It's definitely missing on my side. Let me look into it more, and post later. Thanks.

[gnosygnu]

It's definitely missing on my side. Let me look into it more, and post later. Thanks.

So, it looks like this is an issue with XOWA and Wikidata. More details below.

This can only be fixed with a new build, so it looks like Russian Wikipedia is back on the block again. I was planning to rebuild the wikis to include full-text search, but it may come sooner rather than later.

Out of curiosity, are you seeing a lot of missing images in Russian Wikipedia? The ones I spot-checked looked good, but this is a very small subset (32 pages).

Thanks.

---

• Wikidata returns a JSON document
• https://ru.wikipedia.org/wiki/Module:Wikidata is expecting a property off the root of the JSON document called datatype.
• This property seems to be added recently, as XOWA didn't have it.
• In this example (https://www.iwikdata.org/wiki/iProperty:P18), the datatype should be commonsMedia. Instead XOWA was returning null, which eventually becomes wikidata-item
• The next XOWA build will correctly return the new property

[Ope30]

Out of curiosity, are you seeing a lot of missing images in Russian Wikipedia? The ones I spot-checked looked good, but this is a very small subset (32 pages).

Nah. That was the only missing image I occurred.

This can only be fixed with a new build, so it looks like Russian Wikipedia is back on the block again. I was planning to rebuild the wikis to include full-text search, but it may come sooner rather than later.

No, it's okay. You don't have to reupload a new build of the Wiki. But, if you still want to, you can do that. I feel it's not that needed.

The next XOWA build will correctly return the new property

Alright. Any news regarding the rest of the list (Czech, Haitian)?!

[gnosygnu]

Nah. That was the only missing image I occurred.

Cool. Thanks for the confirm.

No, it's okay. You don't have to reupload a new build of the Wiki. But, if you still want to, you can do that. I feel it's not that needed.

Yeah, I was thinking of regenerating the Russian wikis for either April or May. I'm planning on doing another round through the wikis so that they all have the full-text search indexes.

Alright. Any news regarding the rest of the list (Czech, Haitian)?!

I'm still working on full-text search, and am planning to do a release between this Wed and Fri. Once I get it done, I'll build Czech / Haitian with the next full-text indexes. So hopefully this week, but maybe next...

[gnosygnu]

As we're almost in May, I'm going to mark this issue closed. I'll start a new issue for May / June