Xowa: Bug Report: Bugged Portal (?)

[Ope30]

Please check out de.wikipedia.org/wiki/Portal:Kölner_Dom
What's your result? Because, for me, the page is bugged. (See https://image.prntscr.com/image/A_H-8aA6TQC1-sgA2jvdCg.jpeg)
Has it something to do with https://github.com/gnosygnu/xowa/issues/156?

[gnosygnu]

Hey, thanks for the details.

The problem is that the page uses the User namespace which are not part of the HTML database downloads.

You can build it on your own, but it's a bit complicated. You'd have to do an offline import of home/wiki/Dashboard/Import/Offline using a meta-current dump. For example: https://dumps.wikimedia.org/dewiki/20170801/dewiki-20170801-pages-meta-current.xml.bz2

{{Benutzer:Anitagraser/Textbox
|BILD=Aldus leaf unicode2619.svg
|TITEL=Willkommen im Portal Kölner Dom!
|editpage=Portal:Kölner Dom/Willkommen
|INHALT=Willkommen
|COLOR=#f8f3ec
}}

Let me know if you have any other questions.

Thanks!

[Ope30]

The problem is that the page uses the User namespace which are not part of the HTML database downloads.

You can build it on your own, but it's a bit complicated. You'd have to do an offline import of home/wiki/Dashboard/Import/Offline using a meta-current dump. For example: https://dumps.wikimedia.org/dewiki/20170801/dewiki-20170801-pages-meta-current.xml.bz2

Ah. Damn.. Is there a way to import every single User namespace article, also known as Portal?

{{Benutzer:Anitagraser/Textbox
|BILD=Aldus leaf unicode2619.svg
|TITEL=Willkommen im Portal Kölner Dom!
|editpage=Portal:Kölner Dom/Willkommen
|INHALT=Willkommen
|COLOR=#f8f3ec
}}

Do I have to do something with that code?

Let me know if you have any other questions.

Yeah. Can you explain to me how to download new wikis on my own? Because, that could spare some time for you, and you don't always have to upload when I can do it on my own, you know. If I download the same stuff I get in the Download Central, then I will just do it on my own! Not saying this because you are not uploading recently. I know you're very busy with work, etc!

[gnosygnu]

Ah. Damn.. Is there a way to import every single User namespace article, also known as Portal?

Yeah, you have to do the import manually. Specifically:

• Download a meta-current dump. For example, https://dumps.wikimedia.org/dewiki/20170801/dewiki-20170801-pages-meta-current.xml.bz2
• Go to home/wiki/Dashboard/Import/Offline
• Follow the instructions at the bottom of the page and do an import

Do I have to do something with that code?

Nah, that was just an example.

Yeah. Can you explain to me how to download new wikis on my own?

You can start off with the manual import above. Let me know how it goes, and we'll take it from there.

Thanks!

[Ope30]

I tried to do it on my own today, and it worked. I thought it was way harder, but it's actually pretty easy!

Unfortunately, I forgot a file or something, because after the Import it said " .sql file not found in dir". I think it's referring to full-text-search? However, I know why. I did not download every file that was provided. I only downloaded one specific file that you mentioned. Next time I'm going to download all files.

Though, articles are missing images. I think this has nothing to do with the fact that I did not download the complete set, because just to be sure I downloaded a small wiki with all files, and it was missing images (compared it to the online version). Whatever. Could you include the User namespace Article for future wikis (2017-08 or -09 German, English Commons)? That would be great, I'd look after to download Wikis on my own, but fully working.

Curious about this: Let's say you fix a bunch of bugs in Commons, and I download it on my own, do these fixes still apply? Or do I have to download your set? And what about your download speed when you download wikis at their Dump Site? It takes way longer than downloading it at archives.org

[gnosygnu]

I tried to do it on my own today, and it worked. I thought it was way harder, but it's actually pretty easy!

Cool. Good to hear.

Unfortunately, I forgot a file or something, because after the Import it said " .sql file not found in dir". I think it's referring to full-text-search? However, I know why. I did not download every file that was provided. I only downloaded one specific file that you mentioned. Next time I'm going to download all files.

Yeah, you need a few other files. From the instructions ;)

• Download two additional files for the category system:
  
  
  
  • wiki-date-categorylinks.sql.gz
  
  
  • wiki-date-page_props.sql.gz.
  
  
• Download an additional file for sorting search results by PageRank
  
  
  
  • wiki-date-pagelinks.sql.gz.
  
  

Though, articles are missing images. I think this has nothing to do with the fact that I did not download the complete set, because just to be sure I downloaded a small wiki with all files, and it was missing images (compared it to the online version).

When you use the offline option, you still need to download the images. The offline import doesn't download images. If you do want images, the process is more intricate, as you would need to do command-line: http://xowa.org/home/wiki/Dev/Command-line/Dumps

Whatever. Could you include the User namespace Article for future wikis (2017-08 or -09 German, English Commons)? That would be great, I'd look after to download Wikis on my own, but fully working.

Sorry, but I don't have plans to do this. The User namespace literally doubles the processing time and space requirements. For English Wikipedia, it would jump from 100 GB to 200 GB and would take one week to build. If someone wants to do this, they can try going thru the command-line option above. Otherwise, it really isn't worth the effort.

Curious about this: Let's say you fix a bunch of bugs in Commons, and I download it on my own, do these fixes still apply? Or do I have to download your set?

In most cases, you would have to download another set.

And what about your download speed when you download wikis at their Dump Site? It takes way longer than downloading it at archives.org

archive.org is a free community resource, but unfortunately it is not fast. Unfortunately, there's really nothing to do about it.

[Ope30]

Yeah, you need a few other files. From the instructions ;)

Yoah. The thing is, I downloaded these files when I was downloading the one wiki for testing, you know. I downloaded hi.wikibooks, though I don't have full-text-search. As it says in the instructions, I have to start the dump with "hiwikibooks-20170801-pages-meta-current", which I did. After the dump is complete, these files you listed above get deleted automatically. Maybe the wiki doesn't support full-text-search yet, or so. Not even the original one. Not sure.

When you use the offline option, you still need to download the images. The offline import doesn't download images. If you do want images, the process is more intricate, as you would need to do command-line: http://xowa.org/home/wiki/Dev/Command-line/Dumps

That looks.... Ugh.. Okay. Will look at it

Sorry, but I don't have plans to do this. The User namespace literally doubles the processing time and space requirements. For English Wikipedia, it would jump from 100 GB to 200 GB and would take one week to build. If someone wants to do this, they can try going thru the command-line option above. Otherwise, it really isn't worth the effort.

Really? I did not know it would make such a big difference. But okay. I understand.

Can you include 2017-08 (or 9, up to you) German Wiki as well, along with Commons? I think I'll stop downloading wikis for a while, of course if there's nothing that forces me to redownload!

archive.org is a free community resource, but unfortunately it is not fast. Unfortunately, there's really nothing to do about it.

True.... :(

[gnosygnu]

Yoah. The thing is, I downloaded these files when I was downloading the one wiki for testing, you know. I downloaded hi.wikibooks, though I don't have full-text-search. As it says in the instructions, I have to start the dump with "hiwikibooks-20170801-pages-meta-current", which I did. After the dump is complete, these files you listed above get deleted automatically. Maybe the wiki doesn't support full-text-search yet, or so. Not even the original one. Not sure.

Yeah, full-text search needs to be done through the command-line. There's no GUI for it.

Can you include 2017-08 (or 9, up to you) German Wiki as well, along with Commons? I think I'll stop downloading wikis for a while, of course if there's nothing that forces me to redownload!

Let's try 2017-09 German Wikipedia.

Thanks.

[gnosygnu]

I think this issue was related to the missing User namespace for German Wikipedia. As such, I'm marking this issue closed for now. Let me know if there's anything else, and I'll reopen.

Thanks.