Xowa: Parse arbitrary wikitext to html either like Parsoid or with Parsoid?

[rexdouglass]

Is there a way to pass arbitrary wikitext to Xowa and retrieve html?

Is there a way to do that quickly for a large number of wikitext pieces, without launching an Xowa instance from scratch each time?

Perhaps the solution involves pointing Parsoid to Xowa as a MediaWiki server?

I need to do millions of pages monthly and do not want to hit the wikipedia servers ever; entirely offline with predownloaded files.

I'm already using the app.fetch_page from the commandline as well as the localhost:8080 to retrieve html of existing pages in the wikipedia. I'm next going to try to do a full dump to html with a gfs script.

The part I don't have yet is pass arbitrary wikitext, e.g. just a single key-value pair line from an infobox and retrieve an html conversion of just that one line. I know I can do that with Parsoid, but I do not want to stand up an entire MediaWiki instance from scratch, I'd rather let Xowa be my backend.

Thanks for any help you can provide.

[gnosygnu]

Hi! Thanks for the interest in XOWA!

Regarding your question: if you are comfortable with Java, you can create an XOWA parser object and have it parse wikitext. This will probably be the most performant way to parse data, and is comparable to what I use when I generate Wikipedia dumps. More details in the break below.

If you want a standalone solution, I could try to add some sort of POST call to HTTP Server that will allow you to parse wikitext. Something like http://localhost:8080/en.wikipedia.org/wiki/Special:XowaParse and passing in the wikitext in the request body. The downsides would be:

• It would take me at least a few days for me to code
• It won't be as fast as calling it directly from Java (because of the http intermediary)
• I'm not sure how well it will handle threading (if you try to launch a dozen or so workers that submit client requests). I've heard reports that XOWA server doesn't handle multiple clients well (note the HTTP server support was hand-coded by a contributor and me several years ago)

Finally, regarding Parsoid to XOWA, I'm open to that suggestion, but I've not explored that route, and am not sure well that would go since it would involve some sort of Javascript to Java bridge which wouldn't be technically trivial.

If Java is at all in your wheelhouse, I'd definitely recommend going with the option. Let me know your thoughts / comments / concerns.

Thanks!

---

package gplx.xowa;
import gplx.xowa.addons.parsers.mediawikis.*;

public class Test_class {
    public static void main(String[] args) {
        // create a new manager instance with the root directory of your XOWA installation
        // note that a full XOWA installation is needed, because it needs to load some standalone files (EX: Scribunto .lua files)
        // also note that the directory must end in a "\" if Windows or a "/" if Linux / Mac OS X; EX: "/home/me/xowa/" not "/home/me/xowa"
        Xop_mediawiki_mgr mgr = new Xop_mediawiki_mgr("C:\\xowa\\");

        // create a new worker instance
        // note that a worker cannot be shared between threads. however, each worker can be assigned to one thread and each thread-bound worker will not interfere with the other. This is what the html database builder does and I've tested that it's thread-safe for up to 64 such workers.
        // also note that each worker can only parse pages from one wiki. if you are parsing pages from two different wikis then you'll need two different workers
        Xop_mediawiki_wkr wkr = mgr.Make("en.wikipedia.org");

        // parse some wikitext. the below will print out "<p><i>Hello, world!</i>\n</p>"
        System.out.println(wkr.Parse("SomePageTitleWhichIsRequiredButCanBeAnything", "''Hello, world!''"));
    }
}

[rexdouglass]

Perfect, thank you. This is exactly what we needed to know. I agree the java route looks the best. I have hacked together parsoid solution right now, and I'd like to transition to this Xowa solution in the medium term. When we get resources on this, sometime in the new year, I'll drop by here to give an update.

[desb42]

To me, this smacks of 'edit preview', implemented on the http side of things
(hence issue #315)
My version does this partially

[rexdouglass]

In case it's at all clarifying, our biggest hurdle is template expansion.

Parsoid can handle the wikitext to html conversion offline, but it hits the wikipedia servers to grab templates and probably other things I'm not aware of.

This is bad from our perspective for two reasons (1) it's incredibly rate limiting, we can only make so many requests over long periods of time, plus it's impolite to hit their servers systematically, and (2) we want the option to build against specific dumps of wikipedia, and if a template changes in the live version we'll get different results depending on what day we run the conversion.

We can stand up a whole copy of wikipedia ourselves locally and point Parsoid at it instead, but as you know that's painful and something Xowa automates.

[gnosygnu]

Moving this item to done for now. Let me know if anything else. Thanks.