Pmbootstrap: Wiki migration to full-featured wiki software

[MartijnBraam]

As discussed in IRC, I think its a good idea to move to a wiki with more features like a mediawiki (inspired by the archlinux wiki which has become the main documentation for more than the arch linux stuff) installation. The current wiki is becoming quite a large resource of info about general linux on devices and specific hardware and having some better features for management and searching would be nice.

If we're migrating then its better sooner than later because the longer we wait the more stuff we have to migrate.

[ollieparanoid]

I think migration should be automated with a script, so we don't miss anything and can automatically place redirects.

[ollieparanoid]

Pinging @dhruvkar, because he set up https://postmarketos.org.
Can you also host dynamic content (if so, which languages/software?), or could you make a subdomain and we redirect that to another host?

[dhruvkar]

@ollieparanoid if you want to use something like mediawiki, a subdomain would be best. I'm using something like github pages (netlify) at the moment to host the main site.

I'll setup a subdomain (wiki.postmarketos.org)?

[ollieparanoid]

Okay, then let's do the subdomain 👍
I think we should discuss the software some more. Also: who will be hosting it?

Personally I would be fine with mediawiki (I kind of miss markdown support, but otherwise it seems to be working great for Linux distributions). I think we could create a script with pandoc that converts the pages from markdown to GitHub pages.

[yuvadm]

The only thing I'd be concerned about is what happens to our data. Currently, we know that the wiki data sits on Github (as a git repo!) so we can always clone it on demand. With mediawiki, we have lots of information sitting on a server and no idea about how/where it's being backed up. As a current mediawiki admin for several different organizations, I don't enjoy maintaining those instances myself, and backups are always a hack. So I don't feel too comfortable with that. Also, markdown is a big plus.

To me it feels that we're still a bit early in the game to move to a mediawiki instance, and I'd rather focus our efforts on getting the next website up and running (on my TODO list for the next few days) and maybe have some nice wiki integration into the website (nice to have, not must). But that's just me, so I apreciate any effort that others want to attempt in this regard.

[kskarthik]

we can host wiki in a seperate github repo & use wiki.postmarketos.org. All we need is a good static wiki generator with md support like https://www.staticgen.com/mkdocs & a few others

[yuvadm]

@kskarthik the next branch of the website already has initial wiki integration

[ollieparanoid]

@yuvadm: the website is also a high priority of mine now (not sure if you saw that in the channel).

I think git-mediawiki integration has recently been upstreamed into git, so nowadays you can just git pull a mediawiki site (in theory, maybe you need an add-on on the MediaWiki side or something). Does that solve the backup issue for you?

The current wiki can't be searched through, which is a major downside. We also would not get this feature by statically hosting a copy anywhere (only through JS maybe, but that is a bad hack in my eyes).

I'm with you on markdown, I also like that we can use it on GitHub. But I'd trade that for proper search functionality.

(Signing in with the GitHub account would be convenient, with an option to register on the site.)

I don't think we need to rush this either. But if we can do without much effort and gain a lot of benefits, then we should go for it in my opinion.

[MartijnBraam]

Yeah the point of using mediawiki is that we don't have another static generator and something just something that a lot of people know how to use. Another complete wiki system is also an option.

I could put the mediawiki on one of my servers if you create a DNS record for it.

[craftyguy]

@MartijnBraam are your servers being backed up offsite?

I still remember the Great Gentoo Wiki Outage of (about 10-ish years ago), where lots of data was lost.

[MartijnBraam]

Ofcourse they are backed up, I can't just throw some stuff online and call it hosting. I can probably set up an extra backup if someone else here has a host to push it to so not all data is on my infra.

[craftyguy]

Having those backups accessible by others should be a requirement, I think. Just in case you decide to take an extended holiday if/when the wiki server goes down!

[dhruvkar]

@MartijnBraam @ollieparanoid let me know what the DNS record should look like when you're ready.

[MartijnBraam]

I've made an installation on my server to try to migrate some content. Which is accessable if you create a records in your own hosts file:

37.139.30.192  wiki.postmarketos.org

Not sure if we should add the public dns records already (but it probably couldn't hurt)

[dhruvkar]

added the record, should be accessible now. I'll leave it up, unless someone has a reason not to.

[MartijnBraam]

It should also be reachable on 2a03:b0c0:0:1010::49d:1001

[PabloCastellano]

I'm late to the discussion but my here are my two cents :smiley:

For me, this wiki works as expected as of now. I've also suffered from maintaining mediawikis for multiple projects and totally understand what @yuvadm says :smile:, and also love the markdown format that github provides. I'd not change it but I see that this issue is already forward and the dns got created. In any case, I also want the documentation to be downloadable and for the git-mediawiki plugin looks promising for this matter (first time I hear about it).

The current wiki can't be searched through, which is a major downside. We also would not get this feature by statically hosting a copy anywhere (only through JS maybe, but that is a bad hack in my eyes).

@ollieparanoid You can search in the github wikis (I've recently discovered it!). Just type in the search bar and then choose wiki in the second header (example).

[MartijnBraam]

My argument for mediawiki:

Now you can sort devices (even my what is supported), still needs some css to have colored boxes in the table for component support.

[yuvadm]

@MartijnBraam that table definitely looks sweet!

Not a counter argument, but to add an idea - consider we manage the device status in github wiki, or even somehow in the device-foo/deviceinfo metadata or APKBUILD metadata, and then we can automatically pull and generate device status to present on the main homepage.

[ollieparanoid]

@yuvadm: we would still need to automatize this in some way and it might be a bit complex for the homepage for now - we could also do that if we had git access to the mediawiki. I recommend that we get the new homepage out without automatically generating content from the wiki first.

So with the search feature working in Github like @PabloCastellano explained (thanks!), I do not see that much of an advantage to be honest. Here's my breakdown:

Advantages

• Support for templates
• Can be customized more
• Search is more obvious
• We can support mobile pages (not implemented as I read from the channel?)

Disadvantages (Can we fix them?)

• No (optional) github login -> higher barrier for people to contribute
• Not possible to download the wiki with git (yet, but this should be doable)
  
  
  
  • Ideally we would have a mirror in the postmarketOS organization of the whole wiki!
  
  
• No markdown support (seems like we would have to live with this)

So considering all the work @MartijnBraam has already put into the new wiki, I'd say we should do some research on what we can fix. And if it is enough and we can agree on the new wiki together, we should really migrate everything and put links to the new wiki, preferably with a script.

[ollieparanoid]

TLDR: I'm argumenting against it here, kind of changed my opinion later on (see below, I think you can skip this post).

---

I have put a lot of thought and research into this, and this is what I came up with. Sorry that it got a bit long, you might want to grab a cold drink for this.

Trying to work around the disadvantages

Git integration and public backup

git-mediawiki is integrated into upstream git and you should really be able to:

git clone mediawiki::http://yourwikiadress.com
git commit
git pull --rebase
git push

So while I'm sure that this does not perform as well as using git directly, it should work. This would also make it possible to set up a public mirror of the pages on GitHub, if we somehow make a hook, that pulls the pages from the self-hosted wiki on every update. But this will involve some work to get it right.

Markdown

This is a matter of taste. Almost all other distribution's wikis are written in MediaWikis, so it would also make sense to switch to MediaWiki's syntax.

To me it is very convenient to use the same syntax in (GitHub) issues, PRs, README.md files and similar, and in the Wiki - and from reading this thread it seems like I am not the only one in the project, that prefers Markdown.

One example where I find markdown simpler is embedding source code. We can use three backticks followed by the language instead of: <syntaxhighlight lang="python">...</syntaxhighlight> (source).

I did a bit of research, and I do not think that there is a good solution that allows us to use MD in MediaWiki (this one marked as beta would be the closest one). Ideally we would be able to provide both Markdown and MediaWiki syntax.

Optional GitHub login

The only MediaWiki plugin available for this job is OAuth2Github. But it does not seem to be in good shape, the code repository has "[unmaintained]" at the beginning of the description and a small user-base.

This is the most important point for me. Right now, most users of the project already have a GitHub account and can directly edit the wiki without registering for yet another website. I think that this leads to more contributions and is a reason why the wiki grew so fast.

Conclusion

I don't want to throw the good wiki we have right now over board and replace it with a half-baked solution that can theoretically be better, but needs a high amount of work and custom plugins to get there.

In other words: Being self-hosted is great, but not at the cost of increasing the barrier for contributions.

Alternative: Improve the current wiki

I've also looked at other wiki software, but MediaWiki really seems to be the best choice if we go self-hosted. Okay, then my recommendation is to actually stick with the GitHub wiki until we have something self-hosted, that addresses the points above.

Search feature is hard to find

We can help the user to use the search feature correctly by adding instructions about how to use it in the wiki. It is maybe two clicks more away compared to MediaWiki, but it is still usable.

No proper mobile page

• We can file this as issue upstream in GitHub's support page (although I think that they will reply something like "planned, but not now" - but just to make sure).
• If this is really important to someone, we could also take the effort and mirror the markdown pages on the homepage, as @yuvadm suggested. There we could make it as mobile friendly as we want. (NOTE: I would do that after we launched the new homepage, so that doesn't get blocked!)
• Also, is this really an issue? @MartijnBraam had this as argument in the first place, but then I read in the chat yesterday, that the MediaWiki table would not be supported on mobile because it is too big, and everyone was like "who cares" :p

Prettier devices table

I find the device table (with the combined letter field) quite readable and space saving actually. But I can understand if some people would rather have it sorted and this seems to be the number one reason for @MartijnBraam to introduce the MediaWiki at all.

So the devices table is the only relevant table in the wiki, that we have right now, which would be worth sorting.

How about we use @yuvadm's approach just for the devices page and write a script, that parses the table in the format we have now, and spits out a nice sortable JS table on the homepage? That should not be too hard, I can also assist or do it myself if this helps in resolving this discussion.

The supported devices are a major part of the project, so it would make sense to put them on the homepage directly, and with such a fancy table.

---

What to do right now?

I think more discussion is necessary, and it would be nice if we treated the new wiki as inofficial until we decide together that it's the way we want to go.

I am really interested in what you think about my proposal, everyone. Please note that this is only my opinion, and postmarketOS is a community project. So if the majority of the community prefers the self-hosted MediaWiki, then we should really do that instead of what I proposed above. Let's make a concious community decision!

[MartijnBraam]

The device information could be fully integrated into the postmarketos.org website and have it update using a github hook. Maybe by providing a deviceinfo.yml file for the website with all the info that's in the wiki and parse that into a standard device page and a overview table with the supported devices.

About the mobile tables in the wiki; that could be fixed relatively simple with some css (like the bootstrap reponsive tables). that makes the whole website responsive but the table would scroll in its own container.

[ollieparanoid]

The solutions with the tables sounds good to me, and from IRC it sounds more like people want the self-hosted wiki. So my position on this is neutral now.

[yuvadm]

@ollieparanoid thanks for the detailed overview of our options!

After reading your comment I understand now much better my main criticism against the standalone wiki (be it mediawiki or whatever) - fragmentation. We're still in very early stages of the project and we already have several different resources relevant for project information - Github repos (i.e. code), website, IRC/Matrix, Github wiki, Github issues, reddit.

Of course each of those serves a different function, but they also spread us very thin. At least Github provides us with 3 of those features wrapped into one place. Searching in a repo also searches issues and wiki, not to mention the single sign on.

I'm concerned that we still don't have enough traction to justify spreading ourselves over another resource. All in all, I'm -0 on moving to a standalone wiki at this point in time, I don't think it's neccesary but I won't get up on the barricades to stop the effort if it's in motion.

[ollieparanoid]

@yuvadm thanks for your insights on this! If I counted right, the score is: you and me slightly against it/neutral, the rest in this thread basically wants the MediaWiki. We have a winner! I'll try to put a migration script together. Let's move forward and thanks for discussing this everyone! 👍

[ollieparanoid]

• All pages migrated
• Broken links fixed in the wiki (that was exhausting...)
• Replaced all pages in the old wiki with redirects
• Made the old wiki read-only
• Published raw migration script notes here if someone is interested.
• Added a new wiki repo with remaining issues, the repo should also hold the backups in the future it's working already o/

[ollieparanoid]

One more thing, please register at the new wiki, and look through the pages (especially the device pages you have contributed to) and fix if you see something that broke during the migration. Thanks!

[kskarthik]

Why the new wiki is also not mobile friendly?

[ollieparanoid]

It is now, see https://github.com/postmarketOS/wiki/issues/5.