Julia: moving the manual to Markdown

[StefanKarpinski]

Related: https://github.com/JuliaLang/julia/issues/12489. Since Julia's docstrings are in Markdown format, longer term the manual will also move (back) to being in Markdown. We need a hosting solution for that as well. I'm opening this issue to discuss how to do this.

[tkelman]

11920 ?

[StefanKarpinski]

I think that we should go with a strictly static system that can be hosted on GitHub. Search on read the docs has been spotty at best and I think that static, pre-computed pure JavaScript search functionality could be as good if not better. This will probably require a build step since it seems unlikely that we can get a pure Jekyll setup to do this, but that's ok. We could also consider having a static site and farming our search out to some search engine. A major advantage of having the documentation system be fully static is that you could browse it locally, potentially with just a web browser, including search.

[jakebolewski]

What we really need is a bulleted list of the documentation system as is and the steps we need to do the full conversion. I don't think anyone wants to live in Markdown / RST purgatory.

[StefanKarpinski]

I searched issues for this but didn't find #11920 one since it was closed. Since that was closed with no comment, we can resume the discussion here.

[tkelman]

https://github.com/JuliaLang/julia/pull/11943#issuecomment-125583504 is probably the closest thing to "list of where things stand right now."

Missing pieces that need to be built off the top of my head are

• Website generation and hosting, with versioning and search
• Cross-reference notation in markdown
• Doctest notation in markdown - Lexicon has some of this, may be good enough as is
• Linkcheck from markdown

[StefanKarpinski]

Can we please not put the list in this issue but have an umbrella issue instead?

[tkelman]

Huh? I have no idea what you're going for with "umbrella issue" in terms of how you'd like this to be organized. What is an umbrella issue good for, if not tracking things that need to be done?

[StefanKarpinski]

My point was that this issue is specifically about moving the manual to Markdown, not any of the other stuff. Although I guess depending on how you look at it, all remaining steps can be seen as part of that.

[cormullion]

Something like mkdocs?

[MichaelHatherly]

https://github.com/GitbookIO/gitbook looks kind of cool. Has a mathjax plugin, search, and can build static html, pdf, and epub. From what I can tell Rust is using it for their manual. Doesn't appear to have sphinx-like linking/references though.

Does anyone have experience with it? I've only just found it this evening.

[blakejohnson]

I've never seen gitbook before, but it looks pretty slick.

[StefanKarpinski]

Why can't we just generate some HTML and apply apply some CSS to it? I'm not sure that we really need a 3rd party framework for this.

[rsrock]

Http://github.com/dcjones/Judo.jl ?

[IainNZ]

Judo == Pandoc, essentially. Having said that, I've used Judo and mkdocs lately, I liked both.

Could also just use Mustache.jl and do it at a lower level.

[StefanKarpinski]

The really appealing thing about using Julia packages and code to do all of this is that it's easier for people to get involved in hacking on it and fixing things. Also, fewer external dependencies.

[simonbyrne]

We could build the docs on Travis, use their post-build hooks to push to the results to the gh-pages branch, but at the moment this would require giving Travis commit access to the whole repo.

[StefanKarpinski]

Presumably Judo uses pandoc because @dcjones didn't feel like writing a Markdown parser. But now that we have our own Markdown parser it seems possible to either switch Judo to using it or allow it to use either one, which would give us a pure Julia documentation generation system.

[rsrock]

Is there anything wrong with pandoc, other than having to deal with another external dependancy? I use it a lot, I'm sure others do as well. I never really had a problem with it.

[tkelman]

pandoc's not the easiest thing in the world to get installed across all platforms. For Windows all they've got is an msi, and would rather not make the doc building depend on something that's only available as an msi if we can avoid it.

[StefanKarpinski]

In my experience, external documentation systems are a lot of hassle, especially, as @tkelman says, cross-platform. Given that we have all the pieces to do this ourselves, it seems like a better way to go. The time to fix bugs and other issues in Julia projects is also orders of magnitude faster.

[dcjones]

Presumably Judo uses pandoc because @dcjones didn't feel like writing a Markdown parser.

That's accurate. Judo is a little out of sync with recent developments, namely Markdown.jl and docstrings. I would prefer to use Markdown.jl at this point. First, because the pandoc isn't a simple dependency to provide. Secondly, because any processing of the parsed document (e.g. executing code blocks, inserting or extracting info) relies on pandoc's json serialization of it's AST, which is neither stable nor documented. So it's kind of a pain to update when pandoc changes something, and also requires that the dependency be a pretty specific version of pandoc.

[StefanKarpinski]

It sounds like moving Judo to using Markdown.jl would be a viable plan. Side benefit: Judo would become a widely used (within the Julia ecosystem), pure-Julia system for generating Julia documentation from Markdown. That's a really nice thing to have. If we can get it working well for Base Julia and for packages, then everyone can start using the same thing.

[StefanKarpinski]

In particular, we can solve the document search issue once and for all. I suspect search can be done purely client-side in JavaScript.

[waldyrious]

Just taking the opportunity to suggest @lcw to consider moving jocco to Markdown.jl as well :)

[hayd]

Once the documentation uses markdown docstrings doing this outside of base seems much more sensible to me. I would favor inserting in the latest docstrings into generated markdown files, I have been putting something similar together as part of Lexicon. https://github.com/MichaelHatherly/Lexicon.jl/pull/129

I don't really understand the need to ship the means to build the documentation site/pdf inside Base.

As mentioned once we have them Markdown files there are many options for static hosting e.g. Judo.

[tkelman]

We at least need the build harness and scripts for the documentation to be included in base, since the buildbots, binaries, and distro packagers need to be able to build and include the html docs. It can make use of external Julia packages (or other repositories like the JuliaDoc shim we have now for Sphinx) in doing so, though the distro packagers will have to go out of their way to provide an offline copy of all prerequisites.

[StefanKarpinski]

Ah, jocco looks so nice :-)

[StefanKarpinski]

I don't really understand the need to ship the means to build the documentation site/pdf inside Base.

I don't think we need to, although I do think that it may be something we want as part of the "standard packages" once that's a thing.

[StefanKarpinski]

Which is basically what @tkelman said.

[StefanKarpinski]

Please keep the irrelevant chatter down. My bad for starting it.

[prcastro]

I like the idea of using something written in Julia (even tough gitbook seems very pretty and flexible)

[StefanKarpinski]

@MichaelHatherly, what do you think about resurrecting my PR for using Sphinx to convert RST to Markdown to finally finish off this whole conversion business? At this point it seems like your Markdown syntax for Documenter.jl supports enough features that we need to make the transition lossless without too much complaining. Is the Documenter syntax available in Base Julia?

[MichaelHatherly]

Yes, I've been eyeing this for a little while now, though trying to get it done for 0.5 would be a bit of a squeeze I'd imagine. I really wouldn't want to hold up the release as happened with the docsystem last year.

Regarding feature-completeness of Documenter for use in Base: everything needed is probably there to get it translated with minimal loss, though there are some things we still don't have:

1. Direct to HTML output (currently needs mkdocs, which I'd like to drop before making use of this in Base). @mortenpi is currently working towards that so perhaps a few weeks away from completion I'd think.
2. No PDF output currently. I'm not sure how popular the PDF docs are for users. PDF output is a feature that will get added at some point, but currently that hasn't had any work done towards it.
3. Doctesting works most of the time, but still needs some polishing.

I think it might be reasonable to stick with the current system for the time being, and then make the switch once at least (1) is done. If that happens after or too close to 0.5 being released then we can transition the 0.6 docs to Documenter, see how that goes and then backport it since that should be pretty simple to do.

Is the Documenter syntax available in Base Julia?

Yes, just uses plain old markdown syntax. Just with a few things like cross-reference syntax embedded inside standard markdown link URLs. Won't be a problem as far as I can tell.

[StefanKarpinski]

Excellent. Sounds like we're very close then. I agree that it shouldn't hold up 0.5 in any way. We can potentially always backport it.

[MichaelHatherly]

We can potentially always backport it.

Yes, if it's done very early during 0.6 then backporting should probably been really simple I'd imagine.

(I'm just building #13308 to see how close that gets us to what Documenter needs. I suspect it's pretty close.)

[tkelman]

Would just have to be careful about not backporting doc changes for any code changes that happen on 0.6-dev prior to going through with this.

[MichaelHatherly]

Small update to this:

An initial draft conversion is now available from https://github.com/MichaelHatherly/julia-docs.
The majority of it was automatically converted and _may_ be missing bits in some places,
though having scanned through it manually I've only come across a couple of minor bugs so
far.

This is just a conversion from RST to markdown (with Documenter syntax for cross-references
and docstrings) and has not yet actually been built using Documenter. That'll need to wait
until the HTML rendering is done in the next few weeks, hopefully.

(If anyone does happen to browse through it and find any problems, just open an issue in that
repo so I can track it down, thanks.)

[mortenpi]

There is now a _very preliminary_ Documenter HTML build available too. I shoehorned this onto my normal Documenter build, so the Julia docs are the bottom three sections. Some broken things, but those seem mainly due to Documenter, not the RST to Markdown conversion.

[MichaelHatherly]

Closed by #18588.