Lark: Please use Sphinx for documentation

[chsasank]

Your tutorial here is pretty awesome: http://blog.erezsh.com/how-to-write-a-dsl-in-python-with-lark/

Finding this was not easy. It'd be great if this could be added as tutorial in readthedocs. pandoc can convert this blog's html to rst and can be added to sphinx in less than 10 mins (assuming you don't have the source). If you have the source it's far easier.

[chsasank]

Also link for json tutorial is broken.

[erezsh]

The link is right here, where else should I put it? https://lark-parser.readthedocs.io/en/latest/

Not sure it's worth the trouble of turning it into a rst, but feel free to do so.

Thanks, I'll fix the json tutorial link!

[chsasank]

I just realized that Lark doesn't use sphinx and restructured text for the documentation. Sphinx is vastly superior for any python package's documentation. API documentation is a breeze with sphinx.

You need not export all the existing markdown documents to rst, although this is not hard with pandoc. I can help you set up sphinx and other things if you're interested.

[chsasank]

The link is right here, where else should I put it? https://lark-parser.readthedocs.io/en/latest/

Ideally tutorials should be in left table of contents (TOC)

Not sure it's worth the trouble of turning it into a rst, but feel free to do so.

Do you have markdown version or some source of the post? Working with the source is easier than html.

[erezsh]

I've read the link and I'm not convinced that moving to Sphinx is the right move.

Lark uses markdown, because it's what github uses. It's familiar for everyone, and anyone view and edit the documentation online using github (It will create a PR automatically).

And while I can see the appeal of creating the docs from docstrings, I'm not sure that it will end up improving the quality of the documentation.

I'm not saying no, but I don't see this move as obvious.

[erezsh]

I see now that sphinx can use markdown, so as long as it works with readthedocs, I think I'll be fine with it.

[chsasank]

I've read the link and I'm not convinced that moving to Sphinx is the right move.

Sphinx is the de facto standard for documentation for a python library's documentation. Every major python library - Django, PyTorch, Twisted, you name it - uses sphinx. Sphinx is fairly easy to manage too. As I said before, I'll help you set it up by raising a PR.

Lark uses markdown, because it's what github uses.

Actually github supports restructedtext too. Example. But no matter. You can still keep on using markdown. Sphinx now supports markdown too.

I'm not sure that it will end up improving the quality of the documentation.

This improves the quality because documentation and code will be in one place. See scipy, a huge scientific computing library. It keeps documentation in docstrings of python functions and classes. This allows people to write better documentation.

[erezsh]

But this kind of 100 line docstring is exactly the reason I'm apprehensive about it.

In my experience, if you're already reading the code, most of the information in the docstrings is already trivial and uninteresting. Comments and docstrings should be there to give you an overview and frame of reference, and not duplicate the information that the code already contains clearly.

It can be very frustrating, to try to understand a codebase where every time you jump to a function, you have to scroll down to start reading what it actually does.

And Python's help() doesn't understand RST, and neither do most IDEs (in the context of docstrings), so all of this markup just ends-up as a confusing eye-sore.

[chsasank]

Well, this is the de facto standard in the python world. But this is your project, so whatever works for you :) Anyway, you don't have to use this auto API document generation feature of sphinx. Sphinx can still be useful regardless. Should I work on setting up sphinx skeleton for Lark?

[erezsh]

So, in what ways is it still more useful than mkdocs?

Sure, perhaps a demonstration will help me get it.

[chsasank]

I'll spend some time soon to create a PR with sphinx.

[erezsh]

We are officially Sphinxed!