Incubator-superset: [SIP-47] modernize/facelift superset.apache.org

[mistercrunch]

Motivation

Our current documentation at superset.apache.org has a certain number of issues:

1. look/feel is dated
2. RST/Sphinx/ old templates is a fading/dying standard, markdown seems much more popular
3. can't add rich/dynamic content - it's super static
4. Content is a bit of a patchwork, needs review/restructure/rewrite

This SIP looks to address 1,2 and 3, and hope that those provide a foundation to docs that people want to contribute to and improve, content-wise.

Proposed Change

Migrating the current website from/to:

• framework: moving from Sphinx to Gatsby (45k stars) / DocZ (19k stars)
• language: moving from RST to markdown
• hosting: moving from readthedocs to GitHub Pages (or similar)

Gatsby would allow us to build a fast and modern static site. It provides a lot of flexibility / extensibility if we wanted to do more complex things like exposing a community blog.

DocZ is markdown + jsx, allowing us to integrate arbitrarily complex things in our docs, including things like live examples (embedded Superset components) or anything really.

Markdown is a more common markup language for docs. Most people know it or can learn it very quickly, including tech writers, it's becoming fairly standard.

New or Changed Public Interfaces

N/A

New dependencies

Gatsby / Docz and whatever plugins and things we need there

Migration Plan and Compatibility

Would keep everything under the current main repo under docs/

• individual RST files need to be converted to MD, this can be mostly automated https://cloudconvert.com/rst-to-md , we probably need to do some manual work
• build a new Gatsby / DocZ site, with a good looking theme, maybe with AntD + gatsby-plugin-antd

Rejected Alternatives

• Improving / theming readthedocs
• Other static sites / documentation builder

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[etr2460]

100% in favor of moving off of RST files and onto something markdown driven.

I'm curious what other static sites/documentation builders you considered when evaluating options. I'm not super familiar with the space, but I know Docusaurus is pretty commonly used within the frontend world (Babel, Jest, Redux), also supports adding a blog, and has other features like translated and versioned docs. What tradeoffs did you consider when picking Gatsby + DocZ?

[willbarrett]

+1 :shipit:

[rusackas]

Wondering if it's reasonable to use some CSS-in-JS/Emotion in the mix, in order to leverage the @superset-ui theme package here, and "tie the room together" a bit. This would make sense if there's ever an interest in pulling some of the docs/tutorials back into Superset (i.e. inline help buttons popping up tutorial content in a modal).

[mistercrunch]

@etr2460 I have to admit I didn't go super deep in the research. I think Gatsby is super connected to everything and very extensible. DocZ being markdown + jsx/React seemed perfect.

Check out the Gatsby plugin ecosystem
https://www.gatsbyjs.org/plugins/

DocZ appeared to be top of their doc plugins. I have some experience with both and I think it's hard to top.

[mistercrunch]

@etr2460 Docusaurus looks pretty sweet too, here's their take on how it relates to Gatsby:
https://v2.docusaurus.io/docs/#gatsby

---

Gatsby is packed with a lot of features, has a rich ecosystem of plugins and is capable of doing everything that Docusaurus does. Naturally, that comes at a cost of a higher learning curve. Gatsby does many things well and is suitable for building many types of websites. On the other hand, Docusaurus tries to do one thing super well - be the best tool for writing and publishing content.

GraphQL is also pretty core to Gatsby, although you don't necessarily need GraphQL to build a Gatsby site. In most cases when building static websites, you won't need the flexibility that GraphQL provides.

Many aspects of Docusaurus 2 were inspired by the best things about Gatsby and it's a great alternative.

---

[mistercrunch]

In short:

• Docusaurus is more fitted to a simpler use case
• Gatsby is more extensible / flexible

Key immediate things:

• Docusaurus has better support for versioning, DocZ is working on it, there are ways to do it with DocZ, it's more involved though
• Docusaurus has a i18n "coming soon" thing that could be interesting to us

[etr2460]

I think your pros and cons list is accurate. When I was looking at Docusaurus, it seemed like the v1 branch supported versioning and i18n, but not JS in Markdown. v2 has JS in Markdown (which seems like a really good selling point), but hasn't implemented i18n yet.

To me it all comes down to how much effort we want to spend to start up and maintain the new docs. Docusaurus seems easier to spin up and would free up more time to build Superset features vs. documentation. But at the cost of less overall flexibility (an unknown cost for me since it seems to still support all the use cases we want here).

I'd be happy with either option, but might lean slightly towards Docusaurus as it feels like our competitive advantage should live with the product vs. the documentation (thus we should spend more time building the product and less time building docs).

[mistercrunch]

Looking at the Airflow docs http://airflow.apache.org (rewrite was sponsored by Google I believe), I'm inspired that we could do more eventually, building a blog, events page (gatsby has a meetup.com plugin for instance), maybe even things like a dynamic FAQ or some integrations with StackOverflow, Github and whatnot. Unclear how much flexibility is offered on the Docusaurus to do that.

The beauty of the markdown-based solution is that it becomes very much the meat and it seems like changing the shell should be easy.

---

I terms of effort, we're interested in sponsoring / leading this at Preset. We're not looking for any sort of editorial control on it, just to have a rich, beautiful and modern community website.

[ktmud]

Another thing to note is should we or how can we integrate autogenerated superset Python module/CLI and Swagger API docs.

DocZ have moved to Gatsby, so it seems any plugin available in Gatsby might also have become available for DocZ.

[kristw]

In terms of deployment. Github pages sounds great. Vercal (formerly Zeit) also has native support for Gatsby.

[mik-laj]

Polidea was doing a similar project. We use docsy for main site and sphinx for documentation. On main website, we have blog, community information and other. Sphinx we use for project documentation The site is not versioned, but all documentation is. You can choose the version of Airflow. We also have latest version from master branch available at https://airflow.readthedocs.io/en/latest/

Using the docsy facilitates integration with Sphinx, which uses Jinja as a template engine. We had to use Sphinx and .rst because we generate reference documentation for the Pytthon API

More information:
https://airflow.apache.org/blog/announcing-new-website/
https://github.com/apache/airflow-site

I am happy to share my experiences. If you have any questions, I will be happy to answer.
My e-mail: kamil.[email protected]