Renovate: Build docs site from this repo

[rarkins]

What would you like Renovate to be able to do?

Build docs.renovatebot.com from this repo rather than github.copm/renovatebot/renovatebot.github.io

Did you already have any implementation ideas?

I think we should make the build process JS-based, i.e. deprecating use of mkdocs. Gatsby seems like a good candidate, assuming there's no docs-specialist frameworks with advantages.

I would also like if it's possible use our tags to build one site per-release, so that people can "time travel" to old docs. This is useful for people not on the bleeding edge all the time. Perhaps we have one subdirectory per tag.

Is this a feature you'd be interested in implementing yourself?

Yes, but I'd like help.

[rarkins]

BTW this historical documentation wouldn't be perfect:

• It would reflect whatever content existed at the time of release
• If documentation was missed, we would not go back and change the release/tag to fix it - it would remain as-is

[JamieMagee]

I'm a big fan of GatsbyJS. Their documentation suggests using Docz as a framework for writing software documentation using GatsbyJS.

[rarkins]

I think when I saw Docz I understood it as being targeted at only react components. That's not the case?

[JamieMagee]

I'm pretty sure it's a general documentation tool. It does use MDX files for it's content, which allows you to use React components inline, and it's based on GatsbyJS which uses React, but I don't think it's a storybook replacement.

[rarkins]

I took a deeper look and I'm definitely not convinced by Docz. I think most of the value it adds is beneficial to React components only, or at least something JS/CSS-related. I thought Gatsby already supported MDX: https://www.gatsbyjs.org/docs/mdx/

[viceice]

Another user suggests docusaurus (v2)

ref: renovatebot/renovatebot.github.io#28

[viceice]

docusaurus supports docs versioning https://docusaurus.io/docs/en/versioning

[rarkins]

Thanks @ctison for proposing Docusaurus. We should definitely consider it again.

Versioning would be nice too (although not yet supported for v2 alpha). Ideally we can figure out how that would practically work for us, especially considering how many releases we have. Possibly we could build a new version only for x.y.0 minor releases, but even then we have a lot.

Also, publishing to GitHub Pages. Could we now force push to a specific branch in this repo? I'm not sure it will work though as we want https://docs.renovatebot.com, so we may need to push from here into the gh-pages branch of github.com/renovatebot/renovatebot.github.io ?

[rarkins]

@JamieMagee how would it need to work if we do the site publishing from this repo (using a Github Action of course)? I assume it's not possible to publish to the root of renovatebot.github.io unless it's from the repo named github.com/renovatebot/renovatebot.github.io, or have I mis remembered? In that case, we'd want a solution where:

• The entire logic for building the site is self-contained in this repo
• Contributors can build and view the docs "locally" with a clone of this repo
• Our GitHub Action does the build but then force pushes over to the other repo after each commit?

[rarkins]

Seems the final decision would need to be whether to build it as a DIY Gatsby site versus using docusaurus. User @ctison did a PoC with Docusaurus here: https://renovate-doc.vercel.app/ which seems pretty equivalent in functionality to mkdocs already, sans the search function which should be possible using Algolia's free capability for open source.

Gatsby: More configurable, but will take more time
Docusaurus: More likely to "just work", might continue to get more improvements

Re: versioning, I wonder if it's possible to push a copy of each doc into a subfolder every time a version tag is released? e.g. when tag 23.5.2 is released we build the site off that commit and then push the copy to live in https://renovatebot.renovate.io/23.5.2/ or https://renovatebot.renovate.io/versions/23.5.2/ (perhaps without search?)

[rarkins]

One downside of the current Pages is no ability to create a preview site, but we could maybe hack our way around that if we really need (e.g. creating github.com/renovatebot/docs-preview and pushing to a subdirectory there in gh-pages.

Also btw I know that Docusaurus supports versioning but I think their drop-down approach wouldn't suit our semantic-releasing approach.

[HonkingGoose]

Gatsby: More configurable, but will take more time
Docusaurus: More likely to "just work", might continue to get more improvements

I vote Docusaurus, even I could get that working for my documentation project.
I think it's good enough for what the documentation needs to do, and it's easy to use.

One downside of the current Pages is no ability to create a preview site

I know that Vercel and Netlify both have a pull request preview, so that you can see what is changing.
Both offer plans for open-source projects for free or at a discount.
You can apply for those discounted/free plans by contacting them:

https://vercel.com/pricing (Search for Open Source on this page)

https://www.netlify.com/legal/open-source-policy/

[rarkins]

We switched from Netlify to GitHub Pages not too long ago. Advantages being:

• No worries about using up Netlify's free allowance of "build minutes"
• It's usually good to consolidate to less platforms, not have to worry about duplicating maintainer permissions/team configuration on Netlify

But such things an always be reconsidered. Options:

• Use Docusaurus and GH Pages without previews (need to check it locally if in doubt)
• Use Docusaurus and GH Pages, maybe we can work out an Actions/Pages hack to publish each PR to a subdirectory of a dedicated Pages repo (e.g. renovate-docs-preview.github.com/pr6944/)
• Switch to a platform which supports previews

[HonkingGoose]

No worries about using up Netlify's free allowance of "build minutes"

Yeah the 300 build minutes on the basic Netlify free plan are gone quick, Docusaurus eats a full minute per build for my really simple site.

It's usually good to consolidate to less platforms, not have to worry about duplicating maintainer permissions/team configuration on Netlify

That's true.

I really like Netlify, because I don't have to deal with setting up a build script to update the GitHub Pages and work around the fact that GitHub Pages is tuned for Jekyll websites... With Netlify it was basically set and forget.

If GitHub pages was easier to use for a website that uses a JavaScript based build system, I wouldn't bother with having another platform. So I get where you're coming from.

[rarkins]

I think we should work on the Docusaurus part first, using the existing GitHub Pages approach, and then look at hosting platforms as a later step.

This would mean bringing our documentation generation scripts from https://github.com/renovatebot/renovatebot.github.io into here, and it would be good to convert them to TypeScript too, so we can detect any mistaken or changed assumptions about source code structures.

[HonkingGoose]

User ctison did a PoC with Docusaurus here: https://renovate-doc.vercel.app/ which seems pretty equivalent in functionality to mkdocs already, sans the search function which should be possible using Algolia's free capability for open source.

I got search working on my Docusaurus v2 project with basically no effort using Algolia, it was so easy! It took no effort at all and basically worked right away on the live site as soon as the site was deployed! I'm very impressed with this. I never would have been able to add search to my own project by myself....

---

@rarkins I'm willing to help with the migration to Docusaurus v2, I can maybe do the manual labor to migrate the content once you have the basic Docusaurus v2 skeleton ready on this repository. I'm not very good yet with Docusaurus v2, but I can at least help with the basic migration stuff.

[viceice]

should we really start with docusaurus v2? as it's alpha state. Or should we stay on stable v1? I can send a pr to add the basics

[viceice]

@rarkins maybe we need to refactor this repo a little bit like a monorepo, so we can use yarn workspaces to seperate dependencies for docs / renovate / tools ...

[rarkins]

[HonkingGoose]

@rarkins just posted what I wanted to post... Sneaky ninja! :smile:

---

Keep in mind that there is some work that you need to do when you want to migrate from v1 to v2: Docusaurus v1 -> v2 migration guide.
It seems some of the migration work is automated by a migration command.
I don't know if that command works well enough or not, as I just used v2 right away.

[rarkins]

I'd lean towards using v2 already.

@rarkins maybe we need to refactor this repo a little bit like a monorepo, so we can use yarn workspaces to seperate dependencies for docs / renovate / tools ...

Only if we can't keep all the docusaurus parts insider a regular subdirectory.

[ctison]

I think it is better to use Docusaurus v2 with Vercel because:

1. Docusaurus v2 is easy to implement and documentation is written in MD[X] so it can easily be ported to the next popular documentation engine.
2. Vercel is completly free for open source and is awesome (and is the company behind Next.js (insane SSR React framework), on which Docusaurus is built).

[rarkins]

I'm fine with Docusaurus v2, but not yet with switching hosts. I suggest we implement Docusaurus here and push to GitHub Pages, and then evaluate Netlify and Vercel subsequently.

@JamieMagee if we enable github pages on a repo like github.com/renovatebot/new then it will appear like https://docs.renovatebot.com/new/, right?

Does anyone know if there is anything that prevents Docusaurus running in a subdirectory like this, or must it be at the root?

[HonkingGoose]

I think the Docusaurus v2 docs hint that you can install it in a subdirectory in an existing repository.

Quote from https://v2.docusaurus.io/docs/installation#scaffold-project-website:

The easiest way to install Docusaurus is to use the command line tool that helps you scaffold a skeleton Docusaurus website. You can run this command anywhere in a new empty repository or within an existing repository, it will create a new directory containing the scaffolded files.

[rarkins]

And what's the best name for us to use for the docusaurus source subdirectory in this repo?

[viceice]

I've done some basic testing, looks like it needs a package.json in the docusaurus root, but can be a git repo subdir. That's why I thought we refactor this repo to a monorepo and use yarn workspaces.

[JamieMagee]

@JamieMagee if we enable github pages on a repo like github.com/renovatebot/new then it will appear like https://docs.renovatebot.com/new/, right?

Yep, that's how the helm chart repo works: https://docs.renovatebot.com/helm-charts

[rarkins]

@viceice I still prefer to keep it simpler and non-workspaces unless it's too hard, or we need more

[viceice]

Ok, let's see, how good renovate works with multiple lock files. 😜

My purpose for the first iteration is to use a new folder like website for the new docs and leave the old docs file structure as is until we finished migration.

Also as starter we can publish sh the new docs from this repo as sub gh-pages until we moved to something else and we are happy with the new docs.

We do not need to publish until we are happy with the new layout, as docusaurus provides an easy local dev-server to review any changes.

[viceice]

I've pushed a basic sample at https://github.com/renovatebot/renovate/tree/docs/docusaurus branch.

TODO:

• don't change existing docs, so we can continously deploy existing docs
• remove h1 headers, otherwise docusaurus will show duplicate header, so maybe we should copy all docs to a generated folder and build from there
• move docusaurus to other folder until migration complete?
• add yarn workspaces to have a shared node_modules folder
• add ci config
• need a GH_TOKEN to publish to other github repo (using current repo gh_pages until migration done?)

[HonkingGoose]

How about we host the docs/development/*.md files on the new documentation site as well?
I was thinking we could have a sidebar section called "Contributing guidelines" or something like that, and put the development docs in there.

Or maybe we could have a separate site or subdomain that hosts the contributing guidelines? It would be way nicer to have those nicely formatted in a browser window while you work away.

[rarkins]

I don't mind having the development docs on the site too. Maybe it's a single sidebar entry (non-expandable) "Contributing" and clicking on it then gives an intro/index page of available content and links to it. I don't think we need every option expandable.

[rarkins]

• don't change existing docs, so we can continously deploy existing docs

I don't mind doing a hard switch-over to new file structure if it makes it more nimble for us to get it done with. Unlike run-time code, we should have a fairly good idea of when it's ready or not?

• remove h1 headers, otherwise docusaurus will show duplicate header, so maybe we should copy all docs to a generated folder and build from there
• move docusaurus to other folder until migration complete?

I had originally anticipated this approach for the migration, e.g. both docs and docusaurus co-existing and the build process copies over. But I wasn't sure if that was convenient for docusaurus development.

Depends whether you think it needs to be a migration or just a hard cutover.

• add yarn workspaces to have a shared node_modules folder

I still prefer to avoid workspaces :)

Also it's a definite no to put Renovate into a subdirectory if npm still doesn't support subdirectory dependencies. Otherwise no other project could directly require Renovate based on git, which could be a future requirement.

• add ci config
• need a GH_TOKEN to publish to other github repo (using current repo gh_pages until migration done?)

Again, we need to decide if we just do a hard cutover or not, in which case don't have a temporary docs site at all.

[viceice]

OK, so i think a hard cutover is more easy to handle. we can use the docusaurus buildin dev server to check correctnes.

Also it's a definite no to put Renovate into a subdirectory if npm still doesn't support subdirectory dependencies. Otherwise no > other project could directly require Renovate based on git, which could be a future requirement.

That's a valid requirement against workspaces, so i will see, how to optimize caching for ci.

I'll also have to think about the generated files, because we currently modify some files from docs/usage folder before we generating the final docs. Another issue is, that all docs files need to be in the docusaurus docs (can be renamed) folder. So we need some form of code generation anyways.

[rarkins]

+1 on the unavoidable need for both documentation generation as well as some massaging of docs/ files. Perhaps put docusaurus in website/ and keep docs under docs/ if you think that's most intuitive?

[rarkins]

@viceice would it work to have like:

docs/
docs/usage/
docs/development/
docs/website/

The last could be docs/docsite, docs/docusaurus, etc

[viceice]

sure, but we would need to copy all required files to docs/website/ if this is the docusaurus base dir

[rarkins]

My assumption was that docs/website/ would contain all of docusaurus's committed files and then we would have a combination of copied and generated files from the rest of docs/ which are added somewhere inside docs/website/ but .gitignored. Were you thinking the same?

[viceice]

sure, that's what i thought. I'll update the docusaurus branch to test this.

should i open a draft pr for better review ?

[HonkingGoose]

I think a draft PR is way easier to review. :+1:
I've taken a look at the site already, the basics were working already, and the basic style fits well with the Renovate branding.