Rasa: New docs: Setup build of docs in CI

[m-vdb]

⬆️ Relates to https://github.com/RasaHQ/growth/issues/1436

We should add a new workflow to test the new docs and build them upon release:

• current docs are tested here. We can add a new step to this workflow to test the new docs build
• the docs are built and released in this separate workflow. We should add a new job in this workflow, that will take care of building the new docs.

💡 the current building system works as follows:

• code is checked out, sphinx builds the doc files
• we push the doc files to a docs branch on the same repo
• Netlify reacts (need details) to this and pulls the changes

We could mirror this and create newdocs branch. That way we do not override the existing docs and can still build the new docs. The Netflify part is not handled in this ticket.

[lunelson]

@m-vdb we are at 9 out of 9 members on our Netlify "team", but some of these can probably be removed and you could be added; then I could walk you through the configuration

[m-vdb]

let's wait and see if Tom wants to handle that part. Otherwise we'll figure it out (maybe Rajesh doesn't need access?)

[tmbo]

added you on netlify

[tmbo]

The Netflify part is not handled in this ticket.

I guess it is now? :D

[tmbo]

Regarding the ticket:
@m-vdb before you go ahead and add a build step for the new docs we (@lunelson) should align on how it should look like (let me know if that already happened):

• all of the old docs should stay where they are for now (no change to the docs branch on rasahq/rasa and no change to its netlify config)
• new docs will be pushed to similar setup as we had before:
  
  
  
  • build docs in github ci on push to release branches (releasing docs based on master is fine for now, but won't work when we start working on 2.1)
  
  
  • built artefacts are pushed to rasahq/rasa@documentation (feel free to choose a different name other than documentation, can't use docs though - see above)
  
  
  • netlify deploys that branch
  
  

does that work or did you have something else in mind @lunelson?

[m-vdb]

yes this is exactly what we discussed 👍

[lunelson]

@tmbo @m-vdb yes the existing workflow of building to the docs branch and netlify's configuration to publish from that branch must remain intact; my suggestion for the new build is this:

1. build the v2 docs to a docs2 branch on rasahq/rasa
2. rather than create a new netlify project, I'll configure the existing one to pick up this branch as an _additional "branch deploy context"_
3. the URL for the new build will then be https://docs2--rasahq-docs-rasa.netlify.app/

[m-vdb]

we'll most probably name the branch documentation to avoid a later renaming

[lunelson]

@m-vdb OK let me know when you've named it and I'll update the config in Netlify :+1:

[m-vdb]

@lunelson we'll go with documentation. I'm about to push the first changes soon. Happy to also take a look at Netlify since now I have access

[lunelson]

@m-vdb OK cool. The relevant settings are here https://app.netlify.com/sites/rasahq-docs-rasa/settings/deploys, and you'll see where I configured docs2 as a branch deploy, this should be changed to documentation

[m-vdb]

@tmbo I took a closer look at how to deploy docs properly with docusaurus. If I follow their docs, here is how it works:

• docs live in the docs/ folder on the default branch (master)
• when we want to publish the doc, Netlify listens to changes on the branch and runs the build command
• if we want to generate docs for a new version, we can do docusaurus docs:version x.y.z, and it will create the following structure under the docs/ folder: https://v2.docusaurus.io/docs/next/versioning/#directory-structure (then we need to commit + push)

The drawbacks:

• all versions live inside master, which could impact the size of the repo
• it looks hard to deploy docs from a release branch

The plus:

• we stick to docusaurus philosophy

I also took a look at how sphinx-versioning does it under the hood: it creates a temp folder, git clones the docs branch in it, build the docs and outputs it in the temp folder, commit the changes, pushes it. Our Netlify setup listens to these changes and deploys the docs site.

The approach I propose

• keep everything in a documentation branch, and do something similar to what sphinx did: create a temp dir, git clone the branch inside it.
• at this stage, we move the content of the docs folder
• we run docusaurus docs:version x.y.z to create the new version in the folder
• we commit the changes to the documentation branch and push it
• Netlify listens to the changes on the documentation branch
• done

The drawbacks:

• there is some manual scripting to be done
• we change the setup in Netflify (might require another site to setup a different build command, or we just push a phony package.json inside the current docs branch and be done with it!)

The plus:

• we stick to docusaurus philosophy
• we do not mess with master
• we will be able to push docs for patches

[lunelson]

all versions live inside master, which could impact the size of the repo

I guess it could, but in a standard Docusaurus setup it's only the source files, not the built html

it looks hard to deploy docs from a release branch

yes, Netlify looks for either a specific branch name to deploy, or it can be set to deploy all branches; there is no wildcard matching of branch names 🤷

@m-vdb I'm wondering if the extra control you have with building through Github Actions is worth it here since you have to do a lot of scripting in order to commit built html to the documentation branch so that Netlify will publish them. OTOH if you let Netlify do the build, the built files are never committed anywhere and you could just deploy from master (and PRs against master). Note also, that for local work you can run Netlify's CLI (netlify dev) to test the build in an identical local environment before pushing the source files, to trigger the remote build

[m-vdb]

@lunelson yes 💯 I went for building inside Netlify. There are things that needs to be pre-built inside Github Actions because we need the python context, but after that, everything is handled in the deploy settings of Netlify. There are still some manual things to do though.

I've managed to build the site here: https://rasahq-docsv2-rasa.netlify.app/

~(next: figuring out why I get a 404 on the docs page)~

[lunelson]

@m-vdb Oh cool :+1: Looks good! 

Maybe when I'm back you could walk me through the manual bits ;)

[m-vdb]

sure!