Kube-prometheus: Produce example artifacts on CI runs.

[brancz]

Right now we generate manifests and check them into the repo. This is very tedious when contributing as it's not uncommon to cause merge conflicts and requires rebases, but is also confusing for new contributors who tend to modify the manifests/ directory directly.

If we generated them on CI and published the artifact(s) instead that would remove a lot of this.

Or we make generation _a lot_ faster and just improve the experience of generating manifests (which I think we should probably do in the long run).

cc @lilic @metalmatze @s-urbaniak @paulfantom @dgrisonnet @pgier @simonpasquier

[paulfantom]

we make generation a lot faster

Recently we already made it faster by switching to golang version of jsonnet in https://github.com/coreos/kube-prometheus/pull/482
If we want to make it even faster, then the next thing would be to make jb install faster as this is currently the most time-consuming process

If we generated them on CI and published the artifact

If we put them in the repo, then it doesn't change much apart from moving generation step to CI and making CI more powerful as it would need to commit data into master branch.
If we put those elsewhere, people will still confuse manifests/ directory as they would need to go to some other site to look at examples.

The only thing I can think of that would potentially solve those issues is a static docs site with content generated by CI. This way we:

• don't "pollute" project with generated examples
• don't force contributors to run make generate and commit those changes
• have a clear and commonly used way of presenting examples with the possibility to add more information.

[paulfantom]

It's been a while and during that time I took a closer look at various CD solutions and broader k8s apps management ecosystem. It looks to me the easiest and "cheapest" way of doing this would be to create an additional github repository solely to store generated files pushed by a bot from every merge to every branch in this repository. Such repository would need to have PRs disabled to limit people confusion.

I would propose to have the following directory structure:

.
\ README.md
\ <example-manifests-1>
  \ <yaml-files>
\ <example-manifests-2>
...

Example manifests would be generated from examples/ directory in this repo.

Additionally, I would propose to have the same branches in "generated" repo as in here. This way it would be easy to track manifests in older branches.

Such an approach should allow easy use of tools like flux v2 or argoCD by using plain YAML manifests and thus reducing the entry barrier.

All this would also be possible by doing in this repository, but separating it to another repo allows limiting potential issues with merge conflicts and removes a problem with a bot pushing directly to main branch.

I would like some input from other @prometheus-operator/kube-prometheus-reviewers and if there is a consensus I can implement it (I already have some code for similar things in other repos).

TL;DR:

• Generate manifests from all examples/ in this repo
• Create a separate repo for manifests
• Do not allow PRs in new repo