Metrics-server: Official Helm Chart

[stevehipwell]

What would you like to be added:

An official _Helm Chart_.

Why is this needed:

Because the stable repo is due to be retired and an official _Helm_ chart is really useful for the community.

Prior art:

https://github.com/kubernetes-sigs/descheduler/pull/298
https://github.com/kubernetes/autoscaler/pull/3341

/kind feature

[serathius]

I was a little skeptical for creating separate distribution of manifests in addition to Kustomize. Not everyone uses helm, so Kustomize made more sense as there was Metrics Server chart already maintained by community.

Looking at links you provided, I like the approach taken of separating chart owners and releases from main release. Charts linked are also pretty high quality.

I would be for adding helm charts if 2 maintainers step up to introduce and maintain the chart. Current Metrics Server maintainers do not use helm, meaning that they would not be capable to maintain high quality. We should also figure out how do we maintain both Kustomize and Helm configuration.

/cc @s-urbaniak
What's your opinion?

[olemarkus]

I am one of the maintainers of the helm/stable repo chart and would be happy to maintain the chart as part of this repo instead.

[stevehipwell]

@olemarkus you might want to keep an eye on https://github.com/helm/chart-releaser-action/issues/37 which would allow you to keep the chart name _metrics-server_ but have additional information in the tag name (e.g. _metrics-server-chart-vX.Y.Z).

[scottrigby]

👍 a working demo steps for all the Helm GitHub actions is here: https://github.com/helm/charts-repo-actions-demo

[scottrigby]

I've linked this issue in the table at https://github.com/helm/charts/issues/21103.

@serathius There is also a kubernetes organization on the CNCF Artifact Hub (https://artifacthub.io), which hosts CNCF packages like Helm charts, OLM Operators, Falco Rules, and OPA Policies. Would you and/or other metrics-server maintainers like to be added to that so you can manage the metrics-server charts repo? There's not really much to do after setting it up, verifying, and marking "official". If so, lmk and I can add you, and help with whatever needed there.

[scottrigby]

@olemarkus if you are going to be helping with this – or whoever is – you can look at other previous migrations from stable, such as https://github.com/prometheus-community/helm-charts, https://github.com/grafana/helm-charts, and https://github.com/jenkinsci/helm-charts.

These examples all followed current best practices by importing git history (I've laid out example steps here: https://github.com/prometheus-community/helm-charts/issues/3), and set up Helm GitHub actions for CI (chart testing), and CD (releasing chart version packages to the new chart repo).

Please let me know if you have any questions or would like some help with this.

One important note: this is time-sensitive for end users of the chart. stable and incubator chart repos will be end of life on 13 Nov 2020.

[olemarkus]

Looked easy enough. Filed #606 as a start.

The old merge request PR references are naturally broken in the history, but I see other migrations have ignored that part as well.

[serathius]

PTAL @s-urbaniak

[olemarkus]

The one thing that is a bit unclear here is how to handle chart releases vs metrics-server releases. I guess that is what helm/chart-releaser-action#37 tries to support.

[s-urbaniak]

I agree with the statements of @serathius. We, for instance, do not use helm charts either, but rather rely on jsonnet for managing our manifests. As stated there are also kustomize, etc, hence I am very sympathizing with the following statement:

I would be for adding helm charts if 2 maintainers step up to introduce and maintain the chart. Current Metrics Server maintainers do not use helm, meaning that they would not be capable to maintain high quality. We should also figure out how do we maintain both Kustomize and Helm configuration.

An experience point from maintaining prometheus-operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator). The helm charts are maintained there, in a separate repository, by separate maintainers, treating prometheus-operator releases as "upstream". I suggest a similar model here.

[serathius]

@s-urbaniak I think the problem is that helm/charts repository is being deprecated and those charts don't have place to live. Looks like prometheus-operator was also moved.

[scottrigby]

@serathius I think @s-urbaniak 's point still stands. The https://github.com/prometheus-community/helm-charts/ chart repo is in a separate git repo. One of the side benefits of that is that the github releases aren't listed along with the upstream prometheus projects each chart installs.

Though I understand kubernetes-sigs/metrics-server is a bit different, because a separate charts repo may be confusing in the kubernetes-sigs github org. An example of k8s apps maintaining a chart repo in the same git repo as the upstream project is https://github.com/kubernetes/ingress-nginx/

[bryankaraffa]

I think @scottrigby summed up the two different models that other projects are going with. As a user I have no preference, as long as the chart is published in a proper helm chart repo I can add or declare.

Wherever the chart ends up living, looks like it's easy to "to turn a GitHub project into a self-hosted Helm chart repo":
https://github.com/helm/chart-releaser-action
https://github.com/kubernetes/ingress-nginx/blob/master/.github/workflows/helm.yaml

[stevehipwell]

@olemarkus I've created a PR https://github.com/kubernetes-sigs/descheduler/pull/436 to update the _descheduler_ chart to use the new tag naming template allowing it to live in the same repo as the code.

[baderbuddy]

@olemarkus it look like you were the only one that's offered to maintain this chart. I just wanted to offer that if you need any assistance I'm pretty experienced with Helm so I could help too.

[chadlwilson]

We migrated to https://github.com/bitnami/charts/tree/master/bitnami/metrics-server (which gets regular upstream docker image rebuilds as well) since it was a bit worrying that this didn't seem to be getting the attention it needed (i.e the earlier PR merged/reviewed).

I submitted a PR to remove the main delta I saw from this chart to that one (which was promptly merged); but interested in any thoughts others in the community might have of suitability of one chart over the other.

[olemarkus]

Due to the time this has taken, my employer plan on using the built-in addon in kops 1.19 instead. I may still contribute to an official chart, but I don't think I can commit to being one of two main contributors.

[walkafwalka]

Our company avoids implementing services without official Helm charts. It is a sign of product maturity and we do not have the bandwidth or desire to maintain a chart that is compatibility with version releases. I will probably also look into bitnami's chart especially since it supports 0.4.0 but an official Helm chart is still preferred.

[stefansedich]

Due to the time this has taken, my employer plan on using the built-in addon in kops 1.19 instead. I may still contribute to an official chart, but I don't think I can commit to being one of two main contributors.

Happy to throw my name in the ring to help with the chart @olemarkus @baderbuddy if that eases the burden to get this one out the door.

[stevehipwell]

I'm happy to put myself forwards to help with this. I've already been involved with the descheduler chart directly and have been part of the discussions around the cluster-autoscaler chart.

[scottrigby]

Awesome. PS there's also a kubernetes-sigs org now at artifacthub.io - if maintainers in this org want to access that to add this chart (or @stevehipwell, also the descrheduler chart), lmk.

[stevehipwell]

@scottrigby the descheduler chart repo is currently listed on artifacthub.io. My understanding is that the organisation would take ownership of the repo, do you have any more info on how this would work?

[scottrigby]

@scottrigby the descheduler chart repo is currently listed on artifacthub.io. My understanding is that the organisation would take ownership of the repo, do you have any more info on how this would work?

@stevehipwell Yes. First, you must be a member of the org. By default (unless fine grained permissions are configured for the org using an OPA policy), members can add other members. I can help add whoever is responsible for this repo, and from that point on they can help to add other members as needed too. There is an expectation that trusted people will only add other trusted people (this is where configuring fine-grained permissions can be helpful though, to minimize accidental errors, especially for very large orgs. By default permissions are just flat).

Once a user is part of the org they want to transfer a repo to, they can create an Ownership Claim (see that section in the Artifact Hub repositories guide).

The Control Panel has inline documentation on how to use the repo transfer functionality (allows to transfer a repo between personal and/or any org you're a member of).

Hope that helps! Please lmk if I can help more. Also there is an #artifact-hub channel in CNCF Slack, in case I'm unavailable at any point.

[cpanato]

We can host the helm chart similar to the dashboard is doing https://github.com/kubernetes/dashboard/tree/master/aio/deploy/helm-chart

what do you think?

[scottrigby]

@cpanato Can you summarize the differences between https://github.com/helm/charts-repo-actions-demo and how kubernetes/dashboard is hosting a helm repo? There have been a number of kubernetes projects now hosing their own helm chart, and each do it a little differently, which is ok. But it may be helpful to arrive at and recommend a common pattern for other k8s projects who want to host their own chart repo. If we describe the differences in the existing examples it may help us get closer to that. What do you think?

[cpanato]

@scottrigby makes total sense, lets first check if the way we are trying for will work, then we can suggest a recommended way because I think that pattern will be easier compared with the one I wrote above.

[scottrigby]

@cpanato perfect. @torstenwalter compiled a good step-by-step guide based on work I and he had done on this for various community chart repos: https://github.com/torstenwalter/helm-chart-hosting. This includes how to port over git history using existing tools, and also package history using a tool I wrote (I don't think it includes steps for charts that have already been relocated without git history, but it's very possible to bring that back in too - here is one example I worked on: https://github.com/goharbor/harbor-helm/issues/48), as well as setting up CI/CD with Helm GitHub actions.

Here's a recent example PR I made for a k8s project: https://github.com/kubernetes/kube-state-metrics/pull/1237, if you want to look at how the gh actions workflow files are put together (there were some follow-up PRs as well so check master when looking at the workflows). I know some other k8s projects have done it a bit differently.

I also want to second what @stevehipwell said https://github.com/kubernetes-sigs/metrics-server/issues/572#issuecomment-723935491 about the name template: CR_RELEASE_NAME_TEMPLATE is available as of helm/chart-releaser-action v1.1.0.

[caleb15]

For other people who come across this issue looking for a helm chart, note that linting fails with the helm stable and bitnami chart if you have the newest helm/helmfile versions. If you use helm v3.2.4 and helmfile v0.119.1 then works with the bitnami chart - presumably helm stable chart would work as well.

Related issue: https://github.com/roboll/helmfile/issues/1596

[pierluigilenoci]

I wouldn't bother but I really need an official chart for the metrics server. Is there any news?

[stevehipwell]

As PR https://github.com/kubernetes-sigs/metrics-server/pull/606 has been closed and in the interest of getting ourselves off the deprecated stable repo I'd be willing to create a PR to add a Helm chart to this repo if the maintainers are OK with that?

[cpanato]

@stevehipwell I think that is ok from my side, but I'm not a maintainer

however as soon you open that I will do the infra side of things :)

[stevehipwell]

@cpanato are we looking at the same workflow (GitHub actions) that's used for _descheduler_ and _cluster-autoscaler_ (I've worked on both) or another one?

[stevehipwell]

Are there any maintainers who would like to comment on this?

[jdambly]

@olemarkus

@olemarkus my company also requires that all add-ons be deployed with helm, so I am happy to assist as well. I just need to get plugged into the community. please let me know what I can do to help.

[stevehipwell]

@cpanato due to the lack of comments I'll go ahead and create a PR for the chart and then see where we go from there.

[Sliuza]

@stevehipwell , did you have any progress? I'm also interested on this and I can help with the PR.

[stevehipwell]

I'm planning on creating the PR this week.

[stevehipwell]

I'm waiting on feedback for PR https://github.com/kubernetes-sigs/metrics-server/pull/670.

[pierluigilenoci]

@stevehipwell another two months and no progress. 😢

[stevehipwell]

@pierluigilenoci I know. I'm resisting putting it in my own chart repo as I'd much rather it was alongside the code and maintained as a single unit.