Thanos: An official, community driven Thanos Helm Chart in thanos-community organization.

[daixiang0]

Now doc provide two helm chat, but none is official.
Uses have no idea to choose one to play with.
Some discussion is in slack.

I would like to thanos host an official helm charts like loki does, so that users can easily get official and stable version of helm chart to deploy.

As an official version, not need to update all the time, update it when release(not cover rc) is enough i think.

What do you think? @bwplotka

BTW, it is good to add a group which covert all maintainers to ping in vote case.
@thanos-io/thanos-maintainers

[bwplotka]

cc @thanos-io/thanos-maintainers

As an official version, not need to update all the time,

What's wrong with the unofficial version? What problem are we trying to solve here? Is it only the choice for users; if that's the case what if we would recommend officially one of those? (:

None of the maintainers use helm, so it's hard and not effective for us to maintain such a chart. We are already overloaded. We are definitely happy for someone to help in doing this, but not sure if having the same repository helps here.

We can discuss it on our team sync.

[daixiang0]

Unofficial version means maintainer by person, maybe outdated or something wrong or not support as much as official one does. The official version means most authoritative i think.

Feel free to host a chart or put it in helm/charts as stable/thanos.

I know that many maintainers used to use jsonnet, while more users use helm in fact. I would like to do this if possible.

[brancz]

Prometheus also doesn’t have a chart maintained by prometheus maintainers for the same reasons. I’ve had very bad experiences with having helm “official charts“ in the prometheus operator repository so I’m rather hesitant as well. I think if there are one or more maintainers willing to maintain it then having a separate repo for a chart should be fine.

[domgreen]

I would say that unless it is part of anyone's core workflow for rolling out Thanos with that specific image it would be a burden and become unloved therefore giving less value to the community than one maintained elsewhere.

However, if someone does have Helm as their core workflow and wants to maintain it they should but it would be hard to expect everyone to learn the intricacies of Helm to support it.

[daixiang0]

Any input for this? @thanos-io/thanos-maintainers
Maybe not host by this repo, i would like to add thanos into helm/charts if possible.

[squat]

I am not against having an "official", community-supported thanos chart hosted at helm/charts, just that I don't think the maintainers here will be able to offer support.

[domgreen]

What about having it under a new repo in the org? With a core set of maintainers that are Helm advocates and use it as part of their flow. @daixiang0 do you use helm as part of your daily workflow?

As much as personally do use Helm if users are wanting it it would be better to have something to offer people.

[daixiang0]

@domgreen Yep, i use helm as a part of daily work.
Create a new repo seems a good choice.

[daixiang0]

cc @thanos-io/thanos-maintainers
Any discuss result for this?

[GiedriusS]

AFAIK there's still no consensus here because

We can discuss it on our team sync.

still hasn't happened. My 2 cents: I only deploy certain components of Thanos on K8s so it would be a bit harder for me to maintain them.

[bwplotka]

I am afraid we negative on this side.

[brancz]

I personally also don't see the harm in having 2 helm charts. The two authors could join forces and effectively make it the default choice, without the thanos maintainers having to bless anything when we don't even use it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bwplotka]

Cool, I will put this on the agenda for next Thanos Maintainers Sync.

[daixiang0]

any process about this?

[bwplotka]

Hi @daixiang0

Yes! We discussed this together with the team on our maintainers meeting 12th Feb. We decided together that we can consider maintaining the official helm chart IF there would be a person that will be happy to maintain it AND will meet a bar of being official Thanos maintainer. Since for now, we don't have such a person, we can't reliably maintain it.

Another alternative is to use jsonnet to generate helm charts. This option was for now discarded as it puts another API restrictions on the jsonnet itself which we would like to avoid.

In the meantime, there are many very good helm charts available made by community. It would be awesome if the community would join forces to merge many charts into one. We are happy even to organize single repo for it e.g https://github.com/orgs/thanos-community org that will be maintained by the community, not us.

If you really want to improve helm + Thanos world @daixiang0, do you think you could reach the authors of those (at least 3) Thanos helm chart with this proposition (to combine forces)? You can reference this comment here. (:

[daixiang0]

@bwplotka Oh, it is a really big challenge for me, there are there helm chart repos for thanos and one for thanos-operator(check by https://hub.helm.sh/charts?q=thanos), and i am familiar with none of them. I would file an issue for each one and reference your comment.
Maybe we can have an initial version and invite them as maintainer?

[bwplotka]

The issue with clear explanation is enough. (:

Let's see what their suggestions are first, if they are happy to merge
their version into one under community project. They have to be independent
and motivated to work together. (:

As Thanos maintainers we are happy to be moderators/mentors and advices in
terms of deployment etc if needed.

On Wed, 19 Feb 2020, 08:16 Xiang Dai, [e-mail] wrote:

@bwplotka https://github.com/bwplotka Oh, it is a really big challenge
for me, there are there helm chart repos for thanos and one for
thanos-operator(check by https://hub.helm.sh/charts?q=thanos), and i am
familiar with none of them. I would file an issue for each one and
reference your comment.
Maybe we can have an initial version and invite them as maintainer?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820?email_source=notifications&email_token=ABVA3O7DZCJAQESMEFW56NLRDTTFJA5CNFSM4JTPSZTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMGZDLY#issuecomment-588091823,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O7TU2EDLLTCFIMAPTDRDTTFJANCNFSM4JTPSZTA
.

[carrodher]

Hi, I am Carlos, Bitnami/VMWare engineer. Thanks for contacting us regarding this community project to create an official Thanos chart by unifying the existing ones.

We are happy to collaborate on this project. Before listening to your ideas, please, let me do some clarifications or an introduction about what are we doing in Bitnami and the relation with the stable Helm repository.

Is it possible to add the Thanos chart to the _stable_ repo?
No. The _stable_ Helm charts repository is going to disappear in the following months. So the current centralized, official or _stable_ repository will disappear, in fact, Helm 3 does not include the _stable_ repository by default.
Now the idea is to have different providers hosting their own charts and all of them listed in the Hubs like https://hub.kubeapps.com/ or https://hub.helm.sh/.

What is doing Bitnami?
Bitnami is one of the most important contributors to the _stable_ Helm repository (around 30 of the _stable_ charts are maintained by Bitnami). As _stable_ repo is going to disappear due to this new approach, we are working to move the charts contributed from Bitnami to the _stable_ repository to the Bitnami one.

What users expect?
With the current approach, users expect charts under the _stable_ repository to be secure, well maintained, updated, support responsive, etc. With the new approach when users are looking for a chart they will find different providers and they should choose the one that better fits their requirements (in terms of security, updates, support or whatever).
At this point, if there is an official Thanos chart maintained by the community but approved by Thanos developers, under the Thanos organization or whatever official approach we are discussing here it is more likely that users choose this chart instead of others appearing in the hub.

Bitnami current approach regarding Thanos
We currently have a Thanos Docker image and a Thanos chart (using this image). Everything is integrated into our CI/CD pipeline including tests, updates, etc and we provide technical support using GitHub issues for both (Docker Image and Chart).

I hope the previous items help to clarify the current state of the Helm Charts and how they will be organized in the near future. As I said at the beginning, we are happy to help to create this official Thanos chart. Please feel free to ping me here or schedule a call if you want to talk about this topic.

[bwplotka]

Awesome! Thanks for this. :heart:

Looking forward to seeing response from other Thanos helm charts as well. :muscle:

[monotek]

Maybe not host by this repo, i would like to add thanos into helm/charts if possible.

Helm/charts will be deprecated soon. See:
https://github.com/helm/charts/blob/master/README.md#status-of-the-project

We need some time to answer your question, as some of the Kiwigrid thanos chart developers are on vacation right now.

I think we can answer this in the middle of next week.

[tarokkk]

Hi!

First of all thanks for the effort to help unify the deployment of Thanos. I'm always glad to see when a project organizes itself to be more productive. ;)

As it is already well known the official stable Helm repository will be decommissioned at the end of the year. To be honest we were not a big fan of that monolith repository because it slowed down the collaboration of individual charts. That is one of the reasons why we started our own Banzai Cloud's stable repository as well.

Banzai Cloud provides Kubernetes solutions (as a platform) where amongst other things monitoring is included as well. To be more transparent we are doing as many things as possible as an open-source project. Basically, our charts already work as community charts (with focusing on testing, upgradability, etc...). We are more than happy to help the effort of the official chart, however, we have to consider our customer's requirements as well.

To add a bit more information regarding the Thanos deployment we already started another project called Thanos Operator. We believe the operator approach is more consistent and flexible in the long run.

As @carrodher proposed we can craft a proposal or schedule a call and ping me anytime if you have questions.

[rpahli]

From my POV it would be great to have a single chart or somthing like a operator to install thanos and from kiwiwgrid side(my side) we can also support to to reach this goal.

From my side it would be great to support you.

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[bwplotka]

Ok, let's decide something then. Wonder if some common Google Docs make sense. (: I understand it will be hard or even impossible to merge different helm charts... but maybe it's worth to try? :thinking:

We can definitely create a new project and add anyone who wants to help maintain, but we might need to have first some agreement/design of how merge can happen.

[spencergilbert]

I'd be happy to help with any helm-chart work :+1:

[monotek]

Is there already a repo available for it?

[bwplotka]

Not yet, happy to create ASAP

On Fri, 17 Apr 2020 at 15:08, André Bauer [e-mail] wrote:

Is there already a repo available for it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820#issuecomment-615265164,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O2IT7FP2ITOJR5OC4DRNBPHNANCNFSM4JTPSZTA
.

[bwplotka]

But we might need some idea how we go forward technically with the content
of such repo. Any ideas?

On Fri, 17 Apr 2020 at 16:45, Bartłomiej Płotka [e-mail] wrote:

Not yet, happy to create ASAP

On Fri, 17 Apr 2020 at 15:08, André Bauer [e-mail]
wrote:

Is there already a repo available for it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820#issuecomment-615265164,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O2IT7FP2ITOJR5OC4DRNBPHNANCNFSM4JTPSZTA
.

[spencergilbert]

~the banzai chart is the most comprehensive and up to date chart ive seen for thanos so far, that might provide a good base?~

I guess the question would be how to appropriately merge the existing helm charts in a sensible way.

[monotek]

Hmm... I guess we should compare the charts and use the one with the richest featureset as base?

If we deprecate the other charts some automatic upgradepath would be cool too. If not possible we should at least document how to do it manually.

I'd also like to see Github actions ci in the later repo.

Maybe we should start with the repo and the ci pipeline?

[spencergilbert]

@monotek you mean creating the repo (under thanos or thanos-community?) and start designing the CI pipeline?

We could document the comparison between charts in repo

[monotek]

Yes, that's what I meaned.

Documenting the decision makes sense too :-)

[bwplotka]

Amazing!

Let's start up then :muscle:

I started placeholder repo: https://github.com/thanos-community/charts
.. and Google Doc for collaborations: https://docs.google.com/document/d/18GXxwOm9c2fDK20LV1Sekte0xvK4qXpIGwBcuw_ZYSA/edit#

I described initial information and goals and at this point, I am totally removing myself from this doc. I will only suggest things if needed (I am helm noob after all)

This doc is meant to be community-driven, so I added @spencergilbert @tarokkk as editors (active on this issue). Looks like @monotek and @carrodher you have some nice ideas, so please let me know your email as well (could not find), will add you if you want. Thanks @daixiang0 for starting this effort!

Please DM us if you want to be added as editor and help to shape this project as well! :heart:

Also: Anyone can SUGGEST changes to this doc as well. :hugs:

Help wanted!

[monotek]

Thanks! :)

I would call the repo helm-charts, just to make clear, what kind of charts are saved there...

Edit: I've added a pr for the pipeline: https://github.com/thanos-community/charts/pull/1

[bwplotka]

Good feedback, if there are no objections will rename today. (:

EDIT: Renamed https://github.com/thanos-community/helm-charts

On Mon, 20 Apr 2020 at 11:39, André Bauer [e-mail] wrote:

Thanks! :)

I would call the repo helm-charts, just to make clear, what kind of charts
are saved there...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820#issuecomment-616464380,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O22FGLLXOTKNJHPDTTRNQQ7PANCNFSM4JTPSZTA
.

[spencergilbert]

Nice and specific :+1: looks good to me

[monotek]

pipeline is merged into the repo: https://github.com/thanos-community/helm-charts/pull/1

theoretically we can start adding the chart.

any preferences which chart should be used as the base for the new chart?
if so, why?

[bwplotka]

This is exactly the thing we need to discuss. Since no one wants to start, just propose something @monotek and the doc, and let's see how we can iterate over it :hugs:

cc @spencergilbert @tarokkk @carrodher

[spencergilbert]

Sorry @monotek @bwplotka personal life has been crazy for me the past couple days. I think if we could start a proposal in the doc we can start listing pros/cons of charts and see what matches our expectations the best

[alexgaganov-fiverr]

We rely heavily on community charts for our Thanos setup. Unfortunately, none of the charts I've evaluated fits our needs perfectly. So I would be happy to share some thoughts as well as to help build the new one-chart-fits-all thing.

[spencergilbert]

@alexgaganov-fiverr could you add your thoughts to this google doc? https://docs.google.com/document/d/18GXxwOm9c2fDK20LV1Sekte0xvK4qXpIGwBcuw_ZYSA/edit?usp=sharing

I rolled my own chart to get our thanos usage off the ground, so I haven't worked with the existing charts much. It would be great to get more user feedback before we start implementing anything

[bwplotka]

@spencergilbert are you happy to moderate that doc? 🤗 I think you have edit access.

[spencergilbert]

I'll do my best - hopefully we can grab a few more people though :smile:

[bwplotka]

Hi, :wave:

As per discussions on our public Thanos meeting I created #thanos-helm-charts slack channel in CNCF Slack.

Anyone is welcome to join! @tarokkk @alexgaganov-fiverr @spencergilbert @monotek @rpahli @carrodher @daixiang0 :hugs:

We are slowly moving with our Google doc content for official Thanos helm chart, so feedback is welcome!

[shay-berman]

Question
I saw that prometheus community tend to use jsonnet(vs helm) to deploy in k8s, so they created the kube-prometheus project as the main community deployment method.

So I wonder what is the official deployment method of thanos on k8s? is it kube-thanos or this new helm chart repo?

[bwplotka]

There is no single official one deployment model. It's really up to
whatever your generally use for your other workloads. It's just nice if
every popular deployment way has some good support from Thanos community.
(:

On Sun, 17 May 2020, 14:13 shay-berman, [e-mail] wrote:

Question
I saw that prometheus community tend
https://github.com/coreos/prometheus-operator/issues/3169#issuecomment-624554249
to use jsonnet(vs helm) to deploy in k8s, so they created the
kube-prometheus https://github.com/coreos/kube-prometheus project as
the main community deployment method.

So I wonder what is the official deployment method of thanos on k8s? is it
kube-thanos https://github.com/thanos-io/kube-thanos or this new helm
chart repo?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820#issuecomment-629796044,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O24XVRN3UTHIDD47FDRR7PITANCNFSM4JTPSZTA
.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

We need to get back to this (:

On Tue, 16 Jun 2020 at 21:44, stale[bot] [e-mail] wrote:

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or
needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we
can always reopen an issue if we need!). Alternatively, use remind command
https://probot.github.io/apps/reminders/ if you wish to be reminded at
some point in future.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820#issuecomment-645002022,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O765ULAEY7V4HJR233RW7KULANCNFSM4JTPSZTA
.

[monotek]

Sorry for playing dead.
Not so much time for this at the moment.

[spencergilbert]

Yeah, as soon as I got out from under a bunch of job work - sudden personal life problems pop up :( I'm still around and want to help

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[daixiang0]

still in process.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Still in progress, we are slowly adding more stuff to
https://github.com/thanos-community/helm-charts

Help wanted (:

On Wed, 19 Aug 2020 at 04:00, stale[bot] [e-mail] wrote:

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or
needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we
can always reopen an issue if we need!). Alternatively, use remind command
https://probot.github.io/apps/reminders/ if you wish to be reminded at
some point in future.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1820#issuecomment-675822771,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O5ZDA3IFS3T2N2RV5TSBM56DANCNFSM4JTPSZTA
.

[monotek]

What's blocking us from starting with one of the charts available now, using the most recent thanos version, by voting for it?

[spencergilbert]

What's blocking us from starting with one of the charts available now, using the most recent thanos version, by voting for it?

This is a tricky one. Because there are many already done charts, it’s unfair to choose one. Who would choose the blessed one? Looks like no one wants to just delete their many-weeks-of-work chart and it would be unfair to expect just some people to do so. This means even if we would choose a “blessed” one, there will be some companies/community members who would choose to keep pursuing their own chart and claim it “better” because they did not have an influence on how blessed one was chosen! (: This creates quite an unhealthy situation again, plus again leads to confusion for users. We believe starting new from parts of everyone’s work will create a much healthier attitude and open source, even though it means more work from the start.

It's certainly possible that people could pursue their existing projects regardless of Thanos creating an "official" chart. I'm tired of real work so I'm planning on spending some time working on the proposed chart this weekend.

[monotek]

I'm willing to vote for any other chart, than the one we use @ kiwigrid, just to get this further. Changes can still be made later.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[spencergilbert]

@bwplotka I haven't been able to dedicate a lot of time on this lately, and there doesn't seem to be any other activity. Is the support/desire still there?

[metalmatze]

I think there's still interest generally speaking. :)

[kiefersmith]

The desire is there and would be happy to help support. 🙂

[spencergilbert]

:+1: Good to hear, I was feeling lonely :smiley:

I've swung back around and started to update the query PR I had opened https://github.com/thanos-community/helm-charts/pull/4