Charts: [stable/*] Random passwords change on upgrade. Create and document charts solution

[scottrigby]

I was replying in #3591 and realized this is a much bigger problem that deserves a charts issue to decide whether we should agree on and document a policy for this across charts.

Can we address this at one of the next charts meetings?

Related:

• https://github.com/kubernetes/charts/pull/3591
• https://github.com/kubernetes/charts/issues/4258
• https://github.com/kubernetes/helm/pull/2721
• https://github.com/kubernetes/helm/issues/2196
• https://github.com/kubernetes/charts/issues/5129
• https://github.com/kubernetes/charts/issues/4472
• https://github.com/kubernetes/charts/pull/4965
• https://github.com/kubernetes/charts/issues/2538
• https://github.com/kubernetes/charts/issues/2636
• https://github.com/kubernetes/helm/issues/3053
• https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change
• https://github.com/kubernetes/charts/issues/2488

[unguiculus]

When I became a maintainer I learned it was best practice to auto-generate passwords and have told folks to follow this pattern. I never really questioned this but am not really happy about it either. I appreciate you brought up the discussion.

Auto-generating passwords is kind of nice. It makes it easy to get started and is certainly better than a hard-coded password. But for a real-world installation, I wouldn't rely on this. So, we need at least better documentation.

Thinking about it again, wouldn't it be better not to provide a default password at all? We could use the required function forcing the user to explicitly set a password. For CI, a custom values file could provide a password.

[unguiculus]

cc @kubernetes/charts-maintainers

[alexvicegrab]

This is indeed a problem. Currently I'm overcoming this by specifying a value with --set flag to override sensitive information (e.g. passwords) during the upgrade process.

I tend to save charts on git repositories but without the password. Ideally I neither want to:
1) Cause bloat, by saving one copy of each chart values.yaml for each (e.g.) PostGres chart I install, nor:
2) Cause security issues by saving passwords in Git repositories
3) Cause high complexity (what I currently do) by having to retrieve the passwords from the relevant Helm deployment secrets and resupplying them in the update command.

A way to automate 3 in a K8S/Helm way would go a long way.

[alexvicegrab]

e.g. it could be useful to add an annotation that prevents updating of a secret like this:

"helm.sh/resource-policy": keep

Except this only works for preventing deletion.

[geekuillaume]

I had the same problem. You can create a secret only on install by using a pre-install hook. The secret will be left untouched on upgrades. The only problem is that it's not managed by Helm and so not deleted when you delete the release.

[michaelfig]

I think the issue is not feeding back secrets into a chart upgrade, nor a problem with generating passwords. I think the issue is that if a chart is generating a random password, it should not overwrite an existing secret.

This seems like it should use the following wrapper around the secret definition:

{{- if or .Values.somePassword .Release.IsInstall}}
apiVersion: v1
kind: Secret
...
data:
  {{- if .Values.somePassword }}
  some-password: {{ .Values.somePassword | b64enc | quote }}
  {{- else }}
  some-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}
{{- end}}

Thoughts?

[alexvicegrab]

@michaelfig I like this, either we explicitly specify the password, or we autogenerate it during an install.

Does this work? (i.e. Have you tried running this to check that it does not entirely delete the secret during an update when not finding the secret template?)

[alexvicegrab]

Tried it myself just now, alas Helm will simply wipe out the secret.

[achempion]

The service is writing password to the secrets. Maybe it will be better to check first for existence and use old if a secret has a value already.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[alexvicegrab]

Is there any progress on setting up a way for random secrets to not be reset implicitly during upgrades?

[bacongobbler]

From the dev call, one possible solution to try would be through an install hook that writes out a configmap.

[bacongobbler]

That or checking for the presence of .Release.IsUpgrade.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[yurrriq]

Any update?

[arianitu]

This is pretty annoying, especially with things like Jenkins and Grafana, you're constantly looking up the new password on upgrades.

[giacomoguiulfo]

@arianitu Since Grafana is usually a not mission-critical component, a workaround is to add an annotation to the deployment's spec.template that looks like this:
checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
to upgrade the Grafana pods every time the secret changes (when the password is re-generated). In my experience, this makes the pod be in sync with the secret most of the time.

[pastukhov]

Hi all!
My secrets keep deleting when chart upgrade, even "helm.sh/resource-policy": keep is set. My yaml:

{{ if .Release.IsInstall }}
{{- $graylogPasswordSecret := randAlphaNum 128 }}
{{- $graylogAdminPassword := randAlphaNum 64 }}
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    "helm.sh/resource-policy": keep
  name: {{ template "graylog.fullname" . }}-secrets
  labels:
    configVersion: v1
    app: {{ template "graylog.name" . }}
    chart: {{ template "graylog.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
type: Opaque
data:
  GRAYLOG_PASSWORD_SECRET: {{ $graylogPasswordSecret | b64enc | quote }}
  GRAYLOG_ROOT_PASSWORD_SHA2: {{ $graylogAdminPassword | sha256sum | b64enc | quote }}
  GRAYLOG_ADMIN_PASSWORD: {{ $graylogAdminPassword | b64enc | quote }}
{{ end }}

Any advice pls!

[pastukhov]

Solve this quiz with pre-install hook that check if secret exist
https://gist.github.com/pastukhov/9e894a71e5793cc56a5096a3dc199f5e

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[dtshepherd]

/remove stale

[chrisaige]

Any progress?

[squillace]

I'm curious, @bacongobbler, whether any of the work in Helm 3 will resolve or at least provide a direct workaround to this. thots?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[manics]

This issue is still relevant

[desaintmartin]

There has been some discussion here : https://github.com/helm/charts/pull/10400

[hedayat]

My opinion: The problem is NOT only for randomly generated passwords in helm charts. Even for not random passwords, you need to always pass proper --set params on helm upgrades for passwords, which is very inconvenient and you should always remember the passwords of a specific release. This is also a problem for automatic upgrade tools: they should store passwords somewhere so that they can reuse them for upgrades, while they are already stored by k8s. So, it created redundancy, and also they should secure the passwords they store.

[bacongobbler]

I'm curious, @bacongobbler, whether any of the work in Helm 3 will resolve or at least provide a direct workaround to this. thots?

Hey @squillace, this is just an issue that nobody has documented a working convention in chart development. Helm 3 won't solve this issue necessarily.

Has anyone tried the suggestions made here? If so, does that solve the issue?

[michaelfig]

@bacongobbler I have a working, backward-compatible convention that relies on Helm PR #5290

As per the documentation in the PR, you can just do the following to an existing chart, and it will just work, even for people who upgrade an existing installation:

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "helm-random-secret.fullname" . }}
{{- if not .Values.somePassword }}
  annotations:
    "helm.sh/resource-policy": no-upgrade-existing
{{- end }}
  labels:
    app: {{ template "helm-random-secret.name" . }}
    chart: {{ template "helm-random-secret.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  {{- if .Values.somePassword }}
  some-password: {{ .Values.somePassword | b64enc | quote }}
  {{- else }}
  some-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}

The semantics are:

# Use a specific password.
# If omitted, will use existing password otherwise generate a new random one.
# somePassword: "my Pas$w0rd"

Thoughts?

[bacongobbler]

Dropped a comment in that PR. Let's continue the discussion on that particular feature over there.

[npdeep]

One approach that has been working for us is setting a default value for postgresqlPassword but hiding that value using the secrets plugin so you don't commit the real password to github.

[cgetzen]

If we got something similar to .Release.Time (maybe .Release.Init) that was set only on install (and not upgrades), I think a lot of use-cases would be satisfied. You would be able to create pseudo-random values based on it that do not change on upgrade. It'd be even better if https://github.com/Masterminds/sprig/blob/master/docs/strings.md#randalphanum-randalpha-randnumeric-and-randascii could take a seed value.

[vsliouniaev]

The install time would have to be secret to safely use a deterministic PRNG for a password, which seems awkward. (it would also be subject to brute force attacks)

How about a Helm feature that generates some entropy and stores it in a secret for each release at first install time, and then deriving passwords from that?

[cgetzen]

Good point. I think there are two cases that I would hope could be addressed together: generating random values for secrets, and generating random values that need to be placed in labels, annotations, etc - such as ID's. Instead of a .Release.Init, would a .Release.RandomSeedVal be appropriate for both of these scenarios?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[michaelfig]

Still relevant. Please review Helm PR #5290.

[danielezer]

+1

[cgetzen]

I think https://github.com/helm/helm/pull/5290 is a step in the right direction but doesn't cover all cases. I would like to contribute, but think it would be a waste to work off of master. Are things too in-flux right now / if not, what branch should I work off of?
/cc @adamreese

[michaelfig]

@cgetzen, if you want to take https://github.com/michaelfig/helm/tree/no-upgrade-existing you are welcome to... I'm not working on it any further. Feel free to adopt it in your own repo, and publish it as a new PR against Helm.

[cgetzen]

Thanks. The changes I'm hoping to incorporate (generating some amount of entropy that can be referenced from the Release) would probably be clobbered by Helm V3.
/cc @bacongobbler

[naseemkullah]

put a general note that one should pre generate a secret and call it a day,
or do the checksum thing

[desaintmartin]

To copy what has been previously said elsewhere, here is my personal opinion:

Although random password generation is a very nice way to get started, it should be clearly advertised as DEMONSTRATION PURPOSE only and no complex handle should be done on the chart part in order to keep the chart as simple as possible to manage complexity and understandability. Or to be completely disabled. Some software support having an empty password and that may be an even simpler way to get started.

If there is a native kube/helm feature to store/remember key/values, why not, but handling it through black magic chart tricks seems insane from a pure maintenance and complexity perspective and should be avoided.

tl; dr: I would be in favor of dropping all complex work done in charts for handling passwords generated by the chart, in order to keep complexity low.

[cerw]

Can someone please explain to my, what is the offical best way of passing the DB User/pass/hostname to chart that uses stable/postgrsql as dependencies? example GO application, that needs to connect to it. this is basic stuff but I can not find one answer. I know PSQL stores it in Secret but how shoudl I use it in my custom chart?

thanks

[agronholm]

@desaintmartin It has been suggested to generate secrets as a helm install hook in this issue. Is that not working for you?

[desaintmartin]

It's only my opinion, but I don't really see the point of adding complexity to charts just to generate a password (that, if you do production, should anyway be generated beforehand and stored somewhere else, and if you don't, you don't really care).

[danielezer]

@agronholm but that would make the secret not managed by helm, which will make it harder to make sure its deleted when release is deleted

[agronholm]

@danielezer I see – I thought that adding the appropriate labels for Helm would do the trick.

[danielezer]

@agronholm I can't make it work with helm.sh/hook-delete-policy for secrets because the options are:

1. hook-succeeded that would delete the secret before it can ever be used
2. hook-failed which won't help as well
3. before-hook-creation which will keep the secret after deleting a release.

[llech]

If Helm has no concept for managing secrets with generated random passwords, why encourage people to use helm for generating secrets with random passwords?

Is there any reasonable productive usage for functions like randAlphaNum other than generating random passwords?

You can find numerous tutorials where passwords are auto-generated and following them will bring you to dead end sooner or later...

[tinyzimmer]

I just ran into this also. I was hoping for some sort of functionality along the lines of the ansible ssm parameter store module where you can set overwrite_value: never. That way as long as the parameter exists, if you invoke with a new randomly generated value, it will not be replaced.

I wonder if similar logic could be applied to secrets, but I don't know if that's a k8s scoped thing or a helm scoped thing.

[astorath]

Hi! I'm using the following solution:

1. Create separate autogen secret resource:

kind: Secret
apiVersion: v1
type: Opaque
metadata: 
 name: "consul-autogen"
 annotations: 
  "helm.sh/hook": "pre-install"
  "helm.sh/hook-delete-policy": "before-hook-creation"
data: 
 "admin-token": {{ randAlphaNum 24 | quote }}

1. Create normal secret resource:

kind: Secret
apiVersion: v1
type: Opaque
metadata: 
 name: "consul"
data: 
 "admin-token": {{ .Values.adminToken | quote }}

1. Then reference _generated_ secret value or _normal_ secret value depending on settings.

Comments:

• this solves any problem with helm delete --purge and helm install (because autogen secret gets replaced in the case).
• this is not clean, as after helm delete --purge autogen secret stays. But helm authors don't want to alter hooks behaviour: https://github.com/helm/helm/issues/4434

[zhming0]

This issue happens on helm 3.0.0-beta.3 too. Might be worthwhile to show this caveat somewhere in the doc?

[agronholm]

This issue happens on helm 3.0.0-beta.3 too

Did you expect this to be fixed? It's not a simple bug, it's a design problem.

[vvanouytsel]

Could a workaround be to use the 'pre-install' hook to create a Secret with a generated password using the 'randAlphaNum' function and a 'pre-delete' hook to make sure the Secret gets deleted if the Release is deleted?

Edit: Seems we can only specify manifest files in hooks, so it seems there might not really be a workaround as long as the hook functionality is not altered.

[mattdornfeld]

I hate to criticize, but this really needs to be solved. This thread is almost two years old. Couldn't this be solved by adding in annotation that will leave a resource unchanged in the event of an upgrade?

[PhoneixS]

I hate to criticize, but this really needs to be solved. This thread is almost two years old. Couldn't this be solved by adding in annotation that will leave a resource unchanged in the event of an upgrade?

There was an attempt but failed in the implementation, you can read more about it in helm/helm#5290 and why it wasn't merged. I think that @cgetzen was thinking on taking it but I'm not sure.

[squillace]

From my perspective, the chart chose to generate the password; but we'd like Helm to do something about that? Shouldn't the chart be written differently? I get _wanting_ helm to be something like a password manager, but doesn't that seem.... wrong?

[cgetzen]

@PhoneixS IMO github.com/helm/helm/pull/5290 doesn't fill most requirements of using helm with random values. If anyone is jumping at the bit, I'd love to see additions to Masterminds/sprig that allow you to set a seed for the random values, which could then be fed in from a chart if we exposed either the InstallDate, or a random value that get's attached to the release on install.

[desaintmartin]

From my perspective, the chart chose to generate the password; but we'd like Helm to do something about that? Shouldn't the chart be written differently? I get _wanting_ helm to be something like a password manager, but doesn't that seem.... wrong?

Once again, do we want Helm to act as a vault? Because that it basically what is needed to solve this issue (even if it already stores the manifests... but that's not a vault).
If not, we should not try to handle this case in the first place and recommend that the charts do not generate password for the user (and do not hack with lot'o'hooks to store/get the passwords, complexifying chart maintenance).

[squillace]

Yeah, no, we do not -- at all. Charts should either hard-code dumb pwds so they never change if you're doing dev, or they shouldn't specify anything at all and the status should spit out what to change in order to run properly. I get that charts didn't do the right thing here, that the intent was good. But fixing _Helm_ and not the chart is not the right direction to move, imho. I cannot see any persuasive argument for pressing on this in helm rather than on charts.

[nuwang]

@squillace

I get _wanting_ helm to be something like a password manager, but doesn't that seem.... wrong?

The problem I think is that this issue violates the design intent of Kubernetes secrets. Secrets should store things like database passwords, and be able to store them stably for the desired duration, unless the secret is explicitly overwritten. Otherwise, losing access to the secret means possibly irretrievable data loss. What's being asked for is that helm provide a mechanism to say: "do not overwrite this secret if it already exists". That it can't do this I feel is a design flaw in helm, and not a design flaw in charts.

[nuwang]

Also, the current workaround we are using for this, as others have mentioned before, is to add the following annotations to the Secret, with the attendant caveats:

  annotations:
    # xref: https://github.com/helm/helm/issues/3053
    # xref: https://github.com/helm/helm/pull/5290
    "helm.sh/hook": "pre-install"
    "helm.sh/hook-delete-policy": "before-hook-creation"

But unless 3rd party charts uniformly do this too, password handling is still iffy.

[nuwang]

The other issue is that this comes back to the issue of encapsulation. The chart author is in a better position to determine what values should be secure and can be autogenerated. For example, consider the rabbitMqErlangCookie value in the rabbitmq chart. It is currently randomly generated if not specified. The user of the chart need not ever know what it is as long as the value is stable for the duration of the installation. Forcing the user to deal with this makes something that could have been simple and secure out of the box, now complex and possibly insecure (e.g. poor choice of password).

[squillace]

re:

That it can't do this I feel is a design flaw in helm, and not a design flaw in charts.

Understood. I cannot but disagree. From my POV, kubernetes itself doesn't say or do anything about this issue at all, and neither should Helm. Every time I think about this situation, I think: damn, we wrote charts to do somethign that by design causes problems because it breaks during upgrade. We'd best fix the charts!

If I were writing a chart from scratch, I'd require a pwd, full stop. If I wanted a new feature to do this, I'd write a helm plugin that does https://github.com/helm/charts/issues/5167#issuecomment-583064019 and help everyone use it.

I do not want Helm changing chart behavior en masse because charts are doing something they were designed for for developer usage. It's totally and entirely an anti-pattern for production usage, and to modify the default behavior would potentially open the reverse problem:

1. if you are running in production with an auto-generated pwd, you need to fish that out and replace it explicitly with something random and trusted.
2. you suddenly now get generated pwds and you are now encouraged to think that this is just peach although no one has vetted this for production-grade security in the first place, _least of all you and your organization_. You're just happy no one bothers you.

I cannot support from my own perspective any change that fixes the tool because the charts were created a long time ago when we knew less about usage than we do now.

FWIW, I think your example of the randomly generated value for the rabbitmq cookie is a good example, so long as the system doesn't care if it changes. If it changes, then having randomly generated in the chart is a bad practice, and the chart should be changed.

So, I'm on the opposite side of your argument:

The user of the chart need not ever know what it is as long as the value is stable for the duration of the installation. Forcing the user to deal with this makes something that could have been simple and secure out of the box, now complex and possibly insecure (e.g. poor choice of password).

and believe strongly that that chart is broken for ITS users, and not helm at all. I hasten to add that I definitely appreciate the problem, have experienced it myself, and empathize with the difficulty of modifying the original charts that made this mistake. I just can't agree with fixing "some" charts by modifying the tool that makes use of them. But hey, that's just me.

[nuwang]

@squillace What I gathered from your message is that some people are asking that helm be changed so that a selective group of charts can continue to work as is? In that case, I wholeheartedly agree with what you're saying, it doesn't seem right to change helm and break existing behaviour. I'm not arguing for that at all.

What I'm arguing for is that helm should add features to support this functionality, for some of the afore-mentioned reasons. The key ones are:

1. So as to provide more secure defaults out of the box
2. Encapsulate details that are really not relevant to chart users and thus, can simplify the external interfaces of the charts.

If Helm can provide a good random number/password generation routine coupled with the ability to selectively preserve or overwrite resources, such as by adding a new annotation as mentioned here (https://github.com/helm/charts/issues/5167#issuecomment-385633991), that would be a fairly simple path forward?

That way, users need not ever set passwords unless they really want to. They can always go into the k8s secret and look it up, making a simple helm install stable/postgresql secure by default.

[llech]

I've learned the hard way that random generated passwords in helm chart are something to be avoided.

However, the problem is, that they are in many tutorials / examples. Probably because people couldn't see any other point of functions like randAlphaNum then generating random passwords.

What is the purpose of providing the functions that have no reasonable use? You don't need random-generated passwords for development, admin/nimda do work well enough.

[agronholm]

I've learned the hard way that random generated passwords in helm chart are something to be avoided.

Would you like to elaborate? If there's a good reason to never do this, it might help settle this discussion.

[danielezer]

I want to add that the reality is that a lot of charts out there actually use this option and thus, a lot of users encounter it. Off the top of my head, I can say that the postgresql and mongodb charts use this to generate a random password if none was provided. I'm sure that there are many other charts that use the randAlphaNum function to generate random strings, so, instead of trying to educate everyone to stop using it (which is, IMHO, not really feasible), isn't it easier to introduce an opt-in option to keep the value static?

[squillace]

I totally agree that someone might want this ability, and so creating a helm plugin per https://github.com/helm/charts/issues/5167#issuecomment-583064019 to enable it is the way to go, IMHO. There is nothing stopping anyone from doing this right now. And it'll likely be very popular!

I agree with suggestions that this would be useful. There is nothing stopping anyone from creating this functionality right now!

[squillace]

Even better: have it examine charts for random generation, and prompt to keep, replace, and/or generate one and inject it so that generation occurs outside of the release process.....

[armab]

Hey, it's not only about randAlphaNum, there is also genPrivateKey which can generate private SSH keys for Helm charts. Is it improper use as well? Also, if secrets generation are not favored in Helm, the situation is the same for any autogenerated values and their state.

I don't think anyone is asking about using Helm as a password manager. It's about handling state for the value.

Extracting some of the thoughts from one of the Helm charts that tried to use autogenerated Helm values: https://github.com/StackStorm/stackstorm-ha/pull/62.
Here are discovered problems to be clear:

Helm doesn't provide enough instrumentation to work with autogenerated values and their state. It's not just passwords.

1) It's possible to autogenerate a value, but it will be re-generated every time on Helm upgrade which is undesired
2) It's possible to workaround value generation only once on install and mark it as unmanaged by Helm (so it's not deleted on upgrade), but then there is no way to transition to user-provided data in helm values after that

I've seen other official Helm chart repositories and what they do: allow or autogenerated or predefined secrets without proper transition, meaning if user had autogenerated secret on first install (default), they can't change it to custom with no chance to rotate the secrets afterwards.

The solution for now: a) Don't use autogenerated values at all; b) Hardcode "example/dummy" secrets in Helm values which is more insecure; c) Leave the secrets empty forcing user to fill it, which is a poor user experience. Both Helm core maintainers and Charts maintainers would loose from this situation. We all want better user experience and adoption.

With that missing functionality with autogenerated < > predefined value transition a better solution introduced in Helm itself is something that people are asking.

[squillace]

ANY function can have this behavior, though you're right that most involve secrets. Anything generated by what is essentially an out of band function will create this situation. I remain uninterested in solving this myself and remain of the position that those who want to address this situation have the power to do so already with either a complete implementation, or a plugin. In fact, creating this behavior in a plugin is a great way to vet a solution here generally if it is to be adopted in the future.

[llech]

I've learned the hard way that random generated passwords in helm chart are something to be avoided.

Would you like to elaborate? If there's a good reason to never do this, it might help settle this discussion.

Well, the problem is, that random generated values that they were working on their own.

The secret was always overwritten, even if nothing has changed in the template or values, because helm is unable to detect, that the only 'difference' is the random generated value.

However, the pods using the secret were not started new because helm is unable to detect the fact that the pod is using the secret that has changed.

It created often a situation, when 2 pods, that were using the same secret, were using different versions of the secret, with different passwords etc.

But on the 2nd thought, I'd like to revert what I've stated. The situation above is an example, that it's a bad idea to use helm to generate secrets in the same chart as the deployments using them, because it doesn't work reliable. However, both secrets and random values work fine in install-only scenario, so the solution that might work well is splitting the helm chart to 2 charts.

The first chart contains only secret, and might utilize random functions, to prevent having passwords in the shell history or in the value files. That chart will be normally installed only once, during namespace setup.

The other will use that secrets and will be upgraded in the regular maintanance cycle.

It's not the problem with that functions, but that the documentation is missing to point out the pitfalls of using them.

[agronholm]

The secret was always overwritten, even if nothing has changed in the template or values, because helm is unable to detect, that the only 'difference' is the random generated value.

I would like to point out that this is exactly the problem that this proposal aims to solve: that generated secrets would not be regenerated on subsequent release updates, but unlike pre-install hooks, they would be removed if the release is deleted.

[tahmmee]

This might be a useful workaround. With helm 3.1 there is a lookup function,
which means it's possible to skip upgrading of a secret/configmap if it already exists

{{- if not (lookup "v1" "Secret" .Release.Namespace "my-secret") }}
---
apiVersion: v1
kind: Secret

edit:
This appears to be working, except I had to reuse a specific attribute of the secret rather than entire resource because helm will delete it if the template is empty...

{{- define "cluster.password" -}}

{{- $secret := (lookup "v1" "Secret" .Release.Namespace "my-secret") -}}
{{- if $secret -}}
{{/* 
   Reusing current password since secret exists
*/}}
{{-  $secret.data.password -}}
{{- else -}}
{{/* 
    Generate new password
*/}}
{{- (randAlpha 6) | b64enc | quote -}}
{{- end -}}

Then elsewhere in secret

apiVersion: v1
kind: Secret
data:
    password: {{ template "cluster.password"  .  }}

[herrberk]

Here is an alternative (requires Helm v3) to tahmmee's solution, in which the generated values are stored in template variables and if the resource already exists, the existing values are used:

{{- $mongoRoot := (randAlpha 16) | b64enc | quote }}
{{- $mongoReplicaKey := (randAlpha 64) | b64enc | quote }}

{{- $secret := (lookup "v1" "Secret" .Release.Namespace "mongo-keys") }}
{{- if $secret }}
{{- $mongoRoot = index $secret.data "mongodb-root-password" }}
{{- $mongoReplicaKey = index $secret.data "mongodb-replica-set-key" }}
{{- end -}}

apiVersion: v1
kind: Secret
metadata:
  name: mongo-keys
  namespace: {{ .Release.Namespace }}
type: Opaque
data:
  mongodb-root-password: {{ $mongoRoot}}
  mongodb-replica-set-key: {{ $mongoReplicaKey }}

[movergan]

Please let's have it fixed. It's ok to use workaround for in-house charts, not for 3rd party dependency though.

[mjgallag]

@herrberk Any ideas for rendering $mongoRoot in NOTES.txt on install? Thanks.

[aure-olli]

Hi

I've rewritten kubernetes replicator and added some annotations to deal with this kind of problems: https://github.com/olli-ai/k8s-replicator#use-random-password-generated-by-an-helm-chart

Now can generate a random password with helm, and replicate it only once to another secret thus it won't be change by helm in the future.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: admin-password-source
  annotations:
    k8s-replicator/replicate-to: "admin-password"
    k8s-replicator/replicate-once: "true"
stringData:
  password: {{ randAlphaNum 64 | quote }}

Hope it will help people.

[monotek]

From helm 3.1.0 you should be able to use the lookup function to read a previously created random.password from kubernetes: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/

[llech]

From helm 3.1.0 you should be able to use the lookup function to read a previously created random.password from kubernetes: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/

How can it work, if the docs state, that:

Keep in mind that Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run, so the lookup function will return an empty list (i.e. dict) in such a case.

so exaclty in the cases where that values would be used?

[jglick]

Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run

helm install --dry-run _does_ contact the API server.

If you use this trick from helm template, the chart will not see an existing secret, so it will simply generate a new one each time.

[dudicoco]

@jglick this would still show a diff when using the helm-diff plugin, so still problematic :(