Charts: MySQL root password no longer valid after upgrade

[mstrzele]

It's because the password is random generated inside the template of a secret and after each upgrade it is changed.

$ helm install stable/mysql -n foo
# ...
To get your root password run:

    printf $(printf '\%o' `kubectl get secret --namespace default foo-mysql -o jsonpath="{.data.mysql-root-password[*]}"`);echo
# ...
$ printf $(printf '\%o' `kubectl get secret --namespace default foo-mysql -o jsonpath="{.data.mysql-root-password[*]}"`);echo
SpeSUQshnK
$ helm upgrade foo stable/mysql
$ printf $(printf '\%o' `kubectl get secret --namespace default foo-mysql -o jsonpath="{.data.mysql-root-password[*]}"`);echo
EiKA8TDMp9

[viglesiasce]

I opened the issue above which will need to be fixed before we can react to this.

@prydonius @lachie83 any ideas on other workarounds?

[prydonius]

@viglesiasce currently the only workaround would be to explicitly set the same password as a value when upgrading.

[sstarcher]

@viglesiasce with the above Issue linked from helm is the intention to wrap the secret yaml in a
{{ if .Release.IsInstall }}

{{ end }}

Sensu has this same problem on upgrades as it uses the Redis chart.

[sstarcher]

and if we did the above it would delete the secrets object entirely

[prydonius]

Thanks @sstarcher. I wonder if there's a way to tell Helm not to remove the secret on upgrade?

[mikesplain]

Wouldn't it also be possible to use a job to create the secret and only run that job at install? You could use a hook to run the job before everything else too. I know it's ugly but could work?

[prydonius]

@mikesplain hmm, you can also actually mark Secrets themselves as pre-install hooks, so you might not need a job to do that.

[mikesplain]

@prydonius right but wouldn't that still have the above problem of overriding each time?

[sstarcher]

Per the documentation and a test it looks like pre-install hooks are ran every time even on upgrades.

https://github.com/kubernetes/helm/blob/9665db7d165cc4129bea2456fffa53aaed34af41/docs/charts_hooks.md#the-available-hooks

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[arianitu]

After a helm upgrade, the root password is lost and now I'm having to recover a root password.

It's probably time to stop using Helm.

[GM12Tick]

this just happened to us, how do we prevent this?

[andrewhowdencom]

For those who google this et. al; to prevent this, specify a root (and user) password that overrides the default (random) root password.

[GM12Tick]

@andrewhowdencom - That didn't work on upgrade attempts. (Tried to remove grants table but that also didnt last... )
The solution for us was to:

1. Upgrade values to have skip-grant-tables
   2.backup db
2. Delete and purge the helm chart.
3. Re install with those values, and not use the ROOT creds.

Will that solve my existing deployment issue or i need to reinstall ?