Charts: [bitnami/mariadb-galera] New bootstrap options break upgrade/rollout

[aspettl]

Which chart:
mariadb-galera, 3.1.3 (since 3.0.0)

Describe the bug
The chart version 3.x introduced the force bootstrap feature: https://github.com/bitnami/charts/pull/2727
This is really a needed feature for recovery from a failure state, but it introduced a major regression: A rolling restart of the mariadb-galera pods does not work anymore, which is performed e.g. by a chart upgrade. By default, it will always bootstrap from the pod with index 0. If the cluster is up and running (with pod 1 and 2 that were already upgraded by the rolling restart), pod 0 will want to bootstrap anyway. Of course, safe_to_bootstrap is not set for pod 0 (and it should not be). Therefore it will not start with the error message It is not safe to bootstrap form this node ('safe_to_bootstrap=0' is set in 'grastate.dat') and be stuck in a crash loop.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a new mariadb-galera cluster with the default of 3 nodes and persistence
2. Simulate a rollout of a newer chart version with "kubectl rollout restart statefulset ..."
3. Observe that pod 2 is restarted successfully, it joins the cluster
4. Observe that pod 1 is restarted successfully, it joins the cluster
5. Observe that pod 0 tries to restart, but fails with It is not safe to bootstrap form this node ('safe_to_bootstrap=0' is set in 'grastate.dat')

Expected behavior

• Bootstrap should not be a default option. It introduces the risk of pods that refuse to start (even though they could join the cluster) or even having two "clusters" (split brain).
• If a cluster already is up and running, a new node should never bootstrap. It's better to fail (it does that currently only when persistence is active).
• Nonetheless, it would of course be nice that on initial deployment, the initial bootstrap works. The chart should still be easy to use.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.6", GitCommit:"d32e40e20d167e103faf894261614c5b45c44198", GitTreeState:"clean", BuildDate:"2020-05-20T13:08:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Additional context

Workaround
If the cluster is functional and no bootstrap is necessary, you can set galera.bootstrap.bootstrapFromNode to some large value (larger than the number of nodes). Then no bootstrap wil be tried.

Broken current implementation
The current implementation uses an EmptyDir volume with a .bootstrap/done file to avoid the bootstrap on a pod restart. Unfortunately, this is not enough as a rollout creates new pods. This may be a bit surprising because a StatefulSet is used. But just thinking about the case that the pod could be scheduled on another node (with the same data volume), this makes sense.

Warning for persistence=false
When persistence is deactivated (e.g. for testing), this issue may not be apparent. The reason is that the first node will be able to bootstrap with an empty database - always! So there will be two clusters, one with wsrep_cluster_size=1 and one with wsrep_cluster_size=2. That's even worse.

[rafariossaa]

Hi @aspettl ,
Thanks for reporting this issue.
We will look for a fix for this, I have created and internal task. We will come back as soon as we have news.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[aspettl]

Remove stale

[rafariossaa]

Hi,
I added the on-hold label to avoid the stale bot.
We hope to have news on this issue soon.

[kanersps]

What can I currently do if my cluster is already having this issue?

Would setting galera.bootstrap.bootstrapFromNode and upgrading the helm chart work

[lentzi90]

What can I currently do if my cluster is already having this issue?

Would setting galera.bootstrap.bootstrapFromNode and upgrading the helm chart work

Yes that works for me. I'm installing it like this now (replace passwords of course):

helm install database bitnami/mariadb-galera --set rootUser.password=secret \
    --set galera.mariabackup.password=secret --set db.password=secret
# Change bootstrap node to high number (non-existing) to workaround bug
# See https://github.com/bitnami/charts/issues/2810
helm upgrade database bitnami/mariadb-galera --set rootUser.password=secret \
    --set galera.mariabackup.password=secret --set db.password=secret \
    --set galera.bootstrap.bootstrapFromNode=100

After this has finished, I can delete any one of the pods and they will successfully come back.

[rafariossaa]

Hi ,
Thanks for the workaround, we are currently working on this issue, we hope to have news soon.

[rafariossaa]

Hi,
Just an small update.
You can also upgrade the cluster normally but scaling first to one replica:

kubectl scale --replicas=1 statefulset  mygalera-mariadb-galera