Charts: [bitnami/redis-cluster] Adding a new node to the cluster only add master node

[qi-min]

Which chart:
[bitnami/redis-cluster]

Describe the bug
when I run helm upgrade to add node, the new added nodes are all master nodes, no replicas nodes added

To Reproduce
Steps to reproduce the behavior:

Helm install redis-cluster with default settings
Helm upgrade with the following settings:

helm upgrade --timeout 600s redis  --set "password=${REDIS_PASSWORD},cluster.nodes=8,cluster.update.addNodes=true,cluster.update.currentNumberOfNodes=6,cluster.init=false" bitnami/redis-cluster

Expected behavior
two new nodes added, one is master and one is slave. There should be 4 master and 4 slave node after upgrade.

[javsalgar]

Hi,

Thank you very much! I was able to reproduce the issue. I will open a task for investigation. As soon as we have more news, I will update this ticket.

[rxsto]

We're trying to use a custom scheduler to bind the redis nodes to high available zones and are experiencing the same issue. All the nodes (including the new ones) turn into standalone masters and the job keeps crashing since all the available slots are not covered. We're trying to deploy a high available redis cluster with improved failover handling, further investigation would be appreciated! Thanks :)

[javsalgar]

Thanks for the input! I will forward this to the rest of the team

[rxsto]

So, we've tried to validate the fix that was merged with #3928 but it seemingly still didn't work.

Here's what we've tried:

• Launch a testing kubernetes cluster with 3 nodes and 3 masters.
• Installed a redis cluster with 3 masters and 3 slaves (worked fine, the cluster state was healthy)
• Tested failover, restarted one kubernetes node, redis cluster successfully handled it and elected a new master for the failing pair
• Rebooted the node, manually re-elected the old master, cluster still healthy and working
• Added a new kubernetes node to the cluster
• Changed the values to include 2 new redis nodes in the next helm upgrade
• Upgraded the helm deployment, 2 new nodes were added to the cluster
• The whole statefulset restarted one pod after another just how it's used to act
• After the rollout was done, every single redis node was a single master in a different redis cluster, there was absolutely no communication between the nodes, they seemingly forgot their old role

We're not using a LoadBalancer for the redis cluster service, and we're also not using persistent volumes, since the redis cluster should definitely be able to keep its keys and values while having a minor outage or updating the redis cluster.

We're also using a custom scheduler to make sure the redis nodes are evenly distributed and automatic failover handling actually works. ((podId % k8sNodeCount) + smallestK8sNodeId)

We really hoped the pull request mentioned above would fix this issue, but we're still not able to maintain a healthy redis cluster with this chart.

If we're missing anything, please let us know! That would be highly appreciated!

Thanks

[rafariossaa]

Hi,
Thanks for your thorough testing. It is only failing for you when adding nodes to the cluster ? I mean, Does in the rest of the cases the cluster was with all the nodes ?

[rxsto]

Hey,

I've just been trying to reproduce this issue without adding new kubernetes nodes, and from what I can tell the redis cluster always gets destroyed and ends up in a failing state when upgrading the helm release.

I tried to use the command helm upgrade --timeout 600s redis-cluster --set "password=password,cluster.nodes=8,cluster.update.addNodes=true,cluster.update.currentNumberOfNodes=6" bitnami/redis-cluster -n redis-cluster but ended up getting Error: UPGRADE FAILED: cannot patch "redis-cluster" with kind StatefulSet: StatefulSet.apps "redis-cluster" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. Changing the values.yaml file and upgrading the helm release with the new files forces the cluster to fall apart. New added nodes seem to work at first, but once the rollout finished all of the old nodes are masters in their own redis cluster, just as I described in my earlier comment.

So to sum this up, updating/upgrading the helm release always destroys the redis cluster, with and without adding new kubernetes nodes.

[rafariossaa]

Hi,
Sorry for the long delay.
I am adding all this notes to a internal task to dig deeper into this issue.