Project: Redis cluster with TLS encryption have replication broken.

[denist-huma]

I have issues

I'm submitting a bug report

What is the current behavior?

Obviously, clustering is not working. Neither a slave catch up new key, nor another master.

Verify Redis clustering configured with TLS/SSL

$ kubectl exec -it redis-cluster-shard0-0 -n kubedb-demo -c redis -- sh
/data # ls /certs/
ca.crt      client.crt  client.key  server.crt  server.key
redis-cli --tls --cert /certs/client.crt --key /certs/client.key --cacert /certs/ca.crt
127.0.0.1:6379> cluster nodes
1dddc8fe591bbafc6324a6e90d7cbe4ce9b171e7 10.180.1.14:6379@16379 myself,master - 0 1629108966000 1 connected 0-5460
f5a7c80c1c5e77c578c789700923a7aa1aa4e4b4 10.180.6.49:6379@16379 slave 1dddc8fe591bbafc6324a6e90d7cbe4ce9b171e7 0 1629108969000 1 connected
838e36ecfa7720ba8b1883899ff2d5e764a9b36a 10.180.6.50:6379@16379 slave eed0040e1b8052a6dbc2387d2309a45c9936801c 0 1629108969655 2 connected
d0468e8bbffc73bd089924912493f00cbac5abbc 10.180.2.62:6379@16379 slave 24f06616b89bb35e0932e61e8cbad823324aa1cb 0 1629108968000 3 connected
eed0040e1b8052a6dbc2387d2309a45c9936801c 10.180.1.15:6379@16379 master - 0 1629108968553 2 connected 5461-10922
24f06616b89bb35e0932e61e8cbad823324aa1cb 10.180.0.155:6379@16379 master - 0 1629108968653 3 connected 10923-16383

127.0.0.1:6379> set hello world
OK
127.0.0.1:6379> get hello
"world"
127.0.0.1:6379> 
/data # redis-cli --tls --cert /certs/client.crt --key /certs/client.key --cacert /certs/ca.crt -h 10.180.0.155
10.180.0.155:6379> get hello
(error) MOVED 866 10.180.1.14:6379
/data # redis-cli --tls --cert /certs/client.crt --key /certs/client.key --cacert /certs/ca.crt -h 10.180.6.49
10.180.6.49:6379> get hello
(error) MOVED 866 10.180.1.14:6379

Logs from pods of 3 masters and slave of 0:

denis@L560:~/huma/hpaas-operator$ kubectl -n kubedb-demo logs redis-cluster-shard0-0
1:C 16 Aug 2021 09:39:03.853 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Aug 2021 09:39:03.853 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Aug 2021 09:39:03.853 # Configuration loaded
1:M 16 Aug 2021 09:39:03.855 * No cluster configuration found, I'm 1dddc8fe591bbafc6324a6e90d7cbe4ce9b171e7
1:M 16 Aug 2021 09:39:03.858 * Running mode=cluster, port=6379.
1:M 16 Aug 2021 09:39:03.858 # Server initialized
1:M 16 Aug 2021 09:39:03.858 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 16 Aug 2021 09:39:03.858 * Ready to accept connections
1:M 16 Aug 2021 09:39:15.185 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
1:M 16 Aug 2021 09:39:15.195 # IP address for this node updated to 10.180.1.14
1:M 16 Aug 2021 09:39:20.105 # Cluster state changed: ok
1:M 16 Aug 2021 09:39:35.340 * Replica 10.180.6.49:6379 asks for synchronization
1:M 16 Aug 2021 09:39:35.341 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'dfd89dbfe30eaea9e865e796184dc20bfd838966', my replication IDs are '88a8bab04984f75d2b3cee29ca0a9dfc6fab97b7' and '0000000000000000000000000000000000000000')
1:M 16 Aug 2021 09:39:35.341 * Replication backlog created, my new replication IDs are '3a41d91880d4d884f1441e0b83754b6d97cad647' and '0000000000000000000000000000000000000000'
1:M 16 Aug 2021 09:39:35.341 * Starting BGSAVE for SYNC with target: disk
1:M 16 Aug 2021 09:39:35.342 * Background saving started by pid 66
66:C 16 Aug 2021 09:39:35.346 * DB saved on disk
66:C 16 Aug 2021 09:39:35.347 * RDB: 0 MB of memory used by copy-on-write
1:M 16 Aug 2021 09:39:35.439 * Background saving terminated with success
1:M 16 Aug 2021 09:39:35.440 * Synchronization with replica 10.180.6.49:6379 succeeded
1:M 16 Aug 2021 09:42:03.396 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=23)
denis@L560:~/huma/hpaas-operator$ kubectl -n kubedb-demo logs redis-cluster-shard0-1
1:C 16 Aug 2021 09:39:05.248 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Aug 2021 09:39:05.248 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Aug 2021 09:39:05.248 # Configuration loaded
1:M 16 Aug 2021 09:39:05.252 * No cluster configuration found, I'm f5a7c80c1c5e77c578c789700923a7aa1aa4e4b4
1:M 16 Aug 2021 09:39:05.255 * Running mode=cluster, port=6379.
1:M 16 Aug 2021 09:39:05.256 # Server initialized
1:M 16 Aug 2021 09:39:05.256 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 16 Aug 2021 09:39:05.256 * Ready to accept connections
1:M 16 Aug 2021 09:39:33.442 # IP address for this node updated to 10.180.6.49
1:S 16 Aug 2021 09:39:34.365 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 16 Aug 2021 09:39:34.366 # Cluster state changed: ok
1:S 16 Aug 2021 09:39:35.330 * Connecting to MASTER 10.180.1.14:6379
1:S 16 Aug 2021 09:39:35.330 * MASTER <-> REPLICA sync started
1:S 16 Aug 2021 09:39:35.335 * Non blocking connect for SYNC fired the event.
1:S 16 Aug 2021 09:39:35.337 * Master replied to PING, replication can continue...
1:S 16 Aug 2021 09:39:35.339 * Trying a partial resynchronization (request dfd89dbfe30eaea9e865e796184dc20bfd838966:1).
1:S 16 Aug 2021 09:39:35.343 * Full resync from master: 3a41d91880d4d884f1441e0b83754b6d97cad647:0
1:S 16 Aug 2021 09:39:35.343 * Discarding previously cached master state.
1:S 16 Aug 2021 09:39:35.441 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
1:S 16 Aug 2021 09:39:35.441 * MASTER <-> REPLICA sync: Flushing old data
1:S 16 Aug 2021 09:39:35.441 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 16 Aug 2021 09:39:35.441 * Loading RDB produced by version 6.0.6
1:S 16 Aug 2021 09:39:35.441 * RDB age 0 seconds
1:S 16 Aug 2021 09:39:35.441 * RDB memory usage when created 2.52 Mb
1:S 16 Aug 2021 09:39:35.441 * MASTER <-> REPLICA sync: Finished with success
1:S 16 Aug 2021 09:39:35.442 * Background append only file rewriting started by pid 22
1:S 16 Aug 2021 09:39:35.466 * AOF rewrite child asks to stop sending diffs.
22:C 16 Aug 2021 09:39:35.467 * Parent agreed to stop sending diffs. Finalizing AOF...
22:C 16 Aug 2021 09:39:35.467 * Concatenating 0.00 MB of AOF diff received from parent.
22:C 16 Aug 2021 09:39:35.467 * SYNC append only file rewrite performed
22:C 16 Aug 2021 09:39:35.467 * AOF rewrite: 0 MB of memory used by copy-on-write
1:S 16 Aug 2021 09:39:35.530 * Background AOF rewrite terminated with success
1:S 16 Aug 2021 09:39:35.530 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 16 Aug 2021 09:39:35.530 * Background AOF rewrite finished successfully
denis@L560:~/huma/hpaas-operator$ kubectl -n kubedb-demo logs redis-cluster-shard2-0
1:C 16 Aug 2021 09:39:10.009 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Aug 2021 09:39:10.009 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Aug 2021 09:39:10.009 # Configuration loaded
1:M 16 Aug 2021 09:39:10.013 * No cluster configuration found, I'm 24f06616b89bb35e0932e61e8cbad823324aa1cb
1:M 16 Aug 2021 09:39:10.018 * Running mode=cluster, port=6379.
1:M 16 Aug 2021 09:39:10.018 # Server initialized
1:M 16 Aug 2021 09:39:10.018 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 16 Aug 2021 09:39:10.019 * Ready to accept connections
1:M 16 Aug 2021 09:39:15.186 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH
1:M 16 Aug 2021 09:39:15.301 # IP address for this node updated to 10.180.0.155
1:M 16 Aug 2021 09:39:20.180 # Cluster state changed: ok
1:M 16 Aug 2021 09:39:37.299 * Replica 10.180.2.62:6379 asks for synchronization
1:M 16 Aug 2021 09:39:37.299 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b5c7f4d3733a7a656b0e65ba40710b721e2d735e', my replication IDs are '73af330c9140f0d7800aee294fd6f95d3d831c8d' and '0000000000000000000000000000000000000000')
1:M 16 Aug 2021 09:39:37.299 * Replication backlog created, my new replication IDs are 'edfb5b04ea4f071989d3b72707f2b49f452e6071' and '0000000000000000000000000000000000000000'
1:M 16 Aug 2021 09:39:37.299 * Starting BGSAVE for SYNC with target: disk
1:M 16 Aug 2021 09:39:37.299 * Background saving started by pid 20
20:C 16 Aug 2021 09:39:37.304 * DB saved on disk
20:C 16 Aug 2021 09:39:37.305 * RDB: 0 MB of memory used by copy-on-write
1:M 16 Aug 2021 09:39:37.354 * Background saving terminated with success
1:M 16 Aug 2021 09:39:37.355 * Synchronization with replica 10.180.2.62:6379 succeeded
denis@L560:~/huma/hpaas-operator$ kubectl -n kubedb-demo logs redis-cluster-shard1-0
1:C 16 Aug 2021 09:39:06.567 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Aug 2021 09:39:06.567 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Aug 2021 09:39:06.567 # Configuration loaded
1:M 16 Aug 2021 09:39:06.570 * No cluster configuration found, I'm eed0040e1b8052a6dbc2387d2309a45c9936801c
1:M 16 Aug 2021 09:39:06.573 * Running mode=cluster, port=6379.
1:M 16 Aug 2021 09:39:06.573 # Server initialized
1:M 16 Aug 2021 09:39:06.573 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 16 Aug 2021 09:39:06.573 * Ready to accept connections
1:M 16 Aug 2021 09:39:15.186 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH
1:M 16 Aug 2021 09:39:15.200 # IP address for this node updated to 10.180.1.15
1:M 16 Aug 2021 09:39:20.105 # Cluster state changed: ok
1:M 16 Aug 2021 09:39:36.163 * Replica 10.180.6.50:6379 asks for synchronization
1:M 16 Aug 2021 09:39:36.163 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'bb0f82a8be0af656c166a1c61ec6895fab3157a0', my replication IDs are '5b64f77048f5fa64466b82b703e1ac959d9a9b32' and '0000000000000000000000000000000000000000')
1:M 16 Aug 2021 09:39:36.163 * Replication backlog created, my new replication IDs are '80c4dda3c393349a523f92082d74525cd7ffd08f' and '0000000000000000000000000000000000000000'
1:M 16 Aug 2021 09:39:36.163 * Starting BGSAVE for SYNC with target: disk
1:M 16 Aug 2021 09:39:36.164 * Background saving started by pid 20
20:C 16 Aug 2021 09:39:36.167 * DB saved on disk
20:C 16 Aug 2021 09:39:36.167 * RDB: 0 MB of memory used by copy-on-write
1:M 16 Aug 2021 09:39:36.238 * Background saving terminated with success
1:M 16 Aug 2021 09:39:36.238 * Synchronization with replica 10.180.6.50:6379 succeeded

If this is a bug, how to reproduce it? Please include a code sample if relevant

Use KubeDB Enterprise Edition to provision a Redis cluster with TLS/SSL encryption

• Install KubeDB Enterprise Edition
  
  
  
  • obtain a KubeDB Enterprise Edition license
  
  
• MySQL TLS/SSL Encryption

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=redis/O=kubedb"
Generating a RSA private key
.........................................+++++
.......+++++
writing new private key to './ca.key'
-----
$ kubectl create secret tls my-ca \
     --cert=ca.crt \
     --key=ca.key \
     --namespace=kubedb-demo
secret/my-ca created

$ kubectl apply -f issuer.yaml --namespace=kubedb-demo
issuer.cert-manager.io/redis-issuer created

$ kubectl apply -f demo-tls.yaml --namespace=kubedb-demo
redis.kubedb.com/redis-cluster created

apiVersion: cert-manager.io/v1beta1
kind: Issuer
metadata:
  name: redis-issuer
spec:
  ca:
    secretName: my-ca
---
apiVersion: kubedb.com/v1alpha2
kind: Redis
metadata:
  name: redis-cluster
spec:
  version: 6.0.6
  mode: Cluster
  cluster:
    master: 3
    replicas: 1
  storageType: Ephemeral
  storage:
    resources:
      requests:
        storage: 1Gi
    storageClassName: "standard"
    accessModes:
      - ReadWriteOnce
  terminationPolicy: WipeOut
  tls:
    issuerRef:
      apiGroup: cert-manager.io
      kind: Issuer
      name: redis-issuer

What's the expected behavior?

I should get "hello" from each of 3 masters and slaves

Are you able to fix this problem and submit a PR? Link here if you have already

Environment details

• KubeDB v2021.06.23
• kubedb/operator:v0.19.0
• Kubernetes version: v1.19.10-gke.1700
• redis-cli 6.0.6
• jetstack/cert-manager version: "1.3.1"

Any other relevant info

Installed Community Edition, Migrated to Enterprise Edition KubeDB v2021.06.23. Helm 3.

denis@L560:~/huma/hpaas-operator/demos/redis-cluster-management/kubedb$ helm ls -n kube-system
NAME                    NAMESPACE       REVISION        UPDATED                                         STATUS          CHART                                   APP VERSION
kubedb                  kube-system     2               2021-08-13 13:09:33.079450565 +0300 MSK         deployed        kubedb-v2021.06.23                      v2021.06.23

Operator's log

I0816 09:38:07.539690       1 redis.go:78] Patching Redis kubedb-demo/redis-cluster with {"metadata":{"finalizers":null}}.
I0816 09:39:00.542536       1 redis.go:78] Patching Redis kubedb-demo/redis-cluster with {"metadata":{"finalizers":["kubedb.com"]}}.
I0816 09:39:00.662609       1 service.go:37] Creating Service kubedb-demo/redis-cluster-pods.
I0816 09:39:00.771215       1 secret.go:39] Creating Secret kubedb-demo/redis-cluster.
I0816 09:39:00.783007       1 redis.go:78] Patching Redis kubedb-demo/redis-cluster with {"spec":{"configSecret":{"name":"redis-cluster"}}}.
I0816 09:39:00.843155       1 serviceaccount.go:37] Creating ServiceAccount kubedb-demo/redis-cluster.
I0816 09:39:00.876654       1 role.go:37] Creating Role kubedb-demo/redis-cluster.
I0816 09:39:00.935130       1 rolebinding.go:37] Creating RoleBinding kubedb-demo/redis-cluster.
I0816 09:39:00.996681       1 service.go:37] Creating Service kubedb-demo/redis-cluster.
I0816 09:39:01.050806       1 redis.go:86] wait for all certificate secrets for Redis kubedb-demo/redis-cluster
I0816 09:39:01.123988       1 service.go:77] Patching Service kubedb-demo/redis-cluster with {"metadata":{"annotations":null}}.
I0816 09:39:01.155298       1 redis.go:86] wait for all certificate secrets for Redis kubedb-demo/redis-cluster
I0816 09:39:02.582619       1 redis.go:86] wait for all certificate secrets for Redis kubedb-demo/redis-cluster
I0816 09:39:02.704281       1 statefulset.go:41] Creating StatefulSet kubedb-demo/redis-cluster-shard0.
I0816 09:39:02.722714       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard0
I0816 09:39:02.729475       1 pdb.go:40] Creating PodDisruptionBudget kubedb-demo/redis-cluster-shard0.
I0816 09:39:02.811460       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard0
I0816 09:39:02.916229       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:03.033228       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:04.078098       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard0
I0816 09:39:04.219184       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
E0816 09:39:04.948363       1 health.go:121] Failed to ping the database: kubedb-demo/redis-cluster error: dial tcp 10.184.2.149:6379: connect: connection refused
I0816 09:39:05.041236       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:05.386198       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard0
I0816 09:39:05.489975       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:05.525726       1 statefulset.go:41] Creating StatefulSet kubedb-demo/redis-cluster-shard1.
I0816 09:39:05.542982       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard1
I0816 09:39:05.562015       1 pdb.go:40] Creating PodDisruptionBudget kubedb-demo/redis-cluster-shard1.
I0816 09:39:05.675775       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard1
I0816 09:39:05.761234       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:05.813122       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard1 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:07.104605       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard1
I0816 09:39:08.463605       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard1
I0816 09:39:08.605800       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:08.633625       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard1 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:08.659560       1 statefulset.go:41] Creating StatefulSet kubedb-demo/redis-cluster-shard2.
I0816 09:39:08.671876       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard2
I0816 09:39:08.676374       1 pdb.go:40] Creating PodDisruptionBudget kubedb-demo/redis-cluster-shard2.
I0816 09:39:08.731587       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard2
I0816 09:39:08.853826       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:08.890739       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard1 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:08.923265       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard2 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:11.132276       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard2
I0816 09:39:11.326239       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:11.375281       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard1 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:11.424290       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard2 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:13.107857       1 controller.go:113] Started processing, key: kubedb-demo/redis-cluster-shard2
I0816 09:39:13.298688       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:13.345391       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard1 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:13.383848       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard2 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:39:14.402487       1 cluster.go:82] All redis servers are ready
I0816 09:39:14.402554       1 cluster.go:658] Ensuring new cluster...
I0816 09:39:14.402780       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:39:51.750203       1 cluster.go:308] Ensuring extra slaves be removed...
I0816 09:39:51.750228       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:39:56.432939       1 cluster.go:353] Ensuring extra masters be removed...
I0816 09:39:56.432960       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:40:01.421271       1 cluster.go:474] Ensuring new masters be added...
I0816 09:40:01.421294       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:40:06.102178       1 cluster.go:525] Ensuring slots are rebalanced...
I0816 09:40:06.102266       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:40:11.051881       1 cluster.go:608] Ensuring new slaves be added...
I0816 09:40:11.051931       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:40:16.063165       1 cluster.go:169] Ensuring 1st pod as master in each statefulSet...
I0816 09:40:20.892729       1 statefulset.go:135] Cluster configured
I0816 09:40:20.894189       1 statefulset.go:136] Checking for removing master(s)...
I0816 09:40:20.900946       1 statefulset.go:159] Checking for removing slave(s)...
I0816 09:40:20.936778       1 appbinding.go:38] Creating AppBinding kubedb-demo/redis-cluster.
I0816 09:40:20.961279       1 service.go:140] spec.monitor.agent is not provided by prometheus.io
I0816 09:40:21.077405       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard0 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:40:21.134686       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard1 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:40:21.166340       1 statefulset.go:81] Patching StatefulSet kubedb-demo/redis-cluster-shard2 with {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"redis"}],"$setElementOrder/volumes":[{"name":"data"},{"name":"custom-config"},{"name":"tls-volume"},{"name":"exporter-tls-volume"}],"containers":[{"$setElementOrder/env":[{"name":"POD_IP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":null}}}],"name":"redis"}],"securityContext":null,"volumes":[{"$retainKeys":["name","projected"],"name":"tls-volume","projected":{"defaultMode":null}},{"$retainKeys":["name","projected"],"name":"exporter-tls-volume","projected":{"defaultMode":null}}]}}}}.
I0816 09:40:22.114746       1 cluster.go:82] All redis servers are ready

[Emon46]

Hello @denist-huma,
For connecting as a client with Redis Cluster you need to add flag like -c in your redis-cli command. you can find out it in the official doc Ref.
From Doc,

The redis-cli utility implements basic cluster support when started with the -c switch.

This one should work:

redis-cli -c --tls --cert /certs/client.crt --key /certs/client.key --cacert /certs/ca.crt -h 10.180.0.155

Then try the get hello query from a different master or a slave server.

If you still face the issue, Please let us know.

[denist-huma]

Exactly! My fault: missed "-c". Same as here https://serverfault.com/questions/812156/redis-cluster-error-moved.
My cluster is operational :tada:
Thank you @Emon46 for your support. I was breaking my head for the wrong reason :sweat: