Charts: Redis sentinel is not picking right master after master pod restart

Created on 17 Sep 2020  路  25Comments  路  Source: bitnami/charts

NAME: redis-master
REVISION: 1
RELEASED: Thu Sep 17 07:37:59 2020
CHART: redis-11.0.0

Expected behavior
It should pick the right master after the pod restart.

Helm version:

Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

kubectl version

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Redis Sentinel is not picking the right master IP address after the master pod restart in fact it's picking its own old IP address.

  1. I have deployed a fresh redis-ha
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-master-node-0 2/2 Running 0 2m30s 10.233.121.246 localhost
redis-master-node-1 2/2 Running 0 116s 10.233.121.249 localhost
redis-master-node-2 2/2 Running 0 104s 10.233.121.250 localhost

  1. deleted the master pod 10.233.121.246

  2. Now at this point, other slave [10.233.121.250] became a new master as expected

  3. but the problem here is the deleted master pod came up with new ip [10.233.121.251] and when i see the logs inside that redis container, its trying for the connect the old master.

20:S 17 Sep 2020 07:10:36.234 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:10:36.234 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:11:37.651 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:11:37.651 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:11:37.651 * MASTER <-> REPLICA sync started

=======================================================

Hi,

Please find the simple steps below which I have followed for your reference.

For more logs

sentinel logs of the new pod [10.233.121.251]
admin@localhost:/mnt/onl/sdpon/templates/sdponcharts/charts/redis-ha$ docker logs 605cf63fab95
11:X 17 Sep 2020 07:03:29.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
11:X 17 Sep 2020 07:03:29.380 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=11, just started
11:X 17 Sep 2020 07:03:29.380 # Configuration loaded
11:X 17 Sep 2020 07:03:29.382 * Running mode=sentinel, port=26379.
11:X 17 Sep 2020 07:03:29.382 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
11:X 17 Sep 2020 07:03:29.386 # Sentinel ID is 7df5adeca0eec70d960e5f930e2cb45c3d021529
11:X 17 Sep 2020 07:03:29.386 # +monitor master mymaster 10.233.121.246 6379 quorum 2
11:X 17 Sep 2020 07:04:29.407 # +sdown master mymaster 10.233.121.246 6379

redis logs of the new pod [10.233.121.251]

20:C 17 Sep 2020 07:03:29.073 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
20:C 17 Sep 2020 07:03:29.073 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=20, just started
20:C 17 Sep 2020 07:03:29.073 # Configuration loaded
20:S 17 Sep 2020 07:03:29.075 * Running mode=standalone, port=6379.
20:S 17 Sep 2020 07:03:29.076 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
20:S 17 Sep 2020 07:03:29.076 # Server initialized
20:S 17 Sep 2020 07:03:29.076 * Reading RDB preamble from AOF file...
20:S 17 Sep 2020 07:03:29.077 * Loading RDB produced by version 6.0.8
20:S 17 Sep 2020 07:03:29.077 * RDB age 209 seconds
20:S 17 Sep 2020 07:03:29.077 * RDB memory usage when created 94.77 Mb
20:S 17 Sep 2020 07:03:29.077 * RDB has an AOF tail
20:S 17 Sep 2020 07:03:29.413 * Reading the remaining AOF tail...
20:S 17 Sep 2020 07:03:29.413 * DB loaded from append only file: 0.336 seconds
20:S 17 Sep 2020 07:03:29.413 * Ready to accept connections
20:S 17 Sep 2020 07:03:29.413 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:03:29.413 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:04:30.803 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:04:30.803 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:04:30.803 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:05:31.172 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:05:31.172 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:05:31.172 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:06:32.561 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:06:32.561 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:06:32.561 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:07:33.969 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:07:33.969 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:07:33.969 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:08:34.385 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:08:34.385 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:08:34.385 * MASTER <-> REPLICA sync started
20:S 17 Sep 2020 07:09:35.797 # Timeout connecting to the MASTER...
20:S 17 Sep 2020 07:09:35.797 * Connecting to MASTER 10.233.121.246:6379
20:S 17 Sep 2020 07:09:35.797 * MASTER <-> REPLICA sync started

=================================================

Configuration made in values-production.yaml these are only Highlighted changes made:

command: helm upgrade --install redis-master -f values-production.yaml . --debug

Sentinel:
enabled: true
usePassword: false

networkPolicy:
enabled: false

usePassword: false

securityContext:
runAsUser: 0

command: "redis-server"

disableCommands

- FLUSHDB
- FLUSHALL
command: "redis-server" #for slave

metrics:
enabled: false

configmap: |-
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec

===========================================================

Complete configuration:

[debug] Created tunnel using local port: '43627'

[debug] SERVER: "127.0.0.1:43627"

Release "redis-master" does not exist. Installing it now.
[debug] CHART PATH: /home/admin/redis-work/redis-ha

NAME: redis-master
REVISION: 1
RELEASED: Thu Sep 17 07:37:59 2020
CHART: redis-11.0.0
USER-SUPPLIED VALUES:
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-

Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000

Disable RDB persistence, AOF persistence already enabled.
save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis
tag: 6.0.8-debian-10-r0
master:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands: null
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
priorityClassName: {}
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
loadBalancerIP: null
port: 6379
type: ClusterIP
statefulset:
updateStrategy: RollingUpdate
metrics:
enabled: false
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-exporter
tag: 1.11.1-debian-10-r12
podAnnotations:
prometheus.io/port: "9121"
prometheus.io/scrape: "true"
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ""
rules: []
service:
annotations: {}
labels: {}
type: ClusterIP
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
networkPolicy:
enabled: false
ingressNSMatchLabels: {}
ingressNSPodMatchLabels: {}
password: null
persistence:
existingClaim: null
podDisruptionBudget:
enabled: false
minAvailable: 1
podSecurityPolicy:
create: false
rbac:
create: false
role:
rules: []
redisPort: 6379
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 0
sentinel:
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
downAfterMilliseconds: 60000
enabled: true
failoverTimeout: 18000
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-sentinel
tag: 6.0.8-debian-10-r1
initialCheckTimeout: 5
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
masterSet: mymaster
parallelSyncs: 1
port: 26379
quorum: 2
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
loadBalancerIP: null
redisPort: 6379
sentinelPort: 26379
type: ClusterIP
staticID: false
usePassword: false
serviceAccount:
create: false
name: null
slave:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands:

FLUSHDB
FLUSHALL
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
port: 6379
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
service:
annotations: {}
labels: {}
loadBalancerIP: null
port: 6379
type: ClusterIP
spreadConstraints: {}
statefulset:
updateStrategy: RollingUpdate
sysctlImage:
command: []
enabled: false
mountHostSys: false
pullPolicy: Always
registry: docker.io
repository: bitnami/minideb
resources: {}
tag: buster
tls:
authClients: true
certCAFilename: null
certFilename: null
certKeyFilename: null
certificatesSecret: null
enabled: false
usePassword: false
usePasswordFile: false
volumePermissions:
enabled: false
image:
pullPolicy: Always
registry: docker.io
repository: bitnami/minideb
tag: buster
resources: {}
COMPUTED VALUES:
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-

Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000

Disable RDB persistence, AOF persistence already enabled.
save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis
tag: 6.0.8-debian-10-r0
master:
affinity: {}
command: redis-server
customLivenessProbe: {}
customReadinessProbe: {}
extraEnvVars: []
extraEnvVarsCM: []
extraEnvVarsSecret: []
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
- ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
priorityClassName: {}
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
port: 6379
type: ClusterIP
shareProcessNamespace: false
statefulset:
updateStrategy: RollingUpdate
metrics:
enabled: false
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-exporter
tag: 1.11.1-debian-10-r12
podAnnotations:
prometheus.io/port: "9121"
prometheus.io/scrape: "true"
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ""
rules: []
service:
annotations: {}
labels: {}
type: ClusterIP
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
networkPolicy:
enabled: false
ingressNSMatchLabels: {}
ingressNSPodMatchLabels: {}
persistence: {}
podDisruptionBudget:
enabled: false
minAvailable: 1
podSecurityPolicy:
create: false
rbac:
create: false
role:
rules: []
redisPort: 6379
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 0
sentinel:
customLivenessProbe: {}
customReadinessProbe: {}
downAfterMilliseconds: 60000
enabled: true
failoverTimeout: 18000
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: bitnami/redis-sentinel
tag: 6.0.8-debian-10-r1
initialCheckTimeout: 5
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
masterSet: mymaster
parallelSyncs: 1
port: 26379
quorum: 2
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
service:
annotations: {}
labels: {}
redisPort: 6379
sentinelPort: 26379
type: ClusterIP
staticID: false
usePassword: false
serviceAccount:
create: false
slave:
affinity: {}
command: redis-server
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands:

FLUSHDB
FLUSHALL
extraEnvVars: []
extraEnvVarsCM: []
extraEnvVarsSecret: []
extraFlags: []
livenessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
persistence:
accessModes:
ReadWriteOnce
enabled: true
matchExpressions: {}
matchLabels: {}
path: /data
size: 8Gi
subPath: ""
podAnnotations: {}
podLabels: {}
port: 6379
readinessProbe:
enabled: true
failureThreshold: 5
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
service:
annotations: {}
labels: {}
port: 6379
type: ClusterIP
shareProcessNamespace: false
spreadConstraints: {}
statefulset:
updateStrategy: RollingUpdate
sysctlImage:
command: []
enabled: false
mountHostSys: false
pullPolicy: Always
registry: docker.io
repository: bitnami/minideb
resources: {}
tag: buster
tls:
authClients: true
enabled: false
usePassword: false
usePasswordFile: false
volumePermissions:
enabled: false
image:
pullPolicy: Always
registry: docker.io
repository: bitnami/minideb
tag: buster
resources: {}
HOOKS:
MANIFEST:

Source: redis/templates/configmap-scripts.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-master-scripts
namespace: default
labels:
app: redis
chart: redis-11.0.0
heritage: Tiller
release: redis-master
data:
start-node.sh: |
#!/bin/bash
is_boolean_yes() {
local -r bool="${1:-}"
# comparison is performed without regard to the case of alphabetic characters
shopt -s nocasematch
if [[ "$bool" = 1 || "$bool" =~ ^(yes|true)$ ]]; then
true
else
false
fi
}

export REDIS_REPLICATION_MODE="slave"
if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
  if [[ ${BASH_REMATCH[2]} == "0" ]]; then
    if [[ ! -f /data/redisboot.lock ]]; then
      export REDIS_REPLICATION_MODE="master"
    else
      if is_boolean_yes "$REDIS_TLS_ENABLED"; then
        sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} info"
      else
        sentinel_info_command="redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 info"
      fi
      if [[ ! ($($sentinel_info_command)) ]]; then
         export REDIS_REPLICATION_MODE="master"
         rm /data/redisboot.lock
      fi
    fi
  fi
fi
useradd redis
chown -R redis /data

if [[ -n $REDIS_PASSWORD_FILE ]]; then
  password_aux=`cat ${REDIS_PASSWORD_FILE}`
  export REDIS_PASSWORD=$password_aux
fi

if [[ -n $REDIS_MASTER_PASSWORD_FILE ]]; then
  password_aux=`cat ${REDIS_MASTER_PASSWORD_FILE}`
  export REDIS_MASTER_PASSWORD=$password_aux
fi

if [[ "$REDIS_REPLICATION_MODE" == "master" ]]; then
  echo "I am master"
  if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
    cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
  fi
else
  if [[ ! -f /opt/bitnami/redis/etc/replica.conf ]];then
    cp /opt/bitnami/redis/mounted-etc/replica.conf /opt/bitnami/redis/etc/replica.conf
  fi

  if is_boolean_yes "$REDIS_TLS_ENABLED"; then
    sentinel_info_command="timeout -s 5 10 redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} sentinel get-master-addr-by-name mymaster"
  else
    sentinel_info_command="timeout -s 5 10 redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster"
  fi
  REDIS_SENTINEL_INFO=($($sentinel_info_command))
  REDIS_MASTER_HOST=${REDIS_SENTINEL_INFO[0]}
  REDIS_MASTER_PORT_NUMBER=${REDIS_SENTINEL_INFO[1]}
fi

if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
  cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")

if [[ "$REDIS_REPLICATION_MODE" == "slave" ]]; then
  ARGS+=("--slaveof" "${REDIS_MASTER_HOST}" "${REDIS_MASTER_PORT_NUMBER}")
fi
ARGS+=("--protected-mode" "no")

if [[ "$REDIS_REPLICATION_MODE" == "master" ]]; then
  ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
else
  ARGS+=("--include" "/opt/bitnami/redis/etc/replica.conf")
fi

ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")

touch /data/redisboot.lock
redis-server "${ARGS[@]}"
start-sentinel.sh: |
#!/bin/bash
replace_in_file() {
local filename="${1:?filename is required}"
local match_regex="${2:?match regex is required}"
local substitute_regex="${3:?substitute regex is required}"
local posix_regex=${4:-true}

    local result

    # We should avoid using 'sed in-place' substitutions
    # 1) They are not compatible with files mounted from ConfigMap(s)
    # 2) We found incompatibility issues with Debian10 and "in-place" substitutions
    del=$'\001' # Use a non-printable character as a 'sed' delimiter to avoid issues
    if [[ $posix_regex = true ]]; then
        result="$(sed -E "s${del}${match_regex}${del}${substitute_regex}${del}g" "$filename")"
    else
        result="$(sed "s${del}${match_regex}${del}${substitute_regex}${del}g" "$filename")"
    fi
    echo "$result" > "$filename"
}
sentinel_conf_set() {
    local -r key="${1:?missing key}"
    local value="${2:-}"

    # Sanitize inputs
    value="${value//\\/\\\\}"
    value="${value//&/\\&}"
    value="${value//\?/\\?}"
    [[ "$value" = "" ]] && value="\"$value\""

    replace_in_file "/opt/bitnami/redis-sentinel/etc/sentinel.conf" "^#*\s*${key} .*" "${key} ${value}" false
}
is_boolean_yes() {
    local -r bool="${1:-}"
    # comparison is performed without regard to the case of alphabetic characters
    shopt -s nocasematch
    if [[ "$bool" = 1 || "$bool" =~ ^(yes|true)$ ]]; then
        true
    else
        false
    fi
}

if [[ -n $REDIS_PASSWORD_FILE ]]; then
  password_aux=`cat ${REDIS_PASSWORD_FILE}`
  export REDIS_PASSWORD=$password_aux
fi

if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]]; then
  cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
fi

export REDIS_REPLICATION_MODE="slave"
if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
  if [[ ${BASH_REMATCH[2]} == "0" ]]; then
    if [[ ! -f /data/sentinelboot.lock ]]; then
      export REDIS_REPLICATION_MODE="master"
    else
      if is_boolean_yes "$REDIS_TLS_ENABLED"; then
        sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} info"
      else
        sentinel_info_command="redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 info"
      fi
      if [[ ! ($($sentinel_info_command)) ]]; then
         export REDIS_REPLICATION_MODE="master"
         rm /data/sentinelboot.lock
      fi
    fi
  fi
fi

if [[ "$REDIS_REPLICATION_MODE" == "master" ]]; then
  sentinel_conf_set "sentinel monitor" "mymaster redis-master-node-0.redis-master-headless.default.svc.cluster.local 6379 2"
else
  if is_boolean_yes "$REDIS_TLS_ENABLED"; then
    sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-master-headless.default.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} sentinel get-master-addr-by-name mymaster"
  else
    sentinel_info_command="redis-cli -h redis-master-headless.default.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster"
  fi
  REDIS_SENTINEL_INFO=($($sentinel_info_command))
  REDIS_MASTER_HOST=${REDIS_SENTINEL_INFO[0]}
  REDIS_MASTER_PORT_NUMBER=${REDIS_SENTINEL_INFO[1]}

  sentinel_conf_set "sentinel monitor" "mymaster "$REDIS_MASTER_HOST" "$REDIS_MASTER_PORT_NUMBER" 2"
fi
touch /data/sentinelboot.lock
redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel
Source: redis/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-master
namespace: default
labels:
app: redis
chart: redis-11.0.0
heritage: Tiller
release: redis-master
data:
redis.conf: |-
# User-supplied configuration:
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
appendfsync everysec
#no-appendfsync-on-rewrite no
#save 900 1
#save 300 10
#save 60 10000
# Disable RDB persistence, AOF persistence already enabled.
save ""
master.conf: |-
dir /data
replica.conf: |-
dir /data
slave-read-only yes
rename-command FLUSHDB ""
rename-command FLUSHALL ""
sentinel.conf: |-
dir "/tmp"
bind 0.0.0.0
port 26379
sentinel monitor mymaster redis-master-node-0.redis-master-headless.default.svc.cluster.local 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 18000
sentinel parallel-syncs mymaster 1
Source: redis/templates/health-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-master-health
namespace: default
labels:
app: redis
chart: redis-11.0.0
heritage: Tiller
release: redis-master
data:
ping_readiness_local.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_local.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_sentinel.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h localhost
-p $REDIS_SENTINEL_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
parse_sentinels.awk: |-
/ip/ {FOUND_IP=1}
/port/ {FOUND_PORT=1}
/runid/ {FOUND_RUNID=1}
!/ip|port|runid/ {
if (FOUND_IP==1) {
IP=$1; FOUND_IP=0;
}
else if (FOUND_PORT==1) {
PORT=$1;
FOUND_PORT=0;
} else if (FOUND_RUNID==1) {
printf "\nsentinel known-sentinel mymaster %s %s %s", IP, PORT, $0; FOUND_RUNID=0;
}
}
ping_readiness_master.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_master.sh: |-
#!/bin/bash
response=$(
timeout -s 3 $1
redis-cli
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_readiness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_readiness_local.sh" $1 || exit_status=$?
"$script_dir/ping_readiness_master.sh" $1 || exit_status=$?
exit $exit_status
ping_liveness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_liveness_local.sh" $1 || exit_status=$?
"$script_dir/ping_liveness_master.sh" $1 || exit_status=$?
exit $exit_status
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-3
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-2
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-redis-master-node-1
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
#redis-data-redis-master-node-0
name: redis-data-redis-master-node-0
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
#path: "/mnt/data"
path: "/data"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-2
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-1
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-3
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"
Source: redis/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-redis-master-node-0
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "2Gi"
Source: redis/templates/headless-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: redis-master-headless
namespace: default
labels:
app: redis
chart: redis-11.0.0
release: redis-master
heritage: Tiller
spec:
type: ClusterIP
clusterIP: None
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis
release: redis-master
Source: redis/templates/redis-with-sentinel-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: redis-master
namespace: default
labels:
app: redis
chart: redis-11.0.0
release: redis-master
heritage: Tiller
spec:
type: ClusterIP
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis
release: redis-master
Source: redis/templates/redis-node-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-master-node
namespace: default
labels:
app: redis
chart: redis-11.0.0
release: redis-master
heritage: Tiller
spec:
replicas: 3
serviceName: redis-master-headless
selector:
matchLabels:
app: redis
release: redis-master
role: node
template:
metadata:
labels:
app: redis
release: redis-master
chart: redis-11.0.0
role: node
annotations:
checksum/health: 08ddae951303a7ae9b56253bb4a68c17c05fee1074a472a53302403f9f7bd160
checksum/configmap: 2d8d1cfc2398f046811b068339a9ac33db8b20767cff63bb2a2d09095eab8750
checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
spec:

  securityContext:
    fsGroup: 1001
  serviceAccountName: default
  containers:
    - name: redis
      image: docker.io/bitnami/redis:6.0.8-debian-10-r0
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - /opt/bitnami/scripts/start-scripts/start-node.sh
      env:
        - name: REDIS_MASTER_PORT_NUMBER
          value: "6379"
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
        - name: REDIS_DATA_DIR
          value: /data
      ports:
        - name: redis
          containerPort: 6379
      livenessProbe:
        initialDelaySeconds: 30
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 10
        timeoutSeconds: 10
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 5
      resources:
        null

      volumeMounts:
        - name: start-scripts
          mountPath: /opt/bitnami/scripts/start-scripts
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
        - name: config
          mountPath: /opt/bitnami/redis/mounted-etc
        - name: redis-tmp-conf
          mountPath: /opt/bitnami/redis/etc
    - name: sentinel
      image: docker.io/bitnami/redis-sentinel:6.0.8-debian-10-r1
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsUser: 0
      command:
        - /bin/bash
        - -c
        - /opt/bitnami/scripts/start-scripts/start-sentinel.sh
      env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_SENTINEL_TLS_ENABLED
          value: "no"
        - name: REDIS_SENTINEL_PORT
          value: "26379"
      ports:
        - name: redis-sentinel
          containerPort: 26379
      livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      resources:
        null

      volumeMounts:
        - name: start-scripts
          mountPath: /opt/bitnami/scripts/start-scripts
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath:
        - name: config
          mountPath: /opt/bitnami/redis-sentinel/mounted-etc
        - name: sentinel-tmp-conf
          mountPath: /opt/bitnami/redis-sentinel/etc
  volumes:
    - name: start-scripts
      configMap:
        name: redis-master-scripts
        defaultMode: 0755
    - name: health
      configMap:
        name: redis-master-health
        defaultMode: 0755
    - name: config
      configMap:
        name: redis-master
    - name: sentinel-tmp-conf
      emptyDir: {}
    - name: redis-tmp-conf
      emptyDir: {}
volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app: redis
release: redis-master
heritage: Tiller
component: slave
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"

    selector:
updateStrategy:
type: RollingUpdate
LAST DEPLOYED: Thu Sep 17 07:37:59 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME AGE
redis-master 0s
redis-master-health 0s
redis-master-scripts 0s

==> v1/PersistentVolume
NAME AGE
redis-data-redis-master-node-0 0s
redis-data-redis-master-node-1 0s
redis-data-redis-master-node-2 0s
redis-data-redis-master-node-3 0s

==> v1/PersistentVolumeClaim
NAME AGE
redis-data-redis-master-node-0 0s
redis-data-redis-master-node-1 0s
redis-data-redis-master-node-2 0s
redis-data-redis-master-node-3 0s

==> v1/Pod(related)
NAME AGE
redis-master-node-0 0s

==> v1/Service
NAME AGE
redis-master 0s
redis-master-headless 0s

==> v1/StatefulSet
NAME AGE
redis-master-node 0s

NOTES:
** Please be patient while the chart is being deployed **
Redis can be accessed via port 6379 on the following DNS name from within your cluster:

redis-master.default.svc.cluster.local for read only operations

For read/write operations, first access the Redis Sentinel cluster, which is available in port 26379 using the same domain name above.

To connect to your Redis server:

Run a Redis pod that you can use as a client:
kubectl run --namespace default redis-master-client --rm --tty -i --restart='Never' \

--image docker.io/bitnami/redis:6.0.8-debian-10-r0 -- bash

Connect using the Redis CLI:
redis-cli -h redis-master -p 6379 # Read only operations
redis-cli -h redis-master -p 26379 # Sentinel access

To connect to your database from outside the cluster execute the following commands:

kubectl port-forward --namespace default svc/redis-master-master 6379:6379 &
redis-cli -h 127.0.0.1 -p 6379
on-hold
Source
picture mahanam

All 25 comments

Comments, please!

mahanam picture mahanam on 21 Sep 2020
👀1 👍1

Hi @mahanam, you're right. I've been able to reproduce this issue and will look into it.

marcosbc picture marcosbc on 21 Sep 2020
👍1

Thanks for the confirmation @marcosbc.

Is there any workaround for this issue to move ahead.

mahanam picture mahanam on 21 Sep 2020

If you have important data in the Redis you can backup/restore following this guide: https://github.com/bitnami/charts/tree/master/bitnami/redis#backup-and-restore

After you redeploy you should be able to get a working cluster (until you delete the master node again).

marcosbc picture marcosbc on 21 Sep 2020

This is just a temporary solution and the same we already tried and the cluster is stable.

Currently, can we get something that we can do a workaround by configuration to move ahead?

mahanam picture mahanam on 21 Sep 2020

Unfortunately we don't know what's causing this issue, so we're unable to provide a workaround for now. We'll update if when we find anything.

You could always explore redis-cluster chart as it is configured in a different way.

marcosbc picture marcosbc on 22 Sep 2020

Hi,

I've been digging into this issue and I saw what the problem is. I've thought of some possibilities to fix the initialization code to allow floating IPs but that would not solve the main issue. The problem here is that redis does not work correctly with floating IPs. I believe that, in order to fix, we should force static IPs in the pods, and that could only be achieved by having one service per pod (just like we do with external access). I will open a task for adding that feature.

javsalgar picture javsalgar on 23 Sep 2020

Hello,
Definitively this issue is related to the init code as I had also checked and mainly this block !

export REDIS_REPLICATION_MODE="slave"
if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
  if [[ ${BASH_REMATCH[2]} == "0" ]]; then
    if [[ ! -f /data/redisboot.lock ]]; then
      export REDIS_REPLICATION_MODE="master"
    else
      if is_boolean_yes "$REDIS_TLS_ENABLED"; then
        sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-test-headless.sre.svc.cluster.local -p 26379 --tls --cert ${REDIS_TLS_CERT_FILE} --key ${REDIS_TLS_KEY_FILE} --cacert ${REDIS_TLS_CA_FILE} info"
      else
        sentinel_info_command="redis-cli -a $REDIS_PASSWORD -h redis-test-headless.sre.svc.cluster.local -p 26379 info"
      fi
      if [[ ! ($($sentinel_info_command)) ]]; then
         export REDIS_REPLICATION_MODE="master"
         rm /data/redisboot.lock
      fi
    fi
  fi
fi

when the pod is restarted, the redisboot file is absent. so sentinel is checked to know is master role have been failled over ! When it's a first start the lock file is present and so the role for node 0 is always set as a master one.

alexises picture alexises on 24 Sep 2020

Can I hear some noise :) as it is on-hold state.

mahanam picture mahanam on 28 Sep 2020

I would also love to hear the status of this - this problem basically makes the char unusable. I have noticed many times that after a redeployment the replicas lose sync with each other and we end up with a split-brain cluster with multiple pods believing they are master.

haljin picture haljin on 29 Sep 2020

Hi,

Thanks for the feedback! The approach is still being discussed, but we will work on it during the following weeks. Either fixing the boot script or creating one service per pod, we will have a solution. If you want to speed up the process, feel free to create a PR and we will review and discuss it.

javsalgar picture javsalgar on 29 Sep 2020

Any updates on this?

gnomus picture gnomus on 19 Oct 2020

Faced the same issue today. It worked well locally on k3s-dind but failed after deployment to k8s. The only "workaround" which worked for me is to run kubectl delete for all redis nodes in separate consoles simulatenously.

0x0badc0de picture 0x0badc0de on 19 Oct 2020

Hi,

@rafariossaa is working on it, he will provide more updates when a PR is ready.

javsalgar picture javsalgar on 20 Oct 2020
👍1

Hi @javsalgar and @rafariossaa, any update on this issue? I'm also having this issue and this makes the chart unusable for us.

I have mentioned this issue in this PR as it already solves the problem that once redis-node-0 gets up again with a new IP address, after it was deleted, it sees its new IP address.

DoGab picture DoGab on 9 Nov 2020

Hi,

We are a small team and we had some tasks prioritized, this is currently in our backlog. We will work on this as soon as possible.
I have seen you have a pending change in that PR. Could you take a look to it ?
Will this PR help to this issue ?

rafariossaa picture rafariossaa on 11 Nov 2020

Hi @rafariossaa
Alright, this is fine. I'm sorry but am I missing something? I don't see any new question or task that is for me in this PR? I have tested the latest commit on this PR and do not see any changes in the code since than, that i could test again.

As mentioned in the PR itself, I guess this PR touches the same code that affects this issue. It currently solves the problem that redis-node-0 sees its new IP address if the pod gets recreated. But it still thinks it is the master node and does not register to the sentinels as a slave.

DoGab picture DoGab on 11 Nov 2020

Hi @DoGab

I think related changes were addressed at https://github.com/bitnami/charts/pull/4201
Could you please give a try to the latest chart version?

juan131 picture juan131 on 19 Nov 2020

Hi @DoGab

I think related changes were addressed at #4201
Could you please give a try to the latest chart version?

Hi @juan131 Sure, i'll test it today or tomorrow and will leave you a feedback here.

DoGab picture DoGab on 19 Nov 2020

I'm also getting this issue on v12.0.1

all 3 nodes in my sentinel cluster are stuck trying to connect to an old master IP address.

Manually removing pods doesn't seem to fix the issue. But scaling down to 0 replicas, then back up to 3 does fix it.

config:

    cluster:
      enabled: true
      slaveCount: 3

    sentinel:
      enabled: true
      usePassword: true
      password: password
      image:
        tag: 5.0.10-debian-10-r20
        pullPolicy: IfNotPresent
      masterSet: mymaster
      initialCheckTimeout: 5
      quorum: 2
      downAfterMilliseconds: 30000
      failoverTimeout: 18000
      parallelSyncs: 1
      port: 26379
      staticID: true

    configmap: |-
      # Disable AOF persistence, RDB persistence already enabled.
      appendonly no

      tcp-keepalive 60
      loglevel notice
      logfile /dev/stdout

      save 900 1
      save 300 10
      save 60 10000
      rdbcompression yes
      rdbchecksum yes
      dbfilename dump.rdb
      dir ./

      slave-serve-stale-data yes
      slave-read-only yes
      repl-disable-tcp-nodelay no
      slave-priority 100
      maxclients 10000
      lua-time-limit 5000
      client-output-buffer-limit normal 0 0 0
      client-output-buffer-limit slave 256mb 64mb 60
      client-output-buffer-limit pubsub 512mb 512mb 60
      hz 10

Here are the logs of a sentinel container if it helps. 172.27.213.82 is the old master pod IP

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1:X 19 Nov 2020 18:18:59.766 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 19 Nov 2020 18:18:59.766 # Redis version=5.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 19 Nov 2020 18:18:59.766 # Configuration loaded
1:X 19 Nov 2020 18:18:59.767 * Running mode=sentinel, port=26379.
1:X 19 Nov 2020 18:18:59.767 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 19 Nov 2020 18:18:59.767 # Sentinel ID is 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4
1:X 19 Nov 2020 18:18:59.767 # +monitor master mymaster 172.27.213.82 6379 quorum 2
1:X 19 Nov 2020 18:19:00.854 # +new-epoch 19
1:X 19 Nov 2020 18:19:18.600 * +sentinel sentinel b08a81403f5392f09946c5348cad8a90b00b9779 172.27.160.1 26379 @ mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:19:29.793 # +sdown master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:19:29.865 # +odown master mymaster 172.27.213.82 6379 #quorum 2/2
1:X 19 Nov 2020 18:19:29.865 # +new-epoch 20
1:X 19 Nov 2020 18:19:29.865 # +try-failover master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:19:29.866 # +vote-for-leader 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 20
1:X 19 Nov 2020 18:19:29.869 # 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 20
1:X 19 Nov 2020 18:19:29.870 # b08a81403f5392f09946c5348cad8a90b00b9779 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 20
1:X 19 Nov 2020 18:19:29.949 # +elected-leader master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:19:29.949 # +failover-state-select-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:19:30.020 # -failover-abort-no-good-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:19:30.086 # Next failover delay: I will not start a failover before Thu Nov 19 18:20:05 2020
1:X 19 Nov 2020 18:20:05.914 # +new-epoch 21
1:X 19 Nov 2020 18:20:05.914 # +try-failover master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:20:05.915 # +vote-for-leader 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 21
1:X 19 Nov 2020 18:20:05.918 # b08a81403f5392f09946c5348cad8a90b00b9779 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 21
1:X 19 Nov 2020 18:20:05.920 # 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 21
1:X 19 Nov 2020 18:20:05.986 # +elected-leader master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:20:05.986 # +failover-state-select-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:20:06.057 # -failover-abort-no-good-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:20:06.120 # Next failover delay: I will not start a failover before Thu Nov 19 18:20:42 2020
1:X 19 Nov 2020 18:20:42.669 # +new-epoch 22
1:X 19 Nov 2020 18:20:42.670 # +vote-for-leader 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 22
1:X 19 Nov 2020 18:20:42.676 # Next failover delay: I will not start a failover before Thu Nov 19 18:21:18 2020
1:X 19 Nov 2020 18:21:18.960 # +new-epoch 23
1:X 19 Nov 2020 18:21:18.960 # +try-failover master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:21:18.961 # +vote-for-leader 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 23
1:X 19 Nov 2020 18:21:18.964 # b08a81403f5392f09946c5348cad8a90b00b9779 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 23
1:X 19 Nov 2020 18:21:18.965 # 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 23
1:X 19 Nov 2020 18:21:19.019 # +elected-leader master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:21:19.019 # +failover-state-select-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:21:19.085 # -failover-abort-no-good-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:21:19.140 # Next failover delay: I will not start a failover before Thu Nov 19 18:21:55 2020
1:X 19 Nov 2020 18:21:55.136 # +new-epoch 24
1:X 19 Nov 2020 18:21:55.137 # +vote-for-leader b08a81403f5392f09946c5348cad8a90b00b9779 24
1:X 19 Nov 2020 18:21:55.180 # Next failover delay: I will not start a failover before Thu Nov 19 18:22:31 2020
1:X 19 Nov 2020 18:22:27.839 # +sdown sentinel b08a81403f5392f09946c5348cad8a90b00b9779 172.27.160.1 26379 @ mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:22:31.845 # +new-epoch 25
1:X 19 Nov 2020 18:22:31.845 # +vote-for-leader 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 25
1:X 19 Nov 2020 18:22:31.847 # Next failover delay: I will not start a failover before Thu Nov 19 18:23:07 2020
1:X 19 Nov 2020 18:22:48.365 * +sentinel-address-switch master mymaster 172.27.213.82 6379 ip 172.27.165.19 port 26379 for b08a81403f5392f09946c5348cad8a90b00b9779
1:X 19 Nov 2020 18:23:07.933 # +new-epoch 26
1:X 19 Nov 2020 18:23:07.933 # +try-failover master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:23:07.934 # +vote-for-leader 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 26
1:X 19 Nov 2020 18:23:07.937 # b08a81403f5392f09946c5348cad8a90b00b9779 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 26
1:X 19 Nov 2020 18:23:07.943 # 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 26
1:X 19 Nov 2020 18:23:08.034 # +elected-leader master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:23:08.034 # +failover-state-select-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:23:08.100 # -failover-abort-no-good-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:23:08.152 # Next failover delay: I will not start a failover before Thu Nov 19 18:23:44 2020
1:X 19 Nov 2020 18:23:44.639 # +new-epoch 27
1:X 19 Nov 2020 18:23:44.640 # +vote-for-leader b08a81403f5392f09946c5348cad8a90b00b9779 27
1:X 19 Nov 2020 18:23:44.695 # Next failover delay: I will not start a failover before Thu Nov 19 18:24:21 2020
1:X 19 Nov 2020 18:23:52.470 # +sdown sentinel 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 172.27.116.21 26379 @ mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:23:53.383 # +sdown slave 172.27.116.21:6379 172.27.116.21 6379 @ mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:24:05.989 * +sentinel-address-switch master mymaster 172.27.213.82 6379 ip 172.27.106.41 port 26379 for 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4
1:X 19 Nov 2020 18:24:21.206 # +new-epoch 28
1:X 19 Nov 2020 18:24:21.206 # +try-failover master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:24:21.208 # +vote-for-leader 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 28
1:X 19 Nov 2020 18:24:21.210 # b08a81403f5392f09946c5348cad8a90b00b9779 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 28
1:X 19 Nov 2020 18:24:21.212 # 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 voted for 7cf5d8db5f3d91699f5df2064d059fb5ef63a7e4 28
1:X 19 Nov 2020 18:24:21.274 # +elected-leader master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:24:21.274 # +failover-state-select-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:24:21.357 # -failover-abort-no-good-slave master mymaster 172.27.213.82 6379
1:X 19 Nov 2020 18:24:21.412 # Next failover delay: I will not start a failover before Thu Nov 19 18:24:57 2020
1:X 19 Nov 2020 18:24:57.334 # +new-epoch 29
1:X 19 Nov 2020 18:24:57.334 # +vote-for-leader 8b844a5f3afd0ecfd2392119fdab6aeb71a16ea4 29
1:X 19 Nov 2020 18:24:57.382 # Next failover delay: I will not start a failover before Thu Nov 19 18:25:33 2020
1:signal-handler (1605810302) Received SIGTERM scheduling shutdown...
1:X 19 Nov 2020 18:25:02.833 # User requested shutdown...
1:X 19 Nov 2020 18:25:02.833 # Sentinel is now ready to exit, bye bye...
while-loop picture while-loop on 19 Nov 2020

Didn't #4239 fix that problem? Was it abandoned too quickly? :)

haljin picture haljin on 20 Nov 2020

Hi @juan131 I've tested the new chart (version 12.1.1) and it seems to work. Failover works correctly and a new master is elected after the old one was deleted.

@while-loop I see that you use an old version of the redis-sentinel image (5.0.10-debian-10-r20), however it worked for me with version 6.0.8-debian-10-r55. Also it may be related to your additional configmap.

DoGab picture DoGab on 20 Nov 2020

Hi @DoGab,

Thank you for the feedback!

Let's wait until we have some more positive responses on this. Could you confirm this issue is now solved @mahanam @haljin?

FraPazGal picture FraPazGal on 20 Nov 2020

I've also tried the 12.0.1 and deleting pods as well as redeploying does result in a consistent cluster every time I did it, while with the older version it would very quickly lead to a multi-master cluster. So looks good to me :)

haljin picture haljin on 20 Nov 2020

For me this has not been resolved.

I was running 12.0.0, then I upgraded to 12.1.1. First by upgrading, testing, then deleting the whole namespace and doing a fresh install (in staging) and testing again.

The problem I'm (still) seeing, is that when the sentinel setup is up and running and I go and terminate the master (node-0) (could happen due to GKE node update etc), the other nodes (node-1 and node-2) failover properly and one of the becomes master and all is well for the two nodes (node-1 and node-2).

But the original master node (node-0) still considers it's old pod IP to be the master it should connect to. All logs just spam the old IP and Connection timed out and it cannot recover.

If I go to node-0 and say redis-cli slaveof *ip-of-current-master* port then it joins the sentinel setup and all is well again. Also deleting the node-0 pod again seems to help it recover at least sometimes.

I'm running images

EDIT:

I'm seeing the old deleted Pod IP in node-0 sentinel.conf that is put in an emptyDir. It also lists the now-master as it's slave.

What I have observed with kube is that even after a pod restart, the emptyDir content might not get cleared. It will, if the pod changes kube node where it runs, but if it starts on the same node, it might retain the emptyDir data.

Is this chart based on the fact that the emptyDir should be empty on every pod restart?

EDIT 2:

So this is interesting! I checked the node the master was running on, I cordoned the node (so nothing would start there) effectively forcing the master to another node. Lo and behold, new master was elected, and the old master node joins the cluster just fine.

I'm thinking the problem is the old sentinel.conf from the olden times hanging around after pod restart (when pod restarts on the same node).

ambis picture ambis on 20 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

wantdrink picture wantdrink  路  6Comments
PhoneixS picture PhoneixS  路  6Comments
alesanmed picture alesanmed  路  6Comments
druesendieb picture druesendieb  路  5Comments
Djiit picture Djiit  路  3Comments