Charts: Wrong hostname or port for sentinel when stopped the master sentinel container

Created on 6 Aug 2020  路  7Comments  路  Source: bitnami/charts

The name (and version) of the affected chart
NAME: my-release
REVISION: 1
RELEASED: Thu Aug 6 08:41:17 2020
CHART: redis-10.7.13
USER-SUPPLIED VALUES:

Setup: It's a virtual machine on which am trying to install this redis ha

Updated the configuration file: values-production.yaml as shown below

FYI :Disabled the securityContext

securityContext:
enabled: false

Enabled the sentinel

To Reproduce
Steps to reproduce the behavior

  1. Install the bitnami/redis
    helm install --name my-release bitnami/redis --values values-production.yaml

  2. Now wait for your setup to come up

  3. Delete the master-sentinel container

  4. Slave sentinels are not proceeds for an election to form a new master-sentinel

Expected Behavior:
Slaves should elect a new master-sentinel.

Version of Helm and Kubernetes:
helm- v2.16.1
Kubernetes - v1.16.3

  • Output of kubectl version:
    $ kubectl version
    Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Logs:
docker tries to come up but exiting with this error
* FATAL CONFIG FILE ERROR (Redis 6.0.6) *
Reading the configuration file, at line 24

'sentinel known-sentinel mymaster 10.233.121.197 26379 87febf77983e742f80acd4f2d0c136f20828af14'
Wrong hostname or port for sentinel.
sentinel known-sentinel mymaster 10.233.121.196 26379 9dd049fc09b083ee395fd3c95d5db1bdc67624cbroot@localhost:/ho

Kindly help me what the wrong configuration may could have missed from this side

Thanks in Advance

picture mahanam
👍1

Most helpful comment

Hi,

Have resolved this issue by increasing this timeout .Values.sentinel.initialCheckTimeout from 5 to 10.
So that giving enough window to open the port 26379.

Thanks,
Mahadev

picture mahanam on 11 Aug 2020
🚀1 ❤1 🎉1 👍1

All 7 comments

Hi @mahanam ,
Could you share the values.yml you are using? I trierd to reproduce it but the default values-production.yaml has not the sentinel enabled. Did you perform any other change apart from enable it?

miguelaeh picture miguelaeh on 7 Aug 2020

Thanks for your reply,

As I have already mentioned the only change is the security context is disabled .
/run.sh -> replaced to "redis-server"
sentinel: enabled

It can be reproduced on the above-mentioned version with these changes.

Thanks

mahanam picture mahanam on 7 Aug 2020

Hi,

Please find the more config.

* Redis config *
my-release-redis:6379> info

Server

redis_version:6.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:cc0afcf3cb8ab086
redis_mode:standalone
os:Linux 4.14.151-OpenNetworkLinux x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.3.0
process_id:1
run_id:a0f53a2c7fb96924ccafb43594b42c47c176376e
tcp_port:6379
uptime_in_seconds:300196
uptime_in_days:3
hz:10
configured_hz:10
lru_clock:3204688
executable:/data/redis-server
config_file:

Clients

connected_clients:9
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

Memory

used_memory:2193640
used_memory_human:2.09M
used_memory_rss:8712192
used_memory_rss_human:8.31M
used_memory_peak:2396048
used_memory_peak_human:2.29M
used_memory_peak_perc:91.55%
used_memory_overhead:2055624
used_memory_startup:803216
used_memory_dataset:138016
used_memory_dataset_perc:9.93%
allocator_allocated:2253480
allocator_active:2629632
allocator_resident:5582848
total_system_memory:16791797760
total_system_memory_human:15.64G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.17
allocator_frag_bytes:376152
allocator_rss_ratio:2.12
allocator_rss_bytes:2953216
rss_overhead_ratio:1.56
rss_overhead_bytes:3129344
mem_fragmentation_ratio:4.05
mem_fragmentation_bytes:6561064
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:50958
mem_clients_normal:152874
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1596740051
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:266240
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:92
aof_base_size:92
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

Stats

total_connections_received:120102
total_commands_processed:2997307
instantaneous_ops_per_sec:9
total_net_input_bytes:141884543
total_net_output_bytes:1057741629
instantaneous_input_kbps:0.50
instantaneous_output_kbps:4.78
rejected_connections:0
sync_full:3
sync_partial_ok:3
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:23862
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:592
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

Replication

role:master
connected_slaves:3
slave0:ip=10.233.121.11,port=6379,state=online,offset=82533770,lag=1
slave1:ip=10.233.121.5,port=6379,state=online,offset=82533770,lag=0
slave2:ip=10.233.121.18,port=6379,state=online,offset=82533770,lag=1
master_replid:ba538b62e89f0d8b28b582b71f5daa7b2cbb18d6
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:82533909
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:81485334
repl_backlog_histlen:1048576

CPU

used_cpu_sys:889.081994
used_cpu_user:811.019647
used_cpu_sys_children:0.005246
used_cpu_user_children:0.008702

Modules

Cluster

cluster_enabled:0

Keyspace

my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
my-release-redis:6379>
root@my-release-redis-client:/data#
root@my-release-redis-client:/data#
root@my-release-redis-client:/data# redis-cli -h my-release-redis -p 26379 -a $REDIS_PASSWORD
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
my-release-redis:26379>
my-release-redis:26379>
my-release-redis:26379> info

Server

redis_version:6.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:feeb8ee6bce31d0d
redis_mode:sentinel
os:Linux 4.14.151-OpenNetworkLinux x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:8.3.0
process_id:1
run_id:807e74a7d0b9514c4c9e919d0f91c22ad52ec179
tcp_port:26379
uptime_in_seconds:300189
uptime_in_days:3
hz:15
configured_hz:10
lru_clock:3204701
executable:/redis-server
config_file:/opt/bitnami/redis-sentinel/etc/sentinel.conf

Clients

connected_clients:4
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

CPU

used_cpu_sys:1021.627306
used_cpu_user:1619.720592
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

Stats

total_connections_received:120084
total_commands_processed:1548147
instantaneous_ops_per_sec:3
total_net_input_bytes:79054585
total_net_output_bytes:9273768
instantaneous_input_kbps:0.13
instantaneous_output_kbps:0.02
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:31700
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

Sentinel

sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.233.121.9:6379,slaves=3,sentinels=4
my-release-redis:26379>

* Full Config on Kubernetes:*

[debug] Created tunnel using local port: '37811'

[debug] SERVER: "127.0.0.1:37811"

[debug] Original chart version: ""
[debug] Fetched bitnami/redis to /root/.helm/cache/archive/redis-10.7.13.tgz

[debug] CHART PATH: /root/.helm/cache/archive/redis-10.7.13.tgz

NAME: my-release
REVISION: 1
RELEASED: Thu Aug 6 12:48:47 2020
CHART: redis-10.7.13
USER-SUPPLIED VALUES:
`
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
# Disable RDB persistence, AOF persistence already enabled.
save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker-registry.com:5000
repository: bitnami/redis-cluster
tag: 6.0.6
master:
affinity: {}
command: redis-server
configmap: null
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands:

  • FLUSHDB
  • FLUSHALL
    extraFlags: []
    livenessProbe:
    enabled: true
    failureThreshold: 5
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5
    persistence:
    accessModes:

    • ReadWriteOnce

      enabled: true

      matchExpressions: {}

      matchLabels: {}

      path: /data

      size: 8Gi

      subPath: ""

      podAnnotations: {}

      podLabels: {}

      priorityClassName: {}

      readinessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 5

      successThreshold: 1

      timeoutSeconds: 1

      service:

      annotations: {}

      labels: {}

      loadBalancerIP: null

      port: 6379

      type: ClusterIP

      statefulset:

      updateStrategy: RollingUpdate

      metrics:

      enabled: true

      image:

      pullPolicy: IfNotPresent

      registry: docker.io

      repository: bitnami/redis-exporter

      tag: 1.9.0-debian-10-r19

      podAnnotations:

      prometheus.io/port: "9121"

      prometheus.io/scrape: "true"

      prometheusRule:

      additionalLabels: {}

      enabled: false

      namespace: ""

      rules: []

      service:

      annotations: {}

      labels: {}

      type: ClusterIP

      serviceMonitor:

      enabled: false

      selector:

      prometheus: kube-prometheus

      networkPolicy:

      enabled: true

      ingressNSMatchLabels: {}

      ingressNSPodMatchLabels: {}

      password: null

      persistence:

      existingClaim: null

      podDisruptionBudget:

      enabled: false

      minAvailable: 1

      podSecurityPolicy:

      create: false

      rbac:

      create: false

      role:

      rules: []

      redisPort: 6379

      securityContext:

      enabled: false

      fsGroup: 1001

      runAsUser: 1001

      sentinel:

      configmap: null

      customLivenessProbe: {}

      customReadinessProbe: {}

      downAfterMilliseconds: 60000

      enabled: true

      failoverTimeout: 18000

      image:

      pullPolicy: IfNotPresent

      registry: docker.io

      repository: bitnami/redis-sentinel

      tag: 6.0.6-debian-10-r10

      initialCheckTimeout: 5

      livenessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 5

      successThreshold: 1

      timeoutSeconds: 5

      masterSet: mymaster

      parallelSyncs: 1

      port: 26379

      quorum: 2

      readinessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 5

      successThreshold: 1

      timeoutSeconds: 1

      service:

      annotations: {}

      labels: {}

      loadBalancerIP: null

      redisPort: 6379

      sentinelPort: 26379

      type: ClusterIP

      staticID: false

      usePassword: true

      serviceAccount:

      create: false

      name: null

      slave:

      affinity: {}

      command: redis-server

      configmap: null

      customLivenessProbe: {}

      customReadinessProbe: {}

      disableCommands:

  • FLUSHDB
  • FLUSHALL
    extraFlags: []
    livenessProbe:
    enabled: true
    failureThreshold: 5
    initialDelaySeconds: 30
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
    persistence:
    accessModes:

    • ReadWriteOnce

      enabled: true

      matchExpressions: {}

      matchLabels: {}

      path: /data

      size: 8Gi

      subPath: ""

      podAnnotations: {}

      podLabels: {}

      port: 6379

      readinessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 10

      successThreshold: 1

      timeoutSeconds: 10

      service:

      annotations: {}

      labels: {}

      loadBalancerIP: null

      port: 6379

      type: ClusterIP

      spreadConstraints: {}

      statefulset:

      updateStrategy: RollingUpdate

      sysctlImage:

      command: []

      enabled: false

      mountHostSys: false

      pullPolicy: Always

      registry: docker.io

      repository: bitnami/minideb

      resources: {}

      tag: buster

      tls:

      authClients: true

      certCAFilename: null

      certFilename: null

      certKeyFilename: null

      certificatesSecret: null

      enabled: false

      usePassword: true

      usePasswordFile: false

      volumePermissions:

      enabled: false

      image:

      pullPolicy: Always

      registry: docker.io

      repository: bitnami/minideb

      tag: buster

      resources: {}

COMPUTED VALUES:
cluster:
enabled: true
slaveCount: 3
clusterDomain: cluster.local
configmap: |-
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
# Disable RDB persistence, AOF persistence already enabled.
save ""
global:
redis: {}
image:
pullPolicy: IfNotPresent
registry: docker-registry.com:5000
repository: bitnami/redis-cluster
tag: 6.0.6
master:
affinity: {}
command: redis-server
customLivenessProbe: {}
customReadinessProbe: {}
disableCommands:

  • FLUSHDB
  • FLUSHALL
    extraFlags: []
    livenessProbe:
    enabled: true
    failureThreshold: 5
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5
    persistence:
    accessModes:

    • ReadWriteOnce

      enabled: true

      matchExpressions: {}

      matchLabels: {}

      path: /data

      size: 8Gi

      subPath: ""

      podAnnotations: {}

      podLabels: {}

      priorityClassName: {}

      readinessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 5

      successThreshold: 1

      timeoutSeconds: 1

      service:

      annotations: {}

      labels: {}

      port: 6379

      type: ClusterIP

      statefulset:

      updateStrategy: RollingUpdate

      metrics:

      enabled: true

      image:

      pullPolicy: IfNotPresent

      registry: docker.io

      repository: bitnami/redis-exporter

      tag: 1.9.0-debian-10-r19

      podAnnotations:

      prometheus.io/port: "9121"

      prometheus.io/scrape: "true"

      prometheusRule:

      additionalLabels: {}

      enabled: false

      namespace: ""

      rules: []

      service:

      annotations: {}

      labels: {}

      type: ClusterIP

      serviceMonitor:

      enabled: false

      selector:

      prometheus: kube-prometheus

      networkPolicy:

      enabled: true

      ingressNSMatchLabels: {}

      ingressNSPodMatchLabels: {}

      persistence: {}

      podDisruptionBudget:

      enabled: false

      minAvailable: 1

      podSecurityPolicy:

      create: false

      rbac:

      create: false

      role:

      rules: []

      redisPort: 6379

      securityContext:

      enabled: false

      fsGroup: 1001

      runAsUser: 1001

      sentinel:

      customLivenessProbe: {}

      customReadinessProbe: {}

      downAfterMilliseconds: 60000

      enabled: true

      failoverTimeout: 18000

      image:

      pullPolicy: IfNotPresent

      registry: docker.io

      repository: bitnami/redis-sentinel

      tag: 6.0.6-debian-10-r10

      initialCheckTimeout: 5

      livenessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 5

      successThreshold: 1

      timeoutSeconds: 5

      masterSet: mymaster

      parallelSyncs: 1

      port: 26379

      quorum: 2

      readinessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 5

      successThreshold: 1

      timeoutSeconds: 1

      service:

      annotations: {}

      labels: {}

      redisPort: 6379

      sentinelPort: 26379

      type: ClusterIP

      staticID: false

      usePassword: true

      serviceAccount:

      create: false

      slave:

      affinity: {}

      command: redis-server

      customLivenessProbe: {}

      customReadinessProbe: {}

      disableCommands:

  • FLUSHDB
  • FLUSHALL
    extraFlags: []
    livenessProbe:
    enabled: true
    failureThreshold: 5
    initialDelaySeconds: 30
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
    persistence:
    accessModes:

    • ReadWriteOnce

      enabled: true

      matchExpressions: {}

      matchLabels: {}

      path: /data

      size: 8Gi

      subPath: ""

      podAnnotations: {}

      podLabels: {}

      port: 6379

      readinessProbe:

      enabled: true

      failureThreshold: 5

      initialDelaySeconds: 5

      periodSeconds: 10

      successThreshold: 1

      timeoutSeconds: 10

      service:

      annotations: {}

      labels: {}

      port: 6379

      type: ClusterIP

      spreadConstraints: {}

      statefulset:

      updateStrategy: RollingUpdate

      sysctlImage:

      command: []

      enabled: false

      mountHostSys: false

      pullPolicy: Always

      registry: docker.io

      repository: bitnami/minideb

      resources: {}

      tag: buster

      tls:

      authClients: true

      enabled: false

      usePassword: true

      usePasswordFile: false

      volumePermissions:

      enabled: false

      image:

      pullPolicy: Always

      registry: docker.io

      repository: bitnami/minideb

      tag: buster

      resources: {}

HOOKS:
MANIFEST:

Source: redis/templates/networkpolicy.yaml

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: my-release-redis
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: my-release
heritage: Tiller
spec:
podSelector:
matchLabels:
app: redis
release: my-release
policyTypes:
- Ingress
- Egress
egress:
# Allow dns resolution
- ports:
- port: 53
protocol: UDP
# Allow outbound connections to other cluster pods
- ports:
- port: 6379
- port: 26379
to:
- podSelector:
matchLabels:
app: redis
release: my-release
ingress:
# Allow inbound connections
- ports:
- port: 6379
- port: 26379
from:
- podSelector:
matchLabels:
my-release-redis-client: "true"
- podSelector:
matchLabels:
app: redis
release: my-release
# Allow prometheus scrapes for metrics
- ports:

- port: 9121

Source: redis/templates/secret.yaml

apiVersion: v1
kind: Secret
metadata:
name: my-release-redis
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: "my-release"
heritage: "Tiller"
type: Opaque
data:

redis-password: "SWt3S1Iwd2psRw=="

Source: redis/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: my-release-redis
namespace: default
labels:
app: redis
chart: redis-10.7.13
heritage: Tiller
release: my-release
data:
redis.conf: |-
# User-supplied configuration:
# Enable AOF https://redis.io/topics/persistence#append-only-file
appendonly yes
# Disable RDB persistence, AOF persistence already enabled.
save ""
master.conf: |-
dir /data
rename-command FLUSHDB ""
rename-command FLUSHALL ""
replica.conf: |-
dir /data
slave-read-only yes
rename-command FLUSHDB ""
rename-command FLUSHALL ""
sentinel.conf: |-
dir "/tmp"
bind 0.0.0.0
port 26379
sentinel monitor mymaster my-release-redis-master-0.my-release-redis-headless.default.svc.cluster.local 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 18000

sentinel parallel-syncs mymaster 1

Source: redis/templates/health-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: my-release-redis-health
namespace: default
labels:
app: redis
chart: redis-10.7.13
heritage: Tiller
release: my-release
data:
ping_readiness_local.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_PASSWORD $no_auth_warning
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_local.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_PASSWORD $no_auth_warning
-h localhost
-p $REDIS_PORT
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_sentinel.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_PASSWORD $no_auth_warning
-h localhost
-p $REDIS_SENTINEL_PORT
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
parse_sentinels.awk: |-
/ip/ {FOUND_IP=1}
/port/ {FOUND_PORT=1}
/runid/ {FOUND_RUNID=1}
!/ip|port|runid/ {
if (FOUND_IP==1) {
IP=$1; FOUND_IP=0;
}
else if (FOUND_PORT==1) {
PORT=$1;
FOUND_PORT=0;
} else if (FOUND_RUNID==1) {
printf "nsentinel known-sentinel mymaster %s %s %s", IP, PORT, $0; FOUND_RUNID=0;
}
}
ping_readiness_master.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_MASTER_PASSWORD $no_auth_warning
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ]; then
echo "$response"
exit 1
fi
ping_liveness_master.sh: |-
#!/bin/bash
no_auth_warning=$([[ "$(redis-cli --version)" =~ (redis-cli 5.*) ]] && echo --no-auth-warning)
response=$(
timeout -s 3 $1
redis-cli
-a $REDIS_MASTER_PASSWORD $no_auth_warning
-h $REDIS_MASTER_HOST
-p $REDIS_MASTER_PORT_NUMBER
ping
)
if [ "$response" != "PONG" ] && [ "$response" != "LOADING Redis is loading the dataset in memory" ]; then
echo "$response"
exit 1
fi
ping_readiness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_readiness_local.sh" $1 || exit_status=$?
"$script_dir/ping_readiness_master.sh" $1 || exit_status=$?
exit $exit_status
ping_liveness_local_and_master.sh: |-
script_dir="$(dirname "$0")"
exit_status=0
"$script_dir/ping_liveness_local.sh" $1 || exit_status=$?
"$script_dir/ping_liveness_master.sh" $1 || exit_status=$?

exit $exit_status

Source: redis/templates/headless-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: my-release-redis-headless
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: my-release
heritage: Tiller
spec:
type: ClusterIP
clusterIP: None
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis

release: my-release

Source: redis/templates/metrics-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: my-release-redis-metrics
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: my-release
heritage: Tiller
app.kubernetes.io/component: "metrics"
spec:
type: ClusterIP

ports:
- name: metrics
port: 9121
targetPort: metrics
selector:
app: redis

release: my-release

Source: redis/templates/redis-with-sentinel-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: my-release-redis
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: my-release
heritage: Tiller
spec:
type: ClusterIP
ports:
- name: redis
port: 6379
targetPort: redis
- name: redis-sentinel
port: 26379
targetPort: redis-sentinel
selector:
app: redis

release: my-release

Source: redis/templates/redis-master-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-release-redis-master
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: my-release
heritage: Tiller
spec:
selector:
matchLabels:
app: redis
release: my-release
role: master
serviceName: my-release-redis-headless
template:
metadata:
labels:
app: redis
chart: redis-10.7.13
release: my-release
role: master
annotations:
checksum/health: 5ac73eb1d928a863878d4052ce661e85cef819d1eae39c96f9a3fe98a07d56c8
checksum/configmap: 0a497daa94e4528ef362fc80fc0dd7f2346d29cf3fa66c53cf51e5e889c658d4
checksum/secret: 3167cc7bd6aefeb7d1c9f7e4fc4af24a307f92ea21ab75405bbe233fe369c031
prometheus.io/port: "9121"
prometheus.io/scrape: "true"

spec:

  serviceAccountName: default
  containers:
    - name: redis
      image: docker-registry.com:5000/bitnami/redis-cluster:6.0.6
      imagePullPolicy: "IfNotPresent"
      command:
        - /bin/bash
        - -c
        - |
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
          fi
          if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
          fi
          ARGS=("--port" "${REDIS_PORT}")
          ARGS+=("--requirepass" "${REDIS_PASSWORD}")
          ARGS+=("--masterauth" "${REDIS_PASSWORD}")
          ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
          ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
          redis-server ${ARGS[@]}
      env:
        - name: REDIS_REPLICATION_MODE
          value: master
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
      ports:
        - name: redis
          containerPort: 6379
      livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 5
      resources:
        null

      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath:
        - name: config
          mountPath: /opt/bitnami/redis/mounted-etc
        - name: redis-tmp-conf
          mountPath: /opt/bitnami/redis/etc/
    - name: sentinel
      image: "docker.io/bitnami/redis-sentinel:6.0.6-debian-10-r10"
      imagePullPolicy: "IfNotPresent"
      command:
        - /bin/bash
        - -c
        - |
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then
            cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
          fi
          echo "Getting information about current running sentinels"
          # Get information from existing sentinels
          existing_sentinels=$(timeout -s 3 5 redis-cli --raw -h my-release-redis -a "$REDIS_PASSWORD" -p 26379 SENTINEL sentinels mymaster)
          echo "$existing_sentinels" | awk -f /health/parse_sentinels.awk | tee -a  /opt/bitnami/redis-sentinel/etc/sentinel.conf

          redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel
      env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
        - name: REDIS_SENTINEL_TLS_ENABLED
          value: "no"
        - name: REDIS_SENTINEL_PORT
          value: "26379"
      ports:
        - name: redis-sentinel
          containerPort: 26379
      livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      resources:
        null

      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath:
        - name: config
          mountPath: /opt/bitnami/redis-sentinel/mounted-etc
        - name: sentinel-tmp-conf
          mountPath: /opt/bitnami/redis-sentinel/etc/
    - name: metrics
      image: docker.io/bitnami/redis-exporter:1.9.0-debian-10-r19
      imagePullPolicy: "IfNotPresent"
      command:
        - /bin/bash
        - -c
        - |
          if [[ -f '/secrets/redis-password' ]]; then
          export REDIS_PASSWORD=$(cat /secrets/redis-password)
          fi
          redis_exporter
      env:
        - name: REDIS_ALIAS
          value: my-release-redis
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
      volumeMounts:
      ports:
        - name: metrics
          containerPort: 9121
      resources:
        null

  volumes:
    - name: health
      configMap:
        name: my-release-redis-health
        defaultMode: 0755
    - name: config
      configMap:
        name: my-release-redis
    - name: redis-tmp-conf
      emptyDir: {}
    - name: sentinel-tmp-conf
      emptyDir: {}

volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app: redis
release: my-release
heritage: Tiller
component: master
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"

    selector:

updateStrategy:

type: RollingUpdate

Source: redis/templates/redis-slave-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-release-redis-slave
namespace: default
labels:
app: redis
chart: redis-10.7.13
release: my-release
heritage: Tiller
spec:
replicas: 3
serviceName: my-release-redis-headless
selector:
matchLabels:
app: redis
release: my-release
role: slave
template:
metadata:
labels:
app: redis
release: my-release
chart: redis-10.7.13
role: slave
annotations:
checksum/health: 5ac73eb1d928a863878d4052ce661e85cef819d1eae39c96f9a3fe98a07d56c8
checksum/configmap: 0a497daa94e4528ef362fc80fc0dd7f2346d29cf3fa66c53cf51e5e889c658d4
checksum/secret: 7b8a808782df0fe1fbd8689f7113b6720616794fcf626f26f3db120070127fe6
prometheus.io/port: "9121"
prometheus.io/scrape: "true"

spec:

  serviceAccountName: default
  containers:
    - name: redis
      image: docker-registry.com:5000/bitnami/redis-cluster:6.0.6
      imagePullPolicy: "IfNotPresent"
      command:
        - /bin/bash
        - -c
        - |
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ -n $REDIS_MASTER_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_MASTER_PASSWORD_FILE}`
            export REDIS_MASTER_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis/etc/replica.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/replica.conf /opt/bitnami/redis/etc/replica.conf
          fi
          if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
            cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
          fi
          ARGS=("--port" "${REDIS_PORT}")
          ARGS+=("--slaveof" "${REDIS_MASTER_HOST}" "${REDIS_MASTER_PORT_NUMBER}")
          ARGS+=("--requirepass" "${REDIS_PASSWORD}")
          ARGS+=("--masterauth" "${REDIS_MASTER_PASSWORD}")
          ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
          ARGS+=("--include" "/opt/bitnami/redis/etc/replica.conf")
          redis-server "${ARGS[@]}"
      env:
        - name: REDIS_REPLICATION_MODE
          value: slave
        - name: REDIS_MASTER_HOST
          value: my-release-redis-master-0.my-release-redis-headless.default.svc.cluster.local
        - name: REDIS_MASTER_PORT_NUMBER
          value: "6379"
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
        - name: REDIS_MASTER_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
      ports:
        - name: redis
          containerPort: 6379
      livenessProbe:
        initialDelaySeconds: 30
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 10
        timeoutSeconds: 10
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 5
      resources:
        null

      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
        - name: config
          mountPath: /opt/bitnami/redis/mounted-etc
        - name: redis-tmp-conf
          mountPath: /opt/bitnami/redis/etc
    - name: sentinel
      image: docker.io/bitnami/redis-sentinel:6.0.6-debian-10-r10
      imagePullPolicy: "IfNotPresent"
      command:
        - /bin/bash
        - -c
        - |
          if [[ -n $REDIS_PASSWORD_FILE ]]; then
            password_aux=`cat ${REDIS_PASSWORD_FILE}`
            export REDIS_PASSWORD=$password_aux
          fi
          if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then
            cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
            printf "\nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
          fi

          redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel
      env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
        - name: REDIS_SENTINEL_TLS_ENABLED
          value: "no"
        - name: REDIS_SENTINEL_PORT
          value: "26379"
      ports:
        - name: redis-sentinel
          containerPort: 26379
      livenessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      readinessProbe:
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 5
        exec:
          command:
            - sh
            - -c
            - /health/ping_sentinel.sh 5
      resources:
        null

      volumeMounts:
        - name: health
          mountPath: /health
        - name: redis-data
          mountPath: /data
          subPath:
        - name: config
          mountPath: /opt/bitnami/redis-sentinel/mounted-etc
        - name: sentinel-tmp-conf
          mountPath: /opt/bitnami/redis-sentinel/etc
    - name: metrics
      image: docker.io/bitnami/redis-exporter:1.9.0-debian-10-r19
      imagePullPolicy: "IfNotPresent"
      command:
        - /bin/bash
        - -c
        - |
          if [[ -f '/secrets/redis-password' ]]; then
          export REDIS_PASSWORD=$(cat /secrets/redis-password)
          fi
          redis_exporter
      env:
        - name: REDIS_ALIAS
          value: my-release-redis
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-release-redis
              key: redis-password
      volumeMounts:
      ports:
        - name: metrics
          containerPort: 9121
      resources:
        null

  volumes:
    - name: health
      configMap:
        name: my-release-redis-health
        defaultMode: 0755
    - name: config
      configMap:
        name: my-release-redis
    - name: sentinel-tmp-conf
      emptyDir: {}
    - name: redis-tmp-conf
      emptyDir: {}

volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app: redis
release: my-release
heritage: Tiller
component: slave
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"

    selector:

updateStrategy:
type: RollingUpdate
`
LAST DEPLOYED: Thu Aug 6 12:48:47 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME AGE
my-release-redis 0s
my-release-redis-health 0s

==> v1/NetworkPolicy
NAME AGE
my-release-redis 0s

==> v1/Pod(related)
NAME AGE
my-release-redis-master-0 0s
my-release-redis-slave-0 0s

==> v1/Secret
NAME AGE
my-release-redis 0s

==> v1/Service
NAME AGE
my-release-redis 0s
my-release-redis-headless 0s
my-release-redis-metrics 0s

==> v1/StatefulSet
NAME AGE
my-release-redis-master 0s
my-release-redis-slave 0s

* Pods Description *

admin@localhost:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-release-redis-master-0 3/3 Running 0 3d11h 10.233.121.9 localhost
my-release-redis-slave-0 3/3 Running 2 3d11h 10.233.121.11 localhost
my-release-redis-slave-1 3/3 Running 0 3d11h 10.233.121.5 localhost
my-release-redis-slave-2 3/3 Running 0 3d11h 10.233.121.18 localhost

admin@localhost:~$ kubectl describe pod my-release-redis-master-0 Name: my-release-redis-master-0 Namespace: default Priority: 0 Node: localhost/172.27.172.142 Start Time: Thu, 06 Aug 2020 18:53:29 +0000 Labels: app=redis chart=redis-10.7.14 controller-revision-hash=my-release-redis-master-6556cc749d release=my-release role=master statefulset.kubernetes.io/pod-name=my-release-redis-master-0 Annotations: checksum/configmap: f8f91a4101b4f50819cc1720ecae31ff07e080ce9b04314ce216219194c39abc checksum/health: 888804b652f72551003721e50cf2d48841f77687444e267862c93d766ac67007 checksum/secret: f8d45e1c83d1fac7aaa9b68a95b6f5ecb04109cd8b44c1a4a65ce15edf0fd44f prometheus.io/port: 9121 prometheus.io/scrape: true Status: Running IP: 10.233.121.9 IPs: IP: 10.233.121.9 Controlled By: StatefulSet/my-release-redis-master Containers: redis: Container ID: docker://ff0777553e2042909a8a674f385e5f8417634893405d41ed0d7d3333405a27f5 Image: docker-registry.com:5000/bitnami/redis-cluster:6.0.6 Image ID: docker://sha256:4ab979b05227060a4e050c2100fd9f8ff2cbbe8e7e205e635cee04bc7763ff1d Port: 6379/TCP Host Port: 0/TCP Command: /bin/bash -c if [[ -n $REDIS_PASSWORD_FILE ]]; then password_aux=cat ${REDIS_PASSWORD_FILE}`
export REDIS_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
fi
if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")
ARGS+=("--requirepass" "${REDIS_PASSWORD}")
ARGS+=("--masterauth" "${REDIS_PASSWORD}")
ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
redis-server ${ARGS[@]}

State:          Running
  Started:      Thu, 06 Aug 2020 18:53:32 +0000
Ready:          True
Restart Count:  0
Liveness:       exec [sh -c /health/ping_liveness_local.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness:      exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
Environment:
  REDIS_REPLICATION_MODE:  master
  REDIS_PASSWORD:          <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
  REDIS_TLS_ENABLED:       no
  REDIS_PORT:              6379
Mounts:
  /data from redis-data (rw)
  /health from health (rw)
  /opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
  /opt/bitnami/redis/mounted-etc from config (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-gvh4v (ro)

sentinel:
Container ID: docker://924708fe4e4661accd52b9b0145cb4043e111b396a780b0a446e66f7048ea8ec
Image: docker.io/bitnami/redis-sentinel:6.0.6-debian-10-r10
Image ID: docker-pullable://bitnami/redis-sentinel@sha256:ae9d3b769862a57d3bcb4f97b6d35de2254fe4cb22ab492425112024094bf628
Port: 26379/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -n $REDIS_PASSWORD_FILE ]]; then
password_aux=cat ${REDIS_PASSWORD_FILE}
export REDIS_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then
cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
fi
echo "Getting information about current running sentinels"
# Get information from existing sentinels
existing_sentinels=$(timeout -s 3 5 redis-cli --raw -h my-release-redis -a "$REDIS_PASSWORD" -p 26379 SENTINEL sentinels mymaster)
echo "$existing_sentinels" | awk -f /health/parse_sentinels.awk | tee -a /opt/bitnami/redis-sentinel/etc/sentinel.conf

  redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel

State:          Running
  Started:      Thu, 06 Aug 2020 18:57:27 +0000
Ready:          True
Restart Count:  0
Liveness:       exec [sh -c /health/ping_sentinel.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness:      exec [sh -c /health/ping_sentinel.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
Environment:
  REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
  REDIS_SENTINEL_TLS_ENABLED:  no
  REDIS_SENTINEL_PORT:         26379
Mounts:
  /data from redis-data (rw)
  /health from health (rw)
  /opt/bitnami/redis-sentinel/etc/ from sentinel-tmp-conf (rw)
  /opt/bitnami/redis-sentinel/mounted-etc from config (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-gvh4v (ro)

metrics:
Container ID: docker://33ce4beeb02f422ba8f11be1a5da872e953a94fa54e2e4f4731fa77104365d85
Image: docker.io/bitnami/redis-exporter:1.9.0-debian-10-r19
Image ID: docker-pullable://bitnami/redis-exporter@sha256:b17118a247167540f6953f9d2d5cebe1b9f83bae53a5f0278b796f91c9c4c250
Port: 9121/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -f '/secrets/redis-password' ]]; then
export REDIS_PASSWORD=$(cat /secrets/redis-password)
fi
redis_exporter

State:          Running
  Started:      Thu, 06 Aug 2020 18:53:33 +0000
Ready:          True
Restart Count:  0
Environment:
  REDIS_ALIAS:     my-release-redis
  REDIS_PASSWORD:  <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-gvh4v (ro)

Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
redis-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: redis-data-my-release-redis-master-0
ReadOnly: false
health:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-release-redis-health
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-release-redis
Optional: false
redis-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
sentinel-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
default-token-gvh4v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gvh4v
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
admin@localhost:~$ kubectl describe pod my-release-redis-slave-0
Name: my-release-redis-slave-0
Namespace: default
Priority: 0
Node: localhost/172.27.172.142
Start Time: Thu, 06 Aug 2020 18:53:30 +0000
Labels: app=redis
chart=redis-10.7.14
controller-revision-hash=my-release-redis-slave-64759b74cf
release=my-release
role=slave
statefulset.kubernetes.io/pod-name=my-release-redis-slave-0
Annotations: checksum/configmap: f8f91a4101b4f50819cc1720ecae31ff07e080ce9b04314ce216219194c39abc
checksum/health: 888804b652f72551003721e50cf2d48841f77687444e267862c93d766ac67007
checksum/secret: bec6d8312aff8c9ca7aa7f71f4c4f7e156a603bc4e38ad939c6fed742d4b2301
prometheus.io/port: 9121
prometheus.io/scrape: true
Status: Running
IP: 10.233.121.11
IPs:
IP: 10.233.121.11
Controlled By: StatefulSet/my-release-redis-slave
Containers:
redis:
Container ID: docker://c08f39cc551d379fe124b7cd67bd4539e7f941ec13eb4b2d38a805e0e1225853
Image: docker-registry.com:5000/bitnami/redis-cluster:6.0.6
Image ID: docker://sha256:4ab979b05227060a4e050c2100fd9f8ff2cbbe8e7e205e635cee04bc7763ff1d
Port: 6379/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -n $REDIS_PASSWORD_FILE ]]; then
password_aux=cat ${REDIS_PASSWORD_FILE}
export REDIS_PASSWORD=$password_aux
fi
if [[ -n $REDIS_MASTER_PASSWORD_FILE ]]; then
password_aux=cat ${REDIS_MASTER_PASSWORD_FILE}
export REDIS_MASTER_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis/etc/replica.conf ]];then
cp /opt/bitnami/redis/mounted-etc/replica.conf /opt/bitnami/redis/etc/replica.conf
fi
if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")
ARGS+=("--slaveof" "${REDIS_MASTER_HOST}" "${REDIS_MASTER_PORT_NUMBER}")
ARGS+=("--requirepass" "${REDIS_PASSWORD}")
ARGS+=("--masterauth" "${REDIS_MASTER_PASSWORD}")
ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
ARGS+=("--include" "/opt/bitnami/redis/etc/replica.conf")
redis-server "${ARGS[@]}"

State:          Running
  Started:      Thu, 06 Aug 2020 18:53:33 +0000
Ready:          True
Restart Count:  0
Liveness:       exec [sh -c /health/ping_liveness_local.sh 5] delay=30s timeout=5s period=10s #success=1 #failure=5
Readiness:      exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=10s period=10s #success=1 #failure=5
Environment:
  REDIS_REPLICATION_MODE:    slave
  REDIS_MASTER_HOST:         my-release-redis-master-0.my-release-redis-headless.default.svc.cluster.local
  REDIS_MASTER_PORT_NUMBER:  6379
  REDIS_PASSWORD:            <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
  REDIS_MASTER_PASSWORD:     <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
  REDIS_TLS_ENABLED:         no
  REDIS_PORT:                6379
Mounts:
  /data from redis-data (rw)
  /health from health (rw)
  /opt/bitnami/redis/etc from redis-tmp-conf (rw)
  /opt/bitnami/redis/mounted-etc from config (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-gvh4v (ro)

sentinel:
Container ID: docker://cdde2e36c060230a99de55f1a087223061c9cd4cb008c01ed559a6e29dc20105
Image: docker.io/bitnami/redis-sentinel:6.0.6-debian-10-r10
Image ID: docker-pullable://bitnami/redis-sentinel@sha256:ae9d3b769862a57d3bcb4f97b6d35de2254fe4cb22ab492425112024094bf628
Port: 26379/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -n $REDIS_PASSWORD_FILE ]]; then
password_aux=cat ${REDIS_PASSWORD_FILE}
export REDIS_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis-sentinel/etc/sentinel.conf ]];then
cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "nsentinel auth-pass mymaster $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "nrequirepass $REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
fi

  redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel

State:          Running
  Started:      Thu, 06 Aug 2020 18:53:52 +0000
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 06 Aug 2020 18:53:35 +0000
  Finished:     Thu, 06 Aug 2020 18:53:35 +0000
Ready:          True
Restart Count:  2
Liveness:       exec [sh -c /health/ping_sentinel.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness:      exec [sh -c /health/ping_sentinel.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
Environment:
  REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
  REDIS_SENTINEL_TLS_ENABLED:  no
  REDIS_SENTINEL_PORT:         26379
Mounts:
  /data from redis-data (rw)
  /health from health (rw)
  /opt/bitnami/redis-sentinel/etc from sentinel-tmp-conf (rw)
  /opt/bitnami/redis-sentinel/mounted-etc from config (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-gvh4v (ro)

metrics:
Container ID: docker://86138a06e66471b7a0d5fca8c234f3abc1c8cbdb1eca729c721685744634fa5f
Image: docker.io/bitnami/redis-exporter:1.9.0-debian-10-r19
Image ID: docker-pullable://bitnami/redis-exporter@sha256:b17118a247167540f6953f9d2d5cebe1b9f83bae53a5f0278b796f91c9c4c250
Port: 9121/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -f '/secrets/redis-password' ]]; then
export REDIS_PASSWORD=$(cat /secrets/redis-password)
fi
redis_exporter

State:          Running
  Started:      Thu, 06 Aug 2020 18:53:33 +0000
Ready:          True
Restart Count:  0
Environment:
  REDIS_ALIAS:     my-release-redis
  REDIS_PASSWORD:  <set to the key 'redis-password' in secret 'my-release-redis'>  Optional: false
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-gvh4v (ro)

Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
redis-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: redis-data-my-release-redis-slave-0
ReadOnly: false
health:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-release-redis-health
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-release-redis
Optional: false
sentinel-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
redis-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
default-token-gvh4v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gvh4v
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
`
Thanks
Mahadev

mahanam picture mahanam on 10 Aug 2020

Hi @mahanam

I have been trying to reproduce your issue but was unsuccessful. Is there a reason for you not to be using securityContext? Also, I would like to know which k8s cluster are you using to deploy the Chart (maybe minikube?).

Thanks!

joancafom picture joancafom on 10 Aug 2020

Hi,

It's not Minikube but a proper kubespray based K8S cluster.
Have you reproduced with disabled SecurityContext ?

Thanks,
Mahadev

mahanam picture mahanam on 10 Aug 2020

Hi,

It's a virtual machine where I am trying this scenario.
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial

Am able to reproduce it very easily and this time I have enabled the SecurityContext with runAsUser: 0.
Have used runAsUser:0 because with 1001 UID am facing below issue.

1:C 10 Aug 2020 15:23:47.326 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 10 Aug 2020 15:23:47.326 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 10 Aug 2020 15:23:47.326 # Configuration loaded
1:M 10 Aug 2020 15:23:47.328 # Can't open the append-only file: Permission denied

So I made it runAsUser:0 and skipped that error.

Please try the below steps to reproduce the bug.

  1. Installed redis using below commands
    helm repo add bitnami https://charts.bitnami.com/bitnami # added repo
    helm install --name my-release bitnami/redis --values values-production.yaml # deployed redis ha

  2. Wait for all the pods to come up.

  3. Delete the master sentinel docker container.
    docker rm -f 366d1d92b876

  4. Verify the docker sentinel docker logs of new container as shown below as its keep on crashing due to below failure.
    Getting information about current running sentinels
    Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

sentinel known-sentinel mymaster 10.233.77.12 26379 8c17f6cefe637192fc7913327041fc944481a66f

* FATAL CONFIG FILE ERROR (Redis 6.0.6) *
Reading the configuration file, at line 22

'sentinel known-sentinel mymaster 10.233.77.10 26379 3bc25047104a735b4c351ef3fa1ad6317bd9bb2a'
Wrong hostname or port for sentinel.

Thanks,
Mahadev

mahanam picture mahanam on 10 Aug 2020

Hi,

Have resolved this issue by increasing this timeout .Values.sentinel.initialCheckTimeout from 5 to 10.
So that giving enough window to open the port 26379.

Thanks,
Mahadev

mahanam picture mahanam on 11 Aug 2020
🚀1 ❤1 🎉1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

druesendieb picture druesendieb  路  5Comments
zackijack picture zackijack  路  3Comments
estenssoros picture estenssoros  路  3Comments
PhoneixS picture PhoneixS  路  6Comments
wantdrink picture wantdrink  路  6Comments