Charts: [stable/redis] Persistence is not working (?)

[caleno]

Is this a request for help?:

Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT (maybe)

values-production.yaml

## Global Docker image registry
## Please, note that this will override the image registry for all the images, including dependencies, configured to use the global value
##
# global:v1beta1
#   imageRegistry:

## Bitnami Redis image version
## ref: https://hub.docker.com/r/bitnami/redis/tags/
##
image:
  registry: docker.io
  repository: bitnami/redis
  ## Bitnami Redis image tag
  ## ref: https://github.com/bitnami/bitnami-docker-redis#supported-tags-and-respective-dockerfile-links
  ##
  tag: 4.0.12
  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistrKeySecretName

## Cluster settings
cluster:
  enabled: true
  slaveCount: 2

networkPolicy:
  ## Specifies whether a NetworkPolicy should be created
  ##
  enabled: false

  ## The Policy model to apply. When set to false, only pods with the correct
  ## client label will have network access to the port Redis is listening
  ## on. When true, Redis will accept connections from any source
  ## (with the correct destination port).
  ##
  # allowExternal: true

serviceAccount:
  ## Specifies whether a ServiceAccount should be created
  ##
  create: false
  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the fullname template
  name:

rbac:
  ## Specifies whether RBAC resources should be created
  ##
  create: false

  role:
    ## Rules to create. It follows the role specification
    # rules:
    #  - apiGroups:
    #    - extensions
    #    resources:
    #      - podsecuritypolicies
    #    verbs:
    #      - use
    #    resourceNames:
    #      - gce.unprivileged
    rules: []

## Use password authentication
usePassword: true
## Redis password (both master and slave)
## Defaults to a random 10-character alphanumeric string if not set and usePassword is true
## ref: https://github.com/bitnami/bitnami-docker-redis#setting-the-server-password-on-first-run
##
# password: ""
## Use existing secret (ignores previous password)
# existingSecret:

## Mount secrets as files instead of environment variables
usePasswordFile: false

## Persist data to a persistent volume
persistence: {}
  ## A manually managed Persistent Volume and Claim
  ## Requires persistence.enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  # existingClaim:

##
## Redis Master parameters
##
master:
  ## Redis port
  port: 6379
  ## Redis command arguments
  ##
  ## Can be used to specify command line arguments, for example:
  ##
  command: "/run.sh"
  ## Redis additional command line flags
  ##
  ## Can be used to specify command line flags, for example:
  ##
  ## extraFlags:
  ##  - "--maxmemory-policy volatile-ttl"
  ##  - "--repl-backlog-size 1024mb"
  extraFlags: []
  ## Comma-separated list of Redis commands to disable
  ##
  ## Can be used to disable Redis commands for security reasons.
  ## Commands will be completely disabled by renaming each to an empty string.
  ## ref: https://redis.io/topics/security#disabling-of-specific-commands
  ##
  disableCommands:
  - FLUSHDB
  - FLUSHALL

  ## Redis Master additional pod labels and annotations
  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
  podLabels: {}
  podAnnotations: {}

  ## Redis Master resource requests and limits
  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  # resources:
  #   requests:
  #     memory: 256Mi
  #     cpu: 100m
  ## Use an alternate scheduler, e.g. "stork".
  ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
  ##
  # schedulerName:

  ## Configure extra options for Redis Master liveness and readiness probes
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
  ##
  livenessProbe:
    enabled: true
    initialDelaySeconds: 30
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 1
    failureThreshold: 5
  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 10
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 5

  ## Redis Master Node selectors and tolerations for pod assignment
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
  ##
  # nodeSelector: {"beta.kubernetes.io/arch": "amd64"}
  # tolerations: []
  ## Redis Master pod/node affinity/anti-affinity
  ##
  affinity: {}

  ## Redis Master Service properties
  service:
    ##  Redis Master Service type
    type: ClusterIP
    port: 6379

    ## Specify the nodePort value for the LoadBalancer and NodePort service types.
    ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
    ##
    # nodePort:

    ## Provide any additional annotations which may be required. This can be used to
    ## set the LoadBalancer service type to internal only.
    ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
    ##
    annotations: {}
    loadBalancerIP:

  ## Redis Master Pod Security Context
  ##
  securityContext:
    enabled: true
    fsGroup: 1001
    runAsUser: 1001

  ## Enable persistence using Persistent Volume Claims
  ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
  ##
  persistence:
    enabled: true
    ## The path the volume will be mounted at, useful when using different
    ## Redis images.
    path: /bitnami/redis/data
    ## The subdirectory of the volume to mount to, useful in dev environments
    ## and one PV for multiple services.
    subPath: ""
    ## redis data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    # storageClass: "-"
    accessModes:
    - ReadWriteOnce
    size: 8Gi

  ## Update strategy, can be set to RollingUpdate or onDelete by default.
  ## https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
  statefulset:
    updateStrategy: RollingUpdate
    ## Partition update strategy
    ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
    # rollingUpdatePartition:

  ## Redis Master pod priorityClassName
  # priorityClassName: {}

##
## Redis Slave properties
## Note: service.type is a mandatory parameter
## The rest of the parameters are either optional or, if undefined, will inherit those declared in Redis Master
##
slave:
  ## Slave Service properties
  service:
    ## Redis Slave Service type
    type: ClusterIP
    ## Specify the nodePort value for the LoadBalancer and NodePort service types.
    ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
    ##
    # nodePort:

    ## Provide any additional annotations which may be required. This can be used to
    ## set the LoadBalancer service type to internal only.
    ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
    ##
    annotations: {}
    loadBalancerIP:

  ## Redis port
  # port: 6379
  ## Redis extra flags
  # extraFlags: []
  ## List of Redis commands to disable
  # disableCommands: []

  ## Redis Slave pod/node affinity/anti-affinity
  ##
  affinity: {}

  ## Configure extra options for Redis Slave liveness and readiness probes
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
  ##
  # livenessProbe:
  #   enabled: true
  #   initialDelaySeconds: 30
  #   periodSeconds: 10
  #   timeoutSeconds: 5
  #   successThreshold: 1
  #   failureThreshold: 5
  # readinessProbe:
  #   enabled: true
  #   initialDelaySeconds: 5
  #   periodSeconds: 10
  #   timeoutSeconds: 10
  #   successThreshold: 1
  #   failureThreshold: 5

  ## Redis slave Resource
  # resources:
  #   requests:
  #     memory: 256Mi
  #     cpu: 100m

  ## Redis slave selectors and tolerations for pod assignment
  # nodeSelector: {"beta.kubernetes.io/arch": "amd64"}
  # tolerations: []

  ## Use an alternate scheduler, e.g. "stork".
  ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
  ##
  # schedulerName:

  ## Redis slave pod Annotation and Labels
  # podLabels: {}
  # podAnnotations: {}

  ## Redis slave pod Security Context
  # securityContext:
  #   enabled: true
  #   fsGroup: 1001
  #   runAsUser: 1001

  ## Redis slave pod priorityClassName
  # priorityClassName: {}

## Prometheus Exporter / Metrics
##
metrics:
  enabled: true

  image:
    registry: docker.io
    repository: oliver006/redis_exporter
    tag: v0.28.0
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistrKeySecretName

  service:
    type: ClusterIP
    ## Use serviceLoadBalancerIP to request a specific static IP,
    ## otherwise leave blank
    # loadBalancerIP:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9121"

  ## Metrics exporter resource requests and limits
  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  # resources: {}

  ## Extra arguments for Metrics exporter, for example:
  ## extraArgs:
  ##   check-keys: myKey,myOtherKey
  # extraArgs: {}

  ## Metrics exporter labels and tolerations for pod assignment
  # nodeSelector: {"beta.kubernetes.io/arch": "amd64"}
  # tolerations: []

  ## Metrics exporter pod Annotation and Labels
  # podAnnotations: {}
  # podLabels: {}

  # Enable this if you're using https://github.com/coreos/prometheus-operator
  serviceMonitor:
    enabled: false
    namespace: monitoring
    # fallback to the prometheus default unless specified
    # interval: 10s
    ## Defaults to what's used if you follow CoreOS [Prometheus Install Instructions](https://github.com/helm/charts/tree/master/stable/prometheus-operator#tldr)
    ## [Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#prometheus-operator-1)
    ## [Kube Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#exporters)
    selector:
      prometheus: kube-prometheus

  ## Metrics exporter pod priorityClassName
  # priorityClassName: {}

##
## Init containers parameters:
## volumePermissions: Change the owner of the persist volume mountpoint to RunAsUser:fsGroup
##
volumePermissions:
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: latest
    pullPolicy: IfNotPresent

## Redis config file
## ref: https://redis.io/topics/config
##
configmap: |-
  # maxmemory-policy volatile-lru

## Sysctl InitContainer
## used to perform sysctl operation to modify Kernel settings (needed sometimes to avoid warnings)
sysctlImage:
  enabled: false
  command: []
  registry: docker.io
  repository: bitnami/minideb
  tag: latest
  pullPolicy: Always
  mountHostSys: false

Version of Helm and Kubernetes:
Kubernetes v. 1.12.4
Helm
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Which chart:
stable/redis

What happened:
Persistence is not working. If the master pod is updated the Redis data is not gone and I get a empty Redis. If I delete the cluster and set it up again, it is gone. And yes, the master's persistent volume is
not deleted.

What you expected to happen:
Well, I read this article and it seems like a good approach for setting up a production ready Redis (cluster) and that I would get persistence out of the box but that is not what I am experiencing.
https://engineering.bitnami.com/articles/deploy-and-scale-a-redis-cluster-on-kubernetes-with-bitnami-and-helm.html

I thought the master's PV would retain its data.

How to reproduce it (as minimally and precisely as possible):
I'm Azure AKS (2xStandard B2ms (2 vcpus, 8 GB memory)) (test site)

Deploy using e.g. tag 4.0.11
Then make an entry in the redis database (set foo jalla)
Upgrade the release with tag 4.0.12
Then check if the entry still exists.

So, am I doing something wrong? Have I misunderstood the setup (?) and to get persistence I have to use a manually created PV/PVC ?

Thanks.

[curls]

I've run into this issue as well, how to solve it?

[caleno]

Problem "persist" with a manually created PVC as well.

## Persist data to a persistent volume persistence: {} ## A manually managed Persistent Volume and Claim ## Requires persistence.enabled: true ## If defined, PVC must be created manually before volume will be bound # existingClaim: redis-data

@curls I wish I knew. Have you found out anything more about this?

[caleno]

Here is another BUG.

If I disable cluster the information about how to connect specifies an DNS/host address that is not in use.

```Redis can be accessed via port 6379 on the following DNS name from within your cluster:

redis-test.default.svc.cluster.local
```

2. Connect using the Redis CLI: redis-cli -h redis-test -a $REDIS_PASSWORD

The host is redis-test-master and DNS redis-test-master.default.svc.cluster.local.

[juan131]

So, am I doing something wrong? Have I misunderstood the setup (?) and to get persistence I have to use a manually created PV/PVC ?

No, you shouldn't need to create a PV/PVC. If you AKS cluster has a Dynamic Volume Provisioner you shouldn't need that.

Regarding the persistence error. I install a chart using the same values-production.yaml you shared:

$ helm install stable/redis -f values-production.yaml --name redis

Then I added some data:

$ export REDIS_PASSWORD=$(kubectl get secret --namespace default redis -o jsonpath="{.data.redis-password}" | base64 --decode)
$ kubectl run --namespace default redis-client --rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis:4.0.12 -- bash
I have no name!@redis-client:/$ redis-cli -h redis-master -a $REDIS_PASSWORD
Warning: Using a password with '-a' option on the command line interface may not be safe.
redis-master:6379> set foo bar
OK

Finally, I deleted the Helm chart (WITHOUT REMOVING THE PVC):

$ helm delete --purge redis
release "redis" deleted
$ kubectl get pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-data-redis-master-0   Bound    pvc-69fbe3ed-39a5-11e9-a150-0800273e1ced   8Gi        RWO            standard       10m

Then I installed the chart again (so it uses the same PVC/PV) :

$ helm install stable/redis -f values-production.yaml --name redis
$ export REDIS_PASSWORD=$(kubectl get secret --namespace default redis -o jsonpath="{.data.redis-password}" | base64 --decode)
$ kubectl run --namespace default redis-client --rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis:4.0.12 -- bash
I have no name!@redis-client:/$ redis-cli -h redis-master -a $REDIS_PASSWORD
Warning: Using a password with '-a' option on the command line interface may not be safe.
redis-master:6379> get foo
(nil)

As you mentioned... The data is lost!

What's the problem? You should enable Append Only file (see http://redis.io/topics/persistence). You can do it by adding this to your values-production.yaml:

 ## Redis config file
 ## ref: https://redis.io/topics/config
 ##
 configmap: |-
   # maxmemory-policy volatile-lru
+ # Enable AOF https://redis.io/topics/persistence#append-only-file
+ # Leave default fsync (every second)
+  appendonly yes

Could you please try that and let us know if that fixes the issue?

[caleno]

It worked indeed. One would think that when using values-production.yaml it should imply some sort default backup configured. But that is not the case, one have to configure it. If I understood this correct then one should not be fooled to think Redis saves data to disk (if not instructed to) but in memory, so that's why one has to configuring RDB/AOF.

I now have persistence Redis with using this setting (which I had been using for redis-ha)

configmap: |- # maxmemory-policy volatile-lru appendonly yes appendfsync everysec no-appendfsync-on-rewrite no save 900 1 save 300 10 save 60 10000

Happy day!

Thanks @juan131

[ogavrisevs]

Issue is when i set : slave.persistence.enabled=false

Statefull set has volumeMounts :

  - name: redis-data 
    mountPath: /data            

and is missing volumes part :

  - name: "redis-data"                                              
     emptyDir: {}    

full example :
volumeMounts: - name: health mountPath: /health - name: redis-data mountPath: /data - name: config mountPath: /opt/bitnami/redis/mounted-etc - name: redis-tmp-conf mountPath: /opt/bitnami/redis/etc volumes: - name: health configMap: name: redis-health defaultMode: 0755 - name: config configMap: name: redis - name: sentinel-tmp-conf emptyDir: {} - name: redis-tmp-conf emptyDir: {}