Charts: [redis-cluster] persistence matchLabels not working as expected

[lukejpreston]

I am trying to start a Redis cluster and want to use my own persistence. I am not sure if something is wrong/not working or if I am doing it wrong. Could someone please advise?

This is my PV and PVC on my local machine

apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-data
  labels:
    foo: bar
spec:
  storageClassName: manual
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: <path value>
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: redis-data
  namespace: <namespace value>
  labels:
    foo: bar
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 8Gi
  volumeName: redis-data

This is my values file redis-values.yaml

persistence:
    matchLabels:
      foo: bar
cluster:
  nodes: 1

this is the command I run to install

helm install <name> bitnami/redis-cluster -n <namespace-value> -f redis-values.yaml

The Redis cluster starts but creates it's own volume and uses that. I would like it to use my local volume.

[fifa451]

You may have to set persistence.enabled to false, and use extraVolumes and extraVolumeMounts to define your own local volume.
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#claims-as-volumes

[lukejpreston]

I tried the following

persistence:
  enabled: true
  matchLabels:
    foo: bar
redis:
  extraVolumes:
    - name: backup
      hostPath:
        path: <path value>
        type: Directory
  extraVolumeMounts:
    - mountPath: /bitnami
      name: backup

and it didn't seem to mount these extra volumes

Perhaps if I explain what I am trying to do it might give some context and help resolve this issue. I currently have a redis database, using the redis helm chart. I want to scale redis and see if the redis-cluster will help. I want to migrate my data to the cluster as well as keep a copy of the aof file(s) and any backup so I can restore redis as it is currently running on our own local cluster, which has been unreliable in the past. Using the standard redis chart I can automate startup and backups without needing to manually do anything unless I need to restore the database to a backup.

[fifa451]

@lukejpreston you are right, I did try after that, and there were some bugs for extraVolumes in the old version. I did a pull request for it, the version 3.1.4 should fix the extraVolumes issue.

[lukejpreston]

@fifa451 I will upgrade to 3.1.4 and report back :D

[lukejpreston]

It mounted the volumes but it doesn't seem to be writing as I would have expected

On my machine after install

  |- redis
    |- data
  |- test-file.txt

on each cluster

  |- redis
    |- data
      |- appendonly.aof
      |- dump.rdb
  |- test-file.txt

This feels like a step forward but I am not sure what is missing.

[fifa451]

Maybe you can provide more details of what structure do you want?

The /bitnami/redis/data path is defined in persistence.path for regular Redis data by default. maybe set it to /bitnami-default/redis/data, and then mount your own volume to /bitnami/ ? It seems that all the data has to write into /bitnami/redis/data, from there cluster setup script: https://github.com/bitnami/bitnami-docker-redis-cluster/blob/master/6.0/debian-10/rootfs/opt/bitnami/scripts/librediscluster.sh#L35

[lukejpreston]

Sorry for the delay, I haven't had a chance to try the suggestion until now. I am getting the error

Sorry, the cluster configuration file /bitnami/redis/data/nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration f
iles.

But it looks like it is saving the appendonly.aof file to my local disk which it wasn't doing before. Here are the values I am currently using:

persistence:
  enabled: true
  path: /bitnami-default/redis/data
  matchLabels:
    foo: bar
redis:
  extraVolumes:
    - name: backup
      hostPath:
        path: <path value>
        type: Directory
  extraVolumeMounts:
    - mountPath: /bitnami
      name: backup

[marcosbc]

@lukejpreston Could you describe your configuration further? It looks like you're changing the mount path of the persistence to /bitnami-default/redis/data, and instead making /bitnami be the volume for backups.

Note that the actual configuration for the container will use the /bitnami/redis/data directory for persistence no matter what, it is not customizable (even if it's been added to values.yaml), meaning it would currently be configured to use the backup directory when starting the service.

[lukejpreston]

I have created a repo with my configuration if that helps https://github.com/lukejpreston/redis-cluster-example

Should I change mountPath: /bitnami to mountPath: /bitnami-default/redis/data?

[marcosbc]

In your repository I see it being set to mountPath: /bitnami-default/redis/data. It should be mountPath: /bitnami.

[lukejpreston]

I did this and I now get Invalid value: "/bitnami": must be unique

[marcosbc]

It looks like that is related to there being an extraVolumeMounts referring to /bitnami as well. That should not be needed if the persistence path is kept unmodified.

[lukejpreston]

@marcosbc could you clarify what you mean by persistence path is kept unmodified

I have removed the extraVolumes and extraVolumeMounts but I am back to where I started. I have manually created volumes and no claims but the redis cluster isn't claiming them using the labels, it is instead creating its own volumes. Here are all my volumes after running my latest config. I have pushed to the repo I shared if you want to have a better look.

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                      STORAGECLASS   REASON   AGE
pvc-0e166ba8-7564-4a00-8c13-327376251bc0   8Gi        RWO            Delete           Bound       redis-cluster-example/redis-data-example-redis-cluster-3   hostpath                62s
pvc-373162bc-4129-461c-9956-5e1de8ad74ec   8Gi        RWO            Delete           Bound       redis-cluster-example/redis-data-example-redis-cluster-2   hostpath                63s
pvc-72d6b36e-66b0-4716-bbdf-67127fd7aa72   8Gi        RWO            Delete           Bound       redis-cluster-example/redis-data-example-redis-cluster-5   hostpath                62s
pvc-85bd4bfe-ac5f-482e-939f-24305b2d9f14   8Gi        RWO            Delete           Bound       redis-cluster-example/redis-data-example-redis-cluster-0   hostpath                64s
pvc-892e50dd-c88d-4c9d-b3ec-b07390b764e4   8Gi        RWO            Delete           Bound       redis-cluster-example/redis-data-example-redis-cluster-1   hostpath                63s
pvc-96f697d4-80c7-4ab2-8bb4-4379f6ad0db8   8Gi        RWO            Delete           Bound       redis-cluster-example/redis-data-example-redis-cluster-4   hostpath                61s
redis-data-0                               8Gi        RWO            Retain           Available                                                              manual                  70s
redis-data-1                               8Gi        RWO            Retain           Available                                                              manual                  70s
redis-data-2                               8Gi        RWO            Retain           Available                                                              manual                  70s
redis-data-3                               8Gi        RWO            Retain           Available                                                              manual                  69s
redis-data-4                               8Gi        RWO            Retain           Available                                                              manual                  69s
redis-data-5                               8Gi        RWO            Retain           Available                                                              manual                  69s

[fifa451]

@lukejpreston

I guess the problem is that the when you using your own volume, make sure each Redis node configure is in a different directory, otherwise it will overwrite or lock each other.
I guess This is idea directory structures will:

• /bitnami/redis/data/redis-cluster-0,
• /bitnami/redis/data/redis-cluste-1,
• /bitnami/redis/data/redis-cluste-2,
• /bitnami/redis/data/redis-cluste-3,
• /bitnami/redis/data//redis-cluste-4

however, it seems very difficult to configure because the cluster startup script from docker image has some hardcode variables for the node.conf https://github.com/bitnami/bitnami-docker-redis-cluster/blob/master/6.0/debian-10/rootfs/opt/bitnami/scripts/librediscluster.sh,

one of the option you can try is to add extraEnvVars called REDIS_VOLUME with value bitnami/redis/data/${POD_NAME}, the ${POD_NAME} should be from k8s Downward API, so it will create a different name for each pod, also you need a init container to check and create the "{POD_NAME}" directory.

Sorry for the delay, I haven't had a chance to try the suggestion until now. I am getting the error

Sorry, the cluster configuration file /bitnami/redis/data/nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration f
iles.

But it looks like it is saving the appendonly.aof file to my local disk which it wasn't doing before. Here are the values I am currently using:

persistence:
  enabled: true
  path: /bitnami-default/redis/data
  matchLabels:
    foo: bar
redis:
  extraVolumes:
    - name: backup
      hostPath:
        path: <path value>
        type: Directory
  extraVolumeMounts:
    - mountPath: /bitnami
      name: backup

[lukejpreston]

@fifa451 the volumes are all different directories. Sorry if I am being a bit slow on the intake. From the docs I thought persistence.matchLabels would tell the chart to use these volumes and not create its own, which is what I want it to do.

When the cluster starts up it all looks correct, I just want to control where/how the volumes are saved

For example instead of using volume pvc-85bd4bfe-ac5f-482e-939f-24305b2d9f14 I want to use redis-data-0 then redis-data-0 could be any kind of volume.

I feel like I have missed something obvious but I don't know what that something is. I want it to be the same as https://github.com/bitnami/charts/blob/master/bitnami/redis/README.md#parameters persistent.existingClaim which is perhaps the wrong approach?

[marcosbc]

I see, in that existingClaim is the correct option but the feature is not yet supported in redis-cluster. However, it looks like something that should not require many changes to the Helm chart.

Feel free to submit a PR to implement support for existingClaim, we'd be happy to review it!

[fifa451]

@fifa451 the volumes are all different directories. Sorry if I am being a bit slow on the intake. From the docs I thought persistence.matchLabels would tell the chart to use these volumes and not create its own, which is what I want it to do.

When the cluster starts up it all looks correct, I just want to control where/how the volumes are saved

For example instead of using volume pvc-85bd4bfe-ac5f-482e-939f-24305b2d9f14 I want to use redis-data-0 then redis-data-0 could be any kind of volume.

I feel like I have missed something obvious but I don't know what that something is. I want it to be the same as https://github.com/bitnami/charts/blob/master/bitnami/redis/README.md#parameters persistent.existingClaim which is perhaps the wrong approach?

@lukejpreston I think @marcosbc is right, my previous suggestion isn't the right direction. If the chats add the existingClaim then, we can have volume format like this to reclaim the existing Volume.

e.g.

  volumes:
    - name: redis-data
      persistentVolumeClaim:
        claimName: myExistingRedisVolume

[lukejpreston]

I will have a look at making a PR!

Thanks for the help, I don't think this issue needs to stay open but I will leave that with you to decide if you want to close it.

[marcosbc]

Awesome, thanks! I'll keep this open for now. I'm adding the on-hold label so it doesn't close.