Charts: [stable/redis] persistence does not work

[naseemkullah]

Describe the bug
A clear and concise description of what the bug is.

State is not persisted across reboots although .Values.persistence is enabled and a pvc is mounted.

Version of Helm and Kubernetes:
latest GKE and latest helm

Which chart:
redis 6.4.5 and beyond, have not confirmed behaviour with older versions of chart, i do remember it working as expected in the past however
cluster mode is disabled
persistence is enabled

What happened:

i enable persistence (pvc mounted on /data by chart)
i create a key
i restart the redis-master pod
and the key no longer exists

What you expected to happen:
i expect the key to persist across restarts

How to reproduce it (as minimally and precisely as possible):
install redis as a dependency in a chart with these options

redis:
  enabled: true
  cluster:
    enabled: false
  networkPolicy:
    enabled: true
    allowExternal: false
  serviceAccount:
    create: true
  rbac:
    create: true
  usePassword: true
  metrics:
    enabled: true

Anything else we need to know:
no dump.rdb present in /data

redis-master-0:/data$ ls
lost+found

Also logs do not indicate any bgsave actions

[javsalgar]

Hi,

Did you try executing the save command to confirm that it is not an issue with the pvc mounting?

[naseemkullah]

Thanks @javsalgar .

Save command works:

redis-master-0:/data$ ls
lost+found

then

127.0.0.1:6379> save
OK

and in logs:

* DB saved on disk

then

redis-master-0:/data$ ls
dump.rdb  lost+found

Any idea why it's not automatically saving periodically?

[naseemkullah]

Here is the configmap (comes from the chart)

apiVersion: v1
data:
  master.conf: |-
    dir /data
    rename-command FLUSHDB ""
    rename-command FLUSHALL ""
  redis.conf: |-
    # User-supplied configuration:
    # maxmemory-policy volatile-lru
  replica.conf: |-
    dir /data
    rename-command FLUSHDB ""
    rename-command FLUSHALL ""

[chychkan]

Hi!
I'm facing same issue. According to this doc https://redis.io/topics/persistence persistence can be done either via "snapshotting" or via "append-only file" (AOF). The snapshotting (creating dump.rdb file) is controlled by the save config option. And the AOF is controlled by appendonly yes config option.
When I deploy Redis using this chart, the save config option is set to "" which disables snapshotting and the appendonly is set to no, so effectively persistence is disabled:

127.0.0.1:6379> CONFIG GET save
1) "save"
2) ""

127.0.0.1:6379> CONFIG GET appendonly
1) "appendonly"
2) "no"

I tried enabling snapshotting in a running Redis instance using command like:

127.0.0.1:6379> CONFIG SET save "60 10"
OK

It worked and Redis started saving data to disk, but I didn't find any dedicated option in this chart to configure that. Probably need to put it into a redis.conf file.

[chychkan]

After playing with it a bit more I was able to enable snapshotting by adding top level property to my values.yml file:

configmap: |-
  dir /data
  save 60 10

[juan131]

We should be enabling AOF by default. As you can see, we're already doing it on the default Docker image's setup.sh

https://github.com/bitnami/bitnami-docker-redis/blob/master/5.0/debian-9/rootfs/libredis.sh#L352

However, this script is not being execute in the chart since we're rewriting the Docker image's command:

https://github.com/helm/charts/blob/master/stable/redis/templates/redis-master-statefulset.yaml#L67

I just created a PR to enable AOF by default.

[naseemkullah]

Thanks, is it true that vanilla redis comes with rdb enabled and aof disabled? I feel like mimicking vanilla redis defaults is better

[juan131]

Hi @naseemkullah

In the links below you can find the advantages/disadvantages of using RDB or AOF persistence:

https://redis.io/topics/persistence#rdb-advantages
https://redis.io/topics/persistence#rdb-disadvantages
https://redis.io/topics/persistence#aof-advantages
https://redis.io/topics/persistence#aof-disadvantages

To me the key to use AOF instead of RDB is the one below:

RDB is NOT good if you need to minimize the chance of data loss in case Redis stops working

As we know, pods are ephemeral. Therefore, when running Redis on K8s, whatever configuration is less prone to errors when Redis is suddenly stopped, must be the default configuration.

[naseemkullah]

Good point, I was actually thinking of that after posting my last comment. Just a few follow-up questions if you don't mind:

1. Would a pre-stop hook to write perform a SAVE and thus writing a dump.rdb upon SIGTERM solve this issue?

2. Is it possible to have both RDB and AOF enabled by default? According to documentation links provided it seems feasible.

3. For those of us who use redis as a DB rather than just as a cache, is there anyways to backup data with only AOF enabled?

[juan131]

Would a pre-stop hook to write perform a SAVE and thus writing a dump.rdb upon SIGTERM solve this issue?

Do you refer to a container hook? Please note that Hem hooks won't be suitable for this situation.

Is it possible to have both RDB and AOF enabled by default? According to documentation links provided it seems feasible.

It's not possible. See https://redis.io/topics/persistence#interactions-between-aof-and-rdb-persistence

is there anyways to backup data with only AOF enabled?

You can always run the "redis save" command

[naseemkullah]

Hi @juan131

1) Yes I meant a container lifecycle pre-stop hook.

2) Thanks but I am confused because according to https://redis.io/topics/persistence#ok-so-what-should-i-use you can use both.

The general indication is that you should use both persistence methods if you want a degree of data safety comparable to what PostgreSQL can provide you.

[juan131]

RDB will be ignored if Redis is restarted. I wouldn't enable a second persistence method by default since it can overload Redis.

In the case both AOF and RDB persistence are enabled and Redis restarts the AOF file will be used to reconstruct the original dataset since it is guaranteed to be the most complete

Regarding the pre-stop hook, I'm okay with implementing a way to enable the hook when an optional parameter is set. Please feel free to create a PR adding support for it.

[naseemkullah]

Just to confirm. AOF will be the only form of persistence enabled by default in the chart, and to enable RDB one will have to adjust the chart values?

FWIW in older versions of the helm chart, RDB seemed enable by default.