Horizon: [FORGE INSTALLATION] MISCONF - Redis is configured to save RDB Snapshots.

[brianclogan]

On a forge server, I am getting an error:

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Trace Information

Screenshot of Page

Screenshots of Sentry

Basic Info

Extended Info

Is there something I can do with the forge interface to fix this?

Update 1

It appears to only be happening on the /horizon/api/stats endpoint.

[brianclogan]

@themsaid hopefully it is a small fix, if I have to rebuild the server, that will be a pain, since i have about 300GB of data stored on it in the Database...

[themsaid]

Sorry but this isn't an issue with Horizon or Forge, you'll need to check your Redis configurations and see why it's unable to write to disk. Maybe out of space?

[brianclogan]

@themsaid

Here is the dump from the info memory

forge@server:~$ redis-cli
127.0.0.1:6379> info memory
# Memory
used_memory:1499680608
used_memory_human:1.40G
used_memory_rss:1897467904
used_memory_rss_human:1.77G
used_memory_peak:1741631256
used_memory_peak_human:1.62G
total_system_memory:15371208192
total_system_memory_human:15.80G
used_memory_lua:47104
used_memory_lua_human:46.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.27
mem_allocator:jemalloc-4.0.3
127.0.0.1:6379>

[Lorempn2]

Suddenly got the same error overnight.

[brianclogan]

@themsaid getting the error again, different server, 75GB of space available.

[alexstone]

Also having this issue and can't figure out the cause.

[themsaid]

Can you check your redis logs for more information?

[brianclogan]

@alexstone @themsaid what I found is it is actually a RAM issue, when RAM usage was around 95% it starts to fail, what sucks is you can't clear it easily either. What I found that works is restarting the server, as soon as it is up, going into redis-cli, and doing flushall, granted, you loose historical data, but it fixes the issue.

@themsaid, maybe using the database connection for historical information or snapshots, rather than REDIS would significantly help. When taking snapshots too often, and running tens of thousdands of jobs an hour, it was having issues keeping all the information in memory.

[themsaid]

given that you have total_system_memory_human:15.80G, I wonder how 95% of that was eaten by horizon, we have a heavy test server and it barely reaches 2G

[alexstone]

@darkgoldblade01 @themsaid Yes, upon going through my logs, I found that it's a memory issue too. I also can't figure out what's going on. The processes aren't expensive at all, but there are a lot of them.

[gstoa]

I ran into this same error today and found immediate success as @darkgoldblade01 indicated:

• reboot the server
• redis-cli
• flushall

At least until I loaded up the cache again then it errored again with the same MISCONF message of not being able to persist to disk. I increased the amount of storage on my EC2 volume, so far, so good. If this is the fix, this really has nothing to do with Horizon, and more to do with Redis and their strange error message.

[stevenwaskey]

For those of you arriving here while Googling the Redis MISCONF error listed in OP's initial comment, please see the following comment: GitHub Redis MISCONF

For the sake of brevity, here you are...
Thx to @summerblue

What u should be doing is:

# redis-cli
127.0.0.1:6379> CONFIG SET dir /data/tmp
OK

127.0.0.1:6379> CONFIG SET dbfilename temp.rdb
OK

127.0.0.1:6379> BGSAVE
Background saving started

Please Make sure /data/tmp has enough disk space.

Humble Disclaimer: This is not Laravel, or Horizon, or Forge, or any other code you brought to the party .... This is a system daemon configuration for your Redis Server that's screaming out for your attention.

[jasperf]

I hit this thread looking up information on how Forge sets up Redis normally storage wise. Does it use RDB or AOF, does it do custom x no of snapshots. Ran into missing failed queue error not being able to add to table failed jobs as it was missing and wondered why Redis could not be used. If it is all ephemeral then I understand. If not, why use MySQL tablet to store failed jobs and not Redis where Horizon shows them already?