Horizon: missing TTL in Redis (records kept in Redis forever)

[jankremlacek]

sometimes Horizon Redis entries (_hashes_) are stored without TTL specified. It causes entries are kept in Redis forever. And memory overload.

Do not know real reason what causes it now, but will investigate more. But it's happening repeatedly. Once it starts happening, it happens always until Redis + Horizon restart. Installed on standard Forge server (_no modifications_) ..

Any idea? thank you.

[themsaid]

Can you share some information about your setup? How many workers? where's redis hosted? etc..

[pmartelletti]

I think it depends on your redis configuration. We're having the same issue, but we changed our config to:

maxmemory-policy volatile-ttl

and jobs started to disappear after after their TTL IF memory limit was reached. Or at least that's the workaround we are using ATM.

[jankremlacek]

• Redis is on the same machine as everything else (_just one single machine for all, so no different time etc._)
• processes: 10
• balance: simple

no failed jobs so far ..

but it seems to me more like redis behaviour (setup) when heavy use of RAM (_sorry if I missed something_)

[bryceadams]

@jankremlacek did you ever figure out why the TTL wasn't set? Same issue and had to change the maxmemory policy to be allkeys-lru but would like to figure out why the TTL isn't being set 🙂

[jankremlacek]

@bryceadams, the only I have found is that it happened when Redis came out of RAM and started store keys at disk. For me the only solution was use enough RAM machine to handle amount of jobs. Does not sound like a solution, but at least it's not overloading the machine now.

[bryceadams]

Thanks @jankremlacek! Do you find that the old job data is automatically being cleared out after some time?

[jankremlacek]

@bryceadams the old data seems persistent in Redis, no TTL, you should remove them manually ..

[bryceadams]

Ah, frustrating. No problem, I will just use a policy that removes them
automatically. Hopefully won't lose new/recent data. Thanks!

On Tue, Nov 14, 2017, 19:34 Jan Kremlacek [e-mail] wrote:

@bryceadams https://github.com/bryceadams the old data seems persistent
in Redis, no TTL, you should remove them manually ..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/laravel/horizon/issues/124#issuecomment-344182970,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABdiQsVUruDUR4k0iyIYvqg8YNscG-j3ks5s2VCQgaJpZM4PP7Nc
.

[nyancodex]

This bug is still there? Because my redis service just crashed and I need to manually configure maxmemory-policy to remove expired data.

[halaei]

I think this issue has to do with the fact that a lot of time we set the key in one redis command then in another command we set the expiry time. If for some reason the php script crashes after the set operation but before the expire, then the key will be eternal.
You may think this is a very unlikely scenario, but the lower the timeout for the jobs, the higher the chance of this kind of memory leak.

I guess this issue must be reopened @themsaid @jankremlacek

[carltondickson]

@halaei Thanks for posting this, feels like we're experiencing a similar issue on Heroku.

Because the default maxmemory is noeviction our Redis instance is now full of small 2KB jobs with no expiry time set and clearing them out is proving to be a painful process.

Trying to do this with Redis Insights with bulk actions but it's extremely slow to delete keys with a prefix for the jobs we want to delete (e.g. horizon:143*)

[graemlourens]

@carltondickson are you sure that this is still an issue? the 'trim' settings of current horizon (this thread is pretty old and i'm not sure if those already existed then?) work very well for us, and we process >100k jobs per hour. We do trim very heavily (as we in any case do not rely on those logs, we have our own in an ELK stack).

[carltondickson]

@graemlourens we're on a much older version of Horizon at the moment (upgrade is in progress)
So will have to wait and see if that solves our issues.

[graemlourens]

@carltondickson ok, get back to this issue / me if you're having issues. We heavily use horizon and therefore not only want to know about any issues anybody is having, but also share our insights into high load horizon usage.