Horizon: Laravel horizon with AWS Redis doesn't release queues.

[KevinHN]

Hi all,

I have used Laravel horizon with AWS Redis, it works but got problems as below.

I have created some queues in my code such as:
->onQueue('high') ,
->onQueue('medium') ,
->onQueue('local')

I have tried horizon with the Redis installed on my local, it works fine.
I have tried horizon with AWS Redis, and after Job processed, the queues still exists (i'm using LLEN, LINDEX to check). I have also tried with php artisan queue:listen, all the queues are released. So, the problem is horizon, not aws redis.

How can I resolve this problem, any workaround for this?
Thanks in advance for your help!

[addgod]

I seem to be experiencing something similar.
Only the jobs does not even seem to process, they just seems to end in some limbo state, where I can see them in horizon, but they does not seem to be executed.

[KevinHN]

Hello @taylorotwell
Do you have any idea for this?

[driesvints]

What do you exactly mean with "all the queues are released"? Can you please provide more info? Code samples, a test app, exact AWS setup (Which queue type etc)...

[KevinHN]

@driesvints

I mean that: after process with horizon I have checked on the redis, the queues still existing. This issue happens only with horizon ( queue:listen works fine).

You can simple re-produre by creating a redis service on the AWS ElastiCache page.

1. Goto ElastiCache, select Redis.
2. You will see a button 'Create', click on this button.
3. Select 'Cluster engine' redis, anything else by default.

Now put some queues to above redis, you will see the problem as I said.

[driesvints]

Heya. Thanks for responding. I'll try to have a look at a later time.

[wsamoht]

@KevinHN Are you still having issues with this? I am about to deploy ElasticCache and Horizon and am wondering if I should think twice about using ElasticCache.

[KevinHN]

@wsamoht,
Yes, I'm still having this issue. I hope you check it as soon.

[halaei]

@KevinHN do you have more than one php servers interacting with queues on your aws redis?

[KevinHN]

@halaei yes, but each server interacting with difference queues.
Example: server 01 interacting with queue name 'high', server 02 interacting with queue name 'medium', server 03 interacting with queue name 'low,mail'.

[halaei]

The causes I have in mind (beside an unknown bug in horizon) are those that lead more than one workers picking a job:

1. The servers' times are not in sync. So one server might regard a reserved jobs as expired although it has been picked by one server just a second ago.
2. The amount of time is required for a job to get processed is sometimes more than what you expect, and they get expired and handled by another worker(s).

Maybe you can prove I am wrong if you set tries to 1 and see no MaxAttemptsExceededException exceptions, but the bug is still reproduced.

[halaei]

Something else come to mind is that if you enabled cluster mode or not. I don't think there is any point in enabling that for horizon. Maybe it only introduce bugs.

[KevinHN]

@halaei Sorry for late, I'm quite busy.

I have tried with only 1 worker, and keep only one redis node but the issue is still happen with horizon (not queue:listen).
With an Redis server setup on the EC2 instance (without cluster), it works fine even 6 workers interacting with queues at the same time (~1k CPM)

[halaei]

In case you need to know why horizon might not work well on clusters the answer is hash tags.
If you really want to run horizon on a redis cluster with sharding (without actually any benefit from sharding), make sure config('horizon:prefix') is set to a hash tag, e.g. '{horz}:' instead of 'horz:', then make sure you use the same hash tag for all of the queue names, e.g. '{horz}-high' instead of 'high'.

[mfn]

@halaei wow, that's very interesting!

But can you explain what are the operations which fail iff the keys are sharded, then?

Just trying to get a better understanding here, thanks upfront!

[KevinHN]

@halaei wow, I got it now.

The workaround by adding {key} is NG with my case.

queue:listen (and queue:work) works fine, so I hope that the horizon implementation supports cluster like the queue:listen.

[halaei]

But can you explain what are the operations which fail iff the keys are sharded, then?

Any Redis command that operates on keys in multiple shards will fail. I think the failure is not silent and the server responds with error, unless something is changed in redis behavior that I don't know.

By the way, even without horizon, hash tag for queue names is a MUST because every Redis queue is actually a set of 3 keys which will be accessed by a single LUA script. So if you have a queue system on shards that works fine even without hash tags, that is just because you are lucky enough and the 3 keys are not distributed across multiple shards.

[taylorotwell]

Horizon does not work with Redis Cluster and likely never will. Even if it did work with Clusters, you wouldn't be taking advantage of the clustering because Lua operations can't operate on multiple shards in one operation. And, for Horizon to work properly, our Lua scripts have to perform operations on multiple keys in one atomic transaction.