Horizon: Using Horizon with multiple servers and queues

[crash13override]

Hi,
I'm really looking forward to trying Horizon but I have an issue I can't get my head around.

I have one web server that handles all the users requests and that has its own default queue, but then I also have a "video encoding" server that has the exact same repo like the webserver, but it listens to a different "encode" queue for retrieving the jobs to be done.

So the workflow is the following: a user tells the webserver to encode a video. The webserver adds a couple small tasks in the "default" queue and then it adds the big encoding job to the "encode" queue. The encoding server listens to the "encode" queue and then finally does the job and notifies back the web server via the "default" queue that the job has been completed.

I currently run all this with an Amazon SQS queue driver, so it allows me to keep the queues separated from the actual servers, since they have to be talking to each other. And everything runs smoothly.

The big question is: can I achieve the same thing with Horizon? I tried creating a separate Redis DB hosted by Redislabs (https://redislabs.com/products/redis-cloud/) and I can connect to it without problems, but my problem is: how can I tell the webserver to run just the "default" queue, and the encoding server to run the "encode" queue? Should I run Horizon on both server, or just one of the two? I first thought to run it on both servers, but then I guess some conflicts might happen since they are both sharing the same Redis DB.

Is Laravel Horizon meant to work just with local queues or can it orchestrate more than one server in some way? Am I missing something?

I tried to look into the documentation but it doesn't talk about this aspect.

Thanks a lot and congrats for the amazing work!!

[crash13override]

Never mind, I figured it out in the end.

I use the .env to set the specific queue each server should run and I run "artisan horizon" on every server. At first I didn't get that Horizon would monitor all the queues across all servers, while processing only the queue that is in config file for each server.

Hope it helps for someone else in the same situation I'm in.

[pmartelletti]

@crash13override so what you're doing is in horizon.php the following:

'queues' => env('YOUR_ENV_QUEUE')

?

I guess this only works with a separate redis server, not localhost redis, right? As if not, the dispatched jobs in the main server won't be accesible by other servers.

[crash13override]

yes, and I personalize the processes and the tries as well. Remember to put it in an array as well.

here is my config

'environments' => [
'production' => [
'supervisor-default' => [
'connection' => 'redis',
'queue' => [env('QUEUE_NAME', 'default')],
'balance' => 'simple',
'processes' => env('QUEUE_PROCESSES', 10),
'tries' => env('QUEUE_TRIES', 3),
],
],
'local' => [
'supervisor-default' => [
'connection' => 'redis',
'queue' => [env('QUEUE_NAME', 'default')],
'balance' => 'simple',
'processes' => env('QUEUE_PROCESSES', 10),
'tries' => env('QUEUE_TRIES', 3),
],
],
],

[pmartelletti]

@crash13override and what about horizon dashboard? does the data looks ok? you can look it from any of the servers?

thanks.

[crash13override]

@pmartelletti yes, the data looks all right and I can look it from any server in a consistent way.

The only issue I have is this one #128 , but I don't think that is related to having multiple server sharing the same queue, but more to the fact that the queue jobs take long time.