Horizon: Horizon does not spawn correct number of processes

[duynguyen0511]

Horizon : v2.0.0
Laravel: v5.7.20

This is my horizon config

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['urgent', 'medium', 'low'],
            'balance' => 'auto',
            'processes' => 4,
            'tries' =>1,
        ],
    ],

I have 3 queues and 4 processes. Horizon only created 3 process , 1 for each queue. I expected to see 2-urgent, 1-medium, 1-low

When I change processes to 5, Horizon created 5 process.I get 2-urgent, 2-medium, 1-low.
When I change processes to 6, Horizon created 5 process I get 2-urgent, 2-medium, 2-low.

When I change processes to 7, I get 2-urgent, 2-medium, 2-low. I expected 3-urgent, 2-medium, 2-low
etc ...

No idea why it does not work when processes is 4 or 7 or 10. This behaviour is occurring on Homestead as well as cloud hosting I am using.

[driesvints]

Can you update to the latest Horizon version and see if the problem persists?

[duynguyen0511]

I am running Horizon v2.0.3 and still have this problem

[driesvints]

I currently don't have the time to look into this so appreciating any help with this. Otherwise I'll look into this soonish.

[francislavoie]

This seems like it's a function of how many queues you have defined. The math for balancing out across the queues is probably incorrect.

Seems like the logic for that is over here: https://github.com/laravel/horizon/blob/384c84512cdf929886343c2e498f8dc4180d8728/src/AutoScaler.php#L98-L100

[f3cp]

It its any help, I found that when there were 8 extra processes it would allocate them correctly to my queues on load.

E.g. When I set 14 queues and 22 processes (8 extra processes), 22 processes would be started with 1 process for each queue plus an additional process for the first 8 queues. While if I set 21 or 23 processes (7 or 9 extra processes), for some reason it would only start up 14 processes allocating 1 to each queue.

Testing a few different queue/process combination, 8 extra processes always seemed to work as expected while other amounts didn't.

I should note that if I did have a number of extra processes other than 8 such that only 1 per queue was initially allocated, additional processes would still be added to a queue under high demand, despite the fact that these extra processes weren't initially allocated or showing in the initial process counts, so it seems the auto balancing is still occurring.

[driesvints]

What I can make out from the test below is that the autoscaler attempts to correctly scale the processes on multiple iterations. So I'm left wondering: does in your situations the problem also occurs after a bit of time has passed or only at the initial start?

https://github.com/laravel/horizon/blob/62ba971c2b180fe54dd3e0cde418c7181a481460/tests/Feature/AutoScalerTest.php#L16-L43

[duynguyen0511]

@driesvints the problem occurs at the beginning. The number of processes does not change/update unless I update the config file and restart Horizon.

Updated May/21: laravel/horizon v3.2.1 and laravel/framework v5.7.28 still have this issue

[Gummibeer]

It's the same here. I also have 3 queues and if I define 1 process more than a multiple of 3 it spawns only the max multiple of 3. For example 10 processes will spawn 3 workers for every queue.
The moment my queues get filled and the load is unbalanced it spawns all 10 processes.

For me this isn't bad because it prevents to stop the fourth process of one queue and start it for another one because there is one spare worker available for immediate usage.

So is this intended behavior?

[themsaid]

Hey everyone, I've sent a PR to deal with this issue: https://github.com/laravel/horizon/pull/627

[driesvints]

Fix will be added in the release for today.