Horizon: Autoscaler is too slow (or run too seldom)

[lechbaczynski]

• Horizon Version: 3.3.2
• Laravel Version: 5.8.34
• PHP Version: 7.2.19
• Redis Driver & Version: predis 1.1.1
• Database Driver & Version: mysql

Description:

Supervisor with 6 queues and 1800 processes (3 other supervisors running too, but with not much workers and jobs).
Starts with 300 processes, then the numbers go down.

There is a lot of jobs in one queue, yet the number of workers rises very slowly. It may take over half an hour to an hour to rise. Although all other queues are empty.
(in the image below there were some jobs in other queues, but even if other queues are empty, the number of workers rises slowly - here it rose to 1300, but it took long time)

Steps To Reproduce:

Make a supervisor like

'supervisor-1' => [
                'connection' => 'redis',
                'queue' => [
                    'one', 'two', 'three',
                    'four', 'five', 'six'
                ],
                'balance' => 'auto',
                'processes' => 1800, // even multiply of number of queues
                'min_processes' => 2,
                'tries' => 5,
                'timeout' => 550,
                'memory' => 512,
            ],

The number of workers will decrease from 300.
Add 100.000 jobs in one queue, and make each job run for random(5,200) seconds.
The queue workers number will rise slowly.

[driesvints]

Not sure what's going on here. Maybe @themsaid can help?

[lechbaczynski]

Video with one queue, number of workers and tasks. Other queues were rather empty at that time (animated GIF)

[lechbaczynski]

Another strange thing: 5 queues, 2000 processes, min_processes 50.

            'processes' => 2000, // even multiply of number of queues
            'min_processes' => 50,

3 queues basically idle, two have thousands of jobs.
The number of processes in the bigger queue grow, but in the smaller shrinks - although it is still not using those 2000 allowed processes, only about 800. Shouldn't the number of processes in the smaller queue rise too?

Images in chronological order

1)

2)

3)

[SDekkers]

The function scale() in AutoScale.php is limited to 1 process increase or decrease per 3 seconds, perhaps for these amounts of processes it should be in increments of a bit more..

[tomswinkels]

I have ask the same thing in https://github.com/laravel/horizon/issues/622

[LasseRafn]

Percentage increase/decrease maybe an idea? Example: 5% If only 5 processes it would simply be rounded up to 1 (leading back to default) but if 1000 processes, it would work in intervals of 50.

[themsaid]

Scaling happens every 3 seconds and each time it starts/ends 1 process, is that what you call slow? Not sure what expectation you have.

[taylorotwell]

It's unclear whether there is anything actionable in this issue. It is basically "I don't like the way autoscaling works". If anyone has a better solution it can be PR'd at any time.

[trevorgehman]

What about being able to set the scale amount and/or the time between checking whether to scale in the config?