Horizon: maxProcesses and minProcesses does not work

[tomswinkels]

• Horizon Version: 3.2.3
• Laravel Version: 5.8.23
• PHP Version: 7.2
• Redis Driver & Version: predis/phpredis 1.1.1

We have the next config;

            'Default' => [
                'connection' => 'redis',
                'queue' => ['Default'],
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 8,
                'tries' => 3,
                'sleep' => 1,
                'timeout' => 60,
                'delay' => 0
            ],

There are always running 8 processes for default, even when there are no jobs. On the dashboard and on the server processes list.

[tomswinkels]

When i check the process list on the server i see the min and max settings in the supervisor command. (see screen) But i see always running 8 horizon workers. (other screen)

[ronnievisser]

I got this exact same problem! Is there a solution for already?

[francescomalatesta]

Same problem here :( any ideas?

[SDekkers]

If you look at AutoScaler.php on Line 100, you can see that if the time to clear is 0, it'll always return the max processes

We could add another line so the function would be something like this:

protected function numberOfWorkersPerQueue(Supervisor $supervisor, Collection $timeToClear)
{
    $timeToClearAll = $timeToClear->sum();

    return $timeToClear->mapWithKeys(function ($timeToClear, $queue) use ($supervisor, $timeToClearAll) {
        return $timeToClearAll > 0 && $supervisor->options->autoScaling()
            ? [$queue => (($timeToClear / $timeToClearAll) * $supervisor->options->maxProcesses)]
            : ($timeToClearAll == 0) ? [$queue => $supervisor->options->minProcesses]
                : [$queue => $supervisor->options->maxProcesses / count($supervisor->processPools)];
    })->sort();
}

But I don't know if that would have any side effects.

EDIT: after running my own tests, it scales down to the minimum amount of processes, but it doesn't scale back up, even with 1 hour of jobs in the queue, needs more investigation.

[tomswinkels]

@driesvints check this comment :)

[SDekkers]

I see the CI failed on 2 tests, but the 2 tests that fail:

test_balancing_a_single_queue_assigns_it_the_max_workers
and
test_balance_stays_even_when_queue_is_empty

both assume the amount of processes goes up to the maximum, when there are no jobs to handle, so either this change is undesirable, or the tests should be amended..

[driesvints]

Thanks for investigating. Let's see how the pr goes.

[SDekkers]

I have re-submitted my PR which I'm using on my production environment, this avoids having my queue instances running max processes continuously, while the min processes would suffice under normal load

[ronnievisser]

I would love to see this being merged. It’s quit a bug in my opinion.

Ronnie
On 20 Aug 2019, 09:37 +0200, SDekkers [e-mail], wrote:

I have re-submitted my PR which I'm using on my production environment, this avoids having my queue instances running max processes continuously, while the min processes would suffice under normal load
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[tomswinkels]

@SDekkers Really nice, today i have test it and it is great!

But i have some questions:

• When the balance are false the min and max processes are not worked?

Example:

minProcesses 1
maxProcesses 8

The process is started with 1 process, when there are a lot of jobs than 2 processes, but there are very high processes.

Maybe it is better that change processes directly to the max processes (8)?

Now it is scaling with +1 process.

[SDekkers]

@tomswinkels it's the way Laravel Horizon is built, every 3 seconds it polls to check if the load is high enough to scale up, the function scale() scales processes up or down with 1. If the load is high for lots of time it should eventually scale up to the max processes (in your example, in 21 seconds)

Alternatives could be increasing your minProcesses, or a new PR that allows adjusting the scale up/down increments (from 1 to 3 or 5)

[tomswinkels]

@SDekkers thnx for the reaction.

And my other question? It don't works when the blance = false for the job.

[SDekkers]

Well, if you specify that balance = false, it kind of disables the auto balancer function..

https://github.com/laravel/horizon/blob/3.0/src/SupervisorOptions.php#L125
This function is called to determine if autoscale should work at all