Horizon: No supervisors / processes shown
- Horizon Version: 3.2.1
- Laravel Version: 5.8.19
- PHP Version: 7.3.4
- Redis Driver & Version: predis/phpredis 1.1.1
- Database Driver & Version: mysql 5.7
Description:
No supervisors / processes shown.
Steps To Reproduce:
After upgrading from v3.1.2 to v3.2.1 it fails to display the active supervisors and its associated processes. If I roll back to v3.1.2 it displays it correctly.
I have made sure to run the horizon:assets command, but it does not change anything.
Any suggestions on how to fix this?
pactode
All 12 comments
Heya, thanks for reporting.
I'll need more info and/or code to debug this further. Please post relevant code like models, jobs, commands, notifications, events, listeners, controller methods, routes, etc. You may use https://paste.laravel.io to post larger snippets or just reply with shorter code snippets.
Thanks!
driesvints
on 4 Jun 2019
Hey @driesvints
The horizon config file can be found here:
https://paste.laravel.io/8496fc9d-3a1e-4a04-9c07-a94d9534c6a3
The horizon service provider can be found here:
https://paste.laravel.io/ae853db5-97fe-4133-8966-52d36a0dcea1
I checked the logs and it is filled with this error:
https://paste.laravel.io/94f23bba-30ba-4f18-ba34-3a9ac0ee84c2
The jobs are running fine in the background, but it does not display it correctly on the dashboard. The recent jobs tab works fine. I've tried it on a couple of projects and the results are the same.
pactode
on 4 Jun 2019
UPDATE
I just updated to v3.2.2 to test if that did the trick. Unfortunately, no.
So I spent some time debugging the error below:
[2019-06-04 18:47:13] local.ERROR: A non-numeric value encountered {"exception":"[object] (ErrorException(code: 0): A non-numeric value encountered at /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Collection.php:1766)
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Collection.php(1766): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError(2, 'A non-numeric v...', '/var/www/html/v...', 1766, Array)
#1 [internal function]: Illuminate\\Support\\Collection->Illuminate\\Support\\{closure}(0, Array)
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Collection.php(1501): array_reduce(Array, Object(Closure), 0)
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Collection.php(1767): Illuminate\\Support\\Collection->reduce(Object(Closure), 0)
#4 /var/www/html/vendor/laravel/horizon/src/Repositories/RedisSupervisorRepository.php(130): Illuminate\\Support\\Collection->sum(Object(Closure))
#5 [internal function]: Laravel\\Horizon\\Repositories\\RedisSupervisorRepository->Laravel\\Horizon\\Repositories\\{closure}(Object(Predis\\Pipeline\\Pipeline))
#6 /var/www/html/vendor/predis/predis/src/Pipeline/Pipeline.php(211): call_user_func(Object(Closure), Object(Predis\\Pipeline\\Pipeline))
#7 /var/www/html/vendor/predis/predis/src/Client.php(445): Predis\\Pipeline\\Pipeline->execute(Object(Closure))
#8 /var/www/html/vendor/predis/predis/src/Client.php(396): Predis\\Client->createPipeline(NULL, Object(Closure))
#9 /var/www/html/vendor/predis/predis/src/Client.php(418): Predis\\Client->sharedContextFactory('createPipeline', Array)
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(114): Predis\\Client->pipeline(Object(Closure))
#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(214): Illuminate\\Redis\\Connections\\Connection->command('pipeline', Array)
#12 /var/www/html/vendor/laravel/horizon/src/Repositories/RedisSupervisorRepository.php(139): Illuminate\\Redis\\Connections\\Connection->__call('pipeline', Array)
#13 /var/www/html/vendor/laravel/horizon/src/Supervisor.php(354): Laravel\\Horizon\\Repositories\\RedisSupervisorRepository->update(Object(Laravel\\Horizon\\Supervisor))
#14 /var/www/html/vendor/laravel/horizon/src/Supervisor.php(308): Laravel\\Horizon\\Supervisor->persist()
#15 /var/www/html/vendor/laravel/horizon/src/Supervisor.php(267): Laravel\\Horizon\\Supervisor->loop()
#16 /var/www/html/vendor/laravel/horizon/src/Console/SupervisorCommand.php(92): Laravel\\Horizon\\Supervisor->monitor()
#17 /var/www/html/vendor/laravel/horizon/src/Console/SupervisorCommand.php(67): Laravel\\Horizon\\Console\\SupervisorCommand->start(Object(Laravel\\Horizon\\Supervisor))
#18 [internal function]: Laravel\\Horizon\\Console\\SupervisorCommand->handle(Object(Laravel\\Horizon\\SupervisorFactory))
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(App\\Application), Array, Object(Closure))
#22 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(576): Illuminate\\Container\\BoundMethod::call(Object(App\\Application), Array, Array, NULL)
#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#24 /var/www/html/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#26 /var/www/html/vendor/symfony/console/Application.php(921): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#27 /var/www/html/vendor/symfony/console/Application.php(273): Symfony\\Component\\Console\\Application->doRunCommand(Object(Laravel\\Horizon\\Console\\SupervisorCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#28 /var/www/html/vendor/symfony/console/Application.php(149): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#30 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(133): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#31 /var/www/html/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#32 {main}
"}
Logging the $supervisor->workerStats() returns:
[2019-06-04 18:47:13] local.DEBUG: [{"cpu":0.64,"mem":"root"},{"cpu":0.65,"mem":"root"},{"cpu":0.66,"mem":"root"}]
The error is caused by the following call in Laravel\Horizon\Repositories\RedisSupervisorRepository on line 129:
'mem' => round($workerStats->sum('mem')),
@driesvints I hope this explains the issue in more detail and thanks for looking into it.
pactode
on 4 Jun 2019
What operating system are you using?
Can you paste the results of ps axo %cpu,%mem,command?
bonzai
on 5 Jun 2019
I'm using Docker running Alpine Linux.
The specific image is: phpearth/php:7.3 - Link
Results of ps axo %cpu,%mem,command:
PID USER TIME COMMAND
1 root 0:00 {horizon.sh} /bin/sh ./utils/horizon.sh
15 root 0:00 crond -b
23 root 0:00 php artisan horizon
30 root 0:00 /usr/bin/php7.3 artisan horizon:supervisor 422424f4550b-FABH:supervisor-1 redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --balance=simple --max-processes=3 --min-processes=1 --nice=0
31 root 0:00 /usr/bin/php7.3 artisan horizon:supervisor 422424f4550b-FABH:supervisor-2 redis --delay=0 --memory=128 --queue=notifications --sleep=3 --timeout=60 --tries=3 --balance=simple --max-processes=1 --min-processes=1 --nice=0
65 root 0:00 /usr/bin/php7.3 artisan horizon:work redis --delay=0 --memory=128 --queue=notifications --sleep=3 --timeout=60 --tries=3 --supervisor=422424f4550b-FABH:supervisor-2
71 root 0:00 /usr/bin/php7.3 artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=422424f4550b-FABH:supervisor-1
72 root 0:00 /usr/bin/php7.3 artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=422424f4550b-FABH:supervisor-1
73 root 0:00 /usr/bin/php7.3 artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=422424f4550b-FABH:supervisor-1
459 root 0:00 bash
696 root 0:00 [php]
743 root 0:00 ps axo %cpu,%mem,command
It seems the output of ps is incompatible / unexpected to what flags are given. What flavor/version of ps does Alpine linux use?
The output should look something like this:
%CPU %MEM COMMAND
0.0 0.1 /sbin/init
0.0 0.0 [kthreadd]
0.0 0.0 [kworker/0:0H]
…
mfn
on 5 Jun 2019
Looks like it uses BusyBox v1.29.3
pactode
on 5 Jun 2019
TL;DR
Workaround: Install procps in Alpine Containers:
apk add procps
Full:
I can confirm this on on the official php-alpine image: php:fpm-alpine3.9.
It's the same output and uses the same BusyBox Version as @pactode (but alpine3.9 instead 3.8 based). After installing procps, it surely works. But adding packages seems against the general idea for alpine. Unfortunately, BusyBox's version of ps doesn't support memory or cpu usage right now (Source). Nearest possible solution I found was:
procps 3.3.15-r0:
ps -Ao pcpu,pmem,args
0.1 0.9 /usr/local/bin/php artisan horizon
0.3 0.9 /usr/local/bin/php artisan horizon:supervisor php-worker-85f7c868d7-m2z2n-cx0i:supervisor-1 redis --delay=0 --memory=128 --queue=default,low --sleep=3 --timeout=60 --tries=3 --ba
0.0 1.0 /usr/local/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=lga-php-master-85f7c868d7-m2z2n-cx0i:supervisor-1
0.0 1.0 /usr/local/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=low --sleep=3 --timeout=60 --tries=3 --supervisor=lga-php-master-85f7c868d7-m2z2n-cx0i:supervisor-1
0.0 0.0 ash
0.0 0.0 ps -Ao pcpu,pmem,args
BusyBox v1.29.3 (no support for _pcpu_ and _pmem_ column):
ps -Ao args
/usr/local/bin/php artisan horizon
/usr/local/bin/php artisan horizon:supervisor php-worker-785d997b6b-nq5vk-wkSW:supervisor-1 redis --delay=0 --memory=128 --queue=default,low --sleep=3 --timeout=60 --tries=3 --balance=simple --max-processes=2 --min-processes=1 --nic
/usr/local/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=lga-php-worker-785d997b6b-nq5vk-wkSW:supervisor-1
/usr/local/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=low --sleep=3 --timeout=60 --tries=3 --supervisor=lga-php-worker-785d997b6b-nq5vk-wkSW:supervisor-1
ash
ps -Ao args
kohlerdominik
on 5 Jun 2019
@kohlerdominik Thanks for the workaround. It does work when installing that package.
I think it's okay to install procps, but I suggest adding a note about it in the documentation as it worked without it before adding the latest feature to horizon. Perhaps even in the upgrade notes?
@driesvints What do you think?
pactode
on 5 Jun 2019
@pactode adding a note to the docs is certainly something we can do. But getting this to work in general would be better still. Maybe disable the cpu/memory stuff if no valid ps/procps binary is installed.
Thanks to everyone helping out here 👍
driesvints
on 5 Jun 2019
We've reverted the original PR that implemented the CPU and memory tracking so this problem should be fixed in the next release.
See my answer here: https://github.com/laravel/horizon/pull/589#issuecomment-502085735
driesvints
on 14 Jun 2019
I'm noticing this issue after updating to v3.4.2. Did something change?
shengslogar
on 15 Nov 2019
Related issues
pmartelletti
·
4Comments
meathanjay
·
3Comments
dmitryuk
·
3Comments
vesper8
·
5Comments
RicardoRamirezR
·
3Comments
Most helpful comment
TL;DR
Workaround: Install procps in Alpine Containers:
Full:
I can confirm this on on the official php-alpine image: php:fpm-alpine3.9.
It's the same output and uses the same BusyBox Version as @pactode (but alpine3.9 instead 3.8 based). After installing procps, it surely works. But adding packages seems against the general idea for alpine. Unfortunately, BusyBox's version of ps doesn't support memory or cpu usage right now (Source). Nearest possible solution I found was:
procps 3.3.15-r0:
BusyBox v1.29.3 (no support for _pcpu_ and _pmem_ column):