Debian 9.4 (x86_64)
0.9.8 (amd64), release 21
Complete production server installation including Bind, Nginx, Apache, MySQL, PgSQL and dovecot+exim.
Since yesterday (2018/06/22), Vesta CP panel crashes repeatedly. It reports error 502 - Bad Gateway in nginx. In /var/log/vesta/* there are no errors about this crash reported, only in /var/log/vesta/nginx-error.log, there is mentioned:
2018/06/22 14:35:29 [error] 19059#0: *49 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: 109.238.216.159, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "taira.srv.getmedia.cz:8083"
While I restart vesta service with systemctl restart vesta, I am able to access the panel for a while, however, this is not stable and wont last long.
I don't have any idea while is this happening to me. This bug occurs on all my Vesta CP installations, so whole my cluster (3 nodes) is particullary down.
Do you have any idea what's gone wrong out there?
I've just bumped into this thread on forum https://forum.vestacp.com/viewtopic.php?t=17153
The problem is also mentioned on these:
That is exactly my case.
Hello, Is there any update on this? my server also having exact same issue.
I Can confirm this happens with me too..but only on Apache-less servers that have Nginx + PHP-FPM.
If I restart vesta service, it goes good but then the next day its crashed again. I guess may some cron job is doing this ?
I tried all default Cron Jobs for user admin... all went fine but after I ran
sudo /usr/local/vesta/bin/v-update-user-stats
Vesta Panel threw 502.
I restarted service and tried it again... this time it did not crash. So can't really say it is because of some crons. I have separate cron logs enabled but can't see any error recorded.
/var/log/vesta/nginx-error.log gives
[error] 95978#0: *25 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: _, request: "GET /list/user/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "IP:8908", referrer: "https://IP:8908/login/"
I am running it on port 8908 and not 8083
Same issue, any updates ?
P.S. Start happening couple days ago
I removed nginx to fix the issue, but I am thinking of moving to Virtualmin
From: korsar4eg notifications@github.com
Sent: Sunday, June 24, 2018 8:48 AM
To: serghey-rodin/vesta
Cc: Saurabh Sharma; Comment
Subject: Re: [serghey-rodin/vesta] 502 - Bad gateway on Vesta CP admin (#1625)
Same issue, any updates ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/serghey-rodin/vesta/issues/1625#issuecomment-399725613, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAv8YaOBGMGPtncbFynO2UwkkDAdbA3Dks5t_wUBgaJpZM4UzyCQ.
Same problem
What is a sys load of your servers?
sys load is low. I have got 1 website running on whole virtual server.
Vesta fails even after update.
I think problem is somewhere in the cron jobs, because it fails every x minutes after vesta service restart.
Error from nginx-error.log: 2018/06/24 17:45:22 [error] 29355#0: *29 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: (my ip), server: _, request: "GET /list/updates/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "mydomain.tld:8083", referrer: "https://mydomain.tld:8083/login/"
UPD: when appears 502 BAD Gateway, Vesta service status shows that vesta still active and running fine.
Same problem !
Same logs!
Same as gd-euss
My Vesta works as usual. Hetzner cloud, Ubuntu 16.04, nginx + php7.0-fpm
I can confirm that the problem is caused by this CRON script:
/usr/local/vesta/bin/v-update-user-stats
Everything works until the cronjob is called. Also, as mentioned before, all my servers have really low sys load.
I have the same issue here. Ubuntu 16 AWS (t2.small)
AMI ID: ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180522 (ami-963cecf4)
@treatys @mehargags
Suspending /usr/local/vesta/bin/v-update-user-stats did not help at all.
I suspended later all CRON jobs did not help at all either.
Changing pm from dynamic to ondemand and reboot help me
@Skamasle sorry, but what "PM" means?
@Anton98567, Process manager
Confirmed. I have the same issue on CentOS 7
Freshly updated VestaCP have the same bug.
[2018-06-24] Security fix release 0.9.8-22
Debian 9
I have same problem. Debian 9 vesta 0.9.8
I have the same problem
System:ubuntu 16.04
Same problem
Ubuntu 16.04.4 LTS
vesta 0.9.8-22
Guys, how to fix it?
tail -f /usr/local/vesta/log/nginx-error.log
[error] 1855#0: *44 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: _, request: "POST /api/index.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host:
Thanks for reporting about this issue. The fix will be available soon.
Same as on Ubuntu 16.04
Some error on centOs 7
tail -f /usr/local/vesta/log/nginx-error.log
[error] 816#0: *580 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: , request: "POST /softaculous/index.php?act=cp_api&api=serialize HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: ""
It seems like this issue is caused by default pm.max_requests value.
Please try out the following fix
sudo wget http://c.vestacp.com/tmp/php-fpm.conf -O /usr/local/vesta/php/etc/php-fpm.conf
sudo service vesta restart
I'm going to test it next few hours and push as update if everything is fine.
Thanks
Great. Everything seems to work fine again.
Thx @serghey-rodin
@serghey-rodin
Yes, this definitely works !
@serghey-rodin good job!
Oh no
It show nginx error 502 again
@serghey-rodin
It was commented value, which means that it gets default value, which is 0 according http://php.net/manual/en/install.fpm.configuration.php
You set it now to 0, which means that (if 0 was default value) maybe you changed nothing.
Ok let's try "ondemand" mode. Instructions are the same
sudo wget http://c.vestacp.com/tmp/php-fpm.conf -O /usr/local/vesta/php/etc/php-fpm.conf
sudo service vesta restart
yes, @serghey-rodin , it seems to work again ^_^
but only for a short time...
```
tail -f /usr/local/vesta/log/nginx-error.log
[error] 24539#0: *19 connect() to unix:/var/run/vesta-php.sock failed (111: Connection refused) while connecting to upstream, client: , server: _, request: "GET /list/user/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "", referrer: ""
I'm wondering does anybody else is trying to access vesta on 8083 port?
What nginx-access.log says?
Maybe someone constantly flooding it?
I got access to one machine with this issue.
FPM dies before anything access it through nginx.
It just die... without any log.
Crons are turned off, so it's not cron.
@dpeca can you check this file on your machine? Let me know if problem appears again
https://bugs.vestacp.com/projects/vesta-control-panel/files
Replace it and restart vesta
/usr/local/vesta/php/etc/php-fpm.conf
None of my 30 servers are affected (my vesta is hidden on custom port plus custom URL)
Friend gave me access to his server.
I will check there.
Fall down again.
Can confirm the bug.
system: Debian 9.4
resolves after rebooting vesta.
Happening here too, Centos7, Apache + nginx
With second patch php-fpm.conf again fail.
Don't work.
Ubuntu 16.04
also again 502 with new php-fpm.conf centos 6
I can report same issue, same logs. I have tried the patch posted by @serghey-rodin and will report back.
I had previously also tried some fixes but nothing seems to work. I use a custom port too, but can report this same issue on a friends installation with default port.
Thanks a lot for trying @serghey-rodin, I appreciate it :)
edit: Yeah it keeps dying and going back to 502. Restarting vesta service and accessing a panel link directly instantly crashes the process again, I have to open the root vestacp page to use the panel. And then randomly it dies again.
FPM conf file will not fix this, because vesta-fpm dies randomly, even nothing makes call via vesta-nginx, proccess just dissapear on random time.
Update: Seems to be running now? I did a system reboot after going through all my logs and fixing random issues like in my letsencrypt queue... Not sure why those would affect this issue. Could just be luck. I will report back if it goes offline again.
I have this problem for a week, change that to ondemand and some timeout more and reboot a server solve issue for me, if I not reboot I cant solve this in any way
And I not know if disable some cron also help now I have 2 days all vesta UP
So change it, disable crons for test, reboot server and have 3 days OK
So, for me my workaround working well, just commented line in admin's crontab:
crontab -u admin -e
when will be available the fix? I cannot login my vestacp admin panels.. It must be fixed asap... @serghey-rodin
Guys, what's happening when you just reboot whole server?
In /usr/local/vesta/php/etc/php-fpm.conf leave
pm = dynamic
and then just reboot whole server, and write here if you get "502 bad gateway" again?
This workaround works fine!
So, for me my workaround working well, just commented line in admin's crontab:
crontab -u admin -e
@gentilpaganella
it's not working at our server, we already disabled all crons.
maybe you just restarted whole server or put pm = ondemand ?
centos 7, apache + nginx, php72 fpm
Hi, after updating my VPS OVH Ubuntu 16.04, bug 502 has been fixed. I don't know what fixed it, maybe you'll see something here to help you. I haven't modified anything in Vesta since auto-update 22.
root@domain:~# apt-get update && apt-get upgrade
Hit:1 http://nova.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:3 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:5 http://nginx.org/packages/mainline/ubuntu xenial InRelease
Get:6 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main Sources [310 kB]
Get:7 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [796 kB]
Get:8 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [637 kB]
Hit:9 http://apt.vestacp.com/xenial xenial InRelease
Fetched 2,065 kB in 1s (1,688 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
libssl1.0.0 openssl pollinate
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.8 kB/1,588 kB of archives.
After this operation, 2,048 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 pollinate all 4.33-0ubuntu1~16.04.1 [12.8 kB]
Fetched 12.8 kB in 0s (413 kB/s)
Preconfiguring packages ...
(Reading database ... 113393 files and directories currently installed.)
Preparing to unpack .../libssl1.0.0_1.0.2g-1ubuntu4.13_amd64.deb ...
Unpacking libssl1.0.0:amd64 (1.0.2g-1ubuntu4.13) over (1.0.2g-1ubuntu4.12) ...
Preparing to unpack .../openssl_1.0.2g-1ubuntu4.13_amd64.deb ...
Unpacking openssl (1.0.2g-1ubuntu4.13) over (1.0.2g-1ubuntu4.12) ...
Preparing to unpack .../pollinate_4.33-0ubuntu1~16.04.1_all.deb ...
Unpacking pollinate (4.33-0ubuntu1~16.04.1) over (4.25-0ubuntu1~16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libssl1.0.0:amd64 (1.0.2g-1ubuntu4.13) ...
Setting up openssl (1.0.2g-1ubuntu4.13) ...
Setting up pollinate (4.33-0ubuntu1~16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
root@domain:~# reboot
We have released new vesta-php && and vesta-nginx packages. Please feel free to update your system. Hopefully it will fix 502 issue. At least we don't see it on our test servers anymore.
@serghey-rodin After upgrade vesta-php & vesta-nginx. The problems still happens :(
Yes, also. After update vesta-php and vesta-nginx. Bad gateway again.
Thanks @serghey-rodin !
My VestaCP was showing 502 yesterday, and its configured to auto update, Its working for me today.
502 bad gateway after updating to release-22 on ubuntu 16.04 @serghey-rodin
Did you try to restart whole server aftert upgrade?
I'm try to update whole ubuntu, v-update-sys-vesta-all, v-update-sys-vesta and reboot server, everything was same.
Bad gateway
The frontend not working too after reboot. WELL DONE.
Hi Guys,
after disable the task on cron and full restart on my server my vestacp
front still running without problems.. im using CENTOS.
thanks
Cumprimentos,
Gentil Alves Paganella Filho
mobile - +351 910001155
Em sex, 29 de jun de 2018 às 10:15, wpdaniel notifications@github.com
escreveu:
The frontend not working too after reboot. WELL DONE.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/serghey-rodin/vesta/issues/1625#issuecomment-401298232,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVN2YHkI533PyTPiuq3YIui_d_MCEtdtks5uBfBJgaJpZM4UzyCQ
.
admin@hogyantortent:~$ sudo systemctl status apache2
● apache2.service - LSB: Apache2 web server
Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
Drop-In: /lib/systemd/system/apache2.service.d
└─apache2-systemd.conf
Active: inactive (dead) since Fri 2018-06-29 11:34:03 CEST; 39s ago
Docs: man:systemd-sysv-generator(8)
Process: 3305 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
Process: 3289 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
Jun 29 11:34:03 hogyantortent.com apache2[3289]: * Starting Apache httpd web server apache2
Jun 29 11:34:03 hogyantortent.com apache2[3289]: (98)Address already in use: AH00072: make_sock: could not bind to address 185.187.74.169:80
Jun 29 11:34:03 hogyantortent.com apache2[3289]: no listening sockets available, shutting down
Jun 29 11:34:03 hogyantortent.com apache2[3289]: AH00015: Unable to open logs
Jun 29 11:34:03 hogyantortent.com apache2[3289]: Action 'start' failed.
Jun 29 11:34:03 hogyantortent.com apache2[3289]: The Apache error log may have more information.
Jun 29 11:34:03 hogyantortent.com apache2[3289]: *
Jun 29 11:34:03 hogyantortent.com apache2[3305]: * Stopping Apache httpd web server apache2
Jun 29 11:34:03 hogyantortent.com apache2[3305]: *
Jun 29 11:34:03 hogyantortent.com systemd[1]: Started LSB: Apache2 web server.
@serghey-rodin
Debian server. apt-get update, apt-get upgrade, sudo reboot - seems like solved my problem. Thanks.
I think the problem is .sock in var/run, that was clened when reboot
The socket never clean with restart or stop always is there, in all server when I see this problem reboot solve it.
i think it's not socket, because php-fpm proccess disappear, i watched with 'top'
The problem is complex but old socket might play important role here. In my tests the following command sequence completely resolve the problem
apt-get update
apt-get upgrade
rm -f /var/run/vesta-php.sock
service vesta restart
Complexity comes for the fact that there was no "make clean" procedure during build stage. So when we build new vesta-php package it probably linked to some prebuilded libs (garbage) which caused instability. I'd definitely recommend to remove old .sock file just in case
And can anyone explain how it started to happen on June 20 on all three distributions... and not before?
Oh yeah, .sock problem ?
Debugging this a lot of time :/
I did
apt-get update
apt-get upgrade
rm -f /var/run/vesta-php.sock
service vesta restart
and problem continue.
Ubuntu 16.04 updated.
I did
apt-get update
apt-get upgrade
rm -f /var/run/vesta-php.sock
service vesta restart
and problem did't resolved.
Hello,
thank you @serghey-rodin, everything works perfectly right now. I have no 502 nginx issue since I have updated the server (about 2 days ago). I suppose it's solved now. Good job!
I did this on CentOS 6 (64) and 502 Bad gateway problem persisted.
yum update
yum upgrade
rm -f /var/run/vesta-php.sock
service vesta restart
I reboot the server after restart vesta.
reboot
Right now no 502 Bad gateway message. I will monitor it and report back if the problem persists.
It has been a couple of hours and no 502 Bad Gateway errors appear again.
The same problem since updating from release 20 to 21 and now I'm on release 22 and still, I have the same problem
This problem started from release 21
@serghey-rodin was not too long for fixing a simple bug?
@mahdiyari the problem already fixed. Please read comment from Serghey Rodin
@mahdiyari
did you rebooted server after update?
@anton-reutov No, I already followed that comment
@dpeca I can't restart my server (never ever!)
@mahdiyari well I suppose you have a problem then. I guess you can remove socket file manually.
rm -f /var/run/vesta-php.sock
systemctl restart vesta
However it would be better to reboot the server. What is so critical about your server, you can't reboot for 1 minute?
@treatys Thanks for your comment, but already removed and restarted vesta and still I see 502 error
I can't turn off that server even 0.0001 seconds
@mahdiyari you will need to find a unique solution to your problem then, because as far as I can tell on all the servers I have tested, a hard reboot is required.
Maybe you can try and restart every service manually? What can be so important that you cant reboot the server for even a minute? Surely it is more important to have the server running stable than being scared to reboot?
Wondering what you will do when some hardware die, or even network die for 1 minute...
@ninjao I will try to restart all services. Thank you
I'm running a blockchain node on that server, that node uses RAM as a storage, syncing data after reboot will take 2-3 days!
@mahdiyari I'm not sure about what type of OS you using, but have you tried to upgrade your server? for Debian - apt-get update, apt-get upgrade and then try to restart vesta.
Already updated and upgraded! that is ubuntu 16.04
Update: Restarted all services manually and there are not 502 errors after 20 minutes
Update: again 502 error
@mahdiyari ok lets hope that works out for you! :)
Did you restart only the "vestacp" related services? Or did you literally restart every service on your machine? Would be good to know for future people unable to do a hard reboot.
Good luck
502 again!
I restarted vesta related services!
and deleted unix socket before restart?
Of course!
@mahdiyari Have you tried to have a look on what services have been upgraded and restart them too?
ok, let's then use safe variant that should work for sure
edit /usr/local/vesta/php/etc/php-fpm.conf
find pm=dynamic
set to pm=ondemand
and then
service vesta restart
@gd-euss how can I find upgraded services?
@dpeca Edited and restarted.
@mahdiyari first try dpecas variant and if doesn't work for you then try to find upgrade log in your system logs.
i believe it will work because that way issue will be avoided (whatever it is)
No, again 502 error!
@mahdiyari you should try to reboot your server
does it works even one second?
I can't reboot my server @anton-reutov
I can access admin panel after restarting vesta service, but it shows 502 after a while
@mahdiyari your logs might be in /var/log/dist-upgrade (Not sure because never used Ubuntu)
I restarted all services (except ssh), let's see what will happen!
@mahdiyari you are running out of options :/
I can think of checking all your logs, making sure to resolve everything else... and restarting other services on your machine. I am only thinking of last resort options... restart your network interface? But this might also cause issues with your blockchain node. Also be careful when restarting network interface while using SSH...
I tried so many things to get my server running again, but in the end a reboot was probably what solved the issue.
Edit: Ah I see you have done something, good luck!
Now, I don't have any option and still, there is 502 error
A bug comes with a silent update (release 21) and we can't solve that without a reboot!
if anyone found a working solution (without reboot) please comment here
Thanks for your time:)
Update: No matter what you have in your server! You should reboot to solve. A bad development:)
everything seems to work fine after update, upgrade and reboot.
Thx
@mahdiyari did you solve it? I am still fighting with it.
@niccolomineo you update and reboot ?
If you not reboot server may you cant solve it
Thank you, I restarted NGINX, I hope that sorts it!
edit: Nope, problem's still there.
I did:
rm -f /var/run/vesta-php.sock
service vesta restart
service nginx restart
Try reboot your server.
I did and it is working now. Let's see for how long it lasts (hopefully forever).
apt-get update, apt-get upgrade, reboot will help you.
apt-get update, apt-get upgrade, reboot will help you.
This definitely worked for me. Cheers!
This worked:
rm -f /var/run/vesta-php.sock
service vesta restart
service nginx restart
reboot
Nginx just seems to have a problem in CentOS builds
It still cannot connect
This worked:
rm -f /var/run/vesta-php.sock service vesta restart service nginx restart reboot
This worked for me too!
I can't restart service
nginx-error.log :
2020/06/02 10:18:19 [crit] 647#0: *10 connect() to unix:/var/run/vesta-php.sock failed (2: No such file or directory) while connecting to upstream, client: 176.59.34.108, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "server.domain.ru:8083"
Most helpful comment
It seems like this issue is caused by default pm.max_requests value.
Please try out the following fix
I'm going to test it next few hours and push as update if everything is fine.
Thanks