Ratchet: Websocket server stop responding after some hours.

[pradeeptakhatoi]

Hello. I have used the supervisor concept to check and restart my chat server if it is stopped.

After some time interval my chat not working.When i check the console i didn't find any websocket connection fail error.

Then i use below command to check my server is running or not and see the server still running.

ps aux | grep php

So i am confused where is the issue or bug for which my chat application not working.

[pradeeptakhatoi]

When i kill the chat server process manually by using the "kill" command then the supervisor start the chat server immediately and my chat working fine.

[cboden]

Put the following code in your script and restart it:

error_reporting(E_ALL);
ini_set('display_errors', 1);

Please paste the output of the following command here:

composer show -i

Once the error happens again don't kill the script. Check and paste the latest output of the supervisor stdout and error logs in this issue.

Once the error has happens ssh directly into the server and run the telnet command and enter the following into it (and paste the output here):

GET / HTTP/1.1
Host: localhost

[rohitkhatri]

I'm going through the same problem.

and here is what was shown on "composer show -i" command

cboden/ratchet v0.3.3 PHP WebSocket library
evenement/evenement v2.0.0 Événement is a very simple event dispatching library for PHP
guzzle/common v3.9.2 Common libraries used by Guzzle
guzzle/http v3.9.2 HTTP libraries used by Guzzle
guzzle/parser v3.9.2 Interchangeable parsers used by Guzzle
guzzle/stream v3.9.2 Guzzle stream wrapper component
react/event-loop v0.4.1 Event loop abstraction layer that libraries can use for evented I/O.
react/socket v0.4.2 Library for building an evented socket server.
react/stream v0.4.2 Basic readable and writable stream interfaces that support piping.
react/zmq v0.3.0 ZeroMQ bindings for React.
symfony/event-dispatcher v2.7.1 Symfony EventDispatcher Component
symfony/http-foundation v2.7.1 Symfony HttpFoundation Component
symfony/routing v2.7.1 Symfony Routing Component

I placed these statements at the starting of my server file, but didn't find any.

error_reporting(E_ALL);
ini_set('display_errors', 1);

please help me.

[cboden]

@rohitkhatri What happens when you do the telnet thing after the server has stopped responding?

[cboden]

Closing due to extended period of inactivity. I'll re-open if reporting continues.

[Chaosvex]

Slightly late to the party but I've been looking into Ratchet again and I wanted to check to see if some of the issues that I encountered the last time I tried to use it (two years ago) have been resolved.

This particular issue was a showstopper when it came to using Ratchet for a service that ran continuously and I'm seeing a fair number of reports here with similar symptoms. Ratchet would work fine for a while before becoming unresponsive to new connections, maxing out the CPU and needing a restart. Initially, it'd run happily for several weeks but that ended dropping to needing manual intervention every couple of days.

The problem turned out to be file descriptor leakage and as soon as it hit the maximum (1024 by default, I believe), restarting the process would be required. As the site's traffic grew, it'd end up hitting the limit sooner. I didn't have the time or interest to fix the root problem for a hobby project so I can't offer up a patch but perhaps being aware that there was/is a leak can help get the ball rolling.

Edit: I can see you've suggested raising the limit in another issue (https://github.com/ratchetphp/Ratchet/issues/349#issuecomment-132613596), so I guess you're already aware of the problem. I don't consider increasing the limit to be a robust solution, so is there something that's preventing this from being fixed or a way to ensure the user doesn't run into this problem?

[pinumanu]

pradeepta20 .. got any solution for that...

[pradeeptakhatoi]

Initially i have done the database coding using PDO.Later i change the code to mysqli its working fine for me.
I don't know the exact reason but it was due to some database coding error.

[pradeeptakhatoi]

Use below technique to connect automatically when Websocket server restarded.

var wsUri;
$(function() {
wsUri = $("ws_uri").val();
init();
});
function init() {
websocket = new WebSocket(wsUri);
websocket.onopen = function(ev) { //connection is open //
}
websocket.onmessage = function(ev) {
};
websocket.onerror = function(ev){
};
websocket.onclose = function(ev) {
init();
};
}

[MarcGodard]

I am having a similar issue. The web browser doesn't connect (but it doesnt fail either) and hen I check the websocket logs, everything looks fine and the new connection isn't started. My sockets are basic without a database.

[cboden]

@Chaosvex The file descriptor limit issue is an operating system level feature...you will run into this problem with any language/environment (not just Ratchet/PHP). If you're hosting a WebSocket server 1024 is WAY too low if you expect any traffic at all.

@MarcGodard Please follow the directions from this comment above and report the output so we can help troubleshoot.

[Chaosvex]

@cboden Yes, I know but that has nothing to do with the leaks within Ratchet.

[MarcGodard]

@cboden I figured out the issue minutes after posting that. It was my fault. Thanks for the quick responce.

[andig]

@MarcGodard could you share your finding if its of general interest?

[MarcGodard]

I had an error in my code, that's all.

[Vendin]

The same problem as at @pradeepta20, in several hours sockets cease to answer. In this discussion I haven't found the solution. Whether there is a solution in general, except as restart of sockets constantly?

[grappetite-tahir]

I'm using AWS EC2 server for my application. My chat has stopped 2 times till now automatically but when I go to terminal and use the pgrep command to find the websocket process, its showing me processes, its mean that socket is connected to server. but still chat was not working. Then i kill all two process mnually and my cron job which is set for every 1 minute to start socket its running again and my chat get live again. I faced this issue 2 times. I don't know how things are going on in the background. I didn't find any logs, I have maintained all the logs that user is attached, user msg to this user, blah blah. but there is no error log in my logs.

Further more if mobile user just turned off his/her wifi or mobile data my chat is not firing onClose method. So the thing is that how to mark a user offline in this scenario if someone met this criteria. Which is the way to mark user offline if onClose method not fired.

Thanks!

[Chaosvex]

Considering that neither developer has responded to a showstopper of a bug in several years, you'd be best dropping this project and looking for an alternative.

[WyriHaximus]

We fixed several memory leaks in ReactPHP over the last few years since your last comment. We implemented new event loops and done a ton of performance upgrades. The thing is you need to account for the following when deploying:

• Raise ulimit for your user/application so it can handle more then 1024 open FD's
• Use an event loop extension over the default stream select loop which is hard capped at 1024 FD's
• Monitor your application's connections, memory usage, CPU usage, request/messages per second

we simple cannot debug very deeply into your problem without information like that. But what also matters is what you do when a connection has been establisht, what kind of resources you use and how you connect to databases/storage etc etc.

And last I really want to stress that we cannot change the 1024 thing for you on your deployment. You have to do that yourself. It's baked in hardcoded into stream_select and you need to change the event loop and raise it on ulimit if you want to resolve that. Wheither or not you consider raising the limit a good solution, that is how the OS works, it's entirely out of our hands and into yours to resolve that on your systems.

[Chaosvex]

This was never about memory leaks or even a concurrent handle/descriptor limit, it was about the descriptors leaking so a popular Ratchet-based site would have to be frequently killed off and restarted to prevent it from live-locking and potentially bringing an entire system to a crawl. Concurrent handles was irrelevant.

[WyriHaximus]

File descriptors don't leak, and according to your own description hitting the 1024 limit was/is the root cause of the issue you where having. They can leak over from a parent process into a child process but that's a totally different issue. (And we added tooling against that in react/child-process v0.6.0.)

The thing is we have several users doing more then 50K+ connections in a single process with Ratchet without a hitch. And I don't want to suggest the problem is within your code, but implementing a long running service in PHP is a delicate dance between us the maintainers of Ratchet/ReactPHP and the implementer's of our libraries. This is also on a lower level PHP then your normal request-response cycle. I gladly sit down for a hangouts call with anyone to resolve their problems, but it's going to get technical and diving into tracing, logging and monitoring really fast. And to be really honest I can't do much with "It's locking up please fix". I want to dive into that and preferably try and find a way to reproduce it so we can fix it up stream.

Personally I monitor the memory usage (amongst things) fanatically to stay ahead of issues like this.

[Chaosvex]

Yes, hitting 1024 total over the lifetime of the process, not 1024 concurrent. When those connections are no longer active and there's no longer a handle available, that's when it becomes fair to refer to it as a leak.

[cboden]

Just tested this with StreamSelectLoop, no problems. I suspect you're holding a reference somewhere, holding open 1024 connections, causing your problems.

composer require cboden/ratchet

<?php
    require __DIR__ . '/vendor/autoload.php';

    $app = new Ratchet\App('localhost', 8080);
    $app->route('/echo', new Ratchet\Server\EchoServer, array('*'));
    $app->run();

// In a browser
setInterval(() => {
  let c = new WebSocket('ws://localhost:8080/echo');
  setTimeout(() => c.close(), 250);
}, 500);

---

^ Server still responding with 101 HTTP status code after 1024 connections.

[inri13666]

Given example implements variant with closed connections
What about not closed connections
I'm failed after 125 connections
with

var x = []; for(var i=0; i<1024; i++) {x[i] = new WebSocket('ws://MYHOSTADDERSS/ws');}

[cboden]

@inri13666 What's the error you're getting? The reason I closed connections was because the browser I was testing with didn't allow that many concurrent connections to a single server as a security measure.

[inri13666]

I do not have any errors in the log, the socket not responding and only socket command restart may help :(

[inri13666]

I found the cause for my issue,
_Web browsers allow huge numbers of open WebSockets
The infamous 6 connections per host limit does not apply to WebSockets. Instead a far bigger limit holds (255 in Chrome and 200 in Firefox)._

Chrome Sources
Source Article

[inri13666]

Totally I found the final solution
Never use React\EventLoop\StreamSelectLoop this is the cause of all issues because on Linux it fails after 1023 connections and on windows, it fails after 512 source

To solve the issue described here install one of the following PHP extensions

1. Event extension _My Choice, but for windows, have some issues_
2. libuv wrapper _This loop is known to work PHP 7+._
3. interface to libev library _This loop is known to work with PHP 5.4 through PHP 7+._
4. Libevent - event notification _Not recommended for PHP 7+_ but have An unofficial update
5. PHP libev Extension

P/S
Sample application to check connections
Server: https://github.com/inri13666/web-socket-echoserver
Client: https://github.com/inri13666/web-socket-test

[machomath]

Hi I am sorry for this late comment. I was having a similar issue that server seemingly stopped responding after some time. Tried many solutions but at the end turned out that MySQL was taking timeout after 8 hours of inactivity and disconnecting. So I put a root cron job
0 0,8,16 * * * systemctl restart abc.service
It worked for me.