Laravel-websockets: Concurrent connections limited to ~1000

[rflatt-reassured]

The socket service restarts once we hit about 1000 connections:

I've followed the advice on the deployment page to increase ulimit, and the sysads have confirmed that it has been set to 50000.

I'm going to try the pecl event extension. Once I've installed it, what further configuration is required?

[rflatt-reassured]

We are getting messages about SYN flooding in dmesg. For example:

4248.136617] TCP: request_sock_TCP: Possible SYN flooding on port 6001. Sending cookies. Check SNMP counters.

We have increased tcp_max_syn_backlog from 128 to 4096, then to 8192 (see https://serverfault.com/questions/482277/deactivate-syn-flooding-mechanism). However, the SYN flooding in dmesg kept occurring, and mostly coincided with the service crashing. Next we increased it to something absolutely nuts: sysctl -w net.ipv4.tcp_max_syn_backlog=1000000

That didn't fix the issue, so we set net.core.somaxconn = 128, then 4096. (maximum socket backlog)
https://access.redhat.com/solutions/30453

Which still hasn't fixed the issue.

I'm struggling to even test it - I've tried using siege to hammer the auth endpoint, and thor to open a number of websocket connections, but neither overload it and cause it to crash.

Has anyone else encountered this? Any suggestions for a fix?

[rflatt-reassured]

We've fixed the issue. We installed ev, but needed to do 2 things to ev,so:

1. set the permissions so that it was globally executable.
2. rename it. Apparently php extensions are loaded in alphabetical order, and it had dependencies that were loaded after. Renaming the file to z-ev.so fixed the issue. We're holding steady at ~2000 connections.

[lucasctd]

@rflatt-reassured so after you have installed "ev" extension, you didn't need to do the ulimit thing or you have to do both?
I think the docs are not so clear about it.

[rflatt-reassured]

You need to increase the limit as well. The ev extension allows that to work. Without increasing ulimit, the ev extension doesn't add any benefit.

[lucasctd]

@rflatt-reassured thank you so much.
I have installed "event" on my container and set the ulimit using docker-compose.
I'd like to test it, do you use any testing tool to check how many connections your app supports?
I just thought I could make a JS loop creating connecitions to my server, but I am unsure if it will work XD

[setkyar]

@rflatt-reassured How do you set the permission? I am also facing an issue similar to yours and got stuck at 1500. 😅

Base on your comment https://github.com/beyondcode/laravel-websockets/issues/244#issuecomment-533123015