Phoenix: Performance Testing of Phoenix Channels

[jeregrine]

@Gazler set this test environment up to test how many websockets can connect to a channel.
How to test:

brew install tsung
git clone [email protected]:jeregrine/phoenix_chat_example.git
mix deps.get
MIX_ENV=prod mix do deps.get,phoenix.server
<tab tab>
tsung -f websocket.xml start

We are currently investigating how to push more load to Phoenix and see what is blocking the current throughput (if it is the tool or some configuration we are not aware).

During testing the result's were not great as shown by

sudo cpan Template
cd ~/.tsung/log/<latest_run>/
/usr/local/lib/tsung/bin/tsung_stats.pl ./
open graph.html

I am not sure if we need to change how Tsung is setup or if it's a phoenix issue. MY GUT say's its a tsung configuration issue. But if we get this to work it is super neat because it actually tests the whole system not just the opening of a socket see https://github.com/jeregrine/phoenix_chat_example/blob/master/websocket.xml#L22-L43 for examples.

EDIT: PLEASE NOTE this could be multiple things. This was a super neat, and reproducible way to test phoenix channels.

[jeregrine]

@Gazler mentioned you might need to update your ulimit locally to to do better, but when he did he hit similar bottle necks.

This is a pretty easy setup for doing websocket testing so I figured I'd share here.

[jeregrine]

Also note this assumes OSX.

[jerel]

Might it be something in tsung? I ran a quick test the other day using Thor and hit 40,000 with no problems. (On a Linux laptop). Note that I wasn't sending messages or anything, just opening connections.

thor --workers 4 --amount 40000 --concurrent 40000 http://localhost:4000/socket/websocket

[jeregrine]

This appears to be sending messages

On Thu, Oct 15, 2015 at 2:36 PM, Jerel Unruh [email protected]
wrote:

Might it be something in tsung? I ran a quick test the other day using
Thor and hit 40,000 with no problems. (On a Linux laptop). Note that I
wasn't sending messages or anything, just opening connections.

thor --workers 4 --amount 40000 --concurrent 40000 http://localhost:4000/socket/websocket

—
Reply to this email directly or view it on GitHub
https://github.com/phoenixframework/phoenix/issues/1285#issuecomment-148499135
.

[josevalim]

A thousand connections is way too low. How do you measure: "He was getting a max of approx 1k on his machine. When I ran in dev mode I was maxing around 200." Which limit is being reached? memory? the benchmarking tool?

[jeregrine]

@josevalim it just stopped being able to open new connections

[josevalim]

How did you notice that? Did you get errors? Was something being logged? Timeouts?

[jeregrine]

SORRY everyone this isn't supposed to be a fire: just a reproducible way to performance test phoenix channels in more robust ways. I probably have tsung misconfigured.

[jeregrine]

Reminder to set ulimit in each tab

ulimit -n 1024
ulimit -u 1024

(as high as your machine can handle)

[Gazler]

With a higher ulimit these are the results I am getting locally - just over 6.5k connected. Then something dies - I'm investigating now.

[Gazler]

Ok - so it turns out running tsung and phoenix on the same machine is a bad idea at higher levels. It looks like tsung was causing the phoenix application to crash (erlang crash perhaps?)

Here's a more promising image from an Amazon t2.medium instance:

[chrismccord]

The linked repo does not set buiild_embedded, so protocols are not consolidated by default. Can you verify protocols are consolidated in your latest runs?

[Gazler]

@chrismccord Yeah - I have set them now. That was what was causing the problems at about the 1400 connection mark on my machine.

[josevalim]

Also make sure you are running in production. The only reason for Erlang to
fail really is because you are running out of memory or the OS cannot cope
with the load.

On Friday, October 16, 2015, Gary Rennie [email protected] wrote:

@chrismccord https://github.com/chrismccord Yeah - I have set them now.
That was what was causing the problems at about the 1400 connection mark on
my machine.

—
Reply to this email directly or view it on GitHub
https://github.com/phoenixframework/phoenix/issues/1285#issuecomment-148668083
.

_José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D_

[jeregrine]

So I think @chrismccord and @Gazler took care of this on irc. Was multiple issues, one is do not run tsung on the same machine as you are testing, and various other issues.

[gjaldon]

@jeregrine what were the various other issues? Was just curious and would be great if we could have it as reference in this issue for anyone interested in testing performance of Phoenix channels.

[vaer-k]

Anyone reading this issue would be interested in knowing that this conversation resulted in a popular post about the performance of Phoenix channels; The Road to 2 Million Websocket Connections in Phoenix