Web-frameworks: Skeptical about controversial results

[doanguyen]

Hi,
I had some time to check the benchmark of Laravel and Flask on my computer i7-7600U, 2.8x4 GHz and here is the result:
https://gist.github.com/doanguyen/a81cccb4e884c63422e2c5ae3e846d18

It shows the total different outcomes, Laravel serves only 87 req/s and Flask at best is 8506 req/s, I did some research and it looks similar to some other benchmarks, for instance, https://medium.com/@mihaigeorge.c/web-rest-api-benchmark-on-a-real-life-application-ebb743a5d7a3

Do you have any clue what's happening?

[waghanza]

Hi @doanguyen,

I have not done any benchmark before, but honestly I doubt that we have such differences. In my php adventures I have often to do some configurations that I have NOT to do in python world.

I am currently working on cloudification (on aws) for ruby _based_ frameworks. If you want to make the same for php _based_ frameworks, I'll be glad to push my code and start collaborating with you :heart:

The only way, for me, to be sure of performance is to have an access to the source code / details about technical environments (what I plan to deliver here) :stuck_out_tongue:

[waghanza]

PS : The time I was setting up php here, I had such diff when I was using tcp instead of socket in php

[doanguyen]

Hi @waghanza,

Thanks for fast replying.
I'm not sure you are mentioning about tcp socket and unix socket, if that, I think it doesn't make so much different.
From what I know about Laravel, It is rather slow in comparison with others so maybe I missed something in my configuration that makes the results are so different.

[waghanza]

Do you want to help us running php on cloud ?

I'm results won't be as imagined 😊

[doanguyen]

Sure, I'm happy to do, but how?

[waghanza]

Do you know cloud-init ?

The technical system I use for ruby is some definitions (global, language level and framework level) to define who to setup host. I'll push my code soon and explain how to use it here

[doanguyen]

I've never done it before, the setup that I've made for the benchmark is creating two docker instances and manually benchmark by apache _ab_.

[waghanza]

I understand, but using docker is NOT representative of production-ready (something that produce readable results from which assumptions could be done)

I mean using docker (I think on a local workstation) is NOT the same as :

• using a production env (even on kubernetes)
• using a server (any production grade machine is NOT on the same arch that our workstation)
• ab COULD be useful but we are using wrk here on purpose (as tfb)

Please, be patient :stuck_out_tongue: , I'll keep this issue open while I'll push my cloudify branch so as we can work on both ruby and php together :heart:

[doanguyen]

Hi again,

Really bad news.

I still questioning myself why there is so much different in the result and I finally found out.

I've never written any crystal code so please correct me if I'm wrong. The file tools/src/benchmarker.cr will send, by default, 100k requests with no_cpu+1 threads to the target and collect the result by averaging the 3 endpoints. The problem is, the script does NOT check the return value and Nginx will serve all the inbound connections, forward it to the php process manager without knowing if php is being ready or not. With said, all the incoming request will be served, either return 2xx/3xx response code OR 502 code. So I tested with wrk, and not surprise:

$ wrk -c 1000 -d 10s -t 3 http://172.17.0.3:3000/user/0 -v           
wrk 4.1.0 [epoll] Copyright (C) 2012 Will Glozer
Running 10s test @ http://172.17.0.3:3000/user/0
  3 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    60.58ms  205.00ms   2.00s    93.94%
    Req/Sec    16.58k     7.47k   43.94k    67.68%
  490950 requests in 10.03s, 154.55MB read
  Socket errors: connect 0, read 224, write 0, timeout 233
  Non-2xx or 3xx responses: 490720
Requests/sec:  48958.45
Transfer/sec:     15.41MB

Basically, I sent 100k req, with 3 thread and php rejected >99% of the requests and nginx just return non-2xx or 3xx response.
Reducing the connection, I get:

$ wrk -c 10 -d 5s -t 3 http://172.17.0.3:3000/user/0 -v
wrk 4.1.0 [epoll] Copyright (C) 2012 Will Glozer
Running 5s test @ http://172.17.0.3:3000/user/0
  3 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   133.51ms   32.33ms 239.26ms   76.28%
    Req/Sec    22.51      7.53    40.00     82.64%
  333 requests in 5.01s, 201.97KB read
Requests/sec:     66.44
Transfer/sec:     40.30KB

Now it make perfect sense and very well aligned with other benchmarks.

I know you guys working so hard to perform this benchmark, so please fix it before we have results that can be referenced.

[waghanza]

@doanguyen As I understand, your point is that non http 200 are not checker.

I know the code needs refactoring, but kits check the output (json) of bin/client -u http://google.fr

Error (even socket ones) are taking in count

[waghanza]

@doanguyen however, the errors are not shown on results, we have to choose the right way to display errors

[doanguyen]

It's not about displaying the erros, if it's error then people still thinking you are benchmarking php while in principle, you are benchmarking Nginx. I'm certain no where on earth can get >30k reqs with laravel on single instance, just impossible

[waghanza]

@doanguyen you're concern is aboutphp then.

As I have already said, I have not found anything like gunicorn or puma in php word.

If you know one, feel free to contribute :heart :

Nginx + PHP-FPM is a choice only made because I do not know any appserver on php

[doanguyen]

It's not about server stack, it's about the way you benchmark, collecting results, I think it needs to fix first

[waghanza]

fix what ?

as the README is saying, results are not production-ready (then you can not take any assumptions)

I hope at the end of the year, results will be computed on cloud (so production-ready) and then we can make assumptions

Honestly, results from a local docker and on an isolated machine are NOT the same

[proyb6]

Yes, we need to fix the issues, let’s pull a request if you can contribute.

[waghanza]

sure, anyone could have global knowledge, just name the problem and contribute with a solution :+1:

(ideas, PR ...)

[doanguyen]

@waghanza with your response I really want to ask you what is _benchmark_ stand for?

With that it will come to the point of how to do it properly.

Until now, I have an only recommendation is to use wrk instead of a custom library to perform the benchmark, but the library could be your selling point so either you stick to your solution or move to wrk.

[waghanza]

@doanguyen ah understood, no need to point issues based on assumptions for that (because results are not production-ready) ^^

wrk is used since since june https://github.com/the-benchmarker/web-frameworks/pull/223

I know anything is perfect, but we are trying to do so :heart:

and when we reach production level, we will start a documentation to have more details about implementation, anything is stable as for now, so a documentation will not be helpful

[doanguyen]

I followed the README, not so correct.
I'll close this issue.

[waghanza]

@doanguyen So README was not clear enough for you ?

[doanguyen]

Isn't it wrong? still using the old benchmarker crystal script?

[waghanza]

no, it is NOT wrong

This script, is in fact a custom wrapper around wrk

However, considering it unclear is OK

If you wan to clarify the situation, would you mind list me the questions you are asking yourself, I'll try to maintain a wiki

[doanguyen]

Whatever it is, it's fine by me now.

Aside, https://github.com/giltene/wrk2 provides additional features and DON'T have the problem of taking non-2xx into account of the result.

Is this worth to make the change?

[waghanza]

Yeah, I've created an issue for this https://github.com/the-benchmarker/web-frameworks/pull/183, there is a lot of alternative to wrk

• https://github.com/giltene/wrk2
• https://www.npmjs.com/package/autocannon
• https://github.com/rakyll/hey
• https://github.com/JoeDog/siege
• ...

We decided to go on wrk, but it COULD change

feel free to open a separate discussion to ask for sieger tool change

[doanguyen]

Thanks, @waghanza, things are clear now.

[waghanza]

@doanguyen I have successfully run on aws instances by creating a wrapper here (I'll push soon in my fork)

for sinatra, I have (times are in ms)

| Instance | Average | 50th percentile | 90th percentile | 99th percentile | 99.9th percentile | Standard deviation |
|-------------------------|----------------:|----------------:|----------------:|----------------:|----------------:|----------------:|
| t3.nano (2 cores) | 82714.0 | 82047.0 | 84024.0 | 88330.0 | 135806.0 | 2177.0 |
| c5.2xlarge (8 cores) | 85871.0 | 85187.0 | 88529.0 | 93479.0 | 111095.0 | 2052.0 |

:warning: example
after having proper results like that we can make asumptions, like running c5.2xlarge are useless ...

PS : The above results are for 1 endpoint only -> /

[doanguyen]

It's a bit weird to me. Any chance the wrk instance is the bottleneck? :laughing:

[waghanza]

yeah :stuck_out_tongue_closed_eyes:

this is phase 1

I mean wrk is on my local computer, so with a non-stable conection (even no wireless)

In phase 2, I'll run on a fully stable connection :stuck_out_tongue: cloud -> cloud instead of home -> ... -> cloud

[OvermindDL1]

I mean wrk is on my local computer, so with a non-stable conection (even no wireless)

Lol, yeah the remote connection means you are measuring your possible network load as a maximum rather than the servers once hitting a certain point. ^.^;

[waghanza]

@doanguyen @OvermindDL1 I close this issue, stay tuned on gitter (or here) on cloudification :stuck_out_tongue: