Web-frameworks: Benchmark strategy

[doanguyen]

I was surprised since some PHP full-stack frameworks like Laravel and Symfony outperformed some Micro framework like Flask or Sanic. And then I realized that you put Nginx on top some of your server stack but not the others, so the results might be misleading.

[waghanza]

Hi @doanguyen,

Most of micro frameworks on python could "speak" throught HTTP (using other python modules).

However, php (via php-fpm) could not "speak" HTTP, only FCGI.

To solve this, NginX was used. If you have any better solution, feel free to suggest :heart:

[doanguyen]

I just take a look at the famous TechEmpower benchmarks and they seem to have the same approach as yours, that' explained a lot to me why python frameworks have poorly benchmarked all the time even in reality it's solid.

From my point of view, if we consider php-fpm as process manager talking to nginx, then gunicorn has similar role, and to be _fair_ we also need to put nginx as reverse proxy for python applications as well.

[waghanza]

NginX + php-fpm is for me the equivalente offre gunicorn.

Having NginX on top if gunicorn does not add performance.

NginX is used as a gateway (FCGI -> http)

[doanguyen]

PHP-FPM is just a gateway interface (FastCGI Process Manager) with additional features like process manager, graceful start/stop,.., gunicorn by means is another gateway interface so basically it equivalent to php-fpm but with build in HTTP webserver.

The reason it makes different is that Nginx might cache request and handing by itself. Not to mention Nginx is an event-based model and Gunicorn is a pre-fork base model.

[waghanza]

So what is your proposition for a gateway between php code and http ?

I do not know any equivalent on php world, but probably you do

[doanguyen]

Another stack candidate is Apache prefork-mode works as a proxy layer talking to the php-fpm application.

I could make some PR if you want by the end of the week.

By the way, it still very interesting while keeping the nginx+php-fpm stack and then compare it with others set up, it's not only the framework itself but also the server implementation make difference in real world. Do you think it's outside of your initial scope?

[waghanza]

No matter what is used 😊 php-fpm / nginx, php-fpm / apache, apache / mod_php ... we only target performance

I'll be glad you add a micro comparison un php, as I started here for python https://github.com/the-benchmarker/web-frameworks/issues/292 :heart:

[waghanza]

Also, I have compared Apache / php-fpm vs Nginx / php-fpm, and for this kind of usage NginX / php-fpm outperforms

[doanguyen]

Sure it is.

In my opinion, your benchmark method is meaningful and very appreciate. But in some case, it might mislead. If all people on earth care about speed regardless development time, features, and a bunch of other things then you can make the conclusion about performance. If this, full-stack frameworks have no chance against the micro-framework and duck-typing languages have no chance to defeat static-typing languages. But we are living in a trade-off world so such frameworks like Rails, Django or Laravel have its foothold. Again, I am very grateful to see your results, but I could not choose one and left the others while looking at your data.

[doanguyen]

And comparing Nginx/php-fpm vs gunicorn/django is like comparing apple vs orange, to be honest.

[waghanza]

There is no solution satifying 100% cases of course.

We are juste trying to do our best. For example, hanami is a full Stack framework outperforms against sinatra a micro-framework. So performance, really depends on too much conditions to solve 100% cases.

[waghanza]

The ultimate purpose here is more presenting some results, rather than comparing / debating about how beautil / useful à framework is

[doanguyen]

I agree. Then the methodology must be consistent, or at least we need to specify, for example putting Nginx as a proxy layer on top of anything could lead huge advantage to others not.

[waghanza]

Sure, but what I have experienced, this is theorical.

I mean, in python / ruby (what I know), some framework could run on gunicorn or puma, but some could not due to their design.

For me the best thing we can do is using the more performant implementation for each framework

[doanguyen]

PHP has its own build-in web server, you can use it just by typing php -S localhost:8080but it's dead-simple and for development only.

The smart way a PHP, Python, Ruby,... application can talk to the world by communicating with a webserver is to introduce a GI (Gateway Interface), different languages have its own version of GI, with php it is fastcgi and php-fpm is an implementation of fastcgi with additional process manager, with python it's WSGI and an example of implementation is Gunicorn but with the http webserver build in. Same story for Ruby. I don't know why php-fpm project has been abandoned for quite a long time https://php-fpm.org/, that's why php-fpm lacks a lot of features.

[waghanza]

Sure. If you know a GI manager that is able to "speak" http, please be my guest

I know that php has a built-in webserver, but not designed for production

[waghanza]

ans php-fpm is not abandonned, just introduced on php core 😎

[waghanza]

feel free to perticipate to https://github.com/the-benchmarker/web-frameworks/issues/309

[doanguyen]

I think apache pre-fork / php by mod_php is equivalent to gunicorn/django but you are already benchmarked.

Thanks for the information :fr: :+1:

[waghanza]

I have not try mod_php

[doanguyen]

By definition, it should be slower, in order of magnitude(s).

I love python, so I biased. But comparing Django vs Laravel with those stacks are just not fair.

[waghanza]

Btw, comparing banana to apple is not useful, unless we compare on the same point of view.

It's a matter of comparing usage / same use case, not all advantages 😃

[waghanza]

@doanguyen I close, feel free to re-open