Web-frameworks: Benchmark Usefulness

[bhansconnect]

I was just scanning through this repository and noticed how simplistic the benchmark is. Do you think the benchmark is useful even though it is simply returning a blank response or an id? I feel like for the benchmark to be representative of real world performance it is necessary to do something more complicated than this. For example, if you need to build a full sized webpage with some templating, it could potentially greatly change the results based on which frameworks have the best templating solutions. Should some form of more complexity be added to this benchmark or is this simplicity for a specific reason?

[OvermindDL1]

I do not think this benchmark is a great point of usefulness for the 99% of web apps out there. On that same vein I don't think TechEmpower is useful either because many of its implementations are really really bad and it still mis-tests certain things.

The best benchmark is the one you build to test your very specific usage. Even this repo only tests the network stack of each language and not so much the framework. With raw enough level access you can end up getting identical speed regardless of the language in this repo.

However, this repo is still fun to mess with just for the sake of being fun and trying to micro-optimize a specific use-case that very few care about. ^.^

[waghanza]

@bhansconnect also the goal of this project is more like to be a tool. I mean, a benchmark COULD only be _representative_ if you are comparing YOUR case (not a general one).

What this project is going to be is more a benchmarker than a benchmark :

• Write some implementations => your code
• Define your rules => your configuration
• Launch the benchmark => our code
• Decide which to use facing the results previously found => your results

PS : Of course, this is a long term goal :stuck_out_tongue:

[bhansconnect]

That makes a lot of sense. If your goal is to make a good benchmarker, then what you are benchmarking is not all that important. I was just looking at the library and the benchmark seemed quite odd to me.

[waghanza]

In fact, it could also be useful to have at least one framework per language (to see how this language / framework is deployed)

[waghanza]

I think, I'll (when this project is stable) create a demo, so as readers can have a better understanding of how a benchmark could be done

[waghanza]

I'll make myself clearer. The perfect (as on my own understanding) benchmark is the one made with your implementations, I mean as code you produce.

A usage of this project / tool could be a simple command line like
~
bin/run --directory= --configuration=
~

• MY_CODE : The code you've write for your purpose
• OUTPUT_EXPECT : A definition of what your code should do

Let me explain myself a bit. Image you have to create a simple fizzbuzz api ;-)

• You write code in any language you want
• You put the code produced in a folder
• You write your expectations (openapi defintion or else) of what your api should return
• You run this tool
• You have your result
• You analyze them
• You take a decision based on factual numbers

@the-benchmarker/web-frameworks What do you think ?

PS : The performance depends on too many parameters that any benchmark could never be a universal solution, but we (at least I) want to try to do so ... :heart:

[greed2411]

I thought the same when I made my first contribution. That this is just a simplistic test. But if not for it being single, I wouldn't have made a single contribution.

I aspire the repo to grow in future where we can make a dB call or an external API call. These may introduce other ambiguities because of the external dependencies, still I think we should give it a try in the future.

[iosven]

This performance benchmark should clearly state on the front page what it does benchmark.

Also it should make a statement along these lines: we should all keep reminding ourselves that there are many more non-functional aspects to web frameworks and toolkits, such as

• security
• resilience
• modularity with composability
• clarity, think clean code
• input-validation facilities
• test support
• monitoring support
• integration
• standards (http2, WebSocket, etc.)
• infrastructure compatibility (in times of containers less relevant)
• documentation
• activity around it
• name your aspect

many of which are usually also or even more relevant when making a technology choice.

[waghanza]

sure, we could not pretend to be the ultimate benchmark for 2 aspects :

• we only use 1 scenario (for now) which is not a real-word usage
• performance is only ONE aspect on a technology choice

You can contribute / propose any modification on README :heart: