Web-frameworks: Document methodology

[florimondmanca]

The results exposed in the README are interesting, but they are hard to interpret without context.

I think there should be a dedicated document that explains the methodology used in this benchmark:

• Hardware and infrastructure
• Experiment description: what does each framework has to do during the test?
• Any hypotheses and decisions and how they may affect results.

For example, some (all?) frameworks run in Docker which significantly decreases the results due to network overhead.

There should only be some disclaimers such as the fact that speed does not make or break a framework, or that it can vary greatly in the context of a real world application.

This would only make the benchmark more rigorous and trustworthy by adopting a scientific approach (hypothesis, experiment, results, discussion).

[waghanza]

Yes, you're right. I plan to do this, but in the future. It is probably more frustating for contributors not to have all those kind of informations.

I will create a https://readthedocs.org for this, cause methodology, hardware ... could evolve on each release.

[waghanza]

@florimondmanca There is a list of subject to approach, if you think of anything more ...

• [ ] Hardware used
• [ ] Network used
• [ ] Scenario
• [ ] Difference from https://www.techempower.com/benchmarks

[florimondmanca]

Sounds good! How about software used, too? Thinking about the Python version, the use of Docker (and its limitations? unless this goes under "Network used"), and the actual benchmarking software (wrk I suppose?) but there may be more items.

[waghanza]

I think you have to liste them here