Web-frameworks: How to rank frameworks

[waghanza]

Hi,

As a standard tool, we decide to go on wrk.

This benchmarking tool _give_ us a lot of useful informations.

• Global (per scenario / route to test)
  
  
  
  • Duration
  
  
  • Number of requests
  
  
  • Total bytes received
  
  
  • Number of error (socket / read / write / http / timeout)
  
  
  • Number of
  
  
• Latency
  
  
  
  • Minimum
  
  
  • Maximum
  
  
  • Average
  
  
  • Standard deviation
  
  
• Percentile
  
  
  
  • 50
  
  
  • 90
  
  
  • 99
  
  
  • 99.999
  
  

ALL informations SHOULD be displayed, but COULD be used to determine ranks (I mean display is OK, but to keep it simple now we SHOULD only use 1 metric to rank).

@OvermindDL1 What do you think about talking only the number of requests-per-second to rank ?

Regards,

[OvermindDL1]

All of these values are very useful in different situations, so optimally a section for each sort perhaps with a header jumping to each?

[waghanza]

sure. I think those informations are useful. However, they SHOULD be displayed, not imperatively on the main table, but somewhere ...

In order to keep simple, I suggest choosing 3 or 4 metrics to display on main table :

• Duration
• Requests
• Errors
• Latency (average or deviation, not sure of what)

[OvermindDL1]

Hmm, for active display, perhaps:

• req/s
• average response latency
• 99%ile response latency
• throughput

Honestly errors probably don't need to be shown, if it's anything but 0 ever then a test is broken and it shouldn't be posted anyway

[waghanza]

@tbrand To simplify reading, I think we should display only ONE table of results on README.md (sorted by rank)

@tbrand @OvermindDL1 I have push new results => https://github.com/waghanza/which_is_the_fastest/blob/wrk/README.md#result, computed using our 3 rules

[aichholzer]

It's interesting to see how these results can change, last time this file was updated, rayo was doing ~87K reqs/s and polka around ~86K reqs/s. Now they have both dropped by +10K reqs/s each and the order has shifted.

What kind of consistency are you providing/considering for these tests?
Consistently, in all my tests rayo is still faster.

[waghanza]

Hi @aichholzer, We are adjusting tests parameters, duration, request number ...
I hope to release stable results soon

[OvermindDL1]

@waghanza I'm not sure that adjusting such things should cause such shifting results, that implies to me the server it is running on is either running other code, the network is chatty, it's a cloud machine or VM, etc... etc... My results were fairly stable, less than 1% variance between runs.

[OvermindDL1]

On the last results it shows this:

OS: Linux (version: 4.16.11-100.fc26.x86_64, arch: x86_64)
CPU Cores: 8
threads: 9, requests: 100000.0

Is that the wrk server or is it the server that the servers are running on? Are they running on the same machine (in which case that thread count is super WRONG)? Why is the request count so so very low as many of the servers will not even last a second with such a small amount), and why is it not time based instead of request based (that way you can ensure that the servers warm up properly, especially important for JIT'd languages) and let the siege spool up fully (which with only 100000 requests is too short for many). Also why is it reporting 100000.0 requests when you can only send an integral number of requests? Is the CPU Cores count the number of physical cores or the number of hyperthreads?

[waghanza]

@OvermindDL1 sure, I know it ...

However, I think we SHOULD NOT stopping to run tests ... I know that your 16 cores server is waiting ... I'm trying to find a solution to automate all this things (my idea is to compute results and to display in multi-hardware context, like tfb does, physical, cloud, but in way that COULD be done by others)

[btegs]

What's the easiest way to get wrk installed on Ubuntu 18.04 with dependencies and such?

[aichholzer]

@btegs

sudo apt-get install build-essential libssl-dev git -y
git clone https://github.com/wg/wrk.git wrk
cd wrk
sudo make
# move the executable to somewhere in your PATH, ex:
sudo cp wrk /usr/local/bin

[waghanza]

@tbrand Thinking about it, I am not sure that ranking frameworks has to be done.

Actually, we display informations about ranking only from the req/s perspective

However, each frameworks has (at least) 3 kind of information :

• The number of requests that could be send by the server
  
  
  
  • req/s
  
  
  • latency
  
  
• The bandwidth used between server and client (very useful information as wireless environments)
  
  
  
  • throuput
  
  

@OvermindDL1 what do you think ?

[OvermindDL1]

req/s and latency are definitely big ones to show, as for throughput in relation to wireless and such I think is less big, rather the size of each request where it explicitly lists that this is with their default header set could be useful to list if you want to go that way (easily tested by doing a request on a server and just seeing the total size returned). For throughput to be a useful bit of information overall each server would need to return the exact same information, which is doable though.

[waghanza]

sure, we could optimize all this frameworks (for throughput), but is this very what developers does ?

as I am a developer, I prefer (mostly) use out-of-the-box things (not customising ...) and will it be fare to show results after tweaking (not default ones)

[OvermindDL1]

Heh, exactly my point, thus throughput is probably not useful to rank on then as they are not testing the same thing.

[waghanza]

@OvermindDL1 that's why is propose to remove ranking from README, or having a strong ranking (e.g : using at least req/s and latency)
for example, akkahttp is 16/40 but has a huge latency, more than perfect (which is 17/40)

Language (Runtime) | Framework (Middleware) | Requests / s | Latency | 99 percentile | Throughput
-- | -- | -- | -- | -- | --
scala | akkahttp | 59853.33 | 210753.00 | 4777700.33 | 47.02 MB
swift | perfect | 56951.00 | 17329.67 | 24624.33 | 16.87 MB

[OvermindDL1]

The actor model tends to have a high latency in exchange for throughput, the BEAM languges (erlang/elixir) tend to do the same. But yeah, a strong ranking like that is good, graphs especially to give more visual details.

I was surprised that akka was that high when I saw it though, I wondered why but not hooked up a profiler as of yet... ^.^;

[waghanza]

@OvermindDL1 I think we can remove this ranking (on the actual README), in order to open a clean PR for this

[waghanza]

@tbrand what do you think ?

[tbrand]

@waghanza
Ranking is most valuable metrics for this repository.
I guess that people who give us starts want to know the simple results, not for complex profilings.

Of course we can show the detail results (expanded markdown for ideal) but I want to show a simple ranking at first view as I said at somewhere.

[waghanza]

@tbrand sure, I'm total up this idea
However, the actual ranking, is not accurate (based only on req/s).

For example, with the battle on frameworks in :

• node
• ruby
• php
• python

actual ranking shows :

1. node
2. php
3. python
4. ruby

but, when taking req/s + latency, we have :

1. node
2. ruby
3. python
4. php

My point is NOT to definitively remove the ranking, but prevent showing nasty results, either by removing (temporarily) from the README or using latency

I do not have personally enough time to deal with this.

I think, we SHOULD remove rankin+ having a banner onREADME` that explain why not having ranking infos ?

What do you think ?

[tbrand]

What req/s + latency means? Just sum up them?
For users, req/sec is the most important metrics. Not latency and throughput.

[waghanza]

@tbrand Yes, this example what just a poor / basic operation

rank => (+)req/s (-) latency

latency is important since it represent time between request and response

[tbrand]

But it's duplicated since req/sec includes the latency metrics.
As you says it's too poor since the unit of them are different. (req/sec is [scaler/sec], latency is [sec])

[waghanza]

@tbrand req/s is only https://github.com/wg/wrk/blob/master/SCRIPTING#L108

what I have understood from last weeks (working on this project)

• req/s refer to the number of sent requests (to the server)
• latency refer to the number of received requests (that have been sent)

@OvermindDL1 Am I right ?

PS : For me ranking SHOULD refer to what consumer feels when using an app

[tbrand]

What? So the unit of req/sec is scaler? The name of req/sec is completely wrong!

[OvermindDL1]

Incorrect:

req/sec: Is the number of completed requests per second, as in open the connection, set up the TCP state, send headers, received data back, (traditionally closing the connection is not counted and I don't think it is here either).
latency: Is the amount of time it took to complete a request on average, from connecting to receiving the completed data (which is a single packet each way after setup in these simplified tests). This is why it has things like 99th percentile and so forth as things like 99th percentile shows what the slowest 1% of the slowest requests take, and 99.99th percentile shows what the 0.01% of the slowest requests take, which are highly useful metrics for showing worst-case timings (hence traditionally you have 50, 75, 90, 99, 99.99, 99.9999 or so for the percentiles).

Subtracting latency from req/sec is not a useful metric as they have different magnitudes.

[OvermindDL1]

Thus a library that can do 500k req/sec at an average latency of 300ms and a 99th percentile latency of 1200ms, and another library that can do 200k req/sec at an average latency of 20ms and a 99th percentile latency of 80ms have super different results depending on what you are optimizing for. In general the one with higher latency but higher req/sec is better when you want to optimize for massive connection counts on a single server, and the one with lower req/sec but super low latency is fantastic for low-rate servers that need to return data super-fast (think of an advert server). (And of course some languages/frameworks are great at both.)

[waghanza]

@OvermindDL1 is right, sorry for mis-leading words

[waghanza]

@OvermindDL1 @tbrand So the question is : which metric(s) SHOULD be used to represent to represent the rank (which context) ?

For me, but perhaps because of my knowledge, I think we want to rank in a web context (obviously back). So, I think, taking only req/s is NOT ideal / accurate

[waghanza]

@tbrand If we refer to initial goal, in the README

Measuring response times

I think ranking by latency is more relevant than by req/s

[OvermindDL1]

Ah, I missed that, if Measuring response times is paramount, than latency is what is wanted (at the very least average latency and, say, 99th or 99.99th percentiles or just average and deviation).

[waghanza]

@tbrand We can also display two tables on README :

• 1 sorted by latency
• 1 sorted by req/s

The fact is we are comparing web tools. For me, response time (latency) is an important aspect when we are in a so-called back-end perspective (API), and request time (req/s) is an important aspect when we are in a so-called front-end perspective.

@OvermindDL1 That's why, I think, display two tables (+ explanation) will be accurate

[waghanza]

@OvermindDL1 just for information

With req/s, I found :

• C++
• Go
• Rust
• Java
• Nim

With latency, I found :

• Rust
• Ruby
• C#
• C++
• Java

[OvermindDL1]

Makes sense, the C++ library I was using uses a deferred networking system behind it (libevent), so it tries to maximize throughput to handle the maximum number of connections possible via batching calls. I'm guessing the average Rust library does the opposite, but it would be interesting to see individual projects. Tradeoffs everywhere and it's the kernel calls and such that determines how it will be tuned (with language overhead on top of course). :-)

[waghanza]

Actix (Rust) use the same system than evhtp, but the Rust framework that has the best rank on latency is nickel ;-)

It is this astonishing that Ruby (roda) is better that C# or java

[OvermindDL1]

Ruby is very built for low-latency because of how long it's code can take to run, but that means that it can't overlap calls easily, hence low latency but horrible req/sec.

[waghanza]

sure, but it make me asking myself if having only low latency is the only metric we should use for back-end stuff

some back-end stuff, like API in python are faster than ruby ones
however, in result japronto has a latency of 21209 but roda has a latency of 3166

[waghanza]

@OvermindDL1 @tbrand I was thinking of a display, like this

Ranked by latency

| Language (Runtime) | Framework (Middleware) | Requests / s | Latency | 99 percentile | Throughput |
|---------------------------|---------------------------|----------------:|----------------:|----------------:|-----------:|
| ruby | roda | 38079.00 | 1735.67 | 27182.00 | 12.67 MB |
| ruby | rack-routing | 31005.33 | 2121.33 | 29419.33 | 5.65 MB |
| ruby | flame | 19051.33 | 3362.00 | 36238.00 | 3.50 MB |
| ruby | sinatra | 15901.00 | 4026.67 | 45941.33 | 13.54 MB |
| python | japronto | 82996.67 | 11618.67 | 13816.00 | 33.43 MB |
| ruby | rails | 4154.67 | 15418.00 | 99327.00 | 3.77 MB |
| node | polka | 77130.67 | 19931.33 | 323806.67 | 36.62 MB |
| node | fastify | 66254.67 | 23568.67 | 375535.33 | 65.65 MB |
| node | rayo | 64014.67 | 23681.33 | 396522.67 | 31.35 MB |
| node | express | 47145.33 | 46062.67 | 913263.33 | 39.29 MB |
| python | flask | 18397.00 | 56160.00 | 155783.00 | 14.98 MB |
| python | sanic | 14197.67 | 67893.33 | 132685.00 | 8.62 MB |
| python | django | 10958.00 | 94362.33 | 420568.00 | 10.30 MB |
| php | symfony | 42371.00 | 169928.00 | 2860775.33 | 70.48 MB |
| php | laravel | 30946.33 | 309246.00 | 4397344.67 | 49.90 MB |
| python | tornado | 1878.33 | 528993.67 | 3523509.00 | 1.34 MB |

Ranked by requests per seconds

| Language (Runtime) | Framework (Middleware) | Requests / s | Latency | 99 percentile | Throughput |
|---------------------------|---------------------------|----------------:|----------------:|----------------:|-----------:|
| python | japronto | 82996.67 | 11618.67 | 13816.00 | 33.43 MB |
| node | polka | 77130.67 | 19931.33 | 323806.67 | 36.62 MB |
| node | fastify | 66254.67 | 23568.67 | 375535.33 | 65.65 MB |
| node | rayo | 64014.67 | 23681.33 | 396522.67 | 31.35 MB |
| node | express | 47145.33 | 46062.67 | 913263.33 | 39.29 MB |
| php | symfony | 42371.00 | 169928.00 | 2860775.33 | 70.48 MB |
| ruby | roda | 38079.00 | 1735.67 | 27182.00 | 12.67 MB |
| ruby | rack-routing | 31005.33 | 2121.33 | 29419.33 | 5.65 MB |
| php | laravel | 30946.33 | 309246.00 | 4397344.67 | 49.90 MB |
| ruby | flame | 19051.33 | 3362.00 | 36238.00 | 3.50 MB |
| python | flask | 18397.00 | 56160.00 | 155783.00 | 14.98 MB |
| ruby | sinatra | 15901.00 | 4026.67 | 45941.33 | 13.54 MB |
| python | sanic | 14197.67 | 67893.33 | 132685.00 | 8.62 MB |
| python | django | 10958.00 | 94362.33 | 420568.00 | 10.30 MB |
| ruby | rails | 4154.67 | 15418.00 | 99327.00 | 3.77 MB |
| python | tornado | 1878.33 | 528993.67 | 3523509.00 | 1.34 MB |

[OvermindDL1]

Looks good, though I'd recommend putting what the value is in latency, I'm guessing it is microseconds(µs)?

[waghanza]

@OvermindDL1 good catch, but I prefer display in ms, I find it more human-readable

[OvermindDL1]

@OvermindDL1 good catch, but I prefer display in ms, I find it more human-readable

Likewise, but the values that are there now did not seem to be that, thus conversion necessary. :-)

[lukeed]

TBH, I didn't follow the full thread, but this seems like the appropriate place to leave this concern:

The frameworks should all be deployed on a DigitalOcean droplet with 1 CPU only. A lot of these frameworks (or the languages themselves) naturally take advantage of all available cores, while most others require manual setup / a flag to enable such behavior.

For example, Crystal & Node.js require manual process forking. Rust frameworks are variable, but most will automatically spread across cores. And I believe go/fasthttp fans out across cores as well.

The point is that deploying to a single-core server server to normalize _all_ test subjects without any code changes. And when interpreting results, it's much easier to reason about since everything is forced to be single-core. You get a nice unit of comparison.

The wrk runner can & should run on a separate machine.

[waghanza]

Hi @lukeed,

You have 2 suggestions.

1. Deploy to 1 CPU only droplets

Honestly, I didn't get the point. It's a language advantage to use all cores / CPUs. I do not understand why not taking count on this ?

And, for example, pm2 is in charge of spreading node process here :stuck_out_tongue:

1. wrk

Sure, the metrics shown on README.md are computed from docker host. The metrics will be computed on cloud, of course with a remote sieger

[lukeed]

1. Yes, but if you are running a single-threaded language, you almost always change the deployment configuration to account for it. So that, instead of paying for an 8-core machine (and only using 1), you either
   (A) manually fork the process, if possible;
   (B) use a tool like pm2 to fork for you, if available;
   (C) or load balance a cluster of 8 x 1CPU machines

My point is that it's unfair to report requests/sec & latency for single-core vs multi-core languages _without_ normalizing the behaviors. As mentioned, some frameworks _within the same language_ have different approaches to multi-threaded processing. Running everything on a single core is a good unit of measure & is the easiest way to normalize across all behaviors.

At the end of the day, multi-core server is a convenience. Any server build on any language/framework can utilize 8 cores one way or another. It's just a matter of whether it's built-in and done-for-you or whether it is not. That's why it's unfair to compare 8 against 1.

[waghanza]

@lukeed We use pm2

Here a list of app server in use :

• php => php-fpm / nginx
• node => pm2
• python => gunicorn (async)
• ruby => puma

All app server spread on all cores, it's not a framework feature, but an environment behavior

The point you reveal is right, this is not fair to compare single thread / core languages with multi-cores ones ... However, it's not faire to downgrade multi-core / threads.

I will say that metrics are important, but their are only metrics. We must rank depending on use case, that why I propose to split ranks, latency for back-end stuff and req/s for front-end stuff.

I think it's not fair neither to compare node and haskell, the only comparison is about performance not language, and performance depend on use case

Even if some languages can not utilize all cores, it's a choice of language design (it COULD be either a pro or a con, depending of use case)

[lukeed]

I fully understand. But as a quick retort: Your shortlist is missing Crystal & Rust. And, as a final point, you also introduce overhead which (of course) detracts some % from the overall performance. For example, since long-running uptime doesn't matter for a 5 minute benchmark, you could use this instead of pm2 & the memory usage and latency would improve:

const { cpus } = require('os');
const cluster = require('cluster');
const app = require('./app');

if (cluster.isMaster) {
  let i=0, cores=cpus().length;
  for (; i < cores; i++) {
    cluster.fork();
  }
} else {
  app.listen(3000);
}

The same thing (barring different syntax) can be done in Rust & Crystal.

This is the other approach to "downgrading" the multi-core languages. The trouble is it's more work & requires more knowledge to do it correctly per language (and per framework).

FWIW, I keep saying "per framework" because there are Node.js & Rust frameworks that do the cluster.forking snippet above.

Of course, you're more than welcome to disregard all of this & keep reporting how you'd like 😅

If that's the case, my final request would be that you report the "per-thread" result from wrk and not the totals/averages.

[waghanza]

Your shortlist is missing Crystal & Rust

Rust is actually the first (actix), and crystal is down the list (due to dockerization)
if you want to add other language, feel free to PR

And, as a final point, you also introduce overhead which (of course) detracts some % from the overall performance

Sure, I will take inspiration with tfb to add a more stable results (has a warm up phase before running severals phases*)

The same thing (barring different syntax) can be done in Rust & Crystal

You mean spreading on all cores ?

This is the other approach to "downgrading" the multi-core languages

How ?

[lukeed]

Yes, actix is perfect example. It has multi-threaded built into the framework. But if you used tokio-minihttp, another popular Rust server, it is single-threaded only & you (the developer) are supposed to handle the multi-threading yourself.

Effectively, actix includes my snippet above & tokio-minihttp doesn't. These are just two names, but this happens all the time across Node & Rust -- and I'm starting to see it in Crystal too.

So now you, the benchmark maintainer, need to know of these differences and actively maintain them, or request that PRs that introduce them as a test subject need to do their homework before merging. That's a lot more work!

This is why I initially suggested running all languages / frameworks on a single-core machine. It means you don't have to worry about all the details of each framework. All single-core languages can be clustered to operate as a multi-cored cluster. Similarly, all multi-core languages can operate on a single core just fine. (Most multi-core-oriented languages deal with threads/coroutines anyway, and so are not limited to number of physical cores on the machine.)

As a final note, here are some Polka (Node.js) numbers on my shitty laptop:

• Plain, single-core

  Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency     6.90ms    1.00ms  18.86ms   69.44%
      Req/Sec     3.64k   334.65     4.73k    71.53%
    146229 requests in 10.10s, 13.81MB read
  Requests/sec:  14475.59
  Transfer/sec:      1.37MB
  

• PM2, 6 threads

    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency     1.99ms    2.30ms  73.22ms   94.71%
      Req/Sec    14.04k     1.44k   17.22k    86.00%
    558666 requests in 10.00s, 52.75MB read
  Requests/sec:  55848.44
  Transfer/sec:      5.27MB
  

• Native Cluster, 6 threads

    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency     1.95ms    1.93ms  44.03ms   92.54%
      Req/Sec    14.80k     1.39k   20.93k    83.87%
    593223 requests in 10.10s, 56.01MB read
  Requests/sec:  58727.34
  Transfer/sec:      5.54MB
  

As you can see, pm2 includes overhead that can be measured. You can't and don't want to be held accountable for knowing all the best practices or tricks for each language & framework.

[waghanza]

As for actix, the implementation was done by @fafhrd91, I assume it is a correct as for a rust developper

pm2 is used on all cores, I assume it is considered as a best-practice in node communities.

Neither do I nor @tbrand (nor @OvermindDL1) COULD reasonably maintain ALL implementations, each developer have specialization(s).

Our goal is not to teach the word anything :stuck_out_tongue_closed_eyes: just to learn and to gather communities ;-)

For example, @thiamsantos made several implementations, the PR waiting (wai / haskell) is not mine

I understand your concern about how hard it is to maintain all this, but it not because it is hard, that it is not doable :tada:

If your have any tips or any advice, we are open

The same retort COULD be saied about https://www.techempower.com/benchmarks and any other benchmarking projects

[lukeed]

It _is_ correct, that's my point. And so would any implementation with tokio-minihttp. The problem is that have a mixed & inconsistent set of benchmark targets. Multi is mixed with single & there's no indication of which items are which.

This makes the entire result set meaningless -- sorry.

In the old benchmarks, this wasn't nearly as much of a problem. But now that this repo is reporting latency & throughput metrics, you need to do it right.

[waghanza]

I see your point ... but I think those arguments COULD be used for some projects mature such as https://www.techempower.com/benchmarks

The purpose is to be a representation of some results, but the meaning as to be done with a minimum knowledge about tools (what will it be used for, is it for doing full-stack stuff or else)

I plan to add some categorization, to enable some filtering. I'm aware that ONE language (tool) COULD never be a universal solution

[lukeed]

Techempower benchmarks are useless, that's why I'm hopeful for this project.

Currently the results are a misrepresentation of true performance. They're only a record of various frameworks run on a cloud server w/ no methodology to them.

Let's fix this~! 👍

[alexhultman]

@lukeed Don't even bother with this repo. Clearly they have no idea what they are doing here.

I was blown away by the fact they literally put single-threaded servers head-to-head with servers that automatically use all CPU cores. Just sad to see the incompetence in this repo.

Honestly, I didn't get the point. It's a language advantage to use all cores / CPUs. I do not understand why not taking count on this ?

It's not a language advantage to make use of all CPU cores - any and all languages can do this, you're just too incompetent to set this up. Either you deploy 8 (if 8 is number of logical CPU cores) instances of a single threaded server and compare that to 1 instance of a server that automatically makes use of all CPU-cores, or you limit the hardware to 1 CPU core. Aything else is purely retarded and completely void.

I know servers that in one single thread perform 90% of what Actix Web does in 8 threads. Instead of invalidly showing Actix Web as top server you need to go back to school and learn how to set up a local cluster of single-threaded instances (this is pretty much a 101 thing when it comes to Node.js and similar servers).

Completely incompetent test.

[aichholzer]

@alexhultman -Most of the time I like your writing and your no-bullshit and to-the-point comments, but in this case -I think- the additional harshness was not warranted.

As for the rest, I'm with @lukeed -All tests should be run on a single CPU/core server. Comparing single-core to multi-core frameworks (when not done programmatically) is not a fair meassure.

[waghanza]

@lukeed

Techempower benchmarks are useless, that's why I'm hopeful for this project

Why ? You can with :heart: propose any modification, either here or in tfb

@alexhultman

Clearly they have no idea what they are doing here

Let's give use some ideas then

@aichholzer

All tests should be run on a single CPU/core server

I still do not understand why. The current usage (for production environments) is not to run on mono-thread / mono-core

• Results are non ready to be considered as stable, their are still computed for a local docker host. I intend to have this docker to let developers add some frameworks, while I'm trying to run this in cloud
• Our intention is to measure performance, but performance COULD not be compared the same way for each technology, comparing node to haskell (for example) IS irrelevant (not useful), that's why I have proposed have a use-case comparison (code is not pushed but I'll send a PR in the next days)
• Node is not single-thread

[OvermindDL1]

(has a warm up phase before running severals phases*)

As my benchmark script did.

Techempower benchmarks are useless, that's why I'm hopeful for this project.

Eh, don't, any microbenchmarks like this and techempower are overall useless, I consider this just a fun little thing. It can be useful to test overall network backend handling speed however, which is where I find this to be a bit useful, but still not perfectly accurate (especially since most of the languages use a C socket backend anyway).

This makes the entire result set meaningless -- sorry.

It always has been meaningless except overall differences between when things are implemented properly. But overall the tests should absolutely not be limited to single core, they should at minimum use 4-core if not more like 8 or so (I'm used to dealing with 64 cores). Many frameworks will not hit their full stride and capabilities until they are many-core because of dispatch threads and so forth and thus you are highly mis-representing them. All languages, including the crystal ones and all, should perform proper multi-threading as suggested by the library being tested.

[alexhultman]

@OvermindDL1 utter bullshit.

You have a wiiiiiiiide varity between different servers. Benchmarking them is essential in finding out if they are worth your time. What is important is knowing how to properly benchmark something. This is the issue with this whole repo: it completely lacks any kind of investment per server.

Basically, this repo is the equivalence to benchmarking different cars and drawing the conclustion that automatically geared cars are the fastest. And the reason boils down to the test driver being too incompetent to even shift gear on all the manual cars, so essentally ends up running only on gear 1.

Techempower is worthless for different reasons. Mostly because they mix a bunch of different variables and present them as if they represent 1 single variable, namely the server. This is invalid science for so many reasons (yes, proper benchmarking is a science and follows the same logical conclusions and proofs).

[waghanza]

@alexhultman You seems to have a great knowledge, why not helping (instead of injuring) ?

[alexhultman]

I'm not "injuring" anything or anyone. I'm criticizing fake information. In a healthy and free democracy freedom of critique is extremely fundamental and is how we separate fake bullshittery from actual scientific knowledge. If everyone got help with everything we would not have any kind of evolution or competition whatsoever and we would deteriorate into a socialistic social justice warrior safety net society without any kind of actual skepticism and free thinking. Do not assume everyone is as socialistic and gender-neutral as you, because most people are not.

[waghanza]

@alexhultman You are criticizing this project, I think this _approach_ is good.
However criticizing _alone_ is NOT helpful, in my point of view.

The question here is : How to rank frameworks ? not deciding which framework is better ?
(the comparison here is raw performance, nothing more)

@alexhultman If you have any idea to help, feel free to give you opinion, in which metric could be used, it is an OPEN discusion, then :stuck_out_tongue:

[OvermindDL1]

Benchmarking repo's are inherently broken in many ways and are in no sort ever able to be considered accurate, if you do it for any reason other than "It is fun!", then it is misguided.

If you see issues with a project (of which I see MANY with this one), then PR and discuss to fix it, attacking is generally not productive.

This specific issue is not about how things like the crystal servers are inherently broken here because they are not distributing across cores properly or so, thus issue is about how to rank. The methods of testing (of which I could easily see amount-of-cores to be such a different dimension to the results as I've mentioned in a few other places, but is not implemented currently) is not part of this specific issue. :-)

If everyone got help with everything we would not have any kind of evolution or competition whatsoever and we would deteriorate into a socialistic social justice warrior safety net society without any kind of actual skepticism and free thinking. Do not assume everyone is as socialistic and gender-neutral as you, because most people are not.

Now bringing up things that have not only nothing to do with this issue, but nothing to do with the project or coding in general is attacking. Stick with coding in projects such as this as the information and the merits of the information being discussed and PR'd are what matter.

[alexhultman]

Arguing with random socialists on the internet, yaaaaaay! Basically GitHub is entirely full to the brim with socialists arguing for world peace and gender neutrality.

Criticism alone serves as a pointer for knowing if you're headed in the right direction or not. Sometimes (most of the time) people don't have the time to freely work on a project they from the start consider to be fundamentally broken. In those cases criticism is all you get.

If you don't want criticism then don't post a public project where you claim to know what you're doing. Welcome to reality, grow up and get a job. We can't all be tree hugging hippies spending money we didn't earn ourselves.

[lukeed]

There have been at least three ideas that have been presented in this discussion to fix the approach taken. That _is_ help. Beyond that, you're asking us to commit code changes to your code base, which is something that very few people have the time & luxury to afford.

All suggestions have been met with a "not understood" , a "I disagree", or "they're good as they are, we don't need to be perfect". All are dead ends, which means I have no reason to continue in the discussion.

And, by the way, sending in PRs for the proposed fixes seemingly would also be a waste of time. One can only assume that you'd still have the reasons of disagreement, understanding, or complacency with current setup.

So, sorry but I can't help any further. Please don't claim there was no help offered in this discussion as that's simply not the case. And, if you read passed the flair of Alex's words, he's trying to help too.

[OvermindDL1]

Criticism

Criticism is good, attacking is not.

Arguing with random socialists on the internet, yaaaaaay! Basically GitHub is entirely full to the brim with socialists arguing for world peace and gender neutrality.

Like this is attacking, not criticism (as an aside I and many I know are not socialists by any stretch, personally I'm a subset of transhumanist with a focus on the singularity, of which that nor genderisms I don't see have anything at all to do with programming or functionality testing).

If you don't want criticism then don't post a public project where you claim to know what you're doing.

Criticism is, again, good, look at many of my posts where I point out a rather massive amount of issues with this project along the vein of fixing them and how to progress to such fixes.

There have been at least three ideas that have been presented in this discussion to fix the approach taken.

I saw nothing else stated as to my statement about how 1 CPU would unfairly hobble multiples of the frameworks (anything based on the BEAM for one big example), how would you work around that by using only 1 CPU when every framework can work with multiple CPUs and all can reach their maximum speed in such a situation in addition to 1 CPU servers being highly unusual in the webserving world (I can't say I've ever actually seen it, 4 native cores were the tinyiest ones I've personally seen, and depending on what is being hosted using 8 to 128 cores is common)? Thus your latest post also seems quite hostile without moving the conversation forward?

[lukeed]

Because multi-core languages can still run on a single core, and the argument that maximum speed only is achieved with addition of more cores is moot because that applies to literally every language.

That said, your suggestion is still better than the current setup (albeit flawed, see my reasons above) because you called for consistency. You suggested that all frameworks carefully be run utilizing all four cores, in whichever manner. (4 vs 8 doesn't matter.) That's been exactly the point of my initial post. One way or another, there needs to be consistency. Because even if it's flawed, the consistency allows for a chance of some insight -- the current setup has none.

And again, that's because the current suite has multi-core frameworks pinned against single-core frameworks, with selective attempts at patching them -- and with some mc frameworks pinned against sc frameworks within the same language!

Thus your latest post also seems quite hostile without moving the conversation forward?

FWIW, loosely (and falsely) accusing others' actions as hostile _is_ in itself hostile.

---

This will be my last post. Came with good intentions, still wish this project the best! Hopeful from afar.

[waghanza]

@alexhultman

Arguing with random socialists on the internet, yaaaaaay!

This is quiet a political concern. I'll be glad to discuss about that, but I thing this is not the place to such a conversation, we can use a conversation service if you want (gitter, discord, whats app or else :stuck_out_tongue: )

You have arguments on political opinion, it will be more that ok, if you have the same opinion about our project :heart:

@lukeed

There have been at least three ideas

I do not count if only three :stuck_out_tongue: but all are ideas a taking.

Please see the results re-organization I'm up to, https://github.com/waghanza/which_is_the_fastest/blob/clarify_results/README.md

And, if you read passed the flair of Alex's words, he's trying to help too

sure, but with a huge amount of aggressively

Because multi-core languages can still run on a single core

Yes, but you are not using multi-core languages at their full value if running on a single core

maximum speed

speed or performance ?

That said, your suggestion is still better than the current setup

The current setup, docker was choose to avoid having a bunch of requirements to set-up before playing with this tool :stuck_out_tongue: we are ALL aware that this is not a production-ready tool, like tfb

@lukeed we are going to grow, you can not expect anything to be relevant at the starting point

[OvermindDL1]

Because multi-core languages can still run on a single core, and the argument that maximum speed only is achieved with addition of more cores is moot because that applies to literally every language.

But it doesn't though, there are some frameworks where a core gets owned by a scheduler and it will only run efficiently if multi-core, in some hardware chipsets (like the propeller chips that I work with on occasion) that is even enforced by the hardware design (fascinating little things, wish they took off ^.^).

That said, your suggestion is still better than the current setup (albeit flawed, see my reasons above) because you called for consistency. You suggested that all frameworks carefully be run utilizing all four cores, in whichever manner. (4 vs 8 doesn't matter.) That's been exactly the point of my initial post. One way or another, there needs to be consistency. Because even if it's flawed, the consistency allows for a chance of some insight -- the current setup has none.

That's what I've been saying for over a year now as well, that is why my benchmarker always reported the precise Core count and HyperThread count in addition to the actual threads spooled up and is why I keep saying that frameworks like Crystal here are fixed as (last I checked) they were using only a single core, regardless of how it was spooled up, which is an entirely unrealistic setup. :-)

And again, that's because the current suite has multi-core frameworks pinned against single-core frameworks, with selective attempts at patching them -- and with some mc frameworks pinned against sc frameworks within the same language!

Precisely! Also what I keep saying. :-)

FWIW, loosely (and falsely) accusing others' actions as hostile is in itself hostile.

True it can be, it was just the way I inferred it (which to be honest was biased by the other persons extremely hostile posts) due to parts of it like All suggestions have been met with a "not understood" , a "I disagree", or "they're good as they are, we don't need to be perfect". All are dead ends, which means I have no reason to continue in the discussion. that I've always seen elsewhere in very negative situations (conversation is never worth being cut off, especially with the 'not understood' responses or via things like 'I disagree' but without evidence being shown as to "why").

speed or performance ?

Yeah this has been a sticking point on what to decide, as req/s is more of performance and latency (especially with statistical deviations on high 99.99% values) is more of speed, hence why I'd think both should be reported individually. :-)

The current setup, docker was choose to avoid having a bunch of requirements to set-up before playing with this tool stuck_out_tongue we are ALL aware that this is not a production-ready tool, like tfb

Docker is useful for that, but it also incurs a slight overhead, so that is just one more thing to add to the inaccuracy, but at least it is a 'fairly' stable slowdown across all tests so it seems reasonable. :-)

we are going to grow, you can not expect anything to be relevant at the starting point

+1

Heck, look at when this was set up, the benchmarker itself was actually written in Crystal, while only using a single core, so it couldn't saturate any of the webservers except the really slow ones like rails or flask (not even the crystal web server!), the project has already come quite a bit further. ^.^

[alexhultman]

Heck, look at when this was set up, the benchmarker itself was actually written in Crystal, while only using a single core, so it couldn't saturate any of the webservers except the really slow ones like rails or flask (not even the crystal web server!), the project has already come quite a bit further. ^.^

This alone makes me shiver in disgust. You are biting off more than you can chew. Obviously, very obviously, nobody should trust ANY of these results since you have absolutely no idea what you are doing. You are incompetent beyond repair and should not be spreading such fake information to gullible users.

[btegs]

I'm just curious as to how Sanic keeps on dropping in the rankings. They did remove the code to make it faster (like turning off logging and debug) and that isn't helping it.

[tbrand]

Sorry for the late reply since Im at Guam for vaccination!

I totally agree with the expandable 2 rankings as you showed me.
The latency ranking should be top and expanded by default for ideal since I would like to show a ranking at first view.

[waghanza]

@btegs sanic seem to be low, but as the setup (docker based`), results are not yet stable (fluctuating), we can not take any conclusion, before the next big step -> using a cloud to compute** results

@tbrand glad we are ok

[brandonros]

By what metric is actix-web #1? I must be missing something but it isn't clear based on the table data why it's #1, and it also isn't easily deducible by what margin is it ahead of #2, etc.

[waghanza]

Hi @brandonros,

The results, as presented for now, are totally unclear (until we merge #281).

Actual ranking, https://github.com/tbrand/which_is_the_fastest#ranking-framework, is base on req/s.

The next ranking, https://github.com/waghanza/which_is_the_fastest/tree/clarify_results#results, are separated in two (already ranked) tables :

• one by req/s
• one by latency