Frameworkbenchmarks: Question: regarding c# Frameworks / Names

[surfmuggle]

In the latest techempower benchmark are several "frameworks" for c# but as far as i can tell a few of them are the same:

In this spreadsheet are the c# frameworks listet and also a few dockerfiles were copied.

Framework | JSON seri | Single query | Multiple queries | Fortunes | Data updates | Plaintext
--------------|-------------|----------------|--------------------|-----------|-----------------|-----------
Best framework* | 1,199,614 | 659,613 | 43,168 | 424,339 | 17,811 | 7,025,632
aspcore | 0.798 | - | - | - | - | 0.999
aspcore-ado-pg | - | 0.558 | - | 0.697 | 0.275 | -
aspcore-mw | 0.611| - | - | - | - | 0.441
aspcore-mw-ado-pg | - | 0.449 | 0.489 | 0.571 | 0.262 | -
aspcore-mvc | 0.302 | - | - | - | - | -
aspcore-mvc-ado-pg | - | 0.258 | 0.431 | 0.252 | 0.266 | -
aspcore-mvc-dap-pg | - | 0.248 | 0.427 | 0.239 | 0.262 | -
giraffe | 0.319 | - | - | 0.104 | - | 0.205

*) The best framework contains the value for any framework with the highest result for a certain test type. The values for the dotnet frameworks is the quotient (framework x result divided by best result of best framework)

Could someone help me to understand why they are separated this way?
It would be nice that for example for aspcore-mvc-ado-pg all the test were executed.

[msmith-techempower]

@benaadams Do you know either the answers to this or to whom the question should be asked?

I am closing this issue (as it is not an issue), but feel free to continue the discussion.

[benaadams]

There are non-db tests (plaintext and json) as well as database tests (Single query, Multiple queries, Fortunes, Data updates)

Platform level aspcore

aspcore is against the server apis; no "framework"; and no database (no db tests)

aspcore-ado-pg is platform level, using no ORM and using postgres database (no plaintext/json)
aspcore-ado-my is platform level, using no ORM and using mysql database (no plaintext/json)

Micro-"framework" level aspcore-mw

aspcore-mw only uses aspnet core middleware; and no database

aspcore-mw-ado-pg only uses aspnet core middleware; using no ORM and using postgres db
aspcore-mw-ado-my only uses aspnet core middleware; using no ORM and using mysql db

aspcore-mw-ef-pg only uses aspnet core middleware; using Entity Framework ORM, and using postgres db
aspcore-mw-ef-my only uses aspnet core middleware; using Entity Framework ORM, and using mysql db

aspcore-mw-dap-pg only uses aspnet core middleware; using Dapper ORM and using postgres db
aspcore-mw-dap-my only uses aspnet core middleware; using Dapper ORM and using mysql db

Full-"framework" level aspcore-mvc

aspcore-mvc uses aspnet core MVC; and no database

aspcore-mvc-ado-pg uses aspnet core MVC; using no ORM and using postgres db
aspcore-mvc-ado-my uses aspnet core MVC; using no ORM and using mysql db

aspcore-mvc-ef-pg uses aspnet core MVC; using Entity Framework ORM and using postgres db
aspcore-mvc-ef-my uses aspnet core MVC; using Entity Framework ORM and using mysql db

aspcore-mvc-dap-pg uses aspnet core MVC; using Dapper ORM and using postgres db
aspcore-mvc-dap-my uses aspnet core MVC; using Dapper ORM and using mysql db

giraffe is the F# Giraffe framework ; which uses aspnetcore as a base and then goes its own way; also is more friendly to use in the F# language; as it has different ways of doing things than C#.

ORMs (Object-relational mapper)

No ORM means directly using the ADO database abstractions DbProviderFactory,DbConnection,DbCommand and directly running queries ExecuteReaderAsync etc

Entity Framework Core is made by Microsoft

Dapper ORM is made by StackExchange (the Stack Overflow people)

Why don't they all execute all the tests?

It would be nice that for example for aspcore-mvc-ado-pg all the test were executed.

So for plaintext and json; it doesn't matter what the db, or ORM choice is; as they are not used in the tests; which is why the test has no postfix qualifier. So the db versions would all be the same as the non db version. The database connectivity doesn't play a role in the tests.

So the database tests; its the same level; as the one above, however it also includes the database choice and the database ORM used for that database.

HTH

[benaadams]

There are also aspcore-mono-xxx which is the same as they above but using Mono as the runtime rather than CoreClr; and the Carter and Nancy frameworks, which use aspnetcore as a base, but then go their own way (like giraffe).

There is even peachpie which is a PHP language compiler and runtime that compiles to CIL so runs on CoreClr and sits on top of aspnetcore and Kestrel; which can compile Wordpress to a .NET dll and run it on aspnetcore.

The next round includes aspcore-corert which is using the CoreRT AOT compiler and runtime; rather than CoreClr; also some aspnetcore-vb-xxx variants which are using VB.NET rather than C#.

And finally aspnet-mono-ngx which is the classic aspnet (pre-ASP.NET Core) pipeline; on mono, using ngnix as the frontend server (rather than Kestrel); but its not so good... However, it does serve as a good historic baseline of what performance on Linux used to be with the old stack as you can do a like-for-like comparison of Mono using the old stack and the new stack:

[surfmuggle]

Hello @benaadams ,
your reply

So for plaintext and json; it doesn't matter what the db, or ORM choice is; as they are not used in the tests; ...
So the db versions would all be the same as the non db version. The database connectivity doesn't play a role in the tests.

to my question Why don't they all execute all the tests? made things clear.

the database tests; its the same level; as the one above, however it also includes the database choice and the database ORM used for that database.

So the values could be _copied_ from the corresponding framework name:

• results from aspcore-mw can be _copied_ into aspcore-mw-ado-pg.
• results from aspcore-mvc can be _copied_ into aspcore-mvc-ado-pg .

The results below are from the latest techempower run

Framework | JSON serializ | sgl. qry | mtlpl qrs | Fortunes | Data updates | Plaintext
-----------------|-----------------|-----------|-----------|------------|-----------------|-----------
ac (= aspcore) | 964,268 | - | - | - | - | 7,006,778
ac-ado-pg | _964,268_ | 367,883 | - | 295,535 | 4,883 | _7,006,778_
ac-mw | 740,721 | - | - | - | - | 3,004,847
ac-mw-ado-pg | _740,721_ | 295,958 | 18,481 | 242,882 | 5,019 | _3,004,847_
giraffe | 414,422 | - | - | 38,898 | - | 1,234,323
ac-mvc | 358,847 | - | - | - | - | 1,005,155
ac-mvc-ado-pg | _358,847_ | 170,269 | 18,428 | 106,975 | 4,469 | _1,005,155_

Thanks again for clearing this up.

It would be nice to have the numbers for "asp.net 4 on windows using iis".

[benaadams]

So the values could be copied from the corresponding framework name:

• results from aspcore-mw can be copied into aspcore-mw-ado-pg.
• results from aspcore-mvc can be copied into aspcore-mvc-ado-pg .

Yes, as the TE tests take 70 hrs to run; no point in adding to that by adding an extra 6 variants to the non-db tests (for the db/orm combinations); that should have identical results to the non-db versions.

It would be nice to have the numbers for "asp.net 4 on windows using iis".

It would, and also other frameworks on Windows, and there is a column for OS.

However, the current Linux docker flow is very smooth, and something similar would need to be replicated on Windows and I doubt that would be a small task?