Runtime: HttpClient slow performance on windows and linux

[sergey-brutsky]

Hi guys,

I've noticed that httpclient is performing very slow in dotnet core comparing to golang for example.

For tests I've used VM with 16 virtual cores, and 24Gb of ram based on Intel Xeon E312 2.6Mgz

Tried windows server 2012 and centos 7.2

Code is super simple (you may find it in attached file).
And my test was made via "wrk -c 256 -t 32 http://:5000/ --latency"

Results are very upset.
On this powerful VM dotnet core httpclient performance was about 16 000 reqs/sec on windows and 12 000 reqs/sec on linux.

At the same time solution based on "golang" (check attacked file) shows results like 44 000 reqs/sec

Could you please advice what could be wrong with donet core based solution ?

Thank you
goproxy.zip
Program.zip

[stephentoub]

@sergey-brutsky, I took a quick look at your examples. You call out HttpClient as being slower, but your examples are also involving ASP.NET, making requests to a localhost ASP.NET server that then in turn makes requests with HttpClient. How are you determining that the bottleneck is with HttpClient rather than with the local server? Have you tried narrowing it down to one or the other, such as by just having a simple console app that makes HttpClient requests (no localhost server) or by having a localhost server that just returns a hardcoded string and doesn't make any HttpClient requests?

[sergey-brutsky]

hi @stephentoub,

I've tried 2 scenarios

1) await context.Response.WriteAsync("Hello World");

2) var response = await httpClient.GetStringAsync("http://nginx-web-server/test.html");
await context.Response.WriteAsync(response);

First one testing results
./wrk -c 256 -t 32 -d 30 http://10.3.34.191:5003 --latency --timeout 10

Running 30s test @ http://10.3.34.191:5003
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.84ms 19.49ms 607.69ms 97.62%
Req/Sec 3.93k 620.68 5.09k 65.16%
Latency Distribution
50% 1.67ms
75% 2.73ms
90% 4.21ms
99% 129.88ms
3763677 requests in 30.10s, 441.49MB read
Requests/sec: 125042.55
Transfer/sec: 14.67MB

The second one testing results
./wrk -c 256 -t 32 -d 30 http://10.3.34.191:5003 --latency --timeout 10
Running 30s test @ http://10.3.34.191:5003
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.44ms 8.71ms 256.23ms 96.21%
Req/Sec 379.97 35.92 646.00 70.28%
Latency Distribution
50% 19.66ms
75% 24.80ms
90% 27.08ms
99% 33.87ms
363342 requests in 30.02s, 45.05MB read
Requests/sec: 12102.20
Transfer/sec: 1.50MB

So as you can see 125 000 reqs/sec against 12 000 reqs/sec.

Could you please advice ?

[stephentoub]

So as you can see 125 000 reqs/sec against 12 000 reqs sec.

And what's the difference between the same scenarios with the go client?

[sergey-brutsky]

Tried the same scenarios for go with fasthttp --> https://github.com/valyala/fasthttp

1) fmt.Fprintf(ctx, "Hello, world!")

2) proxy call to http://nginx-web-server/test.html

Results for the first scenario
Running 30s test @ http://10.3.34.191:8080
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.73ms 19.93ms 406.65ms 97.43%
Req/Sec 4.38k 385.28 5.59k 68.01%
Latency Distribution
50% 1.69ms
75% 2.08ms
90% 2.44ms
99% 134.93ms
4195311 requests in 30.10s, 592.14MB read
Requests/sec: 139386.62
Transfer/sec: 19.67MB

Results for the second scenario
Running 30s test @ http://10.3.34.191:8080
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.62ms 12.02ms 217.91ms 99.01%
Req/Sec 1.43k 169.33 3.99k 72.92%
Latency Distribution
50% 5.22ms
75% 6.76ms
90% 8.27ms
99% 18.41ms
1363788 requests in 30.10s, 179.48MB read
Requests/sec: 45309.63
Transfer/sec: 5.96MB

So difference is 140 000 reqs/sec against 45 000 reqs/sec.

Do you have any clue why dotnet httpclient is much slower or it is problem of kestrel web server ?

[stephentoub]

Thanks, @sergey-brutsky.

Do you have any clue

Not yet.

A few more questions:

1. You mentioned initially that you're seeing this difference on both Windows and Linux, but it looks like the most recent numbers you shared are just on Linux. I assume you see the same kinds of numbers for both scenarios on both dotnet and go on both Windows and Linux?
2. For Windows, have you tried running on the .NET Framework vs on .NET Core, and do you see a difference?

[sergey-brutsky]

1) Regarding dotnet core scenarios in Windows and Linux difference wasn't significant.
I have no experience how to run "go" applications on windows (will try to investigate that)

2) If it's possible to run "kestrel" on .net framework on windows I will try and share my results

[sergey-brutsky]

Tried another scenarios under windows 2012 r2 on the same powerful VM (checked attached files)

1) simple hello world web server on .net framework 4.6.1
Running 1m test @ http://10.3.34.191:5004
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.00ms 17.19ms 620.68ms 98.08%
Req/Sec 3.02k 356.86 7.35k 69.00%
Latency Distribution
50% 2.30ms
75% 3.04ms
90% 4.70ms
99% 108.52ms
5776878 requests in 1.00m, 683.15MB read
Requests/sec: 96121.80
Transfer/sec: 11.37MB

2) simple web server + proxy call to nginx web server
Running 1m test @ http://10.3.34.191:5004
32 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 18.35ms 11.42ms 683.03ms 85.52%
Req/Sec 450.25 73.14 1.31k 69.38%
Latency Distribution
50% 16.73ms
75% 22.26ms
90% 29.74ms
99% 43.60ms
858537 requests in 1.00m, 106.44MB read
Socket errors: connect 0, read 2, write 398, timeout 0
Requests/sec: 14298.36
Transfer/sec: 1.77MB

And performance still not good :(

[stephentoub]

cc: @cipop, @davidsh

[karelz]

Triage: We need more deeper dive investigations.

[sergey-brutsky]

@karelz may I help you somehow ? maybe provide more info or do more tests ?

[karelz]

Good question - @stephentoub @geoffkizer is there something that might help us beyond perf investigations?
@sergey-brutsky could you help with the root-cause perf investigations?

[damtur]

Any update on this? We are seeing really poor experience in .net core especially in Linux as well. Is .net core team looking into that?

[karelz]

@damtur do you have a specific repro we could analyze?
If there is no repro, there is not much to look into :(

[karelz]

Given the lack of repro and very high cost, we will have to push this out into Future.
If there is a repro/traces we could look at, please let us know.

[damtur]

@karelz Sorry for a slow response but I wanted to frame it a little. Here you have some details about my problem:
https://goo.gl/a6nKj7
Also:
https://github.com/dotnet/dotnet-docker/pull/226 is one possible fix.

[stephentoub]

@damtur, thanks. Have you tried 2.0 bits? Your blog post description makes me suspect this is the same as was addressed by https://github.com/dotnet/corefx/pull/15354.

[damtur]

@stephentoub I've not tested this on 2.0. I'm struggling a bit to find how.
Moreover it seams to me that the change you are talking about has been ported to 1.1 https://github.com/dotnet/corefx/pull/16895 and it has been merged already (it is in dotnet:release/1.1.0)
I've tested this using .NET Core 1.1 on the newest 1.1.0 image and still POST responses were very slow, but when I updated the libcurl version on that docker image response were fast again.

Libcurl 7.38:

Avarage time:   50.0166165ms
50%-tile time:  49.9941ms
90%-tile time:  52.1015ms
99%-tile time:  55.4909ms
100%-tile time: 61.3775ms

Libcurl 7.53.0:

Avarage time:   5.2403101ms
50%-tile time:  4.2032ms
90%-tile time:  11.2061ms
99%-tile time:  17.5238ms
100%-tile time: 24.5104ms

Both on the same container running Dotnet 1.1.0 Build: 928f77c4bc3f49d892459992fb6e1d5542cb5e86

[stephentoub]

Moreover it seams to me that the change you are talking about has been ported to 1.1 dotnet/corefx#16895 and it has been merged already (it is in dotnet:release/1.1.0)

Yes, but that backport hasn't shipped as part of any 1.1 release... it'll be part of the next one whenever that is.

[damtur]

@stephentoub Can you maybe point me to some guide how to build and run dotnet2 / dotnet 1.1 from the 1.1.0 branch? I'm still trying to find that without any luck. So far I've found information in documentation that it is easy :)
Ideal would be a docker container with everything in in (the newest version of dotnet core 2 / dotnet core 1.1)

Edit:
To install the newest framework on debian:

wget https://dotnetcli.blob.core.windows.net/dotnet/master/Installers/Latest/dotnet-host-debian-x64.latest.deb
dpkg -i dotnet-host-debian-x64.latest.deb
wget vhttps://dotnetcli.blob.core.windows.net/dotnet/master/Installers/Latest/dotnet-hostfxr-debian-x64.latest.deb
dpkg -i dotnet-hostfxr-debian-x64.latest.deb
wget https://dotnetcli.blob.core.windows.net/dotnet/master/Installers/Latest/dotnet-sharedframework-debian-x64.latest.deb
dpkg -i dotnet-sharedframework-debian-x64.latest.deb
wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-dev-debian-x64.latest.tar.gz
tar -zxvf dotnet-dev-debian-x64.latest.tar.gz

Link:
https://github.com/dotnet/cli#installers-and-binaries

[damtur]

@stephentoub I've confirmed that the problem exists on dotnet v2.0:
Libcurl 7.38:

Avarage time:   50.2630825000001ms
50%-tile time:  50.0044ms
90%-tile time:  51.9817ms
99%-tile time:  60.5078ms
100%-tile time: 64.4401ms

dotnet --version:

  Version  : 2.0.0-preview1-001909-00
  Build    : eacbddd1cb2b25b07dc26589b612697ea9c4f30e

(Updating libcurl helped)

[stephentoub]

@damtur, that's the version of the dotnet tool, not of the version of the runtime/libraries (coreclr/corefx) being used, and the latter is where the fix exists.

[stephentoub]

Can you maybe point me to some guide

@damtur, we have the directions at:
https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md
I don't know if these are up-to-date or not; I typically do something very hacky and just overwrite my deployed binaries, but that's because I'm generally not going against nightlies and instead making a change to some binary and wanting to test that specific change locally and immediately.

@danmosemsft or @terrajobst might be able to help more with trying out the 2.0 corefx bits if the above doesn't work for you.

[MichaelSimons]

@damtur - since you are already using Docker, you could make use of the dotnet-nightly repo which has the latest 2.0 bits available via the 2.0 tags.

[MichaelSimons]

I dockerized the test app @damtur created. Running it as is yields the following:

Avarage time:   50.2810424399999ms
50%-tile time:  50.0108ms
90%-tile time:  52.599ms
99%-tile time:  60.0634ms
100%-tile time: 105.8623ms

I then upgraded the app to use 2.0. Running that yields the following results:

Avarage time:   3.31597281000001ms
50%-tile time:  2.4778ms
90%-tile time:  6.0897ms
99%-tile time:  12.6443ms
100%-tile time: 22.5758ms

[stephentoub]

Thanks for confirming, @MichaelSimons!

[iSazonov]

We see this problem in PowerShell Core repo too. It leads to the collapse of our tests and nightly builds randomly. We also see that simple queries take up to 1 second, and the redirection tests are more than 5 seconds! We use .Net Core Runtime 2.0.5 and Kestrel as test HTTP server.

[karelz]

@iSazonov it is not clear if it is the same problem. Do you have a repro without PowerShell involved showing the long delays? We would be interested in looking into that.

[iSazonov]

@karelz The repo from @markekraus use:

1. the temporary .Net Core app based on Kestrel
2. direct CoreFX objects to make web requests. The repo _doesn't envolve_ PowerShell engine and cmdlets in the web requests.
3. different PowerShell versions. Windows PowerShell creates .Net Framework object and get web responses fast. This also confirms that the Kestrel apps works well. PowerShell Core creates .Net Core object and get web responses slow.

.

[markekraus]

thanks @iSazonov

@karelz To repro the issue you can use the following project https://github.com/markekraus/SlowCoreFXRedirects

1. build and run the ASP.NET Core TestServer app
2. Leave TestServer running
3. build and run the SlowCoreFXRedirects .NET Core app
4. build and run the FastFrameworkRedirects .NET Framework app

CoreFX:

Content: {"args":{},"headers":{"Connection":"Keep-Alive","Host":"localhost:65479"},"url":"http://localhost:65479/api/Get/","method":"GET","origin":"127.0.0.1"}
Milliseconds: 4358

.NET Framework:

Content: {"args":{},"headers":{"Host":"localhost:65479"},"url":"http://localhost:65479/api/Get/","method":"GET","origin":"127.0.0.1"}
Milliseconds: 1063

These apps are the same exact code but CoreFX is significantly slower. PowerShell Web Cmdlet tests were set to a timeout of 5 seconds and the redirects related tests hit that time out. but this isn't even related to anything in PowerShell's code base. the slow down is in CoreFX as demonstrated bu this repro.

Another thing to note, if I switch to the WebRequest API I see the same behavior as HttpClient. WebRequest is fast in .NET Framework and Slow in CoreFX.

[davidsh]

We are aware that HttpWebRequest is slow in CoreFx. This is a side-affect of how it is implemented in CoreFx. In .NET Core, It is a layer on top of HttpClient. It isn't able to benefit from HttpClient TCP connection pooling logic, etc. That is because a new HttpClient is created for every new HttpWebRequest.

cc: @karelz @stephentoub

[markekraus]

@davidsh I was vaguely aware of HttpWebRequest being a wrapper for HttpClient, but wasn't certain. So I guess the slowness is in HttpClient then? It appears more pronounced in redirects than other activities, but I haven't done solid analysis on it. I'm also not sure if it's related to the original issue or not. I don't have the time right now to dig into it deeply. It only bubbled up to my level of consciousness when we noticed tests failing for timeout issues relating to redirect tests in the PowerShell project. Discovered the slowness can be reproduced outside of any PowerShell code with simple HttpClient calls in .NET Core, but did not reproduce in .NET Framework.

[karelz]

@markekraus what happens if you don't use localhost in your repro (use directly IP)? Localhost usage on Windows may be hitting dotnet/corefx#24104.

[markekraus]

ah... that does appear to significantly improve the response time, from ~7000ms to ~100ms on my work PC. So it does appear to be hitting dotnet/corefx#24104.

[karelz]

Great, so your repro is not relevant to this issue.

Is there anything else in this issue still pending investigation? Last comment https://github.com/dotnet/corefx/issues/13551#issuecomment-291632907 seems to hint there is some perf improvement in 2.0. @stephentoub @MichaelSimons @damtur do you have more details?

[iSazonov]

@markekraus Previously I saw that the test is slow on Unix too. What you see after using ip address instead of localhost?

[markekraus]

@iSazonov the tests are faster on all platforms after switching to the IP address.

[labsilva]

Same here, inconsistent request timing with HTTP Client when using hostnames instead of IP addresses (on Linux)

[karelz]

@labsilva do you have a repro on .NET Core 2.1 which can be easily transferred? If yes, please file a new issue with details.

The original issues should be addressed to our best knowledge on 2.1, except the case of localhost (#24104).