Runtime: Http Client performance drop in preview-2

[Petermarcu]

@rahul10992 commented on Thu Apr 12 2018

Hello. I was trying out HttpClient benchmarking for net core 2.1 and well, I am seeing a serious drop in the HttpClient provided in preview 2.

for simple Get requests between 2 Windows Server 2016 VMs set up with network acceleration in Azure,

1. for Preview-1 with the "AppContext.SetSwitch("System.Net.Http.UseManagedHttpClientHandler", true);" I get 75,000 requests/second
2. For preview-2 using the SocketsHttpHandler, the performance drops to 43,000 requests/second

Is there a reason for this performance drop?

[Petermarcu]

@geoffkizer @karelz

[karelz]

@rahul10992 can you please share your perf repro?
We are not aware of any intentional perf drops between Preview1 and Preview2.

[rahul10992]

Hello. What I used to test this out is here: https://github.com/rahul10992/net-core-preview-issue
You will have to enter the right IP Address in the HttpClientTester's Program.cs to ping the right endpoint. But apart from that, this should work.

As for my azure environment,
I created 2 DS4_V2 Windows Server 2016 VMs with network acceleration on the same virtual network and had to open up the 5000 port on the side that hosts the endpoint. I also installed preview 1 first for preview 1 testing and then preview 2.

[karelz]

@caesar1995 can you please try the repro locally on our perf machines? Let's see if we can reproduce it.

[caesar-chen]

I have tested the repro with loopback address. The repro is testing SocketsHttpHandler sending GET request to a Kestrel server.

The variance is on the server side: Microsoft.AspNetCore targeting 2.1.0-preview1-final/2.1.0-preview2-final. To be clear, it's NOT about testing the SocketsHttpHandler targeting preview1/preview2. (So if there is a regression, it should not be on the client side). EDIT: That was based on looking at the csprj in repro. I missed the fact that "I also installed preview 1 first for preview 1 testing and then preview 2." Correct comment to reduce the noise.

[stephentoub]

@rahul10992, just a guess, could you try setting UseProxy to false?

[karelz]

@caesar1995 as discussed offline: Can you please test against same server to avoid variance in server stack?
Ideally not localhost to mimic what @rahul10992 did - let's use our perf lab machines (@wfurt @rmkerr can provide details).

[caesar-chen]

Hey Stephen, UseProxy definitely contributes to the regression. (Hopefully dotnet/corefx#29050 resolves it).

FWIW, here is the data from preview 2 with the repro using our perf machine:

The first three run didn't specify UseProxy to false. The average is: (45190+47349+47786)/3 = 46775 requests/second.
The last three run with UseProxy = false. The average is: (56341+55982+58941)/3 = 57088 requests/second.

This accounts for (57088-46775)/57088 = 18% performance regression.

I didn't verify against preview 1 (have some weird setup issue), but from issue description, we have (75000-43,000)/75000 = 42% performance regression. I will continue the preview 1 setup, so that we'll have baseline data.

[davidsh]

@caesar1995 Do the "dotnet run" statements above in the screenshot you showed run with Debug or Release compilation? I think the default is Debug. Maybe you should pass the "-c Release" parameters to "dotnet run" to make sure it compiles in Release mode.

[caesar-chen]

I have got data for preview 1 ready. It's from the exact same perf machines.

The average is: (68850+68873+69048)/3 = 68923.

So based on the last comment, the total regression should be (68923-46775 )/69823 = 31%.

[caesar-chen]

Maybe you should pass the "-c Release" parameters

I will try it and update with the result.

[caesar-chen]

Updated client & server side with Release flag. No differnece. So I guess dotnet run run with Release compilation.

Notice that in order to make sure it's client side issue, on the server side I always target the with preview 2, and didn't switch back and forth. (Client: preview 1/preview 2, Server: preview 2).

From the description/repro, it looks like @rahul10992 is testing:
Client: preview 1 -> Server: preview 1
Client: preview 2 -> Server: preview 2.

I will verify Client: preview 1 -> Server: preview 1 case.

[karelz]

I don't think testing client & server both on Preview 1 is priority. Let's dig deeper into the ~13% remaining difference (against UseProxy=false).
I think it would be best to collect PerfView traces and analyze the diff.

[caesar-chen]

Seems on server side, preview 2 didn't cause regression. Below are the test I did against Server: Preview 1.

Client Preview 1: average 65933 requests/seconds.
Client Preview 2: average 49301 requests/seconds.
Client Preview 2 with UseProxy = false: average 58176 requests/seconds.

Comparing to Server: Preview 2:

Client Preview 1: average 68923 requests/seconds.
Client Preview 2: average 46775 requests/seconds.
Client Preview 2 with UseProxy = false: average 57088 requests/seconds.

No too much difference form different server version. So it's mainly regression in SocketsHttpHandler.

[stephentoub]

Just to make sure it's not an issue we've already fixed, could you also try using the latest from master?

[rahul10992]

Hello. Loks like my message is late. @stephentoub But anyway, I tried it with UseProxy = false and the performance for preview-2 jumps up to 74,000 req/sec.

[caesar-chen]

Hmmm, I did the testing again on the same perf machines, still preview 2 even with UseProxy = false has ~10% regression from preview 1.

But with the latest build from master (2.1.300-preview3-008618), with UseProxy = false, there is no regression from preview 1. (around ~1%)
Also with latest build, without UseProxy = false, there is 14% regression. Probably dotnet/corefx#29050 is not in the 2.1.300-preview3-008618 yet.

[stephentoub]

I tried it with UseProxy = false and the performance for preview-2 jumps up to 74,000 req/sec.

Thanks, @rahul10992. So it sounds like setting that to false basically reversed the regression, right? The cause for that is we added support for automatic proxy discovery. But we implemented a workaround in https://github.com/dotnet/corefx/pull/29050 that should basically behave perf-wise as if UseProxy=false if you don't have a proxy auto-discovery file. And of course you can explicitly set UseProxy to false to entirely avoid any auto-discovery.

[rahul10992]

Thanks @stephentoub! I will keep this in mind for my future tests!

[karelz]

@caesar1995 can you please patch nightly build with fix of dotnet/corefx#29050 as we discussed in standup? (just grab latest binary with the fix and replace it in the latest nightly build)
It will help us double-check that fix for dotnet/corefx#29050 helps in the end-to-end scenario and makes the regression go away. Thanks!

[caesar-chen]

I tried using my local corefx build for the repro, there are some noise which I'm not sure that if dotnet/corefx#29050 resolved the regression.

I will wait until Monday to use the next official nightly build to confirm it.

[caesar-chen]

I can confirm that dotnet/corefx#29050 resolved the regression with 2.1.300-preview3-008627.

Without setting UseProxy = false, 55235 requests/second.
With UseProxy = false, 55878 requests/second.

To entirely avoid any auto-discovery, we can explicitly set UseProxy to false. Closing the issue.

[karelz]

Thanks @caesar1995!