Runtime: Performance regression: WebUtility.UrlDecode 2.5x slower on AMD only

Created on 25 Jul 2019 · 18Comments · Source: dotnet/runtime

For some reason, the System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired benchmark have regressed 2.5x on AMD only.

Repro

git clone https://github.com/dotnet/performance.git
cd performance
# if you don't have cli installed and want python script to download the latest cli for you
py .\scripts\benchmarks_ci.py -f netcoreapp2.2 netcoreapp3.0 --filter System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired
# if you do
dotnet run -p .\src\benchmarks\micro\MicroBenchmarks.csproj -c Release -f netcoreapp2.2 --runtimes netcoreapp2.2 netcoreapp3.0 --filter System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired

System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired

| Conclusion | Base | Diff | Base/Diff | Modality | Operating System | Arch | Processor Name | Base Runtime | Diff Runtime |
| ---------- | ------:| ------:| ---------:| --------:| -------------------- | ----- | ------------------------------------------- | --------------- | --------------------------------- |
| Same | 182.30 | 182.19 | 1.00 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 154.89 | 153.16 | 1.01 | | Windows 10.0.18362 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 309.98 | 305.69 | 1.01 | | ubuntu 16.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 320.10 | 331.77 | 0.96 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 211.93 | 210.11 | 1.01 | | macOS Mojave 10.14.5 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 182.27 | 180.18 | 1.01 | | Windows 10.0.18362 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Faster | 138.34 | 130.48 | 1.06 | | Windows 10.0.18362 | 64bit | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 175.79 | 420.39 | 0.42 | | Windows 10.0.18362 | 64bit | AMD Ryzen 7 1800X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 134.67 | 327.60 | 0.41 | | Windows 10.0.18362 | 64bit | AMD Ryzen 9 3900X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 182.93 | 173.77 | 1.05 | | Windows 10.0.18362 | 32bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 205.06 | 203.17 | 1.01 | | Windows 10.0.18362 | 32bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-17|

/cc @danmosemsft @tannergooding @billwert @DrewScoggins

amd-perf arch-x64 area-System.Net bug tenet-performance

Source

adamsitnik

All 18 comments

@wtgodbe will you be able to take a look?

karelz on 26 Jul 2019

👍1

Sure thing

wtgodbe on 26 Jul 2019

@adamsitnik do you have an estimate of when this regression started? When was the last time you ran these benchmarks?

wtgodbe on 26 Jul 2019

@davidsh do you think this is blocker for 3.0? I have a bit doubts.
It would be nice to understand what caused it (@wtgodbe will look at that), but it is IMO just "nice", not necessary ...

Thoughts?

karelz on 29 Jul 2019

Please keep in mind that the regression affects only AMD hardware. I suspect that some of the methods that UrlDecode uses internally have regressed and this is the outcome of it.

I currently don't have access to AMD machine, so please excuse me for "guessing" instead of providing an answer using a profiler.

From what I can see this method uses Encoding.Utf8.GetBytes() to get the utf8 representation, then does some simple operations and calls Array.Copy :

https://github.com/dotnet/corefx/blob/e23f83e6172ef2d879342e1c4883012749b91ef1/src/System.Web.HttpUtility/src/System/Web/HttpUtility.cs#L180

https://github.com/dotnet/corefx/blob/e23f83e6172ef2d879342e1c4883012749b91ef1/src/System.Web.HttpUtility/src/System/Web/HttpUtility.cs#L186

https://github.com/dotnet/corefx/blob/e23f83e6172ef2d879342e1c4883012749b91ef1/src/System.Web.HttpUtility/src/System/Web/HttpUtility.cs#L201

https://github.com/dotnet/corefx/blob/e72c6c53baddb02a630e8e6592a59d89e30b856b/src/System.Web.HttpUtility/src/System/Web/Util/HttpEncoder.cs#L248-L291

Most probably it's hitting https://github.com/dotnet/coreclr/issues/25505 which affects Array.Copy

@tannergooding is there any chance that you could re-run the Decode_DecodingRequired benchmark and see if https://github.com/dotnet/coreclr/pull/25763 helped?

--filter System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired

@karelz It should definitely not be a blocker for 3.0.

adamsitnik on 29 Jul 2019

Yes, I can try to find some time this week (feel free to ping me if I forget 😄).

tannergooding on 29 Jul 2019

I'll try now with an SDK containing the coreclr fix

wtgodbe on 29 Jul 2019

@davidsh do you think this is blocker for 3.0? I have a bit doubts.
It would be nice to understand what caused it (@wtgodbe will look at that), but it is IMO just "nice", not necessary ...
Thoughts?

The networking stack (HTTP etc.) doesn't use System.Web.HttpUtility. It uses System.Uri for "UrlEncode/Decode" operations. So, I don't think this is 3.0 blocker.

davidsh on 29 Jul 2019

Never mind, it just occured to me that I don't have a machine with an AMD processor 😆. @tannergooding do you?

wtgodbe on 29 Jul 2019

Yes, three of them (two personal, one work). If it isn't high priority though, I will be getting to it in a few days

tannergooding on 29 Jul 2019

👍1

Sounds good, I'll mark this for 5.0

wtgodbe on 29 Jul 2019

❤1

@tannergooding any update on this?

karelz on 12 Sep 2019

No, this slipped off my radar. I have some time today so I've pulled down the repo and will get a profile shortly.

tannergooding on 12 Sep 2019

❤1

Would it be possible to create a vendor specific issue tag so that interested parties could see what kinds of issues are AMD/Intel specific? cc @mangod9

mjsabby on 17 Nov 2020

Looks like I forgot to post my results:

BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19042
AMD Ryzen 9 3900X, 1 CPU, 24 logical and 12 physical cores
.NET Core SDK=3.1.303
  [Host]     : .NET Core 2.1.11 (CoreCLR 4.6.27617.04, CoreFX 4.6.27617.02), X64 RyuJIT
  Job-ALOAVY : .NET Core 2.1.11 (CoreCLR 4.6.27617.04, CoreFX 4.6.27617.02), X64 RyuJIT

| Method | Mean | Error | StdDev | Median | Min | Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------------------ |---------:|--------:|--------:|---------:|---------:|---------:|-------:|------:|------:|----------:|
| Decode_DecodingRequired | 140.2 ns | 0.83 ns | 0.78 ns | 140.1 ns | 139.1 ns | 141.7 ns | 0.1297 | - | - | 216 B |

BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19042
AMD Ryzen 9 3900X, 1 CPU, 24 logical and 12 physical cores
.NET Core SDK=3.1.303
  [Host]     : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT
  Job-MTWJIJ : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT

| Method | Mean | Error | StdDev | Median | Min | Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------------------ |---------:|--------:|--------:|---------:|---------:|---------:|-------:|------:|------:|----------:|
| Decode_DecodingRequired | 126.1 ns | 1.11 ns | 1.04 ns | 125.8 ns | 124.6 ns | 128.7 ns | 0.0258 | - | - | 216 B

BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19042
AMD Ryzen 9 3900X, 1 CPU, 24 logical and 12 physical cores
.NET SDK=5.0.100-rc.2.20480.7
  [Host]     : .NET 5.0.0 (5.0.20.47505), X64 RyuJIT
  Job-HTJILN : .NET 5.0.0 (5.0.20.47505), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 Arguments=/p:DebugType=portable IterationTime=250.0000 ms
MaxIterationCount=20 MinIterationCount=15 WarmupCount=1

| Method | Mean | Error | StdDev | Median | Min | Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------------------ |---------:|--------:|--------:|---------:|---------:|---------:|-------:|------:|------:|----------:|
| Decode_DecodingRequired | 129.4 ns | 1.19 ns | 1.06 ns | 129.5 ns | 127.8 ns | 131.6 ns | 0.0256 | - | - | 216 B |

In short, I don't see any regression. I do however see great variance across multiple runs. Some of the means vary from 126ns up to 150ns.

tannergooding on 19 Nov 2020

@mjsabby have created a amd-perf label for tracking such issues. Though looks like from @tannergooding data above that there doesnt seem to be a regression.