Runtime: [Perf -10%] System.Text.RegularExpressions.Tests.Perf_Regex_Common (2)

[DrewScoggins]

Run Information

Architecture | x64
-- | --
OS | Windows 10.0.18362
Changes | diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark | Baseline | Test | Test/Base | Modality | Baseline Outlier
-- | -- | -- | -- | -- | --
Email_IsNotMatch | 354.11 ns | 380.00 ns | 1.07 | | True
Email_IsMatch | 257.82 ns | 289.67 ns | 1.12 | | True

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*';

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: IgnoreCase, Compiled)

[351.514 ; 362.555) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[362.555 ; 375.809) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[375.809 ; 394.434) | @@@@@@@@@@@@@@@@@@@@@@@@

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: IgnoreCase, Compiled)

[255.611 ; 262.054) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[262.054 ; 267.995) | @@@@@@@@@@@@@@@@@@@@@@
[267.995 ; 274.438) | 
[274.438 ; 279.884) | 
[279.884 ; 286.328) | @@
[286.328 ; 295.294) | @@@@@@@@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[msftbot[bot]]

Tagging subscribers to this area: @eerhardt, @pgovind
See info in area-owners.md if you want to be subscribed.

[danmosemsft]

@DrewScoggins thanks for the tightly bounded diff. IThere were no regex changes since July 16th.

There are only two commits in that diff that are even slightly plausibly related

@davidwrighton Fix CanCastTo behavior (#39915) … 13fa14e
@erozenfeld Fix fgUpdateChangedFlowGraph. (#39878) … b179d69b24b45732537e5170f360851ce4dc1ab3

eyeballing the raw graph, I get a slightly larger bracket. This change may also have been in those bounds:

@davidwrighton Fix issue in type equivalence involving arrays (#39914) … 90989b4

@davidwrighton @erozenfeld can you conceive of either of these causing this regression?

[danmosemsft]

Looking at the 2nd test, I think @DrewScoggins tighter bracket is the correct one.

@DrewScoggins did you see any other regressions in other scenarios during those 3 hours, can you check?

[DrewScoggins]

I believe that this test also regressed during the same gap, https://github.com/dotnet/runtime/issues/40088

[danmosemsft]

OK, I'm inclined to put this in codegen area, but I am interested in @erozenfeld thoughts.

[erozenfeld]

39878 had codegen diffs in System.Text.RegularExpressions so it's possible it caused this regression. I'll investigate.

[danmosemsft]

OK thanks @erozenfeld

[danmosemsft]

Hooray for (1) prompt detection of a performance regression (2) useful issue that allowed us to narrow it down quickly

[danmosemsft]

Looking even closer at the graph and zooming in, we actually could get a diff bounding just 3 hours of time on July 24th:
https://github.com/dotnet/runtime/compare/f3f23b4bde8b0a449d57ef600d99560ac97a784e...6ca600396ee2e80e8bf93c72c2b5439aadc86073