Moq4: Find out why Moq's performance has become worse since version approx. 4.5

[stakx]

It was noted e.g. in https://github.com/moq/moq4/issues/388#issuecomment-334870636 that Moq's performance has suffered in recent releases. It would be nice to find the reason(s) for this, and perhaps apply some optimizations to start reversing that trend before version 4.8.0 gets released.

Suggestion:

• [x] Create some Moq test code that does something non-trivial.
• [x] Profile that code for a couple of Moq releases, going back to approx. version 4.5.0. (Perhaps using PerfView, or another capable, freely available .NET profiling tool.)
• [x] Report back with conclusion what has affected Moq's performance negatively in more recent releases / what are the major performance bottlenecks.

/cc @jatouma, @JohanLarsson

[toha73]

I noticed this since we're using Mock.Of<>() in alot of places in our unit tests.
Since NuGet version 4.7.99 it's extremely slow to use that method.

[stakx]

@toha73: Have you by change done any profiling and found out why that is? Do you possibly have some code / use cases you could share to be used for profiling?

[toha73]

I just did a simple script in LinqPad to test it out between different versions of Moq.

void Main()
{
    var iterations = Enumerable.Range(0, 500).ToList();

    var sw = Stopwatch.StartNew();
    iterations.ForEach(_ => { var x = Mock.Of<IMyInterface>(); });
    Console.WriteLine($"{sw.ElapsedMilliseconds} ms");

    sw.Restart();
    iterations.ForEach(_ => { var x = new Mock<IMyInterface>().Object; });
    Console.WriteLine($"{sw.ElapsedMilliseconds} ms");
}

public interface IMyInterface { string Property1 { get; set; } }

And the results on my machine are:

Moq 4.7.142

9760 ms
3 ms

Moq 4.7.137

9962 ms
3 ms

Moq 4.7.127

9885 ms
3 ms

Moq 4.7.99

10755 ms
4 ms

Moq 4.7.63

287 ms
4 ms

[JohanLarsson]

Ran a quick benchmark comparing with NSubstitute

BenchmarkDotNet=v0.10.9, OS=Windows 7 SP1 (6.1.7601)
Processor=Intel Xeon CPU E5-2637 v4 3.50GHzIntel Xeon CPU E5-2637 v4 3.50GHz, ProcessorCount=16
Frequency=3410107 Hz, Resolution=293.2459 ns, Timer=TSC
  [Host]     : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2116.0
  DefaultJob : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2116.0


 |                       Method |         Mean |      Error |     StdDev | Scaled | ScaledSD |   Gen 0 |  Gen 1 | Allocated |
 |----------------------------- |-------------:|-----------:|-----------:|-------:|---------:|--------:|-------:|----------:|
 |             SubstituteForFoo |     7.754 us |  0.3237 us |  0.9543 us |   1.00 |     0.00 |  1.0529 | 0.0229 |    6.5 KB |
 | SubstituteForFooValueReturns |    14.580 us |  0.2907 us |  0.8105 us |   1.91 |     0.26 |  1.7700 | 0.0305 |  10.94 KB |
 |                   MockOfIFoo |   892.996 us | 19.8067 us | 57.7770 us | 116.97 |    16.61 |  7.8125 | 3.9063 |  50.28 KB |
 |     MockOfIFooWithExpression | 2,138.479 us | 41.3511 us | 44.2452 us | 280.10 |    35.83 | 11.7188 | 3.9063 |  80.59 KB |

    public class MoqBenchmarks
    {
        public interface IFoo
        {
            int Value { get; }
        }

        [Benchmark(Baseline = true)]
        public object SubstituteForFoo()
        {
            return Substitute.For<IFoo>();
        }

        [Benchmark]
        public object SubstituteForFooValueReturns()
        {
            return Substitute.For<IFoo>().Value.Returns(1);
        }

        [Benchmark]
        public object MockOfIFoo()
        {
            return Mock.Of<IFoo>();
        }

        [Benchmark]
        public object MockOfIFooWithExpression()
        {
            return Mock.Of<IFoo>(x => x.Value == 1);
        }
    }

[stakx]

@toha73, thanks for the example. I ran your code for the same versions you mentioned, and also a slightly adapted version with BenchmarkDotNet. Below are my results for each run.

TL;DR: I do not observe that extreme jump (by more than an order of magnitude) that you saw.

Do you by chance have .NET 4.7(.1) installed? If so, are you aware of the performance problems caused by it? (See https://github.com/Microsoft/dotnet/issues/529 and other issues in the same repo.)

BenchmarkDotNet=v0.10.10, OS=Windows 7 SP1 (6.1.7601.0)
Processor=Intel Core i7-4770 CPU 3.40GHz (Haswell), ProcessorCount=8
Frequency=3312695 Hz, Resolution=301.8690 ns, Timer=TSC
  [Host]     : .NET Framework 4.6.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2116.0
  DefaultJob : .NET Framework 4.6.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2116.0

4.7.142

Using your code:

435 ms
3 ms

Using BenchmarkDotNet:

        Method |       Mean |     Error |    StdDev |
-------------- |-----------:|----------:|----------:|
        MockOf | 847.905 us | 9.0737 us | 8.4875 us |
 NewMockObject |   8.747 us | 0.0220 us | 0.0184 us |

4.7.99

Using your code:

467 ms
3 ms

Using BenchmarkDotNet:

        Method |       Mean |     Error |    StdDev |
-------------- |-----------:|----------:|----------:|
        MockOf | 727.259 us | 6.0048 us | 5.3231 us |
 NewMockObject |   8.487 us | 0.0418 us | 0.0391 us |

4.7.63

Using your code:

433 ms
3 ms

Using BenchmarkDotNet:

        Method |       Mean |     Error |    StdDev |
-------------- |-----------:|----------:|----------:|
        MockOf | 592.430 us | 1.8652 us | 1.7447 us |
 NewMockObject |   8.823 us | 0.0203 us | 0.0190 us |

[JohanLarsson]

@stakx Can we have a short private discussion, perhaps you can ping me on gitter? https://gitter.im/DotNetAnalyzers/IDisposableAnalyzers

[stakx]

@toha73 - FYI: If you upgrade to Moq 4.7.145 (or later), which is no longer affected by the mentioned .NET Framework regression (unless you opt back in), execution times should go back down to a normal level. See the changelog for details.

[stakx]

I've spent some time with PerfView. It seems that the biggest performance bottlenecks are:

1. proxy type creation (via Castle DynamicProxy)
2. expression tree compilation (via LambdaExpression.Compile)
3. source file information collection for diagnostic purposes (This has been taken care of with version 4.7.145.)

Those cannot be made more efficient, we can only try to do less of it. Notable hotspots in Moq's codebase are:

• Mock.SetupAllProperties — can cause a lot of proxy type creation
• MatcherFactory.CreateMatcher — can cause a lot of expression tree compilation

Moq's performance would likely be greatly improved by two changes:

1. Make SetupAllProperties lazy. Instead of setting up properties right away, only set them up when they're actually queried. This would reduce the amount of type creation to the level actually required.

2. Require that custom matchers (i.e. methods and properties calling Match.Create) be marked with some kind of attribute, such as [Matcher]. Moq could then rely simply on the presence of that attribute without having to compile & execute one lambda per setup parameter, just to figure out whether they're invoking Match.Create or not.

Of these two changes, (2) would definitely be a breaking change (but worth it). (1) might be done as a non-breaking change if much care is taken.

[jakubozga]

Oh now I can comment it. I couldn`t comment it earlier.
Nevertheless the performance is slow. Is anything that can be done in this case? If not then I will continue be working on older version.

[stakx]

@jakubozga - Yes, work is being done to make Moq faster. I've identified a few ways to significantly improve performance related to the major bottleneck mentioned above, but implementing these improvements will take a lot of care and time; Moq's functional correctness takes precedence over performance.

I've already applied quite a few smaller-scale improvements to both runtime and memory efficiency, but those will probably only become noticeable when you have many thousands of unit tests.

The major performance bottlenecks mentioned in my above post often result from fundamental architectural decisions, and it's not trivial to remove them without causing collateral damage. For example, I have this wonderful draft re-implementation that makes SetupAllProperties lazy. It results in a speedup of Mock.Of<T> by a factor of 23x (which is nearly optimal), but it also changes Moq's behaviour in one specific, fairly important edge case (which is bad), so at this stage I'm back at the drawing board.

I don't make any promises about speed or memory efficiency improvements in particular versions—I'd advise common sense: Just try out new versions as they're being released, and if they work well for you, then that's great.

[stakx]

@jakubozga - I have just found and removed a major, but unexpected performance bottleneck (see referenced PR above). I am fairly confident that Moq 4.8.0 will be noticeably faster than its predecessor versions. Performance should once again be comparable to versions earlier than 4.5.16.

Moq's own unit test suite runs nearly twice as fast now (12.2 seconds instead of 20.2 seconds on the CI server).

[dadhi]

@stakx

1. expression tree compilation (via LambdaExpression.Compile)
   ...Those cannot be made more efficient...

Actually here is FastExpressionCompiler wich speed-ups Expression.Compile ten-fold.

I have just fixed the last three failing tests from Moq with all Compile in Moq code base replaced by CompileFast from FEC. But did not benchmark it yet.

[stakx]

@dadhi - That's some impressive work there! :+1:

Given that Moq 4.8+'s performance is once again comparable to what it used to be in early 4.x versions, I feel that further performance improvements are less of a priority now than approx. half a year ago. At this point in Moq's lifecycle (shortly before Moq 5's release), I think stability and bug fixing is generally the more important goal.

Because of that, I would prefer if Moq 4 did not take on any further hard dependencies on third-party libraries that could introduce new breaking changes. (A small aside: I've learnt the hard way that the code coverage of Moq's unit test suite isn't nearly as complete as it should be, so in your work, I wouldn't rely too heavily on it.)

That being said, what we could do is to introduce an extensibility point that would allow swapping out the expression tree compiler used by Moq. Moq would default to the framework's own Expression.Compile, but users could plug another compiler (such as yours) into that mechanism.

Does that sound fair to you?

Btw., there are still some remaining options for improving performance further:

• One is memoization. If we do introduce an extensibility point for swapping out the expression tree compiler, one could easily implement a memoization (caching) layer.

• Another is #643. There, I'm aiming to improve Moq's handling of multi-dot setup expressions such that it won't use expression tree compilation. This will likely result in noticeable speed gains.

[stakx]

@dadhi, I did some quick prototyping of an extensibility point for custom expression tree compilers, see #647. This would allow you to do this:

// using Moq;

ExpressionCompiler.Instance = new CacheExpressionCompiler(new AlternateExpressionCompiler());

// this implementation might e.g. use your FastExpressionCompiler library: 
class AlternateExpressionCompiler : ExpressionCompiler
{
    public override Delegate Compile(LambdaExpression expression) { … }
    public override TDelegate Compile<TDelegate>(Expression<TDelegate> expression) { … }
}

// this implementation might cache compiled expressions for later reuse:
class CacheExpressionCompiler : ExpressionCompiler
{
    public CacheExpressionCompiler(ExpressionCompiler underlying) {  …  }
    public override Delegate Compile(LambdaExpression expression) { … }
    public override TDelegate Compile<TDelegate>(Expression<TDelegate> expression) { … }
}

Any thoughts? (Would this be enough to integrate with your library?)

/cc @informatorius (FYI because of https://github.com/moq/moq4/issues/188#issuecomment-368027246)

[dadhi]

Moq would default to the framework's own Expression.Compile, but users could plug another compiler (such as yours) into that mechanism.

I think this is good opt-in feature. Less risky as well. The extensibility api as in prototype is fine too.

I would prefer if Moq 4 did not take on any further hard dependencies on third-party libraries that could introduce new breaking changes.

Btw, FEC is just a single cs file. You can drop it in your project. This is exactly how I have tested it with Moq. The only change was a replacing all Compile calls with CompileFast.

A small aside: I've learnt the hard way that the code coverage of Moq's unit test suite isn't nearly as complete as it should be, so in your work, I wouldn't rely too heavily on it.

May you do this already, but in my case, I am gradually adding issue covering tests into solution with each new issue. I have a separate test project for that, which include 3rd party references for real use-cases (issues).

[stakx]

I think this is good opt-in feature. Less risky as well. The extensibility api as in prototype is fine too.

Glad to hear! I might actually merge this, then.

I would prefer if Moq 4 did not take on any further hard dependencies on third-party libraries

Btw, FEC is just a single cs file. You can drop it in your project.

Understood. And apologies if I didn't express myself clearly enough. By "dependencies on third-party libraries", I didn't just mean the form of distribution. Whether it is a NuGet package, or a single source code file doesn't matter all that much. (After all, you could easily use NuGet to distribute your code file as a development-time, source-code only NuGet package, of which Moq already references a few.)

It is more the thought of replacing a framework facility of significant complexity with a reimplementation that very likely won't be nearly as "battle-tested" (hardened against bugs and refined over time) as the framework code is, due to its comparatively short history. (No disrespect intended!)

So I think swapping out the expression compiler should be an explicit opt-in decision made by downstream users; hence the proposed extensibility point.

I have a separate test project for that, which include 3rd party references for real use-cases (issues).

(I have been thinking about setting up a regression test suite that goes beyond Moq's own regression tests. For example, I'd like to run some actual real-world open-source projects' test suites before and after some change in Moq, and see if the two test runs produce different results. If so, that might be an indication that the change causes a regression. Haven't got around to that yet, though.)

[dadhi]

All is fine with me, cause I've just got another ~1000 tests (Moq's) to FEC regression suite, in addition to the biggest tandem of DryIoc, and hmm.. AutoMapper :)