Benchmarkdotnet: Benchmark not run in MS Test test class

[ghost]

Given the following test class in MS Test:

[TestClass]
public class TestBenchmarkDotNet
{
    [TestMethod]
    public void Test()
    {
        BenchmarkRunner.Run<TestBenchmarkDotNet>();
    }

    [Benchmark]
    public void Benchmark()
    {
        Thread.Sleep(5000);
    }
}

Running Test() from the Visual Studio test runner will not trigger the benchmark method at all, neither in Release or in Debug mode. Is this scenario supported?

[AndreyAkinshin]

No, we don't have the unit test support for now. You should create a console application.

[adamsitnik]

@gabisandor If you want to ask us some questions about BenchmarkDotNet please use our chat.

[mattwarren]

You should at least be able to run a test, see the output and report Asserts against the results, see this code for example (Xunit, but the approach should work with MS Test)

[mms-]

Seems like a natural extension to unit tests. Nice to have the benchmarks listed in the IDE as tests currently are.

[adamsitnik]

@mms- the thing is that you should have all applications (including IDE) closed when benchmarking, otherwise it can affect your results ;)

[mms-]

@adamsitnik Agree :) Especially now that VS 2015 seems to be using just as much CPU as the unit test while it's running.

But it still has merit. I'm referring to code refactoring with performance as a goal. Meaning you have old code and new code and want to compare what is faster, all the while you need to maintain correctness. Since tests will be updated anyway when this happens you might as well get feedback on performance.

That is often before one does a serious benchmark on a ready piece of code there is a period (usually long) of experimentation to see what gives better results, a guided search if you will. Accuracy of the benchmark does not matter here just the direction, is it faster or not. Getting this info from the IDE would increase productivity, weather through unit tests or some one other VS panel. Perhaps this is something the Visual Studio team could integrate.

[adamsitnik]

But it still has merit. I'm referring to code refactoring with performance as a goal.

@mms- You are right. @ig-sinicyn is working right now on #155 which will give us performance test runner.

@ig-sinicyn @AndreyAkinshin @mattwarren do you think that we should have our own VS benchmark runner or maybe have (x/n)unit integration for the incoming performance tests runner?

[ig-sinicyn]

@adamsitnik We've integration with nUnit, xUnit and MsTest.
xUnit one is quite lame (waiting for https://github.com/xunit/xunit/issues/908), others two work fine. I'm on vacation this week, so no new commits for some time, sorry :)

About accuracy. As @mms wrote there's actually no need in accuracy in unit tests. The thing you want is not accuracy but repeatability. Same code should produce same results on different runs, different machines and on different environments. If this is false the entire thing is failed. We have our tests running on appveyor CI, local buildserver, dev machines, ultrabooks and even tablets. No failures:)

* Of course, all above is true for "common" benchmarks, those that depend hardly on CPU features, memory available or on IO latency will require same-level hardware to get similar results.