Benchmarkdotnet: Warn the users when running Benchmarks from xUnit with shadow copy enabled

[pardahlman]

Thanks for great work on BenchmarkDotNet 🎉 .

I'm probably doing something obviously wrong, but I can't figure out what.

I'm trying to run a very simple benchmark on .NETFramework 4.6.1. In fact, at this point it doesn't benchmark anything

    [Benchmark]
    public void Do_Nothing()
    {

    }

The code is executed from within an xunit test through the BenchmarkRunner without any custom configuration

BenchmarkRunner.Run<OperationTrackerBenchmarks>();

When running the benchmark I get the following output

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
  Discovering: RaySearch.Core.Benchmarks
  Discovered:  RaySearch.Core.Benchmarks
  Starting:    RaySearch.Core.Benchmarks
Calling global setup
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   OperationTrackerBenchmarks.Do_Nothing: DefaultJob

// Validating benchmarks:
// **************************
// Benchmark: OperationTrackerBenchmarks.Do_Nothing: DefaultJob
// *** Generate ***
// Result = Success
// BinariesDirectoryPath = C:\Users\pardah\AppData\Local\Temp\5ffab0a6-52bc-47e5-a280-b18ca7220111\5ffab0a6-52bc-47e5-a280-b18ca7220111\assembly\dl3\b912a5ae\67e43fd3_ae37d301

// *** Build ***
BuildScript: C:\Users\pardah\AppData\Local\Temp\5ffab0a6-52bc-47e5-a280-b18ca7220111\5ffab0a6-52bc-47e5-a280-b18ca7220111\assembly\dl3\b912a5ae\67e43fd3_ae37d301\fdf9d873-38a8-41e8-aee0-18db9c17dcce.bat
// Result = Success

// *** Execute ***
// Launch: 1 / 1
// Execute: C:\Users\pardah\AppData\Local\Temp\5ffab0a6-52bc-47e5-a280-b18ca7220111\5ffab0a6-52bc-47e5-a280-b18ca7220111\assembly\dl3\b912a5ae\67e43fd3_ae37d301\fdf9d873-38a8-41e8-aee0-18db9c17dcce.exe
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!


// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\OperationTrackerBenchmarks-report.csv
  BenchmarkDotNet.Artifacts\results\OperationTrackerBenchmarks-report-github.md
  BenchmarkDotNet.Artifacts\results\OperationTrackerBenchmarks-report.html

// * Detailed results *
OperationTrackerBenchmarks.Do_Nothing: DefaultJob
Runtime = ; GC =
There are no any results runs

Total time: 00:00:11 (11.11 sec)

// * Summary *

BenchmarkDotNet=v0.10.9, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-7820HQ CPU 2.90GHz (Kaby Lake), ProcessorCount=8
Frequency=2835937 Hz, Resolution=352.6171 ns, Timer=TSC
  [Host] : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2110.0


     Method | Mean | Error |
----------- |-----:|------:|
 Do_Nothing |   NA |    NA |

Benchmarks with issues:
  OperationTrackerBenchmarks.Do_Nothing: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
  Finished:    RaySearch.Core.Benchmarks
=== TEST EXECUTION SUMMARY ===
   RaySearch.Core.Benchmarks  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 13,927s

The way I interpret this is that I can't execute the benchmark (_Benchmarks with issues: OperationTrackerBenchmarks.Do_Nothing: DefaultJob_). I've tried to troubleshoot of find a way to get a more specific error message, without luck. So my question is: how can I troubleshoot this further?

[pardahlman]

Hello again,

I've tried a few things to isolate this further. First of I switched computer 😄 . Further more I created a new solution and referenced BenchmarkDotNet and xUnit (and the console runner for the xUnit.) I have packaged this into a zip file that can be built and run. See attached.

To reproduce this issue:

1. Extract zip file, restore and build solution
2. In the solution folder run the xUnit runner

> .\packages\xunit.runner.console.2.2.0\tools\xunit.console.exe .\BasicBenchmark\bin\Release\BasicBenchmark.dll

This is my exact output from this test project

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
  Discovering: BasicBenchmark
  Discovered:  BasicBenchmark
  Starting:    BasicBenchmark
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   VoidBenchmark.Do_Nothing_At_All: DefaultJob

// Validating benchmarks:
// **************************
// Benchmark: VoidBenchmark.Do_Nothing_At_All: DefaultJob
// *** Generate ***
// Result = Success
// BinariesDirectoryPath = C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301

// *** Build ***
BuildScript: C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301\a18dd1ce-ceec-460f-8b57-e9852b859533.bat
// Result = Success

// *** Execute ***
// Launch: 1 / 1
// Execute: C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301\a18dd1ce-ceec-460f-8b57-e9852b859533.exe
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!


// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report.csv
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report-github.md
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report.html

// * Detailed results *
VoidBenchmark.Do_Nothing_At_All: DefaultJob
Runtime = ; GC =
There are no any results runs

Total time: 00:00:07 (7.19 sec)

// * Summary *

BenchmarkDotNet=v0.10.9, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Xeon CPU E3-1545M v5 2.90GHz, ProcessorCount=8
Frequency=2835937 Hz, Resolution=352.6171 ns, Timer=TSC
  [Host] : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2110.0


            Method | Mean | Error |
------------------ |-----:|------:|
 Do_Nothing_At_All |   NA |    NA |

Benchmarks with issues:
  VoidBenchmark.Do_Nothing_At_All: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
  Finished:    BasicBenchmark
=== TEST EXECUTION SUMMARY ===
   BasicBenchmark  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 8.969s

BasicBenchmark.zip

[pardahlman]

Hang on, looks like this works if I run it as a from a console app 🤔 . Any ideas why that is?

[adamsitnik]

Hello @pardahlman

You need to disable shadow copy. We are currently not supporting it and not planning to change it (due to many consequences, not because we don't like it)

Here you can find the right setting for xunit

[pardahlman]

Hello @adamsitnik - thanks for pointing me in the right direction - I wouldn't have figured this out myself. Everything works as expected now.

Do you think there is any way to detect if shadow copy is performed and warn the user? I'd be glad to help out if you think this is possible or desirable. If not, you can close this issue and keep on providing first class open source! 👌

[adamsitnik]

@pardahlman you are right, we should have provided nice warning. I hope that you did not waste too much time on this.

The users are now going to get following warning:

Assembly Test, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null is located in temp. If you are running benchmarks from xUnit you need to disable shadow copy. It's not supported by design.