Benchmarkdotnet: benchmark / beforeEverything fails when running in docker-win

[mjkkirschner]

I am trying to run a benchmark inside a docker-win container using hyper-v. When I run the benchmark with a default config I see an error like:

BeforeEverythingElse
exitcode != 0

no results were gathered

when I run using the debugInProcess config - the benchmark runs.

I tried the advice here:
https://benchmarkdotnet.org/articles/guides/troubleshooting.html#in-the-same-process

but the execution never seems to make it to my benchmark to even enter the while loop.

debugging the resulting genereted.exe using MDbg.exe just seem to hang forever at the BeforeAnythingElse step.

Is there anything that would prevent the benchmarks from running in docker-win?

[AndreyAkinshin]

@mjkkirschner thanks for the report! Currently, I don't understand why do you have this problem. Could you please share your config? Do you have any enabled diagnosers (e.g., JitDiagnoser)?

@adamsitnik what do you think, could we print some additional information for the exitcode != 0 case? It's pretty hard to investigate a problem by this line.

[adamsitnik]

@AndreyAkinshin I am not sure if we can do something more. Right after we print BeforeEverythingElse we enter a try/catch block and if there is any exception we print it's message to the log.

https://github.com/dotnet/BenchmarkDotNet/blob/12fdbe20430b534bb390d758df821e5b437245d9/src/BenchmarkDotNet/Templates/BenchmarkProgram.txt#L30-L67

@mjkkirschner is that full output? Was there anything else between BeforeEverythingElse and exitcode != 0 ?

[mjkkirschner]

that was the full output with no other information.

[mjkkirschner]

I don't think we have a diagnoser enabled - config is here: https://github.com/DynamoDS/Dynamo/blob/master/tools/Performance/DynamoPerformanceTests/PerformanceTestHelper.cs#L106

[adamsitnik]

@mjkkirschner could you please try to re-run it and check the Windows Event Viewer to see if there are some more details?

If our try block does not work it might be some uncatchable (by user code) exception that is logged by the OS to Event Viewer.

[melanchall]

I have the same problem. My output is:

// BeforeAnythingElse
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

And that's all. At the end of process the benchmark is listed as with issue, but there is no issue information in the log.

[ru-sh]

It would've been a good tool to measure a containerized environment performance.
Hope you will fix it soon.

[onedami]

I'm getting this same error while trying to run Bencmarkdotnet on Microsoft Azure DevOps.
When I run it using [InProcess], it throws an exception "...takes to long to run. Prefer to use out-of-process toolchains for long-running benchmarks".

NB: Both InProcess and out of process run successfully on my local PC.

Has anyone gotten around this?

[melanchall]

The issue is opened more than half a year ago and still no fix. Since BenchmarkDotNet is maintained by MS I suppose it's quite easy to diagnose the problem on Azure Pipelines and solve the issue.

[adamsitnik]

Since BenchmarkDotNet is maintained by MS

It is not. From all the maintainers only I work at MS and I contribute to it in my free time (unless there is a bug|feature blocking my Team). It's part of the .NET Foundation but in reality, it means that we get a logo and CI for free.

I am sorry but I currently don't have the time to dig into that.

takes to long to run. Prefer to use out-of-process toolchains for long-running benchmarks

To avoid this problem you can create a custom instance of InProcessToolchain and change the default timeout to a bigger value.

https://benchmarkdotnet.org/api/BenchmarkDotNet.Toolchains.InProcess.NoEmit.InProcessNoEmitToolchain.html#BenchmarkDotNet_Toolchains_InProcess_NoEmit_InProcessNoEmitToolchain__ctor_System_TimeSpan_System_Boolean_

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Toolchains.InProcess.NoEmit;
using System;

namespace BenchmarkDotNet.Samples
{
    class Program
    {
        static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
            .Run(args, DefaultConfig.Instance.With(Job.Default.With(new InProcessNoEmitToolchain(timeout: TimeSpan.FromHours(1), logOutput: false))));
    }
}

[melanchall]

Thanks @adamsitnik. Yes, it's a workaround. Bit it would be great to fix the bug so we can throw away that custom code. Since several people reporting about the issue, priority of it should be increased I suppose.

Anyway thanks for the library!

[gfoidl]

Similar problem on Azure DevOps for net48 (same for net461) on windows-2019.

With

Get-EventLog -LogName Application -Newest 5

I can see:

MachineName        : fv-az659
Data               : {}
Index              : 8267
Category           : (0)
CategoryNumber     : 0
EventID            : 1026
EntryType          : Error
Message            : Application: DefaultJob.exe
                     Framework Version: v4.0.30319
                     Description: The process was terminated due to an unhandled exception.
                     Exception Info: System.NotSupportedException
                        at BenchmarkDotNet.Engines.ConsoleHost.SendSignal(BenchmarkDotNet.Engines.HostSignal)
                        at 
                     BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(System.String[])
                        at BenchmarkDotNet.Autogenerated.UniqueProgramName.Main(System.String[])


Source             : .NET Runtime
ReplacementStrings : {Application: DefaultJob.exe
                     Framework Version: v4.0.30319
                     Description: The process was terminated due to an unhandled exception.
                     Exception Info: System.NotSupportedException
UserName           : 
Site               : 
Container          : 

So it narrows down to https://github.com/dotnet/BenchmarkDotNet/blob/3223c94a92050147a02482a3810860703f0c5171/src/BenchmarkDotNet/Engines/ConsoleHost.cs#L25-L35

---

On my local dev-machine (win-10) there's no problem. Also there's no problem for .NET Core targets.

[adamsitnik]

@gfoidl thank you for providing more context! it was exactly all I needed to fix this mysterious bug

I've sent a PR with a fix and integration tests: https://github.com/dotnet/BenchmarkDotNet/pull/1490

Our CI has published an updated version of NuGet package. Could you please add our CI feed to your nuget.config file (can be created by running dotnet new nugetconfig) install 0.12.1.1382 and let me know if it works?

<packageSources>
  <add key="bdn-CI" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>

[gfoidl]

@adamsitnik I can confirm that Azure DevOps for net48 produces expected BDN-results. So my issue got solved by 0.12.1.1382.

[adamsitnik]

@gfoidl awesome, thank you!

[adamsitnik]

Fixed by #1491