Benchmarkdotnet: Add a mode to BenchmarkSwitcher that allows to run a method inline for profiling

[pakrym]

It's hard to attach profiler during out of process run so right now we just write a loop inside Program.Main that calls into benchmark method.

Command line would look something like this:

dotnet run -c Release inline method=Class.Method

[adamsitnik]

I like this idea. I could use it also for our Diagnosers (the results would not get affected by our loops)

[DanubeRS]

I couldn't help but notice in #422 it was mentioned this was proposed for 0.10.5. Just wanted to ping to see whether this one has flown under the radar, or whether its still in the vNext milestone backlog.

[adamsitnik]

@DanubeRS @pakrym if the profiler that you want to use has some command line API, then it's possible to integrate it today with BenchmarkDotNet

Idea: have a custom diagnoser (a plugin) which tells BDN to run one more time, attach the profiler before the main run and close it after the main run.

public class SampleIntegrationWithProfiler : IDiagnoser
{
    private Process profiler;

    public IEnumerable<string> Ids => new[] { nameof(SampleIntegrationWithProfiler) };

    // it tells BDN to run benchmarks once again with this diagnoser enabled, do the diagnostics and discard results (they are affected by oveerhead)
    public Diagnosers.RunMode GetRunMode(Benchmark benchmark) => Diagnosers.RunMode.ExtraRun;

    public void Handle(HostSignal signal, DiagnoserActionParameters parameters)
    {
        if (signal == HostSignal.BeforeMainRun)
            profiler = Process.Start("thePathToProfiler.exe", $"somehow tell the profiler to attach to this process {parameters.Process.Id}");
        else if (signal == HostSignal.AfterMainRun)
            profiler.Close();
    }

    public IEnumerable<IExporter> Exporters => Array.Empty<IExporter>();
    public void DisplayResults(ILogger logger) { }
    public IColumnProvider GetColumnProvider() => EmptyColumnProvider.Instance;
    public IEnumerable<ValidationError> Validate(ValidationParameters validationParameters) => Array.Empty<ValidationError>();
    public void ProcessResults(DiagnoserResults results) { }
}

[DanubeRS]

Thanks @adamsitnik. Thankfully I am using dotTrace, which exposes such a thing, will post an example for reference when I have it working to a decent standard.

[adamsitnik]

willll post an example for reference when I have it working to a decent standard.

that would be great!

[grofit]

@DanubeRS Any chance you have that example ready to share?

[DanubeRS]

Sure,

Its not finished and/or clean, but I know it worked well enough for my needs. Largely was an implementation from @adamsitnik and his above code tailored for the dotTrace CLI profiler.

Throwing the diagnoser in a gist for now, but depending on how this issue travels, may become obsolete.

IIRC, one issue I was encountering was that the benchmark was running before the profiler was fully attached, which may lead to some incomplete analytics. Once again, will possibly address this in the future after further investigation.

[adamsitnik]

Implemented in #878 and #964 --profiler ETW --profiler CV