Benchmarkdotnet: Why does BenchmarkRunner generate an isolated project per each benchmark method/job/params?

[Matthew-Bonner]

In the documentation: http://benchmarkdotnet.org/Internals/HowItWorks.htm it states, "BenchmarkRunner generates an isolated project per each benchmark method/job/params and builds it in Release mode."

Can I ask why as the documentation doesn't state why this is the case and once the reason is provided can I add this to the docs as I think others will have the same question.

The reason I'm asking is it takes a lot of time to build each test, and I can't rely on the --no-dependencies argument as each test in the project I'm benchmarking has a particular dependency on a third-party library so all dll's are being pulled in and there are a lot.

Depending on what the answer is to my question, if it is feasible I'd like to create one project containing all the benchmarks to run.

[adamsitnik]

@Matthew-Bonner we generate a new project per Job because it's the easiest way to support all our features (like custom GC mode per job)

I will soon have to change it since I will be using BDN as a CI runner. So I will be generating one project for all benchmarks which share the same job/runtime settings.

[Matthew-Bonner]

Thanks for the clarification, although very informative, this raises another question relating to the original question, which I would be grateful if you could also answer.

When you make this change, will you be keeping support for the current implementation, or will this be a breaking change?

[AndreyAkinshin]

@adamsitnik,

I will soon have to change it since I will be using BDN as a CI runner. So I will be generating one project for all benchmarks which share the same job/runtime settings.

It would be nice to have such option, but I'm not sure about the default behavior.
The original problem which was solved via generated projects is the benchmark isolation. It's pretty easy to write a benchmark which spoils the environment for the next benchmark.

[adamsitnik]

@AndreyAkinshin the idea I have is following: create a dedicated type per benchmark but instead of having separate projects have one (many classes in Program.cs), then when running the benchmark provide the type name to the executable. So we have new process per every benchmark, but we compile only once per given runtime settings.

So if I have 10 000 benchmarks and I want to run them for .NET Core 2.0 and 2.1, I am going to create two big projects with 10 000 benchmarks each, compile once for 2.0 and once for 2.1. But run new process 2x 10 000 times.

@Matthew-Bonner the breaking change will be for those who implemented their custom toolchains.

[AndreyAkinshin]

@adamsitnik, OK, makes sense.
However, could we introduce an option which allows switching to the old behavior? It can be useful for debug sometimes. If it's too complicated, don't worry, we can live without such option. But if we can have it for free, it would be a useful feature for me.

[adamsitnik]

Implemented in #699, will be part of 0.11.0

[Matthew-Bonner]

This will break my toolchain then, so would it be possible to put together some instructions for migrating to the latest version, please?

[adamsitnik]

@Matthew-Bonner the changes are very small on the toolchain level. All methods are now using BuildPartition buildPartition instead of Benchmark benchmark. The BuildPartition class exposes everything you need, adopting your code to the changes should be +-10 lines of code changed