Benchmarkdotnet: Provide a way to change the Artifacts path

[coonmoo]

Hello,

I need to run multiple BenchmarkDotNet instances on my computer in parallel.
Due to automation, I can't put them into seperate folders. So when the Console App is started multiple times, they always overwrite each others results.

So could you provide a property to set the name of the "BenchmarkDotNet.Artifacts" directory manually?

Thank you very much!

[zabulus]

There is option KeepBenchmarkFiles, that places different benchmark projects to separate folders.

[coonmoo]

I tried that, but all instances are running the exact same benchmark project.

[AndreyAkinshin]

@coonmoo, why do you want to run several benchmarks in parallel? It's totally kills the idea of the good precision level.

[nickbabcock]

While I don't share the same use case as OP, I feel like configuring Artifacts path is good when the benchmarks have to be conditionally compiled depending on the environment.

For instance, I'm trying out benchmarking different hash functions across core, clr, and mono. Some nuget packages do not work with core so those have to be conditionally added and referenced in the benchmark code.

After the core benchmark is ran, I have to move those results to another folder so they don't get overridden by the clr benchmark.

[AndreyAkinshin]

I feel like configuring Artifacts path is good

Agree, and we already have such issue: https://github.com/dotnet/BenchmarkDotNet/issues/377 =)

[coonmoo]

@coonmoo, why do you want to run several benchmarks in parallel? It's totally kills the idea of the good precision level.

Thanks guys!
Yeah thats true.

I wanted to leverage BenchmarkDotNet to benchmark network intensive I/O operations. To fully utilize the NIC I need to run these in parallel.

But maybe there are better frameworks for this kind of benchmarking?

[AndreyAkinshin]

@coonmoo, we are working on good multithreading support. Hopefully, will be available soon.
Right now you can perform parallelization inside your benchmark method.

[coonmoo]

Thank you, sounds awesome. Is there an alpha or something for the multhreading support?

Parallelization inside the method would mess up the results when BenchmarkDotNet reports the statistics.

e.g. consider the following methods to benchmark where I want to measure the op/s

void Send1MB()
void Send1MBParallel10()

The statistics for the parallel method should be divided by 10 to get the amount of operations per second sending a 1 MB file.
Is there anything in BenchmarkDotNet where I can override this behaviour?

[adamsitnik]

@coonmoo you can try OperationsPerInvoke (it should work, but I am not 100% sure and can't verify now):

[Benchmark(Baseline = true)]
void Send1MB() => Send(1mb);

[Benchmark(OperationsPerInvoke = 10)]
void Send1MBParallel10() => Parallel.For(1, 10, _ => Send(1mb));

[coonmoo]

Thanks for the fast response.
Yes that does the trick, e.g. it seems to divide the Mean value for Send1MBParallel10 by 10.

But now I noticed from the stats that Send1MBParallel10 "seems" to be more than twice as fast as Send1MB (looking at the Mean or op/s value).
So setting OperationsPerInvoke does not work for me, because in reality Send1MB is usually faster.

It rather needs to sum the time for all the single Send(1mb) calls in the Parallel.For, instead of taking the time the method needs to complete

[AndreyAkinshin]

It rather needs to sum the time for all the single Send(1mb) calls in the Parallel.For

In fact you want to measure the method throughput here. It will be a part of our multithreading support in the future, but not for now, sorry.

[adamsitnik]

Custom paths is being tracked by #377
multithreading support #147