Benchmarkdotnet: We need an option to stop running when the first benchmark fails.

Created on 31 Oct 2018 · 5Comments · Source: dotnet/BenchmarkDotNet

When running benchmarks in the lab, it is ideal to fail fast and avoid using resources unnecessarily. When a benchmark fails we would like to acknowledge the developers as soon as possible.
Currently, the harness keeps executing all benchmarks to completion. Is there a way to tell BDN to fail fast?

Source

jorive

Most helpful comment

@jorive to don't change the current default behavior (breaking change). I like the current one: last night I started the benchmark run for all the benchmarks we have and today in the morning I had the results for few thousands of them. 8 failed had no results but I had the errors in the logs. If it would stop on the first error I would have to fix the problem and rerun all the benchmarks again. With current behavior, I just fixed and re-run 8 of them.

adamsitnik on 31 Oct 2018

👍2

All 5 comments

I think that it would be nice to add such feature.

It could be added to IConfig as StopOnFirstError with default value = false and exposed via console line arguments to override.

@wojtpl2 maybe you would like to give it a try?

adamsitnik on 31 Oct 2018

👍1

Ok next week

On Wed, Oct 31, 2018, 19:13 Adam Sitnik <[email protected] wrote:

I think that it would be nice to add such feature.

It could be added to IConfig as StopOnFirstError with default value =
false and exposed via console line arguments to override.

@wojtpl2 https://github.com/wojtpl2 maybe you would like to give it a
try?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/BenchmarkDotNet/issues/939#issuecomment-434788204,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQh-j4rdJNIUyt1voRgaBfZ5LH7MbUK7ks5uqehigaJpZM4YEowc
.