Benchmarkdotnet: Fastcheck for correctness of benchmark implementations

[ig-sinicyn]

Use case: prevent users from comparing methods that do different things.

It's very easy to leave bug unnoticed in one of the benchmark methods. I just did it and had to spent some time to find that one of the competitors was exiting in the middle.

Proposed solution is fragile, does not _proofs_ that all code is correct but it's still better than nothing.
So, the proposal:

If benchmark method has a return value, collect it and check that all another non-void methods return the same value. The value serves as a checksum, so, if it differs benchmark should fail.
Ensuring that result of the benchmark is calculated correctly and that the result does not produce false positives is responsibility of the author of the benchmark.

I think that this has to be opt-in feature and it's enough to check the results only on first run.

[adamsitnik]

I have some idea, will try to implement it in form of a Validator that will be called before benchmark run.

[adamsitnik]

@ig-sinicyn Some progress: I have implemented ExecutionValidator that checks whether it is possible to run all benchmarks first. If you do not want to continue execution if single benchmark fails validation you need to use ExecutionValidator.FailOnError

The feature is not done yet, I will have to store and compare the results. I need to consider collections as well, but I will most probably only derive CorrectnessValidator from ExecutionValidator .

[ig-sinicyn]

@adamsitnik ok, thanks for it! :)

[mattwarren]

BTW, after using BenchmarkDotNet @fransbouma also asked for this feature, see https://twitter.com/FransBouma/status/723818104501997568

So consider that another +1 for it.

[ig-sinicyn]

@adamsitnik, @AndreyAkinshin

Upping. With #356 merged there's an API that can be easily extended with result reporting.

Initial idea is following: last invocation result should be passed as a member of RunResults and should be exposed via BenchmarkReport. I'd suggest to use Convert.ChangeType or TypeConverter.ConvertFromString(), both should work in .Net Core.

[adamsitnik]

@AndreyAkinshin I believe that we should not implement this feature. We should focus on performance related things, we don't have enough time to implement this one.

People should use unit test frameworks to check the correctness, it's not our responsibility.