Benchmarkdotnet: Should the Engine iterate over and consume IEnumerable and IQueryable results?

[adamsitnik]

Today I have discovered that one of the ML.NET benchmarks returns IEnumerable and takes 16 nanoseconds to execute. It was suspicious to me, so I decided to iterate over the result and consume the result.

[Benchmark]
public IEnumerable<IrisPrediction> Before() => s_trainedModel.Predict(s_batches[0]);

private readonly Consumer _consumer = new Consumer();

[Benchmark]
public void After() => Consume(s_trainedModel.Predict(s_batches[0]));

private void Consume(IEnumerable<IrisPrediction> predictions)
{
    foreach (var prediction in predictions)
        _consumer.Consume(prediction);
}

It turned out that after we consume it, it takes 2.370 ms, not 16 ns.

I would like to start a dicussion here: should BenchmarkDotNet handle IQueryable results and execute them?

@AndreyAkinshin @jorive what do you think?

[redknightlois]

Makes sense. It is also a common error when benchmarking. I am not saying I have been guilty of it, BUT if you try to guess probably you will be right :D.

[adamsitnik]

I had an offline conversation with @jorive and we agreed that most probably a validator with error + a public extension method for consuming the result would be best

[AndreyAkinshin]

should BenchmarkDotNet handle IQueryable results and execute them?

I think it's a good idea to enabled such mode by default. However, it makes sense to introduce an option for this feature (so, users will be able to disable it). Such behavior can be non-obvious for some users, so it would be nice to print a hint about it (in case if we discovered some IQueryable results). We can also create a nice IntroSample for our docs which demonstrates the effect of lazy evaluation.

[adamsitnik]

I added a new Validator which gives a clear error message with explanation + solution. With samples and more info in the docs

[AndreyAkinshin]

@adamsitnik, it looks awesome!