Benchmarkdotnet: Summary & BenchmarkReport: request for additional members

[ig-sinicyn]

Hi!

Can analyzer's conclusions and result returned by benchmarks be added to Summary? Want to warn user if the results do not match or there're important messages from the analyzers.

And some minor suggestions:

• Minor perf issue: use Lazy for ResultStatistics property.
• Use IReadOnlyList<T> or ReadOnlyCollection<T> where's possible. ExecuteResults and AllMeasurements lists as example.
• Add IConfig parameter to the IValidator.Validate(). Will simplify validators that check Job settings (now I had to collect the Jobs from benchmarks arg).

[adamsitnik]

@ig-sinicyn Good ideas, as always! I will try to implement it for 0.10.1

[adamsitnik]

@ig-sinicyn one thing: what do you mean by results? results of the executed benchmarks? like 4 in the folowing example?

[Benchmark]
public int Sample => 4;

[adamsitnik]

I tried to add the Conclusions to the Summary, but when I saw the huge ctor we have there I decided that it would be good to implement #177 instead. Will try to do som PoC

[ig-sinicyn]

@adamsitnik yep, the value stored in the consumer.

As of #177 - great news!
I'd suggest to start with Message struct and with common interface for the engine output, smth alike

interface IEngineOutput
{
  void WriteMessage(Message message);
  void WriteMeasurement(Measurement measurement);
}

It will allow us to isolate Engine implementation from message passing API. Therefore, IEngineOutput should be defined by the Toolchain.

Next thing is to provide host-side API for the messages. I've done something similar but I am not sure my approach will work well with BDN, it's up to you:)

Anyway, here's what I've done so far:

1. Use RunSlots to persist the state during the run (during the runs, as my code could run benchmark multiple times). It is public API and third-party analyzers/diagnosers/validators could use it too.

1. Introduce state object that should be available always (something like Summary, but it's persisted across multiple runs).

2. Messaging API is added as a part of the state object, as example:

        var competitionState = CompetitionCore.RunState[summary];
...
        competitionState.WriteMessage(
            MessageSource.Analyser, MessageSeverity.Informational,
            $"{GetType().Name}: All competition limits are ok.");

1. The messages are exposed as public API and can be used for integration tests.

[ig-sinicyn]

@AndreyAkinshin, @adamsitnik , ouch, forgot about this one:

IEngine.SetupAction and IEngine.CleanupAction properties are not used at all. If it's by design, should these be removed?

[adamsitnik]

I think that it's already implemented, closing