Benchmarkdotnet: Using a benchmark target's return value in a column

[Warpten]

I'm writing a deserialization library that is basically taking a file stream and producing a List<T>:

        [Benchmark(Description = "Achievement (WDC1)")]
        public StorageList<WDC1.AchievementEntry> WDC1()
        {
            using (var fs = OpenFile("Achievement.WDC1.db2"))
                return new StorageList<WDC1.AchievementEntry>(fs);
        }

And it goes on for a couple versions.

However, the execution times don't make much sense since they are a measurement of the total amount of entries per file, which changes per version (I have no control over the data). It would make a lot more sense to display the average time needed to load one single element of each version.

I'd love to be able to have an implementation of IColumn that is able to obtain the return value from a benchmark target and work on it.

I have a local project that is basically a dumbed down performance meter but it has its limitations, namely in regards to results output - producing histograms is a chore.

Is there a way to do that, or am I out of luck? By giving the source a quick glance through ILspy, I don't see much.

[adamsitnik]

hi @Warpten

I am aware of this limitation. The problem is that we run the benchmark in a separate process, so passing any data from one to another is not trivial.

I know that @krzysztofcwalina has faced this issue in ML.NET

@Warpten @krzysztofcwalina would it be enough if I would add a mechanism to return a string and print it in a dedicated column? Sth like:

```cs
[ExtraData]
public string Size() => Serializer(sth).Size.ToString();
````

The method would be executed just once.

[Warpten]

Maybe exposing a method's result as an object would be less limiting? It would be great to have an extra column where I can just do

return (benchmark.Target.Result as IList).Count

Or even more complex things like

// Suppose this is a variation of MeanColumn
return /* mean time here */ / (benchmark.Target.Result as IList).Count;

This is however assuming that Target.Result never changes between executions, since this would only retrieve the last result. However, I think this is fine, since you wouldn't typically want to do this sort of things if you write a benchmark that has its result vary (at least that's not obvious to me).

[adamsitnik]

Maybe exposing a method's result as an object would be less limiting?

but how should I then serialize it and pass from one process to another without introducing any dependencies to BenchmarkDotNet?

[gulbanana]

i think the string is fine, we could implement serialisation on top of that ourselves for advanced cases

[Warpten]

Coming back at this, I didn't know at the time that BenchmarkDotNet was spawning subprocesses (and I also diagonally read your answer, Adam - sorry about that!). So I guess strings are the best way out.

[Konard]

Well, something like this is possible even now. Look at Config class, SQLiteOutput and DoubletsOutput methods.
``` C#
using System.IO;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using Comparisons.SQLiteVSDoublets.Model;
using Comparisons.SQLiteVSDoublets.SQLite;
using Comparisons.SQLiteVSDoublets.Doublets;

namespace Comparisons.SQLiteVSDoublets
{
[ClrJob, CoreJob]
[MemoryDiagnoser]
[WarmupCount(2)]
[IterationCount(1)]
[Config(typeof(Config))]
public class Benchmarks
{
private class Config : ManualConfig
{
public Config() => Add(new SizeAfterCreationColumn());
}

    [Params(1000, 10000, 100000)]
    public int N;
    private SQLiteTestRun _sqliteTestRun;
    private DoubletsTestRun _doubletsTestRun;

    [GlobalSetup]
    public void Setup()
    {
        BlogPosts.GenerateData(N);
        _sqliteTestRun = new SQLiteTestRun("test.db");
        _doubletsTestRun = new DoubletsTestRun("test.links");
    }

    [Benchmark]
    public void SQLite() => _sqliteTestRun.Run();

    [IterationCleanup(Target = "SQLite")]
    public void SQLiteOutput() => File.WriteAllText($"disk-size.sqlite.{N}.txt", _sqliteTestRun.Results.DbSizeAfterCreation.ToString());

    [Benchmark]
    public void Doublets() => _doubletsTestRun.Run();

    [IterationCleanup(Target = "Doublets")]
    public void DoubletsOutput() => File.WriteAllText($"disk-size.doublets.{N}.txt", _doubletsTestRun.Results.DbSizeAfterCreation.ToString());
}

}

You will also need a custom column implementation to fit data from files into the report.
```C#
using System;
using System.IO;
using System.Linq;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;

namespace Comparisons.SQLiteVSDoublets
{
    public class SizeAfterCreationColumn : IColumn
    {
        public string Id => nameof(SizeAfterCreationColumn);

        public string ColumnName => "SizeAfterCreation";

        public string Legend => "Allocated memory on disk after all records are created (1KB = 1024B)";

        public UnitType UnitType => UnitType.Size;

        public bool AlwaysShow => true;

        public ColumnCategory Category => ColumnCategory.Metric;

        public int PriorityInCategory => 0;

        public bool IsNumeric => true;

        public bool IsAvailable(Summary summary) => true;

        public bool IsDefault(Summary summary, BenchmarkCase benchmarkCase) => false;

        public string GetValue(Summary summary, BenchmarkCase benchmarkCase) => GetValue(summary, benchmarkCase, SummaryStyle.Default);

        public string GetValue(Summary summary, BenchmarkCase benchmarkCase, SummaryStyle style)
        {
            var benchmarkName = benchmarkCase.Descriptor.WorkloadMethod.Name.ToLower();
            var parameter = benchmarkCase.Parameters.Items.FirstOrDefault(x => x.Name == "N");
            if (parameter == null)
            {
                return "no parameter";
            }
            var N = Convert.ToInt32(parameter.Value);
            var filename = $"disk-size.{benchmarkName}.{N}.txt";
            return File.Exists(filename) ? File.ReadAllText(filename) : "no file";
        }

        public override string ToString() => ColumnName;
    }
}

But there is a problem with that approach, it works only if ClrJob is used, CoreJob for some reason does not write the files.

The complete example: https://github.com/linksplatform/Comparisons.SQLiteVSDoublets

[sandersaares]

Being able to pass a simple string would be a great start for this functionality.

Being able to pass a Dictionary<string, string> (for multiple columns) would be even better.

Really, I don't care how I get the data from the benchmark into the table - as long as I can show it to the user in a more convenient way than having to search logs for it, I am happy.

A nice stretch goal might be something in line with @konard example - allow benchmark to emit events that are serialized to file and can be processed then by a custom column (e.g. to measure average or count "interesting" events or). But maybe this deviates too much from BenchmarkDotNet's core vision?