Benchmarkdotnet: Strategy for making sure that microbenchmarks are not broken for regular CI builds

[pakrym]

We don't run microbenchmarks during every build/test run because we want to keep them fast. The consequence is that while we still build microbenchmarks projects we don't run them and don't notice when they start failing at runtime.

Is there any guideline on how to properly integrate running couple iterations of every microbenchmark into CI pipeline to validate that they are functional in every build and do it reasonably quickly?

Having something like dotnet run -validate when using BenchmarkSwitcher would be ideal.

[AndreyAkinshin]

Hey, @pakrym.
A short answer: we don't support performance testing for now (here I mean validating that we don't have performance degradation in the CI pipeline), it's a very complicated task (here you can find more info), but we are working on it.

[pakrym]

We don't want to validate performance itself just that benchmarks are functional and able to run.

[AndreyAkinshin]

@pakrym, than you can use a special job for your CI (see http://benchmarkdotnet.org/Configs/Jobs.htm).
E.g. you can mark your benchmark with DryJob or ShortRunJob (see https://github.com/dotnet/BenchmarkDotNet/blob/1670ca349c303b83c373855705ea4f4679f99ad0/src/BenchmarkDotNet.Core/Jobs/Job.cs#L24 and https://github.com/dotnet/BenchmarkDotNet/blob/1670ca349c303b83c373855705ea4f4679f99ad0/src/BenchmarkDotNet.Core/Attributes/Jobs/DryJobAttribute.cs), these kinds of jobs are designed for a quick validation that everything is fine with benchmarks (without actual high accuracy measurements).

[adamsitnik]

The fastest way to run all benchmarks just once would be to use Job.Dry mentioned by Andrey and use our in-process Toolchain implemented by @ig-sinicyn So no compilation and run just once.

private class Config : ManualConfig
{
    public Config()
    {
        Add(Job.Dry
            .With(InProcessToolchain.Instance)
            .WithId("InProcess"));
    }
}

BenchmarkRunner.Run(yourAssembly, new Config());

[adamsitnik]

@pakrym this is how we test that given benchmarks were executed and everything is fine in our integration tests:

var summary = BenchmarkRunner.Run(yourAssembly, new Config());

Assert.False(summary.HasCriticalValidationErrors, "The \"Summary\" should have NOT \"HasCriticalValidationErrors\"");

Assert.True(summary.Reports.Any(), "The \"Summary\" should contain at least one \"BenchmarkReport\" in the \"Reports\" collection");

Assert.True(summary.Reports.All(r => r.BuildResult.IsBuildSuccess),
    "The following benchmarks are failed to build: " +
    string.Join(", ", summary.Reports.Where(r => !r.BuildResult.IsBuildSuccess).Select(r => r.Benchmark.DisplayInfo)));

Assert.True(summary.Reports.All(r => r.ExecuteResults != null),
    "The following benchmarks don't have any execution results: " +
    string.Join(", ", summary.Reports.Where(r => r.ExecuteResults == null).Select(r => r.Benchmark.DisplayInfo)));

Assert.True(summary.Reports.All(r => r.ExecuteResults.Any(er => er.FoundExecutable && er.Data.Any())),
    "All reports should have at least one \"ExecuteResult\" with \"FoundExecutable\" = true and at least one \"Data\" item");

Assert.True(summary.Reports.All(report => report.AllMeasurements.Any()),
    "All reports should have at least one \"Measurement\" in the \"AllMeasurements\" collection");

[adamsitnik]

@pakrym I am closing this one now, please reopen if my answer did not help to solve your problem.

[pakrym]

What we end up doing is creating shared source package containing default runner that does verification: https://github.com/aspnet/Common/blob/dev/shared/Microsoft.AspNetCore.BenchmarkRunner.Sources/Program.cs#L28

And a custom attribute https://github.com/aspnet/Common/blob/dev/shared/Microsoft.AspNetCore.BenchmarkRunner.Sources/ParameterizedJobConfigAttribute.cs that allow us having config set per class but replace base job for it (https://github.com/aspnet/Common/blob/dev/shared/Microsoft.AspNetCore.BenchmarkRunner.Sources/Program.cs#L76)

In Test target of our build we run benchmarks with --validate-fast argument https://github.com/aspnet/BuildTools/blob/e4496e21e1661355670bd7fef34a7d723ab8cd5b/files/KoreBuild/modules/vstest/module.targets#L69