Benchmarkdotnet: Question: Any official way to benchmark same method between different assembly versions?

[Shazwazza]

Hi,

I can't seem to find anything specifically about what I'd like to do, but if this has already been discussed some other place please let me know :)

Otherwise, what I'd like to do is benchmark the same method of an assembly but with different assembly versions. I know I can accomplish this in various ways and use reflection to load in the type/method based on different assembly files but am wondering if there is any official way to do this?

[adamsitnik]

H! @Shazwazza !

Currenly we don't support it, but I am planning to add such feature.

My initial goal was to enable comparison for for assemblies that are available on the nuget. Sample scenario: what's the difference between current implementation and latest version available on nuget?

What would be your scenario?

[Shazwazza]

yes that is exactly what I was planning on implementing :)

I haven't written anything quite yet but wanted to try to get something going as early as next week. The idea that I had was something along the lines of:

• During the benchmark initialization, have some code config to point to either a referenced assembly or a nuget alias + version
• Use Nuget API (or whatever) to go fetch the configured Nuget assembly versions from whatever Nuget sources are configured
• Then load in the class/method from either the referenced assembly or the Nuget fetched assembly
• Run the benchmark

[adamsitnik]

One question: are you using .csprojs or .xprojs? I have an idea how dotnet cli could do the nuget restore for us.

If you are using csprojs then I think that the best thing to do would be to extend our RoslynToolchain. Here you can see how we build the code with Roslyn. I think that the solution would be to extend our Job class with list of assembiles/nugets that would be used by the RoslynGenerator

then it would a matter of executing all benchmarks in two configurations (by using two separate Jobs): without custom assemblies and with custom assemblies.

[Shazwazza]

For the current project I'm working with it would be csproj. Thanks for the pointers, I'll fork the project and have a look and hopefully i can find some time :)

[GeertvanHorrik]

I am currently working on this for my own purposes. I will push this to the (open source) repository soon:

https://github.com/catel/catel.benchmarks

What I have in mind is to create separate console apps / version (so we can target specific versions without problems). Then I have a "wrapper runner" that runs the console apps one by one and stores the results inside results / version.

Once this is done, I "only" have to combine the results.

[GeertvanHorrik]

Here is the full repository:
https://github.com/catel/catel.benchmarks

Example of a generated comparison markdown:
https://github.com/Catel/Catel.Benchmarks/blob/develop/results/summary.md

In theory we could use a single app and load them into separate AppDomains. However I am not sure if defining the different versions via NuGet is the way to go (it makes testing against local builds harder).

For now the combiner runs all the different benchmarks and combines the results (reading all measurements and grouping them by version) which are easy to use for exporting purposes.

[Rbn3D]

Any updates on this? I'm trying to benchmark two versions of the same project (one as project reference and the other as nuget packageReference).

My purpose is to measure optimizations i'm making to that project (so I can compare them with the current version published in nuget easily).

[GeertvanHorrik]

@Rbn3D did you check out the project I mentioned? It does do benchmarks against each version and creates a fully summary of the delta (top 10 improvements and top 10 stuff becoming worse).

[Shazwazza]

The PR for this is here https://github.com/dotnet/BenchmarkDotNet/pull/805

[Shazwazza]

Updated PR is here https://github.com/dotnet/BenchmarkDotNet/pull/922