Benchmarkdotnet: Switch back from project.json and xproj to csproj, support dotnet cli preview 3

[AndreyAkinshin]

@adamsitnik, it's time to get rid of the project.json files!

[adamsitnik]

I definitely plan to implement this, but currently there are few blocking issues:

1. It works only with Visual Studio 2017 which is still RC, no support for VS 2015.
2. Moving from xprojs to csprojs is something new, probably easy to do, the thing is that I will need to rewrite our Core Toolchain once again. This might be tricky and time consuming.

I would like to wait a little bit (at least few weeks) to see if our "main" .NET Core users will move to csprojs to see if anybody needs that.

[adamsitnik]

Results of my twitter form with 421 answers:

[adamsitnik]

@AndreyAkinshin there is one good thing. Roslyn is cross platform now, so I will try to have a single Toolchain for both classic and Core .NET ;) There is a chance that we will not need a strong dependency to dotnet cli anymore ;)

[AndreyAkinshin]

Sounds great!

[mattwarren]

I think it's amazing that there are over 400 people who like BenchamrkDotNet enough (and presumably use it), to vote in a Twitter poll, that's fantastic!!!

[adamsitnik]

@mattwarren I was surprised too! I got so many answers mostly because Ben and then @dotnet have retweeted my tweet ;)

[bojanrajkovic]

This would be nice to have sooner rather than later. I was able to run the generated project.json-based code by using dotnet migrate and some manual hackery of both the project.json and generated .csproj, but of course, that didn't give me nice output. :)

[roji]

+1 on this, since updating my project to the new csproj format I can no longer run benchmarks... I really hope you guys get around to doing this soon!

[adamsitnik]

@roji Good news! I have experimental version for you ;)

1. Add our custom CI nuget feed to your nuget sources https://ci.appveyor.com/nuget/benchmarkdotnet
2. Upgrade your .NET App to netcoreapp1.1, remember to also update the Microsoft.NETCore.App nuget package to 1.1.0
3. Use the custom toolchain, sample config below. The key is .With(Job.Default.With(CsProjToolchain.NetCoreApp11))

var config = ManualConfig.CreateEmpty()
        .With(Job.Default.With(CsProjToolchain.NetCoreApp11))
        .With(DefaultConfig.Instance.GetLoggers().ToArray())
        .With(DefaultConfig.Instance.GetColumnProviders().ToArray())
        .With(MarkdownExporter.GitHub)
        .With(HtmlExporter.Default)
        .With(MemoryDiagnoser.Default);
//        .KeepBenchmarkFiles(); enable in case of troubleshooting

BenchmarkRunner.Run<ClassThatContainsTheBenchmarks>(config);

This is limited, I could not do the full testing:
limitations:

• can not set process priority and/or affinity (needs research and implementation)
• can not break with ctrl+c (needs research and implementation)
• does not support custom GcMode settings (needs research and implementation)
• does not support x86 settings (needs research and implementation)
• path to the .csprojs with benchmarks deifnitions supports only the simplest scenario (now it's
• <ProjectReference Include="..\$EXECUTINGASSEMBLYNAME$\$EXECUTINGASSEMBLYNAME$.csproj" />

Edit: feedback is most welcome!

[roji]

@adamsitnik wow thanks, that was pretty quick :) I'll try to test ASAP and provide feedback.

[bleroy]

Worked for me. I was able to upgrade my project to the new tooling with this build. No problem to report yet. Thanks!

[adamsitnik]

@bleroy thanks for the feedback!

I am in Redmond now, I have met @cesarbs and we improved the toolchain. Currently I am removing last problems, it will be 100% production ready very soon,

[adamsitnik]

@bojanrajkovic @roji @bleroy @cesarbs It's ready. Could you give it a try and let me know if it works or not? You can download the latest nuget package from our CI feed <add key="appveyor-bdn" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />, please download the 0.10.2.333 version.

Btw. You don't need to set up the toolchain manually anymore. We detect the stuff for you.

[cesarblum]

@adamsitnik Awesome. I'll git it a try within the next couple of hours.

[cesarblum]

Works beautifully 😄

One remark though. In Kestrel, I see binaries/ directories created in src/Microsoft.AspNetCore.Server.Kestrel/ and test/Microsoft.AspNetCore.Server.Kestrel.Performance/. Why is that? Before, everything was limited to the BDN.Generated directory.

Actually, couldn't the generated files be put in a temporary directory?

[adamsitnik]

Works beautifully

@cesarbs great! thanks for the feedback!

I see binaries/ directories created

When we had early .NET Core support at some poin the dotnet cli/dnx changed the default output directory. It was an issue for us because it broke our Executor which could not find the exes/dlls. Then I decided to use --output binaries option so we always know where the dlls ends up. The problem is that with new msbuild based dotnet cli it has changed the behaviour.

Let's say that we have project A & B, B references A.
Previously:
dotnet build .B\project.json --configuration Release --framework netcoreapp1.1 --output binaries was producing:

A\bin\Release\netcoreapp1.1\{dlls}
B\binaries\{dlls}

now with the new msbuild dotnet cli we get:

A\binaries\{dlls}
B\binaries\{dlls}

we cleanup our stuff (B\binaries), but we don't remove A\binaries (we don't even know about it's existence).

Actually, couldn't the generated files be put in a temporary directory?

They could, but it would not solve the issue + some users want to analyze the compiler output. I will check if we can stop using the --output binaries option

[mattwarren]

The problem is that with new msbuild based dotnet cli it has changed the behaviour.

Ahh the joys of the project.json -> msbuild changes, @adamsitnik I'm surprised you're still a .NET dev after all the work you've had to do!! If I was you I'd have given up and started working on Go or Ruby :-)

[adamsitnik]

@adamsitnik I'm surprised you're still a .NET dev after all the work you've had to do!! If I was you I'd have given up and started working on Go or Ruby :-)

hahaha, @mattwarren I am not sure how to respond to that ;) It was neither easy or pleasant but I feel somehow responsible for making it work since I ported BenchmarkDotNet to .NET Core. I can also see that when we give our users great benchmarking tool they are improving overall .NET performance. And I am a very big fan of that! On the other hand, I am a very stubborn perfectionist and I really don't like broken things.