Benchmarkdotnet: Migrate from custom build scripts to Cake (C# Make)

[Ky7m]

Cake (C# Make) is a cross platform build automation system with a C# DSL to do things like compiling code, copy files/folders, running unit tests, compress files and build NuGet packages.
I can help migrate from custom build scripts to CakeBuild.

Does it make sense?

A few examples of real usage:

• IdentityServer4
• Polly

[AndreyAkinshin]

Hey @Ky7m. Sounds interesting. May I ask what exactly are you going to change?

[Ky7m]

@AndreyAkinshin, my vision is following:

1. Create build.cake script to replicate all custom steps from appveyor.yml and custom scripts from build directory
2. Test and verify that new build approach works identically to the existing one.
3. Once verification is completed switch to the new build approach (based on cakebuild)

Modifications:

• Add 3 new files: cake.build, build.ps1, build.sh
• No modifications to C# source code
• Remove custom build scripts (only when migration will be completed)

Let me know if I forgot something.

[AndreyAkinshin]

@adamsitnik, what do you think?

[adamsitnik]

@Ky7m recently we have simplified our building process when I ported BDN to the new .csproj format. Today we have "only" three problems:

• dotnet build run from console can't build a .csproj that references old .vbproj/.fsproj. VS can build it, it handles the paths in some different way.
• VS can't build a .csproj that references new .fsproj (.NET Core)
• VS can't build new .fsproj (.NET Core).

As of today our VB and F# integration tests are turned off due to that. The problem is on MS Build/VS side, I am not sure if it's possible to get it up and running today, it's truly bleeding edge ;(

My conclusion: I am not sure if this is possible for today. But big thanks for offering your help @Ky7m ! Very few contributors want to play with CI and build scripts!

[Ky7m]

@adamsitnik thank you for your detailed answer.
Let me build a quick prototype to verify if it is possible or not to address current problems.
Once I finish I am back with results.

[Ky7m]

A quick update. I have started working on migration and I was able to replicate exisiting build steps as a first step. Now process is described in one file build.cake and I started an integration testing with appveyor.
Once I reproduce build steps on Windows I switch to Linux/Mac (integration with TravisCI). WIll keep you posted.

[adamsitnik]

@Ky7m great, thanks!

[Ky7m]

1. Windows build (AppVeyor) is replicated https://ci.appveyor.com/project/IgorFesenko/benchmarkdotnet.

2. macOS build is partially (build+unit tests) ready https://travis-ci.org/Ky7m/BenchmarkDotNet.
   Issues:

   • vbproj cannot be compiled and on build it is excluded
   • Integration tests require additional investigation (dotnet test runner's issue, probably)

3. Build on Linux is next

[adamsitnik]

wow! running our tests for Mac and Linux would make our life much easier!

[Ky7m]

Partial build pipeline works on Linux now.
By "partial build pipeline" I mean:

1. restore-packages
2. build-solution (w/o vb project)
3. run unit tests (BenchmarkDotNet.Tests)

Quick summary:

• AppVeyor build is replicated.
• Travis CI Linux and MacOS builds are done without integration tests.

Next step is to spend some time and run integration tests on Linux and MacOS (trying to find a root of problem).

[Ky7m]

Now there are "green" builds on 3 different platforms:

• Windows
• Ubuntu
• macOS

Each build contains next steps (with some platform specifics):

• Build all projects from solution (except VB project on Unix platforms)
• Run unit tests (Windows - all targets, non-Windows - netcoreapp1.1)
• Run integration tests (Windows - all targets, non-Windows - netcoreapp1.1)
• Create nuget packages (only on Windows)

I think builds and test suites on non-Windows platforms are absolutely required (one platform specific issue was caught).

What is the next step for this migration?

In addition, I see some possible operations that would be good candidates to be automated as well.

[AndreyAkinshin]

@Ky7m, awesome, thank you. Your next step is a pull request. My next step is working on the Travis integration.

[Ky7m]

Great, will prepare pull request. Let me know if you need any support/guidance with Travis CI app/integration.

[AndreyAkinshin]

We use the .NET Foundation account for appveyor, probably we will use the same approach for Travis. I already wrote a letter to @jongalloway, waiting for a response.
One quick question: is it possible to check the list of tests in a nice form? (like this one: https://ci.appveyor.com/project/IgorFesenko/benchmarkdotnet/build/0.10.8.30/tests)

[Ky7m]

It seems to me that is not possible to get a nice and separate view for tests. I didn't see it on other projects too. So I will look to other approach how to create more usable test summary.

[AndreyAkinshin]

What do you think about Jenkins? Will it be hard to create a corresponded configuration?

In this case, we can probably migrate to https://ci.dot.net/ (default CI for many .NET Foundation projects). @jongalloway, is it possible?

[Ky7m]

Current implementation of build process itself can be easily adapted for Jenkins (and not only for Jenkins). Need to check only environment/container specific things and we are ready to go.
I think that all information can be taken from this repo https://github.com/dotnet/dotnet-ci
As far as I know @jongalloway is at NDC Oslo this week.

[AndreyAkinshin]

Still no news from @jongalloway =(
I guess we can merge #475 and solve the Travis CI issue later. @adamsitnik, is it ok for you?

[adamsitnik]

I guess we can merge #475 and solve the Travis CI issue later. @adamsitnik, is it ok for you?

I think it's a good idea. I don't expect that @jongalloway will ever answer any email ;(

[SeanKilleen]

What about using Appveyor? I believe they'd be free for this and work with Cake. I'll try to confirm.

Blog post for reference: http://www.michael-whelan.net/continuous-delivery-github-cake-gittools-appveyor/

[AndreyAkinshin]

@SeanKilleen, we already use Appveyor for Windows builds, but I don't see how to make it work for Linux and MacOS.