Benchmarkdotnet: Multiple build/publish failure with `--coreRun` toolchain

[MarcoRossignoli]

If we run benchmarks with CoreRun toolchain and --no-build switch we get bechmarks build errors.
This PR https://github.com/dotnet/BenchmarkDotNet/pull/1001 fix the issue adding some build/publish attempts https://github.com/dotnet/BenchmarkDotNet/pull/1001/files#diff-83177b9866652943fa5d5bb8f80d847aR89
I found this issue using new performance https://github.com/dotnet/performance repo.
To repro:
clone https://github.com/dotnet/corefx and build from root

build -release

clone dotnet/performance and run script(CoreRun.exe will be generated by corefx repo build):

d:\git\performance\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter *.Perf_Dictionary.* --corerun D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --bdn-arguments="--join"

It works and we can see the "attempts" from output

[2019/01/07 20:13:52][INFO] $ pushd "D:\git\performance\src\benchmarks\micro"
[2019/01/07 20:13:52][INFO] $ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages D:\git\performance\artifacts\packages --runtimes netcoreapp3.0
[2019/01/07 20:14:01][INFO] // Validating benchmarks:
[2019/01/07 20:14:01][INFO] // ***** BenchmarkRunner: Start   *****
[2019/01/07 20:14:01][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2019/01/07 20:14:02][INFO] // start dotnet restore  /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:04][INFO] // command took 2.16s and exited with 0
[2019/01/07 20:14:04][INFO] // start dotnet build -c Release  /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:29][INFO] // command took 25.36s and exited with 1
[2019/01/07 20:14:29][INFO] // start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:32][INFO] // command took 3.2s and exited with 0
[2019/01/07 20:14:32][INFO] // start dotnet publish -c Release  /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:44][INFO] // command took 12.04s and exited with 1
[2019/01/07 20:14:44][INFO] // start dotnet publish -c Release  --no-build /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:46][INFO] // command took 1.53s and exited with 0

I did some investigation here https://github.com/dotnet/performance/issues/209#issuecomment-451759735
Could be useful understand if we can fix the issue without "attempts" saving time.

/cc @adamsitnik @jorive(https://github.com/dotnet/performance/pull/217#discussion_r245769189)

[WojciechNagorski]

@adamsitnik I can help but I need information on what to do it. There https://github.com/dotnet/BenchmarkDotNet/pull/913#issue-223126912 you wrote that dotnet restore --no-dependencies and dotnet build --no-restore --no-dependencies are dengerous. Do you have some idea how to fix this issue or do I have to come up with a solution?

Maybe benchmarkDotNet should have parameters --no-build,--no-dependencies, --no-restore and then pass them to DotNetCliCommand during build?

[MarcoRossignoli]

@wojtpl2 I spent some time on it https://github.com/dotnet/performance/issues/209#issuecomment-451759735 I think that could be great understand why build and publish fail.
Here https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs#L81-L94 Adam fixed code adding attempts with --no-dependencies and --no-build
You should try re-enable the issue and understand why msbuild SDK take so different steps with and without --no-build of benchmark project(in this case MicroBenchmarks.csproj).
To do that clone https://github.com/dotnet/performance remove https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs#L91-L92 and try to run with your local(with code removed) source of BDN updatating MicroBenchmarks.csproj in this way

<ItemGroup>
    <ProjectReference Include="D:\git\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj" />
    <ProjectReference Include="D:\git\BenchmarkDotNet\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj" />
...

After this changes you should get errors(fixed by Adam's retry).
To generate msbuild logs I updated local BDN to emit log adding /bl:path(https://github.com/Microsoft/msbuild/blob/master/documentation/wiki/Binary-Log.md#creating-a-binary-log-during-a-build) here https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs#L125
And after generation I read with http://msbuildlog.com/

[MarcoRossignoli]

As @jorive said could be worth understand if this is an issue related to BDN+CoreRun toolchain or with dotnet core cli sdk(build phases...in particular why build wants move files that are clearly used/locked)
More info if you don't use --corerun it works i.e.:

d:\git\performance\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter *.Perf_Dictionary.* --bdn-arguments="--join"

So I think it's related only to CoreRun toolchain behaviour.

[MarcoRossignoli]

If someone wants more info on repro reach me on gitter or DM me on twitter.

[adamsitnik]

I can help but I need information on what to do it. There #913 (comment) you wrote that dotnet restore --no-dependencies and dotnet build --no-restore --no-dependencies are dengerous. Do you have some idea how to fix this issue or do I have to come up with a solution?

Maybe benchmarkDotNet should have parameters --no-build,--no-dependencies, --no-restore and then pass them to DotNetCliCommand during build?

@wojtpl2 I think that the best option would be to detect that user is running dotnet run with those arguments and re-apply them when building the auto-generated code.

[WojciechNagorski]

@adamsitnik There is not possible to detect that user run dotnet run with arguments.

I can't reproduce this problem. It's working for me 😄
Maybe my enviroment is different:

λ dotnet --version
3.0.100-preview-009812

I've cloned fxcore:

c:\Work\corefx (master -> origin)
λ git show -s --format=%H
a20e52392cfb80be0bc6aa29e84697690f254a9a

And I've built it build -release without errors and warning.

Next, I've cloned performance repo:

c:\Work\performance\src\benchmarks\micro (master -> origin)
λ git show -s --format=%H
9f7d4413af2ce110cdaddb11d988595bc10cc9a2

I've changed MicroBenchmarks.csproj:

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.11.3-develop" />
    <PackageReference Include="BenchmarkDotNet.Diagnostics.Windows" Version="0.11.3-develop" />

And I've changed NuGet.config to use my local feed. In this location I had changed BenchmarkDotNet.

    <add key="benchmarkdotnet-cli" value="c:\Work\BenchmarkDotNet\artifacts" />

After build in bin directory I see:

I've changed DotNetCliCommand:

But my log looks like:

λ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-build -- --coreRun C:\Work\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages D:\git\performance\artifacts\packages --runtimes netcoreapp3.0
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7f9a667e-3fd0-4fd3-9202-a0466b7f6ddd
// command took 4s and exited with 0
// start dotnet build -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7f9a667e-3fd0-4fd3-9202-a0466b7f6ddd
// command took 5.89s and exited with 0
// start dotnet publish -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7f9a667e-3fd0-4fd3-9202-a0466b7f6ddd
// command took 3.34s and exited with 0

I can't reproduce attempts.

@MarcoRossignoli Do you have any idea?

[MarcoRossignoli]

@wojtpl2 can you share your branch?I would clone and try on my local.

[WojciechNagorski]

Do you mean branch of BenchmarkDotNet?

[adamsitnik]

@wojtpl2 could you try the dotnet run with --no-restore option?

[MarcoRossignoli]

Do you mean branch of BenchmarkDotNet?

Yes I don't found issue_1002 on yours

I can't reproduce attempts.

Have you tried with py script?

d:\git\performance\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter *.Perf_Dictionary.* --corerun D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --bdn-arguments="--join"

[WojciechNagorski]

@adamsitnik The same result:

c:\Work\performance\src\benchmarks\micro (master -> origin)
λ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-build --no-restore -- --coreRun C:\Work\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages D:\git\performance\artifacts\packages --runtimes netcoreapp3.0
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\1cebdde0-cf2f-4a18-85e2-624d1a48f4cd
// command took 4.39s and exited with 0
// start dotnet build -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\1cebdde0-cf2f-4a18-85e2-624d1a48f4cd
// command took 6.67s and exited with 0
// start dotnet publish -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\1cebdde0-cf2f-4a18-85e2-624d1a48f4cd
// command took 4.05s and exited with 0

@MarcoRossignoli I've pushed https://github.com/wojtpl2/BenchmarkDotNet/tree/issue_1002

Now I'll try to use benchmarks_ci.py.

[WojciechNagorski]

Ok. I reproduced problem 😃

When I used benchmarks_ci.py I got error:

[2019/01/11 12:37:01][INFO] $ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun C:\Work\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages C:\Work\performance\artifacts\packages --runtimes netcoreapp3.0


[2019/01/11 12:37:10][INFO] // Validating benchmarks:
[2019/01/11 12:37:10][INFO] // ***** BenchmarkRunner: Start   *****
[2019/01/11 12:37:10][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2019/01/11 12:37:14][INFO] // start dotnet restore  /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:00][INFO] // command took 45.54s and exited with 0
[2019/01/11 12:38:00][INFO] // start dotnet build -c Release  /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:29][INFO] // command took 29.49s and exited with 1
[2019/01/11 12:38:29][INFO] // start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:34][INFO] // command took 5.45s and exited with 0
[2019/01/11 12:38:34][INFO] // start dotnet publish -c Release  /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:48][INFO] // command took 13.78s and exited with 1

and now I can reproduse this error without this script. When I type the commands in the order from script, I got 'attemps':

cd "C:\Work\performance\src\benchmarks\micro"

dotnet restore MicroBenchmarks.csproj --packages C:\Work\performance\artifacts\packages

dotnet build MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore

dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun C:\Work\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages C:\Work\performance\artifacts\packages --runtimes netcoreapp3.0

// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 4.02s and exited with 0
// start dotnet build -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 18.57s and exited with 1
// start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 5.27s and exited with 0
// start dotnet publish -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 13.57s and exited with 1
// ***** Done, took 00:00:43 (43.58 sec)   *****
// Found benchmarks:

I have some Idea how to solve this problem.

[MarcoRossignoli]

@wojtpl2 great!Can you share you "idea"? I thought something with Environment.GetCommandLineArgs() but I don't know if it makes sense with dotnet run... , or maybe something with parent process?

[WojciechNagorski]

@MarcoRossignoli I've pushed fix #1013

[adamsitnik]

Fixed in #1013