Benchmarkdotnet: Execution of benchmark on Windows Mobile 10 device

[zabulus]

I need benchmark code to be executed on WM device. Execution of a benchmark is a part of bigger process, that I don't want, or it would be impossible to port on UWP.

Is there some guide about implementation of execution part of BDN to clarify some questions?
For example, how DBN.Generated.exe process communicates with parent process.

[AndreyAkinshin]

BenchmarkDotNet has a flexible architecture which allows to define own toolchains and specify behavior for generating, building, and executing. You could look at predefined toolchains here: https://github.com/dotnet/BenchmarkDotNet/tree/master/src/BenchmarkDotNet.Core/Toolchains Probably, we have to define another one for executing benchmarks on a device.

@adamsitnik, what do you think about the subject?

Related issue: https://github.com/dotnet/BenchmarkDotNet/issues/103

[zabulus]

Ok, I think starting point is quite clear.
I'll start with implementing custom Toolchain for new Runtime.UAP.

[zabulus]

Just leaving this as reference.
Deploying UWP Applications via SDK:
https://blogs.windows.com/buildingapps/2015/07/09/just-released-windows-10-application-deployment-tool/#Tdbu6MiJsUVZIDPl.97

[zabulus]

@AndreyAkinshin I'm currently researching minimal UWP application for execution process.
Do execution process should provide additional information in any manner (console output, pipes and so on), or it could be just execute benchmark and exit process?

[adamsitnik]

@zabulus here you can find the code of the auto-generated console that we are using today.

The flow is following:
Parent => generate, build and execute child process
Child => let the parent know that has started, create engine, pre allocate, run setup, warmup the jit, let diagnoser know that setup is done, run the benchmarks, let the diagnoser know that is before cleanup, run cleanup, print the results.
Parent => parse the results

[zabulus]

The flow is following:
Child => let the parent know that has started, create engine, pre allocate, run setup, warmup the jit, let diagnoser know that setup is done, run the benchmarks, let the diagnoser know that is before cleanup, run cleanup, print the results.

Ah I found, it's SynchronousProcessOutputLoggerWithDiagnoser class on host side. Something similar should be implemented via IPC.

EDIT: It's just support for custom diagnosers. Time measurements is dumped with https://github.com/dotnet/BenchmarkDotNet/blob/f1f2317dafc663a2fa854ea566ede8befc68ea6d/src/BenchmarkDotNet.Core/Reports/Measurement.cs#L49, and parsed below https://github.com/dotnet/BenchmarkDotNet/blob/f1f2317dafc663a2fa854ea566ede8befc68ea6d/src/BenchmarkDotNet.Core/Reports/Measurement.cs#L67

[zabulus]

I've tried to continue working on porting to UAP, but at the end I came to conclusion that an additional library should be extracted as separate assembly. Let's name it Support-library. This Support library should be used only by code that is _benchmarked_, and should be as portable as possible. Problem that would be solved is that benchmark target code depends on i.e. code of diagnosers, toolchains and so on.
Currently dependency is following, based on the Samples project:
Samples.Host (code for starting benchmark) -> BenchmarkDotnet.Core
Samples.Benchmark (code that is benchmarked) -> Samples -> BenchmarkDotnet.Core,

With new library it would be:
Samples.Host (code for starting benchmark) -> BenchmarkDotnet.Core -> BenchmarkDotnet.Support
Samples.Benchmark (code that is benchmarked) -> BenchmarkDotnet.Support

Suggestion is pretty raw, but I hope I've made it clear in general.

[zabulus]

I've made the library to compile for UAP10.0 profile. It was not so hard TBH.
I've also added UapToolchain and got implementation done to UapBuilder.
Main problem is about messing with host/target libraries used by runner/benchmark code. Currently BDN expects that everything works on same platform and existing toolchains just set up location for assemblies from run-time object Assembly.Location. In case different host/target, the Builder needs access to all pre-build libraries for target platform. For my purpose, I'll just patch paths for my needs, but for integration the code to BDN this should be reconsidered.

For UAP I've adopted existing BenchmarkProgram.txt template, with changes regarding Application model of UAP apps. And here place where other problem revealed. Runnable code doesn't compile well, with error of missing field. Haven't researched this yet but I guess that root cause is reflection call from here:
https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet.Core/Templates/BenchmarkProgram.txt#L40.

[AndreyAkinshin]

@zabulus, awesome! It would be really cool to have UAP support.
@adamsitnik, could you look at the problem with assemblies?

[zabulus]

@AndreyAkinshin, I also hope so ).
From here I see other issue to consider about.
Host/Target communication currently implemented via Console output and parsing. In case UAP this should be fully implemented via remote IPC: HttpServer/HttpClient, Raw Sockets, you name it. Currently System.Console is still used on target, but when I'll compile it, UapRunner won't have access to a deployed process, not mention that System.Console seems not working on UAP, just stab (I can be wrong here).
I had other issues to consider about, but currently can't remember, I'll add them later.

[adamsitnik]

For my purpose, I'll just patch paths for my needs, but for integration the code to BDN this should be reconsidered.

we'll look for some universal solution

Runnable code doesn't compile well, with error of missing field.

it's most probably a warning, could you check?

In case UAP this should be fully implemented via remote IPC: HttpServer/HttpClient, Raw Sockets, you name it.

we used Console for classic desktop .NET, Mono & .NET Core because it was common ground for all of them. Since UWP is Windows-only the choice should be easier. The simpler the solution, the less dependencies the better.

@zabulus great job on doing this! sorry for my late response, I am on holidays in place where WiFi is 18kbps.... ;)

[zabulus]

I think I've resolved previous issue.
Changed dependency on package "Microsoft.NETCore.UniversalWindowsPlatform" from "5.1.0" to "5.2.2".
Now Autogenerated project build fails with error:
C:\Program Files (x86)\MSBuild\Microsoft\.NetNative\x64\ilc\IlcInternals.targets(936,5): error : Internal compiler error: Exception of type 'System.OutOfMemoryException' was thrown.

[zabulus]

I've got auto-generated project to be fully automatically compile on my PC. Though workaround for UAP libraries paths is needed.

Now I'm starting working on automatic deploy and run of a compiled application. I've found client library package for Windows Device Portal which allows many things, deploying and launching app including.
But this introduces additional dependency to this package. I think it would be better to package UAPToolchain in separate package in future.

References:

1. https://msdn.microsoft.com/en-us/windows/uwp/debug-test-perf/device-portal
2. https://github.com/Microsoft/WindowsDevicePortalWrapper/
3. https://www.nuget.org/packages/WindowsDevicePortalWrapper/

[AndreyAkinshin]

I've got auto-generated project to be fully automatically compile on my PC.

Great news!

I think it would be better to package UAPToolchain in separate package in future.

Totally makes sense to me. We could call it BenchmarkDotNet.Toolchains.Uap.

[zabulus]

BenchmarkDotNet=v0.10.1-develop, OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Core(TM) i7-6500U CPU 2.50GHz, ProcessorCount=4
Frequency=2531253 Hz, Resolution=395.0613 ns, Timer=TSC
  [Host] : Clr 4.0.30319.42000, 64bit LegacyJIT-v4.6.1586.0DEBUG [AttachedDebugger]

Job=Uap  Runtime=Uap  Allocated=0 B  

Method | Mean | StdDev | Tool | Gen 0 |
------------------ |------------------- |---------------- |------- |--------- |
UtcNowLatency | 236.5281 ns | 2.6017 ns | UtcNow | - |
NowLatency | 7,208.8181 ns | 149.1966 ns | Now | - |
UtcNowGranularity | 15,222,587.4565 ns | 75,678.6683 ns | UtcNow | - |
NowGranularity | 15,227,980.1897 ns | 115,696.4019 ns | Now | 108.3333 |

Full console log:
https://gist.github.com/zabulus/61bdf97f7daff57789bb059b97b9415b

Of course, reports contains not accurate information regarding hardware, but it is work for another day :)

[zabulus]

One more thing to consider.
Device Portal needs PIN or Cookies for pairing with API.
These data should be specified in some .config file, that won't be a part of benchmark binaries.

[AndreyAkinshin]

Awesome!

[adamsitnik]

@zabulus once again big thanks for your amazing work!

Since Windows Mobile 10 is discontinued I am closing the issue.