Benchmarkdotnet: Support CopyToOutput

[xoofx]

I'm having an issue with native dlls (I guess things changed in a recent 0.9.x release?)

In the test I'm running, I have some native DLLS, that are part of the build as a CopyToOutput, but it seems that BenchmarkDotNet is not copying these DLLs to the subfolders where it runs the tests. Do you know why?

Also not sure I like of recopying assemblies to a subfolder. What was the reason behind this change? (That makes things less straightforward when you want to override a file locally - a text file for example used by the test, but not copied by the BenchMarkDotNet process...)

[xoofx]

Looking at the code ClassicGenerator.cs, hm, it creates a new project but only add references to .NET assemblies but not other project files... but what if the original project had more stuffs copied?

Couldn't this just build the runner assemblies and copy it in the current folder instead? I'm really worried that the way it is working now doesn't allow any custom build (native dlls, content to output...etc.)

[xoofx]

Actually, found the problem. I was using a library that was using ClearScript Google V8. The problem is that their system for loading the native DLL is not that great, because they are trying to load it from some specified paths (PATH variable, path of assembly) but they are not actually able to load it from CurrentDirectory, which made the native dlls to fail...

So as I understand, when running a BenchMarkDotNet program from folder XXX\Bin\Release

• Environment.CurrentDirectory: XXX\Bin\Release
• Location of assembly: XXX\Bin\Release\BenchmarkDotNet.Artifacts\bin\YYY_METHOD_TO_TEST...etc

My only grip with this is that sometimes you have some resources attached to an assembly in the same folder, and the location of the assembly is used to load the resources from the disk (and not CurrentDirectory)

[adamsitnik]

Hello @xoofx

Right now we are switching from MSBuild to Roslyn. What we will offer in 0.9.8 is that we will copy all referenced dlls that are not in GAC to the benchmark-specific bin folder. Code

I am not sure how we could support native dlls or content to output. We would most probably need to parse the csproj and get the data from there and so far we tried to avoid that. We use benchmark.Target.Type.Assembly.GetReferencedAssemblies() to get the referenced dlls.

The other option is that we could extend our config with some list of files to copy or some regular expression that would tell us which ones are needed. But I don't like this solution too.

Or maybe we should simply copy all non *.cs files from CurrentFolder to bin?

I don't like these limitations but I simply do not see good solution for now.

[xoofx]

Hi @adamsitnik, thanks for the details

Precisely, my question is: why generating a sub-folder? Why not just build the runner assembly, drop it in the normal output folder, remove it when the bench is done, and continue? This way we avoid any problems with attached content (native dlls in sub-folders, content attached to an assembly...etc.), avoid to copy referenced assemblies...etc.

[adamsitnik]

@xoofx I like this idea! I don't know why, but I guess that @AndreyAkinshin is analyzing the compiled output from time to time and wanted to have it in separate folders for better readability.

@mattwarren @AndreyAkinshin what do you guys think? I can implement it in the next week if you accept that.

[AndreyAkinshin]

LGTM. There were some MSBuild related reasons for separated sub-folders. However, now we are going to use RoslynToolchain, so, we can move all the file to the root bin folder.

[xoofx]

So @AndreyAkinshin were you finally able to remove the need for sub folders?

[AndreyAkinshin]

Yes! With help of new RoslynToolchain: we don't have anymore a separated .csproj project, only .cs file in your bin directory without subfolders (+app.config+script file+resulted executable file). New package is already available on NuGet: https://www.nuget.org/packages/BenchmarkDotNet/0.9.8

[xoofx]

amazing, thanks!

[AndreyAkinshin]

@xoofx, also now we have a nightly NuGet feed:

<add key="appveyor-bdn" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />

(based on artifacts from appveyor: https://ci.appveyor.com/project/PerfDotNet/benchmarkdotnet/history )
So, if we will make some fixes, you shouldn't wait for an official release on NuGet (or build BenchmarkDotNet manually), you can use the nightly feed.