Benchmarkdotnet: Copy custom setting from app.config

[kedde]

I'm trying to test the performance of different ways to query my database, my configuration is given in a configuration file(app.config). I couldn't figure out how to set the configuration file, is there a way to specify the path to an app.config?

[adamsitnik]

BDN creates new project for each Benchmark, compiles it and executes as separate process. New app.config file is also created, which probably is cause of the problem here.

So I would say that your benchmarks should not rely on app.config files, which is DBN limitation.

[kedde]

ok, thanks for the quick answer. I'll try to get the Entity framework to work without the config file.

[joebarbere]

I was able to get around this limitation by implementing the AppConfig class outlined here: http://stackoverflow.com/questions/6150644/change-default-app-config-at-runtime

Steps are:
Copy the AppConfig.cs file to your project
Set the App.config file to "Content, Copy if newer"
Add the following to your benchmark code:

[Setup]
public void Setup()
{
        // Set the app.config to the one we want to use, override the base one that BenchmarkDotNet uses
        AppConfig.Change($"{Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"..\"))}App.config");
}

This might not be inline with what the BenchmarkDotNet author has in mind. So please take with a grain of salt.

[FransBouma]

I'd like to ask for this issue to be reopened: it's a showstopper for me currently as I can't port RawDataAccessBencher over to BenchmarkDotNet due to this limitation: benchmarking e.g. database related stuff relies on app.config files. If the app.config file of the exe the [Benchmark] marked code is in is found, it should be simply copied to the source being generated, included in the csproj and it is then properly used with the dll.

[FransBouma]

I've looked into the code, but it's a bit hard to fix this without breaking things. The main fix I had in mind was in GeneratorBase.GenerateProject to call an extra virtual method which would be overriden in the ClassicGenerator and which would use the CurrentDirectory with the active calling process to see whether there's a .config file of the format .config, but that would break situations where people have explicitly asked for a different jit in the IConfig instance, _and_ at the same time the exe used has a config file too.

Alternatively IConfig could be adjusted but that too is a breaking change (albeit in theory), and the value only makes sense on clr/mono.

I have no idea whether the breaking changes I ran into above are really breaking changes for you, I can't judge that. So I didn't proceed with adding code for this yet, as it might be for nothing ;).

Any ideas how you wanted to fix this?

[heemskerkerik]

It's also an issue when you have to use bindingRedirect to fix version conflicts. The result is that the benchmarked process crashes and the following error message gets logged:

No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

[mattwarren]

but that would break situations where people have explicitly asked for a different jit in the IConfig instance, and at the same time the exe used has a config file too.

I think that the best/safest option for this feature is that if you ask for us to use your existing app.config file we use that as is. If we go down the route of merging your config with our own settings (to set the JIT, GC, etc), things may get messy. We can then throw and error if you ask us to use your app.config and create a job that would require us to re-write it, with a message explaining that it's one or the other.

@FransBouma, @heemskerkerik, @kedde as you are the ones asking for this, do you want to use your own app.config and have BenchmarkDotNet set the JIT (i.e. Jit.LegacyJit, Jit.RyuJit) OR would you be happy for us to copy your app.config as is?

I have no idea whether the breaking changes I ran into above are really breaking changes for you, I can't judge that.

At the moment we're pre-1.0, so we're a bit more open to breaking changes, although at the same time we don't want to break everything for our existing (loyal) users.

[heemskerkerik]

My problem would be solved when app.config is copied as is, and opt-in for that sounds logical.

[FransBouma]

Vs.net creates an app.config file after you've simply changed the framework version, so in many cases the app.config file is there, so opt-in is indeed best: the user then deliberately chooses to go for this route.

Merging the jit with the copied app.config file might not be that difficult though, the file is xml after all. Some xpath voodoo to find the node for the jit (don't forget the namespace name ;)) and if it's there, alter it, and if it's not there, add it, shouldn't be too hard, but a little edge case (only if people need different jit testing AND have to provide their own app.config).

[adamsitnik]

I think that we must merge custom user settings (like connection strings, assembly binding redirects) with the runtime settings: JIT and GC flags. It should be simple: take everything from the source app.config that is not <runtime>, generate <runtime> like we used to do and if any binding redirects are defined in app.config, then add them to <runtime>

[mattwarren]

@adamsitnik thanks for picking this up and yes I agree, if we can merge everything together that would be great.

Are you planning to add this as a setting to Config, like KeepBenchmarkFiles and will it default to the current behaviour, i.e. off/false?

[adamsitnik]

@mattwarren I was thinking about making it new, default, non-configurable behaviour. Is that OK for you?

[mattwarren]

My only thought was that if people have existing benchmarks that don't copy across the app.config and then we start copying it across by default, it may break some benchmarks when they upgrade (because of some settings in app.config, that is now copied across).

But it's a pretty obscure scenario, so maybe I'm worrying about nothing.

[FransBouma]

It's a breaking change, for the people who have an appconfig file which is used in the exe but not in the benchmark and now it is used in the benchmark too. I'd opt for a config setting or otherwise a setting which is off by default.

[adamsitnik]

@FransBouma but how could it affect the execution of the benchmarks? I can't see a scenario where it breaks something. As I said I will cut off every runtime setting different than assembly binding redirects.

[FransBouma]

You're right indeed: the benchmarks currently don't read anything from the config file, and if there's one there after the change, it won't read from the config file as well. So indeed no breaking change!

[FransBouma]

Great news @adamsitnik!

So this issue can be closed I think?

[adamsitnik]

@FransBouma I am glad you like it! It should solve few problems (custom assembly redirects was always pain with F#). As for the closing we keep issues open as long as we don't release new nuget package

[kolektiv]

Really pleased to see a fix for this is coming :) Is there any expectation of when 0.9.8 might land? Is there anything I can do to help?

[adamsitnik]

Hi @kolektiv !

Is there any expectation of when 0.9.8 might land?

It's up to @AndreyAkinshin , but my guess is within next few days (3-7).

Is there anything I can do to help?

Thanks for asking! The last thing we need to check is Mono support after we switched to Roslyn from MSBuild in 0.9.8. But I know that it's non trivial and requires a lot of work:

• get the code from our Roslyn branch
• compile it (we require VS 2015 with update 2 and dotnet cli installed)
• test it for Linux/Mac OS

[gdoron]

Hi, is there a doc/example somewhere showing how to use app.config for init?
Thanks.

[adamsitnik]

Hello @gdoron !

We just take app.config file from the project that defines/executes benchmarks (your console app) and copy them to the app.config that we create for the project that we generate, build and run on the fly.

So in your [GlobalSetup] method you can just use app.config as you want to.

[gdoron]

@adamsitnik Thanks Adam, that explains why I couldn't find how to use it, BECAUSE IT JUST WORKS... ;)
Thanks a lot, awesome library!