Nunit-console: Cannot load specified configuration file

[CharliePoole]

@JornWildt commented on Tue Nov 08 2016

I am trying to get nunit3-console.exe to load a specific configuration file as described here: https://github.com/nunit/docs/wiki/Configuration-Files. The idea is to use on single config file for many assemblies through the use of a .nunit project file.

I have "ServerTests.nunit" and "ServerTests.nunit.config" and the console runner is started as:

nunit3-console.exe ServerTests.nunit

or

nunit3-console.exe /process:Separate /domain:Single ServerTests.nunit

But apparently it is not loading the ServerTests.nunit.config file in any way. What could be wrong?

---

@JornWildt commented on Tue Nov 08 2016

I am trying to use configFile="ServerTests.config" in the .nunit project file too - without any success.

---

@rprouse commented on Tue Nov 08 2016

I have assigned a confirm label on this for the team to look at. The feature is not used much and we don't have integration tests for this, so it may have become broken.

---

@CharliePoole commented on Tue Nov 08 2016

IIRC, we no longer use the default "project.nunit.config" in NUnit 3, which would make the doc page we copied from NUnit V2 incorrect at this point. This needs to be confirmed.

The nunit-project-loader extension does return a package with the configuration file set from what is in the project but we need to check to see whether the framework honors what is passed.

If we find that the framework is honoring what is passed to it, then this issue should be moved to either nunit-console or to the nunit-project-loader extension.

---

@CharliePoole commented on Tue Nov 08 2016

One possibility here would be to allow an override of the config file on the command-line - although we should honor what's in the nunit project file if not overridden.

@JornWildt has also asked about this on SO, where he mentions the nunit adapter as well. Of course, the adapter can't load nunit project files but only loads what VS tells it to load. We could add a setting to the .runsettings file to specify the config file.

---

@JornWildt commented on Wed Nov 09 2016

One possibility here would be to allow an override of the config file on the command-line

That would certainly work too. What ever works, I'll take it :-) It is rather troublesome to maintain one test X.config for each assembly X - when it could all be put into one single shared config file.

And, yes, the nunit adapter is an issue too and I do see that it can be difficult here. If only the new GUI runner could use .nunit project files like the old one then I would be happy - running from inside VS 2015 is not as got an experience as the the old GUI runner provided.

---

@JornWildt commented on Wed Nov 09 2016

We could add a setting to the .runsettings file to specify the config file.

That would help, but unfortunately we do have solutions that contain many different test projects that would need different .runsettings files (or, rather, different .config files).

---

@CharliePoole commented on Wed Nov 09 2016

@JornWildt Bear in mind that there is as yet no production release of an NUnit 3 GUI at all! Whatever features it had or doesn't have, if we the developers are telling you it's not ready for production use, I think it makes sense to pay attention!

WRT the adapter, we are limited by the VS architecture, which has no provision for separate setting files other than the nunit per-assembly default. However, your initial question was about how to have a single configuration for all assemblies and an entry in the run settings file would give you that.

---

@JornWildt commented on Wed Nov 09 2016

Bear in mind that there is as yet no production release of an NUnit 3 GUI at all!

I know :-) I did compile it myself from GitHub to see how it worked.

So the conclusion is that NUnit 3 has no way of supporting shared config files as it is today, right? The GUI-runner is not ready and neither the console runner nor the nunit adapter supports it. That is unfortunate but not a show stopper. We _really_ need NUnit 3 for testing async code :-)

---

@JornWildt commented on Wed Nov 09 2016

I'll see if some creative app.config copying in the build process solves it.

---

@rprouse commented on Thu Dec 01 2016

@JornWildt did you solve your problem? Can we close this issue?

---

@JornWildt commented on Thu Dec 01 2016

I worked around the issue by copying the config files as part of the build process. It would still be nice to have a, per project/assembly, setting that defines which config file to load.

---

@rprouse commented on Thu Dec 01 2016

Thanks @JornWildt. Based on that, I am making this an idea for discussion.

@nunit/contributors do you think we should promote this from an idea to the backlog or close as is?

---

@CharliePoole commented on Thu Dec 01 2016

I'm confused as to what the issue actually is after reading through this. It seemed to start out with wanting a __common__ config file and then evolved into a separate config per assembly.

As it now stands, you can specify a common config file either in an NUnit project file or on the console runner command-line (I think). If you want each assembly to have a separate config, then you have to use the naming convention and not specify a common config.

---

@JornWildt commented on Thu Dec 01 2016

It seemed to start out with wanting a common config file and then evolved into a separate config per assembly.

No - it evolved into a setting per assembly that points to the common config file :-)

As it now stands, you can specify a common config file either in an NUnit project file or on the console runner command-line (I think).

But I could not get a common config file to work with the console runner (see my initial comments) - have you fixed that? And it would still be a problem to use a common config file from inside Visual Studio as there currently is no way to specify the config file here (except by naming convention which does not allow assemblies to share config files).

---

@ChrisMaddock commented on Fri Dec 02 2016

Seems like there's a possible bug to be investigated with the engine/project-loader-extension, as to why the config file isn't being picked up. I had a quick look, and there seems to be code with the intention of doing that. @JornWildt, would be great if you would be able to dig in to it and see what's going on! From @CharliePoole's comments, it sounds like we also need to clarify the documentation about what we support.

If there's also a feature request to add this functionality to the vs adapter, I guess that should be a separate issue. @JornWildt - if you're still interested in this part, can you create an issue over at https://github.com/nunit/nunit3-vs-adapter? 😄

@rprouse - I've been closing/reclassifying some of the issues you've flagged as I look at them - please feel free to undo anything you disagree with!

---

@ChrisMaddock commented on Fri Dec 02 2016

@JornWildt - Just had another thought. There was a mistake in the documentation at the time you first tried this, the attribute for the project file should be configfile, rather than configFile. Would that work?

https://github.com/nunit/docs/issues/80

@nunit/core-team - The mix of camelCase and lowercase in the project file attribute names is a bit of a trip-wire. To keep backwards compatibiltiy, can we make them all case-insensitive?

---

@ChrisMaddock commented on Fri Dec 02 2016

Ok, I'm confusing myself at this point, so here's a list of distinct issues, as I see it! 😄

1. NUnit project files _should_ allow setting a configfile. This may work, and the issue may just have been caused by incorrect docs already resolved as https://github.com/nunit/docs/issues/80

2. The docs currently state that MyProject.nunit.config will be picked up automatically when running MyProject.nunit - @CharliePoole thinks that may not be the case. We should look in to that and decide if we want to support it.

3. We should consider making project file consts case-insensitive. It's not ideal, but imo would be an improvement on the mix of casing we have at the moment. (Created https://github.com/nunit/nunit-project-loader/issues/6)

4. There's possibly a separate feature-request for specifying a config file for the VS adapter, which can be a separate issue. 😄

Have I missed anything?

---

@CharliePoole commented on Fri Dec 02 2016

@ChrisMaddock There's no backward compatibility issue around casing of tag names. The tags have been the same since they were first created. There was a typo in the docs, which were recently created, but it's only coincidental that it was a casing error - it could easily have been a misspelling, in which case we wouldn't be talking about changing the XML format to match.

---

@ChrisMaddock commented on Fri Dec 02 2016

@CharliePoole - sure, that's not what I was suggesting at all. :-)

I'm suggesting that we should standardise the casing going forwards. And rather than just make everything lowercase, would could make it case insensitive, _so that_ we maintain backwards compatibility.

---

@CharliePoole commented on Fri Dec 02 2016

@ChrisMaddock Sure. But there are a lot of other things we have wanted to change about this format, but didn't change because of backward-compatibility. That's why we continued to use the V2 nunit project format in NUnit 3, in the expectation that it would eventually be replaced. When we do a new format, with capabilities like nested projects and per-assembly config files, we'll be about 10 or 15 years smarter than we were when the first format was designed, so we ought to do a better job. :smile:

Given that the same project format is used by V2 and V3 and that the project editor is shared by both versions, this could get a bit complicated.

---

@CharliePoole commented on Fri Dec 02 2016

@ChrisMaddock On your four points...

1. NUnit project files do support setting a common config file. There is no indication this stopped working at any point. Certainly our tests of it continue to pass.

2. We don't want to support that naming. It was talked about as being overly confusing during the V2 era and we decided to drop it in NUnit 3. This is simply a case where the old docs were copied and never changed.

3. As I explained on the issue you created, I don't think this is a good idea.

4. Yes, that would be a separate request.

---

@CharliePoole commented on Fri Dec 02 2016

@JornWildt So let's go back to your original request...

1. You tried to use a config file called "ServerTests.nunit.config" to match your nunit project. That's no longer supported, in spite of what the docs say.

2. You indicate you tried to specify the config file in the NUnit project file. That didn't work for you. It's supposed to work, which is why this issue is marked confirm. Can you provide a sample of an NUnit file that failed to use the config you specified so we can try to reproduce the problem?

3. As you explained, our side trip into individual configs for each assembly was merely a workaround for the original problem, so let's set that aside for purposes of this issue and try to resolve item 2.

---

@CharliePoole commented on Fri Dec 02 2016

Correction: There is no configfile command-line option. I guess it was discussed but never added. So the only current way to set a non-standard config file name is through an nunit project.

---

@JornWildt commented on Mon Dec 05 2016

Correction: There is no configfile command-line option. I guess it was discussed but never added. So the only current way to set a non-standard config file name is through an nunit project.

Okay. I'll try to see if I can get it to work with a nunit project file and the console runner.

But ... in the end it does not help much if it is not supported from the VS adapter/runner as I would end up having to maintain (make copies of) the identical config files anyway to be able to develop and test from VS. And if I have to maintain multiple identical files anyway then I won't need the nunit project file setting as it will not add anything extra.

---

@CharliePoole commented on Mon Dec 05 2016

@JornWildt Actually, it matters a great deal for us to know why it isn't working. The only thing the adapter can do is to tell the engine to use a particular config. If the engine doesn't pay attention, then it's not going to work for __any__ runner at all!

You filed this issue against "nunit", which is the framework. The framework doesn't set up any AppDomains, but just runs in the one that it's loaded into by the engine. I'm moving this to the "nunit-console" project because (1) that's where the engine is and (2) you originally complained about the console runner. I'll add more comments there.

---

@JornWildt commented on Mon Dec 05 2016

Sure :-)

[CharliePoole]

Moved this from nunit/nunit#1893 and changed title to indicate that it refers to a "specified" config file, not the default-named config file.

The issue has three aspects:

1. If the package provided to run an assembly contains the name of a config file, is it actually used.
2. Does the console runner need to provide a way for specifying the config file?
3. Does the nunit project loader extension provide a config file to the package when one is specified in the project?

If the third point turns out to be a problem, we will need to create an issue against the project loader extension.

This issue will not deal with other runners besides the console. If we need a way to specify a config file in any other runners, such as the gui or the vs adapters, separate issues should be filed. However, making sure the engine and the nunit project loaded work correctly in this regard is a prerquisite to doing anything else.

[JornWildt]

I can confirm that assigning a config file from the .nunit project file works when running via the console runner (nunit3-console.exe). Here is a working example that reads configuration settings from "MyTestSetup.config":

<NUnitProject>
  <Config name="Debug" configfile="MyTestSetup.config">
    <assembly path="SomeTests\bin\Debug\SomeTests.dll"/>
  </Config>
</NUnitProject>

Using configfile instead of configFile fixed it.

[CharliePoole]

So that gives us a "Yes" to item 3 in my list:

1. Does the nunit project loader extension provide a config file to the package when one is specified in the project?

Additionally, since the way the extension provides the name of the config file is as a package setting, item 1 is a "Yes" as well:

1. If the package provided to run an assembly contains the name of a config file, is it actually used.

That leaves the question of item 2:

1. Does the console runner need to provide a way for specifying the config file?

A command-line option to set the config file would allow specifying a config file to be used when no nunit project is involved. Do we see a need for this?

[JornWildt]

A command-line option to set the config file would allow specifying a config file to be used when no nunit project is involved. Do we see a need for this?

From my point of view, no.

I do not know if the command-line runner allows multiple assemblies to be tested in the same run. If this is the case then one could argue that being able to supply a config file on the command line would be smart. On the other hand - why run run multiple assemblies without a .nunit project file in the first place?

[CharliePoole]

It's actually pretty common to run multiple assemblies on the command line.

[ChrisMaddock]

As far as I can see - everything here was resolved.

The idea of specifying a config file on the command line has just been re-raised here: https://github.com/nunit/nunit-console/issues/246. Given the digression on this issue, I'm going to close it in favour of the new one - although I think the conclusion here was that @JornWildt could get by without the feature. 🙂

Please comment to reopen if I've missed anything!

[CharliePoole]

Even though the newer issue would usually get the duplicate label, I think this makes sense since that one is more closely targeted to the command-line. That is, it seems to be - I'm waiting on a reply from the OP.