Nunit-console: nunit3-console 3.11.1 with --domain=Multiple and single nunit project of multiple assemblies uses only 1 AppDomain

[dmason-mis]

I've just upgraded a solution to use the NUnit3 console runner 3.11.1, up from 3.10.0.

The tests are configured to run with each test assembly in a separate AppDomain, using the --domain=Multiple command line option. The tests fail when a single AppDomain is used.

This works with v3.10.0, and the OneTimeSetUp for each test assembly reports a different AppDomain id and a FriendlyName : "domain-_hexchars_-_dllname_"

With the same command line in v3.11.1, the OneTimeSetup for each test assembly reports AppDomain id 2 and "domain-" as the AppDomain FriendlyName.

Full command line:

"..\packages\NUnit.ConsoleRunner.3.11.1\tools\nunit3-console.exe" "..\Framework.Testing.nunit" --config="Debug" --result="TestResults.xml" --process=Single --domain=Multiple --framework=net-4.5

Installed by nuget (Manage nuget packages in VS2017).

nunit.framework 3.12.0

Using --inprocess instead of --process=Single makes no difference

[ChrisMaddock]

As a workaround, would --process=Separate work?

[ChrisMaddock]

Also, can I ask what your usecase us, for using the process-single and domain-multiple options, rather than the default process isolation? I ask because we were considering deprecating those in future...precisely because of the maintenance problems we face such as this. 😞

[CharliePoole]

@ChrisMaddock If memory serves, there are two issues of deprecation here.

We already deprecated ProcessModel.Single in favor of ProcessModel.InProcess because the name is confusing to users. They get it mixed up with ProcessModel.Separate.

We have also discussed deprecating ProcessModel.InProcess and removing it in the future because of difficulties of getting in-process execution to work effectivelly.

[CharliePoole]

@dmason-mis Can you try substituting the individual assemly names for the nunit project in your command-line? I think it may create only one AppDomain because there is only one file.

This seems similar to issue #382, except there it's processes rather than AppDomains.

[rprouse]

@dmason-mis I expect that @CharliePoole is correct in his workaround, but I'd also be interested in why you want to run your tests in separate AppDomains rather than separate processes which gives you much better isolation.

I ask because AppDomain itself has been deprecated in .NET Core and will not be in .NET 5 either, https://docs.microsoft.com/en-us/dotnet/core/porting/net-framework-tech-unavailable#appdomains. Because of this, we haven't given the NUnit AppDomain code much attention recently.

[dmason-mis]

@ChrisMaddock and @rprouse : I believe the single process, multiple AppDomains approach was configured to avoid inheriting global configuration data present in memory from the running of other test assemblies. I'm sure there's another way to do that - I don't believe we need AppDomains.

@ChrisMaddock : --process=Separate did not work, --process=Multiple did work.

@CharliePoole : I may try out this eventually, but for now the most convenient workarounds for me are to continue using 3.10.0, or use --process=Multiple.

Thanks for your help everyone.

[CharliePoole]

@dmason-mis Trying out separate assemblies on the command-line isn't intended as a convenient workaround, but as a way to point the nunit team in the right direction. So, if you get a chance... :smile:

[dmason-mis]

@CharliePoole : Specifying all the test DLLs along with --process=Single --domain=Multiple works.

[CharliePoole]

Thanks @dmason-mis !

@ChrisMaddock In that case, it seems like this is related to an issue we have discussed before, where the runner is selected based on number of files rather than number of assemblies. The case that breaks things is a single project file containing more than one assembly, of course.

In our prior discussion we were talking about multiple vs single processes and IIRC that issue was already resolved. This is basically just the same thing WRT ApplicationDomains.

I'll try to find time to run some tests.