Nunit-console: 3.10 release

[ChrisMaddock]

Thanks to @nvborisenko's work on #524 - I think we're about ready to go ahead with the 3.10 release! @rprouse - how are you fixed for time on this? I'm happy to help out, if you'd like.

@nunit/engine-team and anyone else - are there any outstanding issues you think should be resolved before we do 3.10? I'd like to merge #572 in before the release - it just needs a simple change to get finished.

We should be aware that this release contains potential breaking changes with the re-packaging of the engine, which allows the .NET Framework and .NET Standard builds to be distributed together.

[rprouse]

Now would be a good time for me. This is a pretty big change, so we should give it a good test, especially in areas where we have seen the bigger changes. I could use help defining what we should test and doing the tests with the latest builds. The .NET Standard build won't be consumed immediately, but we should test it in the adapter. I'm hanging out with @OsirisTerje this week, so we can do that. I'll resurrect my adapter branch and test it.

Plugin discovery also changed, so we should test some of the extensions. The console should also be tested in various modes like in-process, separate process, app domains? @ChrisMaddock can you take a pass at this?

[ChrisMaddock]

Sure! I'm travelling at the moment, and can't build on my surface, but will try and get this done this week. 😄 Thanks Rob!

[rprouse]

I've updated my PR to the adapter project that uses these changes. I built a test release of the console, engine and adapter based on that. I then updated my full framework test project and my test project that targets as many different versions of the framework and .NET core. So far, I've had no problems. This tests,

• Running full framework tests using the console
• Running full framework and .NET Core tests using visual studio
• Running full framework and .NET Core tests using dotnet test

Next I'm going to test;

• Using extensions from the console as a NuGet package.
• Using extensions from the console installed from an MSI

[rprouse]

I tried using the NUnit 2 result writer and it looks like NuGet extensions aren't working.

NUnit Console 3.9

λ  .\packages\NUnit.ConsoleRunner.3.9.0\tools\nunit3-console.exe --list-extensions
NUnit Console Runner 3.9.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.17763.0
  CLR Version: 4.0.30319.42000

Installed Extensions
  Extension Point: /NUnit/Engine/NUnitV2Driver
    Extension: NUnit.Engine.Drivers.NUnit2FrameworkDriver (.NET 2.0)
  Extension Point: /NUnit/Engine/TypeExtensions/IService
  Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
    Extension: NUnit.Engine.Listeners.TeamCityEventListener (.NET 2.0) (Disabled)
  Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
  Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
    Extension: NUnit.Engine.Services.ProjectLoaders.NUnitProjectLoader (.NET 2.0)
      FileExtension: .nunit
    Extension: NUnit.Engine.Services.ProjectLoaders.VisualStudioProjectLoader (.NET 2.0)
      FileExtension: .sln .csproj .vbproj .vjsproj .vcproj .fsproj
  Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter
    Extension: NUnit.Engine.Addins.NUnit2XmlResultWriter (.NET 2.0)
      Format: nunit2

NUnit Console 3.10

λ  .\packages\NUnit.ConsoleRunner.3.10.0\tools\nunit3-console.exe --list-extensions
NUnit Console Runner 3.10.0 (.NET 2.0)
Copyright (c) 2018 Charlie Poole, Rob Prouse
March 21, 2019 1:34:58 PM

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.17763.0
  CLR Version: 4.0.30319.42000

Installed Extensions
  Extension Point: /NUnit/Engine/NUnitV2Driver
  Extension Point: /NUnit/Engine/TypeExtensions/IService
  Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
  Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
  Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
  Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter

[ChrisMaddock]

Bugger. Thanks rob, I hoping to get round to looking at this at the weekend.

[rprouse]

The above was for old style CSPROJ projects with the console runner in the packages directory in the solution root. I tested using an CSPROJ that uses PackageReference where the NuGet packages get put in the user directory and it works there. It looks like we messed up the probing of extensions with the different directory structure between the two.

Good to know it is probably just a minor bug... 😸

3.9

λ  C:\Users\rob\.nuget\packages\nunit.consolerunner\3.9.0\tools\nunit3-console.exe --list-extensions

NUnit Console Runner 3.9.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.17763.0
  CLR Version: 4.0.30319.42000

Installed Extensions
  Extension Point: /NUnit/Engine/NUnitV2Driver
    Extension: NUnit.Engine.Drivers.NUnit2FrameworkDriver (.NET 2.0)
  Extension Point: /NUnit/Engine/TypeExtensions/IService
  Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
    Extension: NUnit.Engine.Listeners.TeamCityEventListener (.NET 2.0) (Disabled)
  Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
  Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
    Extension: NUnit.Engine.Services.ProjectLoaders.NUnitProjectLoader (.NET 2.0)
      FileExtension: .nunit
    Extension: NUnit.Engine.Services.ProjectLoaders.VisualStudioProjectLoader (.NET 2.0)
      FileExtension: .sln .csproj .vbproj .vjsproj .vcproj .fsproj
  Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter
    Extension: NUnit.Engine.Addins.NUnit2XmlResultWriter (.NET 2.0)
      Format: nunit2

3.10

λ  C:\Users\rob\.nuget\packages\nunit.consolerunner\3.10.0\tools\nunit3-console.exe --list-extensions
NUnit Console Runner 3.10.0 (.NET 2.0)
Copyright (c) 2018 Charlie Poole, Rob Prouse
March 21, 2019 1:39:35 PM

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.17763.0
  CLR Version: 4.0.30319.42000

Installed Extensions
  Extension Point: /NUnit/Engine/NUnitV2Driver
    Extension: NUnit.Engine.Drivers.NUnit2FrameworkDriver(.NET 2.0)
  Extension Point: /NUnit/Engine/TypeExtensions/IService
  Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
    Extension: NUnit.Engine.Listeners.TeamCityEventListener(.NET 2.0) (Disabled)
  Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
  Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
    Extension: NUnit.Engine.Services.ProjectLoaders.NUnitProjectLoader(.NET 2.0)
      FileExtension: .nunit
    Extension: NUnit.Engine.Services.ProjectLoaders.VisualStudioProjectLoader(.NET 2.0)
      FileExtension: .sln .csproj .vbproj .vjsproj .vcproj .fsproj
  Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter
    Extension: NUnit.Engine.Addins.NUnit2XmlResultWriter(.NET 2.0)
      Format: nunit2

[rprouse]

Got it, easy fix, PR coming. An extra ..\ got added to the addins file, probably while fixing the v3 format.

[ChrisMaddock]

Hey @rprouse - got a bit of time to work on this today. Going to start with your comment above:

Plugin discovery also changed, so we should test some of the extensions. The console should also be tested in various modes like in-process, separate process, app domains? @ChrisMaddock can you take a pass at this?

Sounds like you've been able to look at plugin discovery already, so I'll test some process/app domain modes. I also have some of the leg work in copyrights etc done while I was offline, which I'll start a PR for, and finally, close that last PR I wanted to get in. 👍

[ChrisMaddock]

Ok, I'm currently seeing problems running anything out-of-process that targets >.NET 3.5 This doesn't look promising. :-/

NUnit.Engine.NUnitEngineException : An exception occurred in the driver while loading tests.
  ----> System.BadImageFormatException : Could not load file or assembly 'nunit.framework, Version=3.11.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.
--NUnitEngineException
An exception occurred in the driver while loading tests.

Server stack trace:
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Agents.RemoteTestAgent.Run(ITestEventListener listener, TestFilter filter)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in C:\Users\Chris\Documents\git\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 170
--
BadImageFormatException
Could not load file or assembly 'nunit.framework, Version=3.11.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.
   at System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)

[ChrisMaddock]

Diagnosis, my agents appear to be ignoring COMPLUS_Version, and always starting up as .NET 2.0.

12:58:55.651 Info  [ 1] NUnitTestAgent: Running under version 2.0.50727.9035, .NET 3.5
12:58:55.651 Debug [ 1] NUnitTestAgent: Agent process has the COMPLUS_Version environment variable value "NULL" overridden with "v4.0.30319", restoring the original value.

I think this must be machine specific, otherwise we'd be seeing it in CI. But...what?! 😕

[ChrisMaddock]

Ok...having spent the last 40 minutes trying to work out why the COMPLUS_Version was never working on my machine, I then thought to check NUnit Console 3.9, which is working fine for me. Time to diff instead...

[ChrisMaddock]

Ok, I'm totally confused, and out of time. Afraid I've made zero progress on this in the last 3 hours...

As it stands, I can't run tests in any .NET 4.0+ assemblies. (Unless specifying --inprocess).

• Building Master myself fails to run them
• Appveyor artefacts of master fails to run them
• The 3.9 artefacts (zip/msi) from the GitHub Downloads page CAN run them
• If I try and re-build 3.9 myself, it fails to run them

I'd appreciate anyone's thoughts on what might cause that odd combination of results. 🤔

[rprouse]

When you say that building master and building 3.9 fails for you, do you mean that it fails to build, or that the tests fail after you build it with the BadImageFormatException?

[rprouse]

@ChrisMaddock I am not sure what is going on or what is different for you. I ran all of the NUnit tests fine. Here is the start of the net45 run on my machine;

λ  C:\Users\rob\.nuget\packages\nunit.consolerunner\3.10.0\tools\nunit3-console.exe .\net45\nunit.framework.tests.dll
NUnit Console Runner 3.10.0 (.NET 2.0)
Copyright (c) 2018 Charlie Poole, Rob Prouse
March 23, 2019 5:24:47 PM

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.17763.0
  CLR Version: 4.0.30319.42000

Test Files
    .\net45\nunit.framework.tests.dll

=> NUnit.Framework.Assertions.AssertThrowsTests.AssertThrowsDoesNotDiscardOutput

I tried running from within the test directory, from the parent directory to the tests and from the bin directory for the NUnit Console. In other words, I tried tests in current dir, console in current dir and both absolute and relative paths to tests.

Can you think of something different on your machine?

[rprouse]

I've seen problems with having an older version of NUnit Console installed as an MSI and having it in the PATH conflicting with newer versions so I tried that too, but it works fine 😕

[ChrisMaddock]

, or that the tests fail after you build it with the BadImageFormatException

The exception sorry, yes. I thought of the msi interfering too, but no luck...I’m really not sure what might be different on my machine to cause this. ☹️

I’ll try on my work machine tomorrow, and see if it repros there or not...

[ChrisMaddock]

@rprouse - Good news! I think I've pinpointed the problem.

---

I set up the framework CI to run on the master console build, and it fails in the same way I'm seeing here. See below:

https://ci.appveyor.com/project/CharliePoole/nunit/builds/23317175

At this point, I was more confused by how yours WAS working! 😅

Because I couldn't see any changes to how complus_version was working, the only other thing I could think that might affect this was app.config files. I checked these yesterday too, but yesterday I checked the versions in source. Turns out something in the build process is inserting the below line into the agent configs, which is restricting them to run in .NET 3.5 max.

    <supportedRuntime version="v2.0.50727" />

My money is that it's this change, which caused the problem: https://github.com/dotnet/sdk/issues/2319#issuecomment-396409673

I'm also assuming you happen to have an older msbuild installed that I do right now, which is why you couldn't repro the problem, and why I couldn't rebuild a working v3.9.0.

I think the only solution is to add explicit supportedRuntime lines to the agent app.config, although we have an explicit comment in the file NOT to do that! I'm going to try now and see if that still allows our previous complus_version behaviour to work, or if it will override it.

[ChrisMaddock]

Update: the above does not work, as the supportedRuntime precedence overrides complus_version. There is however a csproj control to disable this behaviour however. PR incoming.

[CharliePoole]

I remember adding the config comments after fixing a similar bug many years ago on V2. However, that was due to our own supportedRuntime elements, not the tooling creating them.

[rprouse]

@ChrisMaddock good work tracking this down. That is a tough one to find! I believe I have the latest MSBUILD version installed. I only have Visual Studio 2019 RC 3 on this machine. That is why I had to do the 2019 PRs on the adapter and NUnit projects.

I checked my build and supportedRuntime was not added to the agent configurations.

[rprouse]

@ChrisMaddock is there anything else you would like to test or get into the 7.10 release? I can start preparing it today. Maybe get #575 by @jnm2 merged and into this release? The other PRs are way too big to do this late in the release.

[ChrisMaddock]

I wonder if VS 2019 contains a later MSBuild, with a fix. 🤔 FYI, this is the version of MSBuild I have:

Microsoft (R) Build Engine version 15.9.21+g9802d43bc3 for .NET Framework

Rob - afraid after all that, I haven't had any chance to test anything. :-| My next real opportunity to do so will be Tuesday, however if you're happy, please go ahead! The only think I think may be worth testing is running nunit projects etc, as there were some changes there.

[rprouse]

I'm using Microsoft (R) Build Engine version 16.0.452+g01bae6224c for .NET Framework

[rprouse]

NUnit Projects are working fine too. I may not release today, but I'm going to start pulling everything together.

[rprouse]

Release is out! 🎉

[ChrisMaddock]

Awesome! Thanks Rob! 🥳