Nunit-console: Unable to test net 3.5 assembly if there's incompatible extension installed

[ig-sinicyn]

We have an failing build on appveyor server due to extension that does not support net2.0 framework.
related issue.

Output from agent trace file:

12:51:40.231 Error [ 1] ServiceManager: NUnit.Engine.NUnitEngineException: cC:\Tools\NUnit3\addins\Appveyor.NUnit3Logger.dll targets v4.0, which is not available.
   at NUnit.Engine.Services.ExtensionService.FindExtensionsInAssembly(ExtensionAssembly assembly)
   at NUnit.Engine.Services.ExtensionService.StartService()
   at NUnit.Engine.Services.ServiceManager.StartServices()

There's a fix for similar problem but it looks it works only for netstd2+ projects.

How to reproduce:

1. Use a machine with nunit3 extension. Extension should not support test target framework.
   As example: appveyor build machine; extension does not support net20 target fw.
2. Run a test over assembly that targets incompatible framework
3. Test fails with NUnitEngineException : Unable to acquire remote process agent

Example:

PS C:\projects\codejam\CodeJam.Main.Tests\bin\Release\net20> nunit3-console.exe .\CodeJam.Tests.dll --trace=Verbose

NUnit Console Runner 3.11.1 (.NET 2.0)
...
Unhandled Exception: NUnit.Engine.NUnitEngineException: Remote test agent exited with non-zero exit code -532459699
...
1) Error :
NUnit.Engine.NUnitEngineException : Unable to acquire remote process agent
--NUnitEngineException
Unable to acquire remote process agent
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunner() in 
...



md5-27420343a4fdd7bb80c508c2b28a1681



12:51:40.168 Info  [ 1] ExtensionService: Scanning C:\Tools\NUnit3\addins\Appveyor.NUnit3Logger.dll assembly for Extensions
12:51:40.231 Error [ 1] ServiceManager: Failed to initialize ExtensionService
12:51:40.231 Error [ 1] ServiceManager: NUnit.Engine.NUnitEngineException: Extension C:\Tools\NUnit3\addins\Appveyor.NUnit3Logger.dll targets v4.0, which is not available.
   at NUnit.Engine.Services.ExtensionService.FindExtensionsInAssembly(ExtensionAssembly assembly)
   at NUnit.Engine.Services.ExtensionService.StartService()
   at NUnit.Engine.Services.ServiceManager.StartServices()

Expected behavior:
There should be a way to ignore uncompatible extensions on test run.

[ChrisMaddock]

Ack - this took me a bit of thinking, but I think I understand what's going on. I assume CodeJam.Tests.dll tagets either .NET 2.0 or .NET 3.5?

The console/engine itself can load the extension fine, but then it starts up a .NET 2.0 agent to run the tests on. The agent then _also_ tries to load the extension and that's the point it fails. Hmpf!

Is Appveyor.NUnit3Logger.dll something you've opted in to on Appveyor, or is it there by default? I'm trying to work out why you're the first person to encounter this, as it isn't new behaviour on NUnit's side, and I'm surprised if you're the first person to try running .NET 2.0/3.5 tests on Appveyor's default test runner! Maybe those older frameworks are less common that we think. 🙂

Two suggested workarounds:

1. If you would be happy to temporarily run your tests on the v4 CLR rather than v2, pass the --framework=net4.0 flag to force that. That should allow everything to run happily on the v4 CLR.

2. Alternatively, delete the file C:\Tools\NUnit3\appveyor.addins on each run.

By the way, nice bug report! If you'd be interested in contributing a fix for this, happy to point you in the right direction?

cc @CharliePoole - this is another great reason to work out how we stop the agents needing to load extensions!

[ig-sinicyn]

I assume CodeJam.Tests.dll tagets either .NET 2.0 or .NET 3.5? ...
The agent then also tries to load the extension and that's the point it fails. Hmpf!

Yes, this is very correct explanation of what's going on.

Is Appveyor.NUnit3Logger.dll something you've opted in to on Appveyor, or is it there by default?

It's a built-in thing that is preinstalled on appveyor build image.

to temporarily run your tests on the v4 CLR rather than v2

We've used this approach and we are fine with it. TBH, the primary issue is on appveyor side and I'm not sure if it worth fixing. All in all, these old frameworks are really dead:)

nice bug report

Took a while to discover the trace log for the agent, after that diagnostic was very easy.

[ChrisMaddock]

I'm not sure if it worth fixing. All in all, these old frameworks are really dead:)

Aye - but the same bug will bite us when we add newer platforms I think, which is all in the pipeline. 🙂

For anyone who might be interested in fixing this, some thoughts...

• I suggest as a sticking plaster, we restrict the agents to just trying to load the type of extensions they care about. I _think_ that's just DriverExtensions, but someone should check that!
• In the long run, then engine will need to check it has a valid agent/driver combination, before launching the agent. That is if the driver is only available on .NET 4.0, and the agent should be .NET 2.0, then we can check for that, and exit nicely.
• In the longer longer run, it would be nice if the main engine did all the extension discovery and loading, and just passed the agent a driver that it new worked. (Mechanics behind that 'passing' tbc!)

[ig-sinicyn]

As an idea (I'm not sure, if it is a good one:) ).

The console runner has --skipnontestassemblies arg that allows to run tests even if there are non-test assemblies in input files list.

So, current issue may be solved with --skipbadextensions (todo: better name here) switch.

[ChrisMaddock]

My opinion: I don't think there's a need for a flag here, more that it just shouldn't crash! 😄

We need to work out exactly what that means when "you" try to load an extension for a platform it isn't valid on. I have "you" in inverted commas, as in this case, it's Appveyor. Who are running the console on .NET 4.x, and (rightly) believe their .NET 4.x extension should run fine!

[CharliePoole]

@ChrisMaddock You're right that the agents only need to check for driver extensions. Problem is that you have to inspect every potential extension assembly to see what it contains.

[ChrisMaddock]

@CharliePoole Am I right that we can inspect assemblies from a lower platform, we just can't load them?

Another sticking-plaster-level fix, wondering about passing a filter to the ExtensionService in the agents, to only attempt to load extensions with the DriverFactoryPath

[CharliePoole]

Yes, by using either Cecil or the metadata assembly I created to replace it. In a few places, we may have leftover stuff that uses reflection. If so we should root that out.

Re filtering: right now, the only filter we have is what assemblies we check. I believe that's fixed in the nunit engine but modifiable in testcentric. I could be wrong. But in any case, we check the entire API assembly and that's where most extension points are defined. We __could__ split the API assembly, but I'm not a big fan of that. I'll think about ways of filtering.

[CharliePoole]

I created #760 to cover "filtering" and other ways of limiting what the agent does with extensions.