Nunit-console: TypeLoadException thrown when changes are made to the API assembly, with multiple versions of the engine available

[jnm2]

I'm getting a persistent failure to run tests in ReSharper's test runner (ReSharper 2020.1.4 on VS 16.6.5) with this error in the log:

ERROR System.TypeLoadException: Method 'get_AssemblyPath' in type 'NUnit.Engine.Extensibility.ExtensionNode' from assembly 'nunit.engine, Version=3.10.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' does not have an implementation.
System.TypeLoadException: Method 'get_AssemblyPath' in type 'NUnit.Engine.Extensibility.ExtensionNode' from assembly 'nunit.engine, Version=3.10.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' does not have an implementation.
at NUnit.Engine.Services.ExtensionService..ctor()
at NUnit.Engine.TestEngine.Initialize()
at NUnit.Engine.TestEngine.GetRunner(TestPackage package)
at JetBrains.ReSharper.UnitTestRunner.nUnit30.NUnitRunner.Run(String assembly, IDictionary`2 testParameters) in /Product.Root/Psi.Features/UnitTesting/nUnit/Runner30/Src/NUnitRunner.cs:line 41
at JetBrains.ReSharper.UnitTestRunner.nUnit30.BuiltInNUnitRunner.WithExtensiveErrorHandling(IRemoteTaskServer server, Action action) in /Product.Root/Psi.Features/UnitTesting/nUnit/Runner30/Src/BuiltInNUnitRunner.cs:line 73

This is what the test project looks like:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net48</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.6.1" />
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.17.0" />
  </ItemGroup>

</Project>

This is all it takes to repro the problem:

using NUnit.Framework;

public class Repro
{
    [Test]
    public void Test() { }
}

Test Explorer was also acting up by skipping all tests. After git clean -xdf on the whole project, Test Explorer started working and ReSharper did not.

Let me know if there's anything else I can help with.

[ChrisMaddock]

I...have absolutely no idea! 😅

What a weird stack trace, I can't even work out how constructing the ExtensionService (empty ctor) would end up calling get_AssemblyPath on ExtensionNode. And even if it did, it's a bog-standard property...how can it not have an implementation?

I haven't used the resharper running in a good while. Is there any way to get logs out of it? You always used to be able to point the runner to a custom version of the console, can you still do that, and try it on the latest release?

[ChrisMaddock]

Ah, here's a thought.

v3.17.0 of the adapter is bringing in engine version 3.11.1. Resharper seems to be looking for 3.10.0. D'ya think the assemblies are getting mixed up somehow? Isn't ReSharper also built on top of the adapter these days?

Does removing the adapter (and cleaning etc) make ReSharper start working?

[ChrisMaddock]

Ok, I Googled it and found the below:

https://resharper-support.jetbrains.com/hc/en-us/community/posts/360009425440-NUnit-Tests-Not-Running

It looks like ReSharper have recently implemented their new test runner, which I'm guessing is based on the adapter for both netfx and .NET Core. (They describe it as "unifying the test runner", and I know previous the old netfx one worked through the console, and .NET Core via the adapter.)

What I'm guessing has happened, is that now it's based on the adapter, it needs to copy the relevant files over to the bin directory for the adapter to work. If, as you have here, you have a different version of the adapter installed in your project to the version resharper is relying on, both assemblies get copied into bin, and you get the clash.

Workarounds tricky, as you basically need to keep resharper's version and the version in all your projects in sync. Not sure if ReSharper provides any settings to set the adapter used, which might make that easier?

I'm going to close this issue, as I don't think there's any change we could make in the engine to fix this - this is down to JetBrains's implementation, I think. Let me know if you think I've missed something! 👍

[ChrisMaddock]

(Should also point out that this is apparently fixed in EAP v6. I haven't tested that.

[estrizhok]

Hi @ChrisMaddock,

I am from ReSharper team, working on the test runner. The root of the problem is that somewhere along the lines in NUnit nunit.core.api.dll was changed, but it's version remained the same. Namely, two new properties AssemblyPath and AssemblyVersion were added into NUnit.Engine.Extensibility.IExtensionNode, and NUnit3TestAdapter 16.1 comes with nunit.core.api.dll without additional properties whereas NUnit3TestAdapter 3.17 already has them in its nunit.core.api.dll. Both dlls have the same version 3.10.

Neither old nor new R# runner need NUnit3TestAdapter for it to work, but because standard nunit test project template has a package reference to NUnit3TestAdapter, dlls from that package are copied into /bin dir anyway. Say NUnit3TestAdapter 3.17 is referenced. Inside R# test runner, when binder resolves dependencies of our adapter for nunit, it tries to load them from GAC, then from AppBase (set to test's /bin dir), and after that using our custom resolver. First it tries to resolve nunit.core 3.10 (because this is the version R# adapter was compiled against), and not finding it neither in GAC nor AppBase, loads it using our custom resolver. Then 'nunit.core.api.dll' 3.10 (as referenced by nunit.core.dll) is being resolved, this time a dll with matching identity is found in AppBase directory, and is loaded from there. But this is not the same nunit.core.api.dll , and NUnit.Engine.Extensibility.ExtensionNode does not implement the new NUnit.Engine.Extensibility.IExtensionNode. (One of the properties is missing completely, the other one is not virtual which is necessary to qualify as implementing interface). Hence, weird exception.

The problem that is described in here is only affecting the test runner before R# 2020.2 (which is not released yet, EAPs are available). But only because a test adapter from the new runner was compiled against newer nunit.core.dll, referencing new nunit.core.api.dll. If you are to change either library without changing its version, it will likely break again.

[CharliePoole]

Interesting scenario! It seems to break our assumption that it's safe to add properties, methods, etc. to the interfaces without changing the version number of the api assembly.

Of course the break requires accidental collusion across two different "third-party" (I'm counting the adapter here) assemblies, but even so we ought to give it a think for the future.

[ChrisMaddock]

Whoops - please disregard my last (now deleted) post, I've finally figured things out. Yes - I can absolutely see how that would cause an issue!

Thanks for getting in touch @estrizhok - appreciate the pointers. 😄

@CharliePoole - do you agree we need to start incrementing the API assembly from the next version of the engine?

I have to say, I'm a little confused as to why this has never come up before, given we've been working this way for 5 or so years now. Anyone have any ideas? 🤔

[CharliePoole]

@ChrisMaddock I noticed that @estrizhok had the assembly names wrong, but I thought I knew which he meant. Now I'm not so sure. Maybe we should wait for him to clarify before taking any action.

I __think__ that what we are doing with the API assembly is safe, so long as people use packages to install. If you install 3.11 of the engine package, then you get the correct api assembly to go with it even though that assembly has version 3.0 (and file version 3.11). It's possible folks could mess up by downloading assemblies and mixing and matching the versions but I don't think we need to take responsibility for that.

Even with packages, however, if the user references the adapter, which brings in an engine version, and also uses an application like ReSharper that brings in a different engine version, something is bound to break.

[ChrisMaddock]

Is there a particular advantage to keeping the API assembly version at 3.0 however, which I’m missing?

Agreed the packages ‘isolate’ around this problem, but it does seem to leave us open to issues.

[CharliePoole]

It was part of the notion that there would be a central installation of the engine on a dev machine, which could be updated without changing any runners. As it stands, we distribute an engine with each runner anyway, so it may no longer make sense.

I guess it depends on what you want the future to look like. :smile:

[estrizhok]

@CharliePoole Yeah, you are right. I got the names and versions of assemblies wrong. I was writing from memory. Let me reiterate:

NUnit3TestAdapter 3.16.1:
nunit.engine, Version=3.10.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
NUnit.Engine.Extensibility.IExtensionNode has neither AssemblyPath nor AssemblyVersion

NUnit3TestAdapter 3.17.0:
nunit.engine, Version=3.11.1.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
nunit.engine.core, Version=3.11.1.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
NUnit.Engine.Extensibility.IExtensionNode has both AssemblyPath and AssemblyVersion

Old R# Adapter:
nunit.engine, Version=3.10.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
NUnit.Engine.Extensibility.IExtensionNode has neither AssemblyPath nor AssemblyVersion

The logic described in resolve still applies.

[ChrisMaddock]

It was part of the notion that there would be a central installation of the engine on a dev machine, which could be updated without changing any runners. As it stands, we distribute an engine with each runner anyway, so it may no longer make sense.

I'm not too sure the "centrally installed engine" concept ever really took off. My personal feeling is that the engines sufficiently lightweight that deploying it as an assembly with each runner isn't an issue. Having runners interact with multiple unknown versions of the engine would also make our overall architecture much more complicated, and I'm struggling to see sufficient benefit to justify it here.

I think we should go ahead with the versioning change, if nobody is strongly opposed. I've broken that out into https://github.com/nunit/nunit-console/issues/801 - I'd like to get this in the next release. 🙂