Nunit: NUnit Attribute referencing different version of NUnit causes tests to not run

[zastrowm]

Creating a custom attribute that derives from TestActionAttribute, then using that attribute from a test library that uses a newer version of NUnit will result in NUnit not executing the test because it could not load the older NUnit library. Using an app.config to redirect to the newer NUnit version fixes the issue and tests then run as expected.

The issue is non-obvious because normal tests appear to not run (status: inconclusive). This was the cause of NUnit.ApplicationDomain Issue #25: Unable to run with VS2017 test runner and I discovered the root of the problem only because one of the parameterized test fixtures gave the following stack trace:

An exception was thrown while loading the test.
System.IO.FileLoadException: Could not load file or assembly 'nunit.framework, Version=3.7.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'nunit.framework, Version=3.7.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb'
   at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeType type, RuntimeType caType, Boolean inherit)
   at NUnit.Framework.Internal.TypeWrapper.GetCustomAttributes[T](Boolean inherit)
   at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.GetFixtureBuilderAttributes(ITypeInfo typeInfo)
   at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

  Exception doesn't have a stacktrace

The underlying issue - referencing an older version of NUnit for attributes - might be closed as "will-not-fix", but I think it might be useful for users to have some sort of error message rather than having the tests seemingly refuse to execute.

I've created a reproduction solution over @ _repo_NUnitAttributesTargettingDifferentVersion

[CharliePoole]

I agree with the conclusion: the fact that custom attributes are locked into the version of NUnit that they extend is not new, but also not easily discoverable.We should review the docs to make sure this is clear and also try to figure out a way to detect conflicting framework versions and give an error message.

Also, to the extent it's possible, we should document how to write custom attributes that will be backward-compatible. I believe that relying solely on our unchanging interfaces and using a binding redirect in the config file will do the trick, but somebody should try that out first.

[rprouse]

I am marking this as both docs, so we can review them and enhancement, to investigate detecting conflicting versions and possibly recommending a binding redirect.

[stevenaw]

I'm less well versed on the Roslyn side, but is this something an analyzer could detect? Or would this have to be a runtime check (improved error messaging)