Nunit-console: Loading a multi-targeting csproj using nunit3-console gives NRE exception

[FransBouma]

I have the following project file (one of 9 in this .sln, they all are multitargeting)

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp2.0;net462</TargetFrameworks>
    <AssemblyName>AsyncLinqTests</AssemblyName>
    <RootNamespace>LinqTestsAsync</RootNamespace>
    <DebugType Condition=" '$(TargetFramework)' == 'net462'">Full</DebugType>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
    <PackageReference Include="NUnit" Version="3.11.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.11.0" />
    <PackageReference Include="System.Data.SqlClient" Version="4.5.1" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'net462'">
    <Reference Include="$(SolutionDir)RuntimeLibraries\SD.LLBLGen.Pro.DQE.SqlServer.dll" />
    <Reference Include="$(SolutionDir)RuntimeLibraries\SD.LLBLGen.Pro.ORMSupportClasses.dll" />
    <PackageReference Include="SD.Tools.OrmProfiler.Interceptor.NET45" Version="1.5.20150715" />
    <Reference Include="System.Transactions" />
    <Reference Include="System.Windows.Forms" />
    <Reference Include="$(SolutionDir)ExpressionTreeVisualizer\ExpressionTreeVisualizer\bin\Debug\ExpressionTreeVisualizer.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\InheritanceTwo.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\InheritanceTwo.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\InheritanceTwo.SelfServicing.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\NW26Async.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\NW26Async.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Net452\NW26Async.SelfServicing.dll" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp2.0'">
    <Reference Include="$(SolutionDir)RuntimeLibraries\NetStandard2.0\SD.LLBLGen.Pro.DQE.SqlServer.dll" />
    <Reference Include="$(SolutionDir)RuntimeLibraries\NetStandard2.0\SD.LLBLGen.Pro.ORMSupportClasses.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\InheritanceTwo.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\InheritanceTwo.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\InheritanceTwo.SelfServicing.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\NW26Async.Adapter.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\NW26Async.AdapterDBSpecific.dll" />
    <Reference Include="$(SolutionDir)ReferenceSupportDLLs\Netstandard2.0\NW26Async.SelfServicing.dll" />
  </ItemGroup>
</Project>

Using Rider (latest) or VS 2017 and this .sln runs the tests fine (with the nunit adapter).

However using: nunit3-console AllUnitTests.sln

gives:

NUnit Console Runner 3.9.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    AllUnitTests.sln

System.ArgumentException : Invalid project file format: AsyncLinqTests.csproj
  ----> System.NullReferenceException : Object reference not set to an instance of an object.

--ArgumentException
Invalid project file format: AsyncLinqTests.csproj
   at NUnit.Engine.Services.ProjectLoaders.VSProject.ThrowInvalidFormat(String projectPath, Exception e)
   at NUnit.Engine.Services.ProjectLoaders.VSProject.Load()
   at NUnit.Engine.Services.ProjectLoaders.VSSolution.Load()
   at NUnit.Engine.Services.ProjectLoaders.VisualStudioProjectLoader.LoadFrom(String path)
   at NUnit.Engine.Services.ProjectService.ExpandProjectPackage(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.ExpandProjects(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.EnsurePackagesAreExpanded(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.EnsurePackagesAreExpanded(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.InitializePackage()
   at NUnit.Engine.TestEngine.GetRunner(TestPackage package)
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)
--
NullReferenceException
Object reference not set to an instance of an object.
   at NUnit.Engine.Services.ProjectLoaders.VSProject.TryLoadDotNetCoreProject()
   at NUnit.Engine.Services.ProjectLoaders.VSProject.Load()

Nunit3-console is the latest (downloaded it 2 hours ago), used the .msi installer.

Loading the csproj instead of the sln gives the same error. Running nunit3-console with the assembly works, but is less ideal as there are multiple targets and that would be more work to set up (I'm looking at running the tests on the command line as running the tests inside an IDE is slow: 3800 tests over 2 platforms take over 8 minutes combined)

[CharliePoole]

This makes sense to me. The console gets a package containing a list of assemblies from any project loader extensions that are installed. In this case, for either the solution or the csproj file, it would be the VS Project Loader extension. The code for the solution loader simply loads every project, so the outcome is the same in either case.

I don't think the project loader has ever been updated to deal with this format. In particular, I believe it looks for a Configuration property as part of the Conditions on each item group. I could be wrong, however, since it's a long time since I worked on it and changes have probably been made.

[FransBouma]

Hmm, if the project loader isn't up to it, then it indeed won't get very far :) I was under the impression it did work, as nunit's tests themselves are in a csproj with multi targeting ;) https://github.com/nunit/nunit/blob/master/src/NUnitFramework/tests/nunit.framework.tests.csproj#L4 Isn't the nunit adapter for vs using the vs project loader? (No idea haven't checked).

[ChrisMaddock]

The VS project loader doesn't currently support mu,ti-targeted csproj's, no - there's an open issue for that here. https://github.com/nunit/vs-project-loader/issues/26

Even if it did, there's not yet a build of the NUnit console which can run both net core and netfx tests. This is on the roadmap, but hasn't happened yet.

Your best bet with the current tooling would be to do dotnet test - it should work to do dotnet test AllUnitTests.sln, I think.

I'm going to close this issue, as I think all the problems here are covered in other open issues. Shout up if you think it should be reopened!

[FransBouma]

Thanks for the info, indeed that makes it impossible. I tried the nunit console runner as dotnet test starts 5 runners at once and I can't seem to find info how to limit it so it runs more tests in parallel than that it should, causing tests to fail (as they're integration tests hammering a database they need to be run 1 assembly at a time).

[CharliePoole]

You might consider creating an NUnit project that tracks the test assemblies in your solution. A few hints:

1. A config in an NUnit project is an arbitrary string that calls out a specific set of test assemblies. The obvious way to use it is to define configs that match the solution, but a more creative approach would be possible in your case.

2. Relative assembly paths in the project file are based on the location of the .nunit file itself. Especially important to know if you are creating the XML by hand.

3. NUnit will attempt do whatever you tell it to, whether it makes sense or not, so don't try to load incompatibly targeted assemblies into the same AppDomain or Process.

4. You are still subject to the limitations of the console runner in terms of which targets are supported.

[FransBouma]

Thanks for the pointers, will take them into account while solving this problem! :)

[ChrisMaddock]

. I tried the nunit console runner as dotnet test starts 5 runners at once and I can't seem to find info how to limit it so it runs more tests in parallel than that it should

It should be possible to control this via a runsettings file. Can’t remember the options off the top of my head, I’m aftaid. 😊

[FransBouma]

Heh :smile: I thought about that, but there's little I can find about configuring it through a runsettings file. I'll do some more digging. Thanks :)

[CharliePoole]

Because VS only calls the NUnit adapter for one assembly at a time, none of the NUnit-specific runsettings will be of any help. You're looking for a VS setting for running multiple test assemblies in parallel. That appears to be MaxCpuCount, which I have never used personally.

[FransBouma]

No it's not from within vs, it's on the command line, using dotnet test solutionfile. That command starts 5 test runners by default and makes the test fail left and right. I can't seem to limit it to use just 1 test runner. Anyway, that's outside the scope of your project, so don't worry about it :)

[CharliePoole]

Ah sorry... I'm thinking of the IDE. However, the principle is the same... NUnit is called 5 different times. I'm surprised there is nothing to limit concurrency in the dotnet test command.

[FransBouma]

Surprised me too, to be honest. I'll see if I can find info about it, and post back here for completeness.

[FransBouma]

This turned out to be super simple :) dotnet test is actually using msbuild to run the tests. So it's configurable using msbuild parameters:

dotnet test AllUnitTests.sln -v q -m:1

limits it to 1 testrunner at a time, using the -m:n flag.

Hope this helps other people with the same problem :) An additional bonus here is that it runs the tests on all target platforms in the csprojs so I have 2 platforms per csproj and it runs all tests, using nunit on all targeted platforms, on the command line. Only downside is the configuration of e.g. the output. The nunit console runner has more options to configure things.