Nunit-console: 'Failed to start remote test agent' when using nunit.engine.dll in custom runner

[kyle-v-51]

• Method of installation: NuGet
• Version of the NUnit Engine: 3.6.1

I'm writing a custom runner app (.NET Framework v4.6.2) and have added the nunit.engine and nunit.engine.api NuGe packages to the project. My project only references nunit.engine.api. I'm following this doc here: https://docs.nunit.org/articles/nunit/technical-notes/nunit-internals/Test-Engine-API.html

Here's my Program.cs:

internal class Program
{
    public static void Main(string[] args)
    {
        // Get an interface to the engine
        ITestEngine engine = TestEngineActivator.CreateInstance();

        // Create a simple test package - one assembly, no special settings
        TestPackage package = new TestPackage("tests.dll");

        // Get a runner for the test package
        ITestRunner runner = engine.GetRunner(package);

        var testEventListener = new MyTestEventListener();

        // Run all the tests in the assembly
        XmlNode testResult = runner.Run(testEventListener, TestFilter.Empty);
    }
}

When I execute the program I get this:

Unhandled Exception: NUnit.Engine.NUnitEngineException: Failed to start remote test agent.
   at NUnit.Engine.Services.TestAgency.OnAgentExit(Process process, Guid agentId)
   at NUnit.Engine.Services.TestAgency.<>c__DisplayClass7_0.<LaunchAgentProcess>b__0(Object sender, EventArgs e)
   at System.Diagnostics.Process.OnExited()
   at System.Diagnostics.Process.RaiseOnExited()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)

Then a second later I get:

Unhandled Exception: System.TypeLoadException: Could not load type 'NUnit.Engine.NUnitEngineUnloadException' from assembly 'nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb'.
   at NUnit.Common.ExceptionHelper.FlattenExceptionHierarchy(Exception exception)
   at NUnit.Common.ExceptionHelper.BuildMessageAndStackTrace(Exception exception)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
   at Zzz.Program.Main(String[] args) in C:\Zzz\Program.cs:line xx

I tried running my tests.dll with nunit3-console and it seems to work fine so I don't think the tests assembly is at fault.

[CharliePoole]

Please indicate the versions of the NUnit engine and api you are using as well as the runtme target of your custom runner and test assemblies.

What you are doing generally sounds good except for separately including the API package. Since the engine package includes the API assembly, that's a bit odd, although I would be surprised if that's what is causing the problem.

[kyle-v-51]

Tried removing the separate reference to the API package but the issue persists.

nunit.engine.api.dll = 3.1.6.0 according to the dll file properties but in Rider the assembly reference says nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb (???)
nunit.engine.dll = 3.6.1.0 (version matches file properties on this one)
Runtime target of custom runner project = .NET Framework v4.6.2
Runtime target of test assembly = .NET Framework v4.6.2

[ChrisMaddock]

I think this is a stacktrace we've seen before in #424. Does your project also reference the NUnit3TestAdapter, by any chance?

Try updating the engine to the latest version, and the adapter if it's present. Also, Charlies right - just reference the nunit.engine package, not both the engine and separate API. That could potentially introduce a version conflict between two different API versions.

[kyle-v-51]

No projects in the whole solution reference NUnit3TestAdapter. I'll try updating the engine.

[ChrisMaddock]

nunit.engine.api.dll = 3.1.6.0 according to the dll file properties but in Rider the assembly reference says nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb (???)

That assembly had different assembly versions and product version up until the most recent beta release, which is the reason for the version difference you see. (Those versions are the expected ones, too!)

[kyle-v-51]

Looks like updating the engine/API NuGet package to latest fixed the issue.

As an aside, I see my report comes back as XML -- is there a way to get all the same output I see when running a test via the nunit console too?

[ChrisMaddock]

Short answer - no. 😅 It's the console itself that creates that text, based on the standard XML report/fragments which is passed back to all runners.

Any particular piece of info you were after?

[kyle-v-51]

The reason i'm making a custom runner is to query a Web API to decide what tests to run and it seemed like a good opportunity to also get some hooks into the runner. I think it actually won't work out though if I don't get that standard console reporting.

[CharliePoole]

@kyle-v-51 The console output is pretty straightforward based on the XML result as well as various XML-formatted events that are fired while the test runs. Take a look at the code in the console project or in the NUnitLite assembly (different code that does basically the same thing)