Nunit-console: "ERROR" written to output makes runner exit with exit code -1

[lbergnehr]

I cannot reproduce this with a simple solution, but I have a big one with a lot of tests running which I'm currently converting from NUnit 2.6.4 to NUnit 3 using the NUnit.Console.3.6.1 NuGet package as the runner.

All tests pass (some are ignored) but the runner still exits with -1, which in turn fails the build script. It prints a message which comes from our code base which contains _ERROR:_. If I change the message written to stdout to e.g. _ERRR_, then the runner exists with 0 exit code when running the tests.

This is an excerpt from the build log related to NUnit. Note that the overall result is Warning:

Run Settings
    BasePath: C:\teamcity\work\271dbcd976b167dd\Common\Tests\UnitTests\bin\Debug\
    DisposeRunners: True
    DefaultTimeout: 60000
    WorkDirectory: C:\teamcity\work\271dbcd976b167dd
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.5.2
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 4

Test Run Summary
  Overall result: Warning
  Test Count: 7273, Passed: 6848, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 425
    Skipped Tests - Ignored: 4, Explicit: 421, Other: 0
  Start time: 2017-06-14 08:33:12Z
    End time: 2017-06-14 08:37:18Z
    Duration: 245.853 seconds

Results (nunit3) saved as TestResult.xml
The command "packages\NUnit.ConsoleRunner.3.6.1\tools\nunit3-console.exe --teamcity --timeout=60000 <A couple of .csproj files as arguments>" exited with code -1.

[jnm2]

Yeah, that's weird! 😄
I can't find any code in this repo or https://github.com/nunit/teamcity-event-listener which searches for the substring "ERROR". Because you said you couldn't reproduce with a simple solution, I'm suspicious that maybe there's a hidden factor at play in your full solution. Instead of starting by building the repro from the ground up, how about tearing down the full solution until you identify the problem well enough to create the repro from the ground up? I'm not sure if we can do anything without that.

[lbergnehr]

I'm very sorry, but this was simply not an issue at all with NUnit. The MSBuild Exec task has the default behavior of scanning the output and looking for patterns matching a specified error pattern. As I mentioned, our code produces ERROR: lines, which likely is what MSBuild picks up.

The weird thing to me is that MSBuild reports it as the executed process exiting with exit code -1, which is simply not true. I debugged the whole thing with the NUnit source code and it exits with 0. MSBuild is a filthy liar. The scanning behavior can be disabled with the MSBuild IgnoreStandardErrorWarningFormat property set to true, which is what I'm doing now.

Thanks for the help in pointing me in the right direction and sorry again for the bad issue.

[jnm2]

No problem at all. Thanks for looping back!