Nunit-console: NUnit Console Reports Successful Exit Code When there is an Exception on Dispose

[AnthonySimmons]

Summary
The NUnit Console app returns an exit code of zero, when a test fixture throws an exception on dispose.

Steps to Produce

1. Create a test fixture that throws an exception on dispose.
2. Run the tests using the NUnit Console Runner

Example:
This error can be demonstrated using this example.
To produce this error:

1. Clone the repo.
2. Open the solution
3. Compile in Release mode.
4. Execute the RunTests.bat script.

Result
The test passes, the exception is displayed in the console output, the overall result is "Failed", but the console runner returns an exit code of zero, indicating a success.

Example:

C:\workspace\NUnitTest\NUnitTearDownBug>packages\NUnit.ConsoleRunner.3.8.0\tools\nunit3-console.exe Tests\NUnitDisposeError.Test\bin\Release\NUnitDisposeError.Test.dll
NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.17134.0
  CLR Version: 4.0.30319.42000

Test Files
    Tests\NUnitDisposeError.Test\bin\Release\NUnitDisposeError.Test.dll


Errors, Failures and Warnings

1) TearDown Error : NUnitDisposeError.Test.NUnitDisposeErrorTest
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
--TearDown
   at NUnitDisposeError.Test.NUnitDisposeErrorTest.Dispose() in C:\workspace\NUnitTest\NUnitTearDownBug\Tests\NUnitDisposeError.Test\NUnitDisposeErrorTest.cs:line 26
   at NUnit.Framework.Internal.Commands.DisposeFixtureCommand.<>c.<.ctor>b__0_0(TestExecutionContext context) in C:\src\nunit\nunit\src\NUnitFramework\framework\Internal\Co
mmands\DisposeFixtureCommand.cs:line 52

Run Settings
    DisposeRunners: True
    WorkDirectory: C:\workspace\NUnitTest\NUnitTearDownBug
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 4

Test Run Summary
  Overall result: Failed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2018-09-02 21:38:19Z
    End time: 2018-09-02 21:38:21Z
    Duration: 1.648 seconds

Results (nunit3) saved as TestResult.xml

C:\workspace\NUnitTest\NUnitTearDownBug>echo %ERRORLEVEL%
0

[mikkelbu]

Sounds quite similar to #282 (if not a duplicate).

[CharliePoole]

Although this behavior is according to spec, it doesn't make sense!

The console returns a negative value if there is an unexpected error of some kind - such as a file not found. If no tests fail it returns zero otherwise it returns a positive value indicating the number of tests (i.e. test cases) that failed.

I think we should return UNEXPECTED_ERROR for this situation, or invent a new negative code covering any kind of fixture closure or teardown error.