Nunit-console: Test run exits with an exit code of 0 if a multiple of 256 tests fail

[rolfbjarne]

Test case:

using NUnit.Framework;

class X {
    [Test]
    public void D ([Range (1, 256)] int value)
    {
        Assert.Fail ("Ops: {0}", value);
    }
}

Project file:

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net461</TargetFramework>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <DisableTransitiveProjectReferences>true</DisableTransitiveProjectReferences>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit.ConsoleRunner" Version="3.11.1" />
  </ItemGroup>
</Project>

Build:

$ dotnet build /nologo /v:quiet

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.57

Execute and check the exit code:

$ mono ~/.nuget/packages/nunit.consolerunner/3.11.1/tools/nunit3-console.exe bin/Debug/net461/test.dll         
NUnit Console Runner 3.11.1 (.NET 2.0)
Copyright (c) 2020 Charlie Poole, Rob Prouse
Thursday, May 7, 2020 8:26:59 AM

Runtime Environment
   OS Version: MacOSX 19.4.0.0 
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    bin/Debug/net461/test.dll


Errors, Failures and Warnings

1) Failed : X.D(1)
Ops: 1
  at X.D (System.Int32 value) [0x00000] in <ffca60e77a7b4747bd44aa6cb2a6233b>:0 

[...]

256) Failed : X.D(256)
Ops: 256
  at X.D (System.Int32 value) [0x00000] in <ffca60e77a7b4747bd44aa6cb2a6233b>:0 

Run Settings
    DisposeRunners: True
    WorkDirectory: /Users/rolf/test/nunitbug
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    RuntimeFramework: mono-4.0
    NumberOfTestWorkers: 16

Test Run Summary
  Overall result: Failed
  Test Count: 256, Passed: 0, Failed: 256, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 256, Errors: 0, Invalid: 0
  Start time: 2020-05-07 06:27:00Z
    End time: 2020-05-07 06:27:01Z
    Duration: 0.923 seconds

Results (nunit3) saved as TestResult.xml

$ echo $?
0

The problem is that:

• The exit code is capped to 255 on unix systems
• NUnit returns the number of tests that failed
• If a multiple of 256 tests fail, the exit code will be truncated to the lower 8 bits, which will be 0.

[CharliePoole]

Amazing that nobody saw this before! IIRC we put in that return code around 2004.

[ChrisMaddock]

Hahahaha! Thanks @rolfbjarne, nice bug! 😅

Let’s just cap it at 255. Any chance you’re interested in contributing a fix? If not, I’ll try and write up some pointers for a first-timer to contribute this at the weekend.

[jnm2]

Love it!

[rolfbjarne]

Any chance you’re interested in contributing a fix?

No, sorry, but I'm (fortunately!) quite busy at the moment :smile:

[jnm2]

@ChrisMaddock Why is it that we can't use exit codes 256, 257, etc?

[ChrisMaddock]

I’m just basing this on Rolf’s explanation above. 😊 I don’t use unix enough myself to know for sure.

The problem is that:

• The exit code is capped to 255 on unix systems

What I didn’t think, when I first commented, was that the negative exit codes we use on windows to indicate ‘error’ will also roll around unsigned on unix. We should probably cap at 100 or so instead, so our existing -100 code is still distinguishable from test failure on unix.

Once you have over 100 tests failing, what’s an extra hundred, right? 😛

[jnm2]

Ah, I see. Would we want to cap on Windows too?

[CharliePoole]

I think so... in order to keep the explanation simple. We could go 127, but 100 seems like a round number. I'll push a PR.

[CharliePoole]

@rolfbjarne I'm surprised it's not a multiple of 128, actually. Aren't the 8 bits taken as signed?

[rolfbjarne]

Why is it that we can't use exit codes 256, 257, etc?

http://man7.org/linux/man-pages/man3/exit.3.html: "The exit() function causes normal process termination and the least significant byte of status (i.e., status & 0xFF) is returned to the parent"

I'm surprised it's not a multiple of 128, actually. Aren't the 8 bits taken as signed?

No, they're unsigned. Returning -1 from the managed Main method makes the process return 255 as the exit code (and -2 -> 254, etc).

[CharliePoole]

I suppose folks running nunit console runner through the years just knew that. Somewhere we document special meanings for -1, -2, etc. Anyway, 100 now means 100 or more failures.

[jnm2]

We return -100 for one error which makes 155 the absolute maximum. 100 is better as a maximum.