Nunit: Proper test result exception message when hitting `TimeoutAttribute`

[Prodigio]

Hey guys,

lately I was expetimenting with the TimeoutAttribute and ran into a few timeouts for my tests. I only saw a Exception doesn't have a stacktrace as a test result message, which confused me at first sight. Then I realized, the test were hitting the timeout.

I tried to dig into the code to see, if implementing a timeout exception message is possible, but havn't found anything so far.

So I'd like to suggest a better test result message when hitting the timeout.

Code to reproduce:

[Test]
[Timeout(1000)]
public async Task Timeout()
{
    // Arrange

    // Act
    await Task.Delay(2000).ConfigureAwait(false);

    // Assert
    Assert.Pass();
}

Thanks and kind regards

[mikkelbu]

Hi @Prodigio

How are you running the tests (VS, dotnet test, or the console runner), which framework are you targeting, and which versions of NUnit, NUnit3TestAdapter, and console runner are you using?

Some initial skimming of the code could indicate that for dotnet core we don't set the message, but only set the label. Compare (framework flow)
https://github.com/nunit/nunit/blob/b34eba3ac1aa6957157857bddd116256c634afab/src/NUnitFramework/framework/Internal/Commands/TimeoutCommand.cs#L83
against (dotnet core flow)
https://github.com/nunit/nunit/blob/b34eba3ac1aa6957157857bddd116256c634afab/src/NUnitFramework/framework/Internal/Commands/TimeoutCommand.cs#L110-L113

Also compare the difference in the tests TimeoutCausesOtherwisePassingTestToFailWithoutDebuggerAttached(). Dotnet framework version
https://github.com/nunit/nunit/blob/b34eba3ac1aa6957157857bddd116256c634afab/src/NUnitFramework/tests/Attributes/TimeoutTests.cs#L297-L299
dotnet core version
https://github.com/nunit/nunit/blob/b34eba3ac1aa6957157857bddd116256c634afab/src/NUnitFramework/tests/Attributes/TimeoutTests.cs#L367-L369

I'm unsure if this difference is on purpose, but it looks like an mistake. Since we don't write anything in the message for dotnet core, the resulting XML does not contain any information and thus it cannot be presented for the user. On dotnet framework the test will have the following error message Test exceeded Timeout value of 1000ms

[Prodigio]

Hey @mikkelbu,

sorry for the missing information. Here it is:

• Locally I'm executing our tests with Resharper within VS 2019
• Our CI server is TeamCity, where we execute the "bare" tests on the one hand with dotnet test and on the other with the help of NUKE and dotnet test, for some coverage reports
• Targeting .NET Core 3.1 in all test projects

I do see the same message Exception doesn't have a stacktrace locally and on TeamCity.

I'm not using the console. Other NUnit versions are:

• NUnit v3.13.2
• NUnit.Engine v3.12.0
• NUnit3TestAdapter v4.0.0-beta.2

I noticed, that besides NUnit we also have Microsoft.NET.Test.Sdk v.16.9.4 installed as NuGet package. Could this be an issue?

Thanks!

[mikkelbu]

Hi @Prodigio

Thanks for all the information. I'm quite sure that the missing information is since you are targeting .NET Core 3.1 and we have made an mistake in nunit/nunit#3190.

That being said I don't think that we record stacktrace when the test fail due to a timeout, as we don't know how far in the code we are (so where exactly should the stacktrace point to). The two calls to context.CurrentResult.SetResult above have a third argument which is the stacktrace, but we don't fill it in.

But I think we should fix the .NET Core flow, so we also report Test exceeded Timeout value ... as a message.