Omnisharp-vscode: No output or errors from nunit tests - 1.10.0-beta2

Created on 14 May 2017 · 17Comments · Source: OmniSharp/omnisharp-vscode

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.4)

Product Information:
 Version:            1.0.4
 Commit SHA-1 hash:  af1e6684fd

Runtime Environment:
 OS Name:     debian
 OS Version:  8
 OS Platform: Linux
 RID:         debian.8-x64
 Base Path:   /usr/share/dotnet/sdk/1.0.4

VS Code version:
C# Extension version: 1.12.1

omnisharp-vscode: 1.10.0-beta2

Steps to reproduce

Here is a repo which exactly reproduces this https://github.com/tomzo/bug-nunit-no-output-vscode

Just run

dotnet restore
dotnet test DefaultTests/DefaultTests.csproj

Then try the same in vscode by clicking on run test on one of the tests.

Expected behavior

There are tests which are failingand there is standard output printed in one of the tests.

[Test]
        public void Test1()
        {
            new Class1 ();
            Console.WriteLine("Test body running..");
        }

        [Test]
        public void Test2()
        {
            Assert.AreEqual(2,26);
        }

I would expect to see both the stdout and especially failed asserts from nunit test cases.

Actual behavior

In VsCode .NET Test Log output just counts the failed tests

NUnit Adapter 3.8.0.0: Test discovery starting
NUnit Adapter 3.8.0.0: Test discovery complete
NUnit Adapter 3.8.0.0: Test execution started
Running selected tests in /ide/work/nunit-repro/DefaultTests/bin/Debug/netcoreapp1.0/DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete

Total tests: 1. Passed: 0. Failed: 1. Skipped: 0

The output when running with dotnet is fine:

$ dotnet test  DefaultTests/DefaultTests.csproj 
Build started, please wait...
/usr/share/dotnet/sdk/1.0.4/Sdks/Microsoft.NET.Sdk/Sdk/Sdk.targets(40,3): warning MSB4011: "/usr/share/dotnet/sdk/1.0.4/Microsoft.CSharp.targets" cannot be imported again. It was already imported at "/ide/work/nunit-repro/ExampleLib/ExampleLib.csproj (28,3)". This is most likely a build authoring error. This subsequent import will be ignored. 
Build completed.

Test run for /ide/work/nunit-repro/DefaultTests/bin/Debug/netcoreapp1.0/DefaultTests.dll(.NETCoreApp,Version=v1.0)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
NUnit Adapter 3.8.0.0: Test execution started
Running all tests in /ide/work/nunit-repro/DefaultTests/bin/Debug/netcoreapp1.0/DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete
Failed   Test2
Error Message:
   Expected: 2
  But was:  26

Stack Trace:
at DefaultTests.UnitTest1.Test2() in /ide/work/nunit-repro/DefaultTests/UnitTest1.cs:line 20


Total tests: 2. Passed: 1. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 1.8351 Seconds

Source

tomzo

All 17 comments

Hello @tomzo!
Are you using the alpha version of NUnit?

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0"/>
    <PackageReference Include="NUnit" Version="3.6.1"/>
    <PackageReference Include="NUnit3TestAdapter" Version="3.8.0-alpha1"/>
  </ItemGroup>

bernardbr on 15 May 2017

Hi @bernardbr ,

Exactly, as you can see here

tomzo on 15 May 2017

Hello @tomzo!
I cloned your example and changed only .csproj and after this worked fine. Could you check if it resolves your problem?

bug-nunit-no-output-vscode.zip

bernardbr on 15 May 2017

Oh, I also created the config files launch.json and tasks.json

Build started, please wait...
Build completed.

Test run for c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\bin\Debug\netcoreapp1.1\DefaultTests.dll(.NETCoreApp,Version=v1.1)
Ferramenta de Linha de Comando de Execução de Teste da Microsoft (R) Versão 15.0.0.0
Copyright (c) Microsoft Corporation. Todos os direitos reservados.

Iniciando execução de teste, espere...
NUnit Adapter 3.8.0.0: Test execution started
Running all tests in c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\bin\Debug\netcoreapp1.1\DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete
Test2 com falha
Mensagem de erro:
   Expected: 2
  But was:  26

Rastreamento de pilha:
at DefaultTests.UnitTest1.Test2() in c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\UnitTest1.cs:line 22


Total de testes: 2. Aprovados: 1. Com falha: 1. Ignorados: 0.
Falha na Execução de Teste.
Tempo de execução de teste: 1,4842 Segundos

bernardbr on 15 May 2017

I cloned your example and changed only .csproj and after this worked fine

It does not help for me, we must have something different in environment?

I downloaded your example. Ran dotnet restore then opened in vscode, and this is the output I see after clicking on run test above Test2:

NUnit Adapter 3.8.0.0: Test discovery starting
NUnit Adapter 3.8.0.0: Test discovery complete
NUnit Adapter 3.8.0.0: Test execution started
Running selected tests in /ide/work/nunit/DefaultTests/bin/Debug/netcoreapp1.1/DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete

Total tests: 1. Passed: 0. Failed: 1. Skipped: 0

Other than simplifying the csproj, you have changed target framework to netcoreapp1.1, which was netcoreapp1.0, do you think this would have any impact? I am running all this on top of official docker image microsoft/dotnet:1.0.5-sdk

I will add you as collaborator to that repo. Please push to some branch, instead of the zips.

tomzo on 15 May 2017

@tomzo In my machine, your project won't works with DebugType tag on DefaultTests.csproj
I Just removed it and the tests ran!

I Pushed to https://github.com/tomzo/bug-nunit-no-output-vscode/tree/without-debug-type

bernardbr on 15 May 2017

In my machine, your project won't works with DebugType tag on DefaultTests.csproj
I Just removed it and the tests ran!

Tests are running fine in my environment either with or without DebugType. But the issue is about the test output in vscode.
Test2 is failing because of Assert.AreEqual(2,26); and the error is not shown. Can you post your output from visual studio code when you run that test?

tomzo on 15 May 2017

But in my machine the output for test task works fine:

Build started, please wait...
C:\Program Files\dotnet\sdk\1.0.4\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.targets(40,3): warning MSB4011: "C:\Program Files\dotnet\sdk\1.0.4\Microsoft.CSharp.targets" cannot be imported again. It was already imported at "c:\Projetos\testes\bug-nunit-no-output-vscode\ExampleLib\ExampleLib.csproj (28,3)". This is most likely a build authoring error. This subsequent import will be ignored. 
Build completed.

Test run for c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\bin\Debug\netcoreapp1.0\DefaultTests.dll(.NETCoreApp,Version=v1.0)
Ferramenta de Linha de Comando de Execução de Teste da Microsoft (R) Versão 15.0.0.0
Copyright (c) Microsoft Corporation. Todos os direitos reservados.

Iniciando execução de teste, espere...
NUnit Adapter 3.8.0.0: Test execution started
Running all tests in c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\bin\Debug\netcoreapp1.0\DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete
Test2 com falha
Mensagem de erro:
   Expected: 2
  But was:  26

Rastreamento de pilha:
at DefaultTests.UnitTest1.Test2() in c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\UnitTest1.cs:line 20

Total de testes: 2. Aprovados: 1. Com falha: 1. Ignorados: 0.
Falha na Execução de Teste.
Tempo de execução de teste: 2,5723 Segundos

Let's compare our enviroment settings:

Dotnet info:

.NET Command Line Tools (1.0.4)

Product Information:
Version: 1.0.4
Commit SHA-1 hash: af1e6684fd

Runtime Environment:
OS Name: Windows
OS Version: 6.1.7601
OS Platform: Windows
RID: win7-x64
Base Path: C:\Program Filesdotnet\sdk\1.0.4

VsCode version:

Version 1.12.1
Commit f6868fce3eeb16663840eb82123369dec6077a9b
Date 2017-05-04T21:26:50.689Z
Shell 1.6.6
Renderer 56.0.2924.87
Node 7.4.0

Omnisharp-vscode version:

1.10.0-beta2

Nunit version:

3.8.0-alpha1

In my case I'm running on Windows. I'll try run on Ubuntu.

PS: With DebugType tag the test didn't work.

bernardbr on 15 May 2017

@bernardbr thank you for helping with this. But I think that you keep invoking a different action.
When I run a test task like you just did, then I also get the correct output:

Starting test execution, please wait...
NUnit Adapter 3.8.0.0: Test execution started
Running all tests in /ide/work/nunit/bug-nunit-no-output-vscode/DefaultTests/bin/Debug/netcoreapp1.0/DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete
Failed   Test2
Error Message:
   Expected: 2
  But was:  26

Stack Trace:
at DefaultTests.UnitTest1.Test2() in /ide/work/nunit/bug-nunit-no-output-vscode/DefaultTests/UnitTest1.cs:line 20
Total tests: 2. Passed: 1. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 1.8279 Seconds

But that runs all tests.

There is no output however when I click on run test above the test name. Like on this screenshot

tomzo on 15 May 2017

Ooh! Ok. I'm sorry, I didn't understand it before!

I don't know, but for me, in this case, the nunit does the "focus" only in "ran test" ignoring the other ones. You can try run the Test1 and see this output:

Running test DefaultTests.UnitTest1.Test1...
Microsoft (R) Build Engine version 15.1.1012.6693
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Program Files\dotnet\sdk\1.0.4\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.targets(40,3): warning MSB4011: "C:\Program Files\dotnet\sdk\1.0.4\Microsoft.CSharp.targets" cannot be imported again. It was already imported at "c:\Projetos\testes\bug-nunit-no-output-vscode\ExampleLib\ExampleLib.csproj (28,3)". This is most likely a build authoring error. This subsequent import will be ignored. 
  ExampleLib -> c:\Projetos\testes\bug-nunit-no-output-vscode\ExampleLib\bin\Debug\netcoreapp1.0\ExampleLib.dll
  DefaultTests -> c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\bin\Debug\netcoreapp1.0\DefaultTests.dll

Build succeeded.

C:\Program Files\dotnet\sdk\1.0.4\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.targets(40,3): warning MSB4011: "C:\Program Files\dotnet\sdk\1.0.4\Microsoft.CSharp.targets" cannot be imported again. It was already imported at "c:\Projetos\testes\bug-nunit-no-output-vscode\ExampleLib\ExampleLib.csproj (28,3)". This is most likely a build authoring error. This subsequent import will be ignored. 
    1 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.31


NUnit Adapter 3.8.0.0: Test discovery starting
NUnit Adapter 3.8.0.0: Test discovery complete
NUnit Adapter 3.8.0.0: Test execution started
Running selected tests in c:\Projetos\testes\bug-nunit-no-output-vscode\DefaultTests\bin\Debug\netcoreapp1.0\DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0

I miss an option to run and debug all test fixtures, but I'm afraid this doesn't exist on VsCode.

Maybe @DustinCampbell and @rprouse could help us to clarify it!

bernardbr on 15 May 2017

I miss an option to run and debug all test fixtures, but I'm afraid this doesn't exist on VsCode.

That's not a big problem.

You can try run the Test1 and see this output:

Yes, I can see that output from the very begging.
To be clear - my point is that for Test2 it should contain Error message and stack trace:

NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete
Failed   Test2
Error Message:
   Expected: 2
  But was:  26

Stack Trace:
at DefaultTests.UnitTest1.Test2() in /ide/work/nunit/bug-nunit-no-output-vscode/DefaultTests/UnitTest1.cs:line 20

But now it is just

NUnit Adapter 3.8.0.0: Test discovery starting
NUnit Adapter 3.8.0.0: Test discovery complete
NUnit Adapter 3.8.0.0: Test execution started
Running selected tests in /ide/work/nunit/DefaultTests/bin/Debug/netcoreapp1.1/DefaultTests.dll
NUnit3TestExecutor converted 2 of 2 NUnit test cases
NUnit Adapter 3.8.0.0: Test execution complete

Total tests: 1. Passed: 0. Failed: 1. Skipped: 0

tomzo on 15 May 2017

👍1

I agree with you!

I miss it too... Always when a single test fails, I need run all tests to get a stack trace...

But I don't know if it is a fail of OmniSharp or a limitation of NUnit test adapter!

bernardbr on 15 May 2017

Looks like this is an NUnit issue. The NUnit adapter has only ever been used in Visual Studio before. In Visual Studio the errors are displayed in a test results window, not in the output. I created an xUnit project and they output the results to the output so we should do the same.