Omnisharp-vscode: XUnit projects cannot be launched under the debugger

[aggieben]

The symptoms are related to those described in #1253, but wasn't solved with the thing the closed that one. Environment info is the same.

λ dotnet --info
.NET Command Line Tools (1.0.0-rc4-004771)

Product Information:
 Version:            1.0.0-rc4-004771
 Commit SHA-1 hash:  d881d45b75

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.0-rc4-004771

@gregg-miskelly:

Do you always see the program has exited with code 0 or is it intermittent?
If you always get it --

• What is program set to in launch.json?
• Can you send me the output you get in your console window?

Yes, the issue is 100% repeatable. In this case, the problem is with an Xunit project, called Saddleback.AspNetCore.Features.Test. The project file as as follows:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>netcoreapp1.0;netcoreapp1.1</TargetFrameworks>
    <NetStandardImplicitPackageVersion>1.6.1</NetStandardImplicitPackageVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.TestHost" Version="1.1.0"/>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="../../src/Saddleback.AspNetCore.Features/Saddleback.AspNetCore.Features.csproj"/>
    <ProjectReference Include="../Saddleback.Logging.Xunit/Saddleback.Logging.Xunit.csproj"/>
  </ItemGroup>

</Project>

The launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": ".NET Core Launch (console)",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            "program": "${workspaceRoot}\\test\\Saddleback.AspNetCore.Features.Test\\bin\\Debug\\netcoreapp1.0\\Saddleback.AspNetCore.Features.Test.dll",
            "args": [],
            "cwd": "${workspaceRoot}",
            "externalConsole": false,
            "stopAtEntry": false,
            "internalConsoleOptions": "openOnSessionStart"
        },
        {
            "name": ".NET Core Attach",
            "type": "coreclr",
            "request": "attach",
            "processId": "${command:pickProcess}"
        }
    ]
}

the console output:

Microsoft (R) Build Engine version 15.1.545.13942
Copyright (C) Microsoft Corporation. All rights reserved.
  Saddleback.AspNetCore.Features -> d:\Users\ben\proj\sb\SaddlebackFeatures\src\Saddleback.AspNetCore.Features\bin\Debug\netstandard1.4\Saddleback.AspNetCore.Features.dll
  Saddleback.Logging.Xunit -> d:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.Logging.Xunit\bin\Debug\netstandard1.4\Saddleback.Logging.Xunit.dll
  Saddleback.AspNetCore.Features.Test -> d:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.0\Saddleback.AspNetCore.Features.Test.dll
  Saddleback.AspNetCore.Features.Test -> d:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.1\Saddleback.AspNetCore.Features.Test.dll
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:07.60

and the debug console output:

--------------------------------------------------------------------------------
You may only use the Microsoft .NET Core Debugger (clrdbg) with Visual Studio
Code, Visual Studio or Visual Studio for Mac software to help you develop and
test your applications.
--------------------------------------------------------------------------------
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.3\System.Private.CoreLib.ni.dll'. Cannot find or open the symbol file.
Loaded 'd:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.0\Saddleback.AspNetCore.Features.Test.dll'. Symbols loaded.
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.3\System.Runtime.dll'. Cannot find or open the symbol file.
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.3\mscorlib.dll'. Cannot find or open the symbol file.
The program 'd:\Users\ben\proj\sb\SaddlebackFeatures\test\Saddleback.AspNetCore.Features.Test\bin\Debug\netcoreapp1.0\Saddleback.AspNetCore.Features.Test.dll' has exited with code 0 (0x00000000).

[gregg-miskelly]

@aggieben I think the issue here is that XUnit projects aren't runnable in this way -- it either doesn't run your tests at all, or it runs them in a child process. You can confirm or Deny this theory by adding a Console.WriteLine to one of your tests to see if it runs. If it does run have it print its process id and then pause (ex: Thread.Sleep(10000);). You can then use ps -A to find out if it is the process that the debugger launched.

[gregg-miskelly]

I should say, in theory the right way to debug a test is to use the little button next to the test method. But from https://github.com/OmniSharp/omnisharp-vscode/issues/1100 I gather that that is currently broken for .csproj-based projects.

[aggieben]

@gregg-miskelly indeed, the test launch links decorating the methods do not work for me. If I click run, I get the "Failed to run test because null" error, and if I click debug, nothing at all happens.

[DustinCampbell]

@aggieben: The "Run Test" and "Debug Test" CodeLens links don't work yet for .csproj-based .NET Core projects. This is tracked by https://github.com/OmniSharp/omnisharp-vscode/issues/1100.

[jflepp]

also doesnt work for me:
csproj <ItemGroup> <PackageReference Include="xunit" Version="2.3.0-beta2-build3683" /> <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.0-beta2-build3683" /> </ItemGroup>
debugger doesnt hit breakpoints when xunit-cli tool is added

works again when commenting out from csproj, removing bin/ and obj/ folder and restoring

full csproj
````

<PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp1.1</TargetFramework>
</PropertyGroup>

````

I also tryed to reinstall omnisharp, c# extension and dotnet core
i got this issue on two different computers

[gregg-miskelly]

@jflepp I might suggest opening a new issue. But either way, a few questions:

• What do you see in the debug console?
• You are on C# extension 1.11.0?
• What OS are you on?

[jflepp]

@gregg-miskelly I created a new issue: #1630

[kamranayub]

I'm having similar problems with the new .NET 2.0 (I never tried with earlier SDKs).

I had no launch.json and so VS Code created a new one for me but I see the same thing where tests aren't run, the program just exits. Is there something special I need to initiate the running of tests?

My repo is public if you want to test: https://github.com/kamranayub/GiantBomb-CSharp/tree/net20

I'll edit/update if I end up figuring it out, I wish it wasn't so hard 😢

[DustinCampbell]

The fact that you can F5 an xUnit project is really a bit strange. To be honest, C# for VS Code really shouldn't offer to create a launch.json file for them, but xUnit projects are created as "netcoreapp2.0" with no entry point.

The trick to debugging unit tests is to open a file containing unit tests and then click the "debug test" annotation above the method:

[kamranayub]

Yeah but that still doesn't work with whatever I have set up :( You can
clone my net20 branch and see that too.

On Fri, Aug 18, 2017 at 10:16 AM Dustin Campbell [e-mail]
wrote:

The fact that you can F5 an xUnit project is really a bit strange. To be
honest, C# for VS Code really shouldn't offer to create a launch.json file
for them, but xUnit projects are created as "netcoreapp2.0" with no entry
point.

The trick to debugging unit tests is to open a file containing unit tests
and then click the "debug test" annotation above the method:

[image: image]
https://user-images.githubusercontent.com/116161/29465232-71edbe00-83ed-11e7-9fd0-0ccc66b462c0.png

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/OmniSharp/omnisharp-vscode/issues/1292#issuecomment-323381663,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiaa9rVqyps7d-NSf0faC4lIgn2pGqNks5sZarUgaJpZM4MVA2F
.

[DustinCampbell]

Here's what I see with your branch:

Debugging method 'GiantBomb.Api.Tests.Games.game_resource_should_return_game_for_33394'...

Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  GiantBomb.Api -> c:\Projects\GiantBomb-CSharp\GiantBomb.Api\bin\Debug\netstandard1.2\GiantBomb.Api.dll
  Successfully created package 'c:\Projects\GiantBomb-CSharp\GiantBomb.Api\bin\Debug\GiantBomb.Api.2.4.0.nupkg'.
  GiantBomb.Api.Tests -> c:\Projects\GiantBomb-CSharp\GiantBomb.Api.Tests\bin\Debug\netcoreapp2.0\GiantBomb.Api.Tests.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:05.29


An exception occurred while test discoverer 'VsTestRunner' was loading tests. Exception: Object reference not set to an instance of an object.
Debugging complete.

Is that the same thing you're seeing?

[DustinCampbell]

FWIW, I think this is a completely different problem than what this issue is tracking.

[kamranayub]

Yes but I already figured the debug test link doesn't work (I think there are other issues tracking that). My issue is the regular debug launch in VS Code (using start button) doesn't seem to work and ends up with the same output as @jflepp above.

Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.0\System.Private.CoreLib.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Loaded 'e:\Development\Contrib\GiantBomb-CSharp\GiantBomb.Api.Tests\bin\Debug\netcoreapp2.0\GiantBomb.Api.Tests.dll'. Symbols loaded.
Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.0\System.Runtime.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
The program '[34748] GiantBomb.Api.Tests.dll' has exited with code 0 (0x0).

I am not sure if it's by design to not work but I'm looking for the same experience I get in Visual Studio 2017 where debugging tests by launching the debugger does work.

[DustinCampbell]

As I mentioned earlier, I wouldn't expect that to work and its somewhat of a bug that we create a launch.json at all for xUnit test projects. Pressing F5 on test project is akin to launching it with 'dotnet run'. It doesn't launch it with the test framework.

[DustinCampbell]

FWIW, the issue I'm seeing looks like it _might_ be a bug introduced in xUnit. If I use the version of xUnit that dotnet new xunit generates (2.2.0), run and debug tests works fine. If I moved to the latest xUnit preview that you're using (2.3.0-beta4-build3742), it fails with An exception occurred while test discoverer 'VsTestRunner' was loading tests. Exception: Object reference not set to an instance of an object.

[kamranayub]

@DustinCampbell Interesting! I just followed the latest docs of xunit, I'll try the earlier version maybe... it said the latest beta 4 adds .NET Core 2.0 support.

[DustinCampbell]

The latest beta 4 definitely should be much better. IIRC, it fixes issues with running tests in VS. I'm debugging this now to see if I can figure out where the issue might be.

[DustinCampbell]

Here's the exception that's happening:

TpTrace Error: 0 : 16628, 7, 2017/08/18, 13:00:27.519, 129122016173, testhost.dll, Exception: System.NullReferenceException
    Message: Object reference not set to an instance of an object.
    Stack Trace:    at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.ValidateRuntimeFramework() in C:\projects\xunit\src\xunit.runner.visualstudio\VsTestRunner.cs:line 724
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestDiscoverer.DiscoverTests(IEnumerable`1 sources, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink) in C:\projects\xunit\src\xunit.runner.visualstudio\VsTestRunner.cs:line 81
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscovererEnumerator.LoadTestsFromAnExtension(String extensionAssembly, IEnumerable`1 sources, IRunSettings settings, IMessageLogger logger)
    BaseExceptionMessage: Object reference not set to an instance of an object.

[DustinCampbell]

The issue appears to be right here:

```C#
///

/// Validates the runtime target framework from test platform with the current adapter's target.
///

/// True if the target frameworks match.
static bool ValidateRuntimeFramework()
{

if NETCOREAPP1_0

        var targetFrameworkVersion = RunSettingsHelper.TargetFrameworkVersion;

        return targetFrameworkVersion.StartsWith(".NETCore", StringComparison.OrdinalIgnoreCase) ||
               targetFrameworkVersion.StartsWith("FrameworkCore", StringComparison.OrdinalIgnoreCase);

else

        return true;

endif

    }

```

The problem is that our test runner passes RunSettings to the dotnet vstest that do not include a TargetFrameworkVersion and xUnit expects it to be there. The change was introduced with https://github.com/xunit/xunit/commit/d22f7c1d5a6b3a1058a57fa94e97cee32f0f2e9a.

[DustinCampbell]

Yup, that's the issue. If I change OmniSharp to pass a run settings file containing <TargetFrameworkVersion>.NETCoreApp, Version=2.0</TargetFrameworkVersion> it works fine. I don't think xUnit is at fault here, but we need to do a bit more work to make it work right. I've filed https://github.com/OmniSharp/omnisharp-roslyn/issues/944 to track this work.

[kamranayub]

@DustinCampbell Great, thanks!

[Neutrino-Sunset]

Any idea when the fix will go live. Unless I've misunderstood it's currently impossible to debug either xunit or mstest test which is a major showstopper.

[DustinCampbell]

Not yet, but there is a beta release that you can use in the meantime: 1.13.0-beta3. You can use the instructions here to install a beta release.

[Neutrino-Sunset]

Awesome, thanks Dustin. I've also just worked out that I can run an xunit test, but I just have to enable this horrid codelens thing. Looking forward to the time I can turn that back off.