Sdk: Add support to net451 to dotnet-test-xunit

[user4794170]

I'm trying to run xUnit tests using dotnet test. I'm getting the following error: dotnet-test Error: 0 : System.InvalidOperationException: c:\Test\bin\Debug\dnx451\Test.dll\project.json does not exist. The tests work fine if run in Visual Studio using the Test Explorer.

Repro Steps

1. Windows 8.1
2. Install .NET CLI 1.0.1.001200 dev (from https://dotnet.github.io/getting-started/)
3. Run the batch file below
4. Update the project.json in the relevant step

repro.bat

cd c:\
mkdir Test
cd Test
dotnet new
echo Update project.json now to the provided project.json
start /wait notepad project.json
dotnet restore
dotnet build
dotnet test
pause

project.json

{
  "version": "1.0.0-*",
  "compilationOptions":
  {
    "emitEntryPoint": false
  },

  "dependencies":
  {
    "xunit": "2.1.0",
    "xunit.runner.dnx": "2.1.0-rc1-build204"
  },

  "commands":
  {
    "test": "xunit.runner.dnx"
  },

  "frameworks":
  {
    "dnx451":
    {
    }
  }
}

[livarcocc]

This is not how we run tests anymore. We don't use dnx or the dnx runner. Your project.json needs to be something like below.

Also, to run, you do dotnet restore, then you need to do a dotnet build and then dotnet test.
project.json

{
  "dependencies": {
    "NETStandard.Library": "1.0.0-rc2-23728",
    "xunit": "2.1.0",
    "dotnet-test-xunit": "1.0.0-dev-45337-57"
  },
  "frameworks": {
    "dnxcore50": {
      "imports": "portable-net45+win8"
    }
  },
  "testRunner": "xunit"
}

[user4794170]

Hi livarocc.
Thanks for your reply.

• I'm getting that error message even if I don't change the project.json from it's defaults
• Using your project.json, the restore step fails with warn : Dependency specified was NETStandard.Library (>= 1.0.0-rc2-23728) but ended up with NETStandard.Library 1.0.0-rc2-23804. and error: Unable to resolve dotnet-test-xunit (>= 1.0.0-dev-45337-57) for DNXCore,Version=v5.0.
• Using your project.json, I got several error messages on the build step, including c:\Test\project.json(3,28): error NU1001: The dependency NETStandard.Library >= 1.0.0-rc2-23728 could not be resolved. and c:\Test\project.json(5,26): error NU1001: The dependency dotnet-test-xunit >= 1.0.0-dev-45337-57 could not be resolved..
• After the failed compilation, during dotnet test, I get the following error: dotnet-test Error: 0 : Microsoft.DotNet.Cli.Utils.CommandUnknownException: No executable found matching command "dotnet-test-xunit"
• I'm using .NET 4.5.1 because one of the libraries I need is not available for .NET Core yet so I can't use dnxcore50
• I'm also using Visual Studio Enterprise to open this same project and run the tests through the Test Explorer (I'm trying to setup the CLI process so that I can also use VS Code in addition to VS Enterprise). According to xUnit documentation, I need xunit.runner.dnx to have tests show up in the Tests Explorer so I don't think I can get rid of it.

[livarcocc]

Right now, you won't be able to use dotnet test from within VS. This is something that we are working to support still.

You can leave your test project as dnxcore50 and take a dependency on a project built for net451. That should work.

Also, you need some extra feeds in your nuget.config to be able to resolve the extra dependencies:

Extra feeds in NuGet.config

    <add key="dotnet.myget.org dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="dotnet.myget.org dotnet-cli" value="https://dotnet.myget.org/F/dotnet-cli/" />
    <add key="coreclr-xunit" value="https://www.myget.org/F/coreclr-xunit/api/v2" />
    <add key="AspNetCIDev" value="https://www.myget.org/F/aspnetcidev/api/v3/index.json" />
    <add key="nugetbuild" value="https://www.myget.org/F/nugetbuild/api/v3/index.json" />

[user4794170]

You can leave your test project as dnxcore50 and take a dependency on a project built for net451. That should work.

• If I updated my Logic and Test project.jsons to dnxcore50 I get the error message: Error NU1002 The dependency Selenium.WebDriver 2.50.1 in project Test does not support framework DNXCore,Version=v5.0.
• I updated my logic project to use dnx451 and my test project to use dnxcore50 and can't reference my logic project, I get the error message The project Logic has a target framework that is incompatible...

---

Both Logic and Tests set to dnxcore50

Both Logic.xproj and Tests.xproj to this, results in Error NU1002 The dependency Selenium.WebDriver 2.50.1 in project Test does not support framework DNXCore,Version=v5.0.

{
  "dependencies":
  {
    "NETStandard.Library": "1.0.0-rc3-23811",
    "Newtonsoft.Json": "8.0.2",
    "Selenium.WebDriver": "2.50.1",
    "Selenium.Support": "2.50.1"
  },
  "frameworks":
  {
    "dnxcore50":
    {
      "imports": "portable-net45+win8"
    }
  }
}

---

Logic Set to dnx451 and Tests set to dnxcore50

Results in The dependency Logic does not support framework dnxcore50.

logic project.json

{
  "dependencies":
  {
    "Newtonsoft.Json": "8.0.2"
  },

  "frameworks":
  {
    "dnx451":
    {
      "dependencies":
      {
        "Selenium.WebDriver": "2.50.1",
        "Selenium.Support": "2.50.1"
      },

      "frameworkAssemblies":
      {
        "System": "4.0.0.0",
      }
    }
  }
}

tests project.json

{
  "dependencies":
  {
    "NETStandard.Library": "1.0.0-rc3-23811",
    "xunit": "2.1.0",
    "xunit.runner.dnx": "2.1.0-rc1-build204",
    "Logic": ""
  },

  "commands":
  {
    "test": "xunit.runner.dnx"
  },

  "frameworks":
  {
    "dnxcore50":
    {
      "imports": "portable-net45+win8"
    }
  }
}

[livarcocc]

cc @pranavkm

We don't support net451 in the runner yet. We will soon.

[user4794170]

Sorry, I'm a bit confused.

You can leave your test project as dnxcore50 and take a dependency on a project built for net451. That should work.

We don't support net451 in the runner yet. We will soon.

Can you please clarify how I should correctly setup my projects so that I can use a Nuget package that isn't dnxcore50 compatible and still run tests both through the Test Explorer and via dotnet test? Or are you saying that this isn't possible yet? Thanks!

[livarcocc]

You cannot run your tests using dotnet test through Test Explorer yet. You must use the command line.

[user4794170]

Is it possible for me to setup my project so that I can both run tests in the the Test Explorer using xunit.runner.dnx and on the command line using dotnet-test-xunit?

Also, can you elaborate on referencing things that are not dnxcore50 compatbile in dnxcore50 projects? (Based on your comment: You can leave your test project as dnxcore50 and take a dependency on a project built for net451. That should work.). What I tried didn't work.

[livarcocc]

In this case, I don't think you can. We are working on adding net 4.5.1 to the runner, which is why I CCed @pranavkm above. He is working on it. At this point, either you port the Selenium Webdriver to dotnet core, or you wait for the runner to support net 4.5.1.

And I don't think you can set it up to run for both VS and the command line. We are also working to add support for it in VS.

[user4794170]

Thanks @livarcocc.

Selenium dotnet-core support might be a little while: https://github.com/SeleniumHQ/selenium/issues/1587, so I'll wait for runner support in the mean time. Is there an estimated date for completion of the feature (i.e. with dotnet-core RTM?) that you or @pranavkm could give me?

Thanks again for the assistance!

[Pete-PlaytimeSolutions]

@livarcocc Is there a time frame for when net451 will be supported? For either via the command-line or test explorer?

[livarcocc]

@pranavkm Is this something that you will take care of or should I take over?

[pranavkm]

Soryr, I haven't been paying attenting to this. For our scenarios in AspNet, we're using the desktop xunit runner after publish.

[livarcocc]

Tracking dotnet-test-xunit support here.

[qooroo]

Is net451 now supported? This is a blocker for those upgrading from RC1 when xunit worked OK. Specifically running into dotnet/cli#3069

[qooroo]

There is also a separate test runner not found issue when running on ubuntu/mono - details here: https://github.com/xunit/xunit/issues/848

[bragma]

Probably not a surprise, but I have the same problem on a project.json containing:

"frameworks": {
"net461": { }
}

[alberto-riggi]

same with net452

[pranavkm]

This is an issue with test runner - https://github.com/xunit/xunit/issues/843. Try the instructions from - https://github.com/xunit/xunit/issues/843#issuecomment-219868947

[am11]

@alberto-riggi, with RC2, we can follow this structure in project.json:

{
  "version": "1.0.0-*",

  "dependencies": {

    "<the app you are testing>": "<its version>",

    "xunit": "2.1.0",
    "xunit.runner.visualstudio": "2.1.0",
    "dotnet-test-xunit": "1.0.0-*"
  },

  "frameworks": {
    "net452": { }
  },

  "testRunner": "xunit"
}

Then in VS Test Explorer, you will see the tests after the build.

Or from CLI:

cd test-dir
dotnet restore

dotnet test

Note that `dotnet-test-xunit` has not landed on NuGet yet, so before `dotnet restore`'ing, add a file `NuGet.Config` in your solution directory with xUnit's nightlies MyGet feed URL key:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="xunit" value="https://www.myget.org/F/xunit/api/v3/index.json" />
  </packageSources>
</configuration>

*(in future, when `dotnet-test-xunit` will land on nuget.org as stable package, we will just delete this `NuGet.Config` file)*

(see comment below by @thecodejunkie)

[alberto-riggi]

It worked!

Thanks!

[thecodejunkie]

@am11 @alberto-riggi

Note that dotnet-test-xunit has not landed on NuGet yet,

Not quite. The rc2 version, 1.0.0-rc2-build10015 can be found on NuGet https://www.nuget.org/packages/dotnet-test-xunit/1.0.0-rc2-build10015 but it contained a bug (https://github.com/xunit/xunit/issues/843) which had to be fixed and it's the fixed version that is only available on MyGet at the moment

[am11]

Great! So I removed the NuGet and changed the dotnet-test-xunit version from "1.0.0-rc3-*" to 1.0.0-*. It restores 1.0.0-rc2-build10015 (visible in project.lock.json file) and dotnet test still successfully executing for net452 framework. Thanks @thecodejunkie!