Sdk: Unable to test non-Core code on Linux

Created on 17 May 2016  路  32Comments  路  Source: dotnet/sdk

(Apologies if this is the wrong repo for this; it's not clear to me whether this is xUnit-specific or the expected behaviour when testing on Linux.)

Background

I have projects which don't target core yet, but I still want to test on Travis. They work fine on RC1 (with dnx test and the old xUnit setup) and work fine with an updated project.json on Windows, but fail to run tests on Linux.

It's not clear whether the non-core runtimes/frameworks are expected to work on non-Windows platforms. Using dotnet run works fine for a console app with a TFM of net451, so it's clearly got _some_ knowledge... but dotnet test fails.

Similar issues: dotnet/sdk#4833, dotnet/sdk#5172.

Steps to reproduce

Create xUnit test project with a framework of net451, and run dotnet restore, dotnet build, dotnet test.

Sample project.json:

{
  "dependencies": {
    "xunit": "2.1.0",
    "dotnet-test-xunit": "1.0.0-rc2-build10015",
  },

  "testRunner": "xunit",

  "frameworks": {
    "net451": { }
  }
}

Sample C#:

using Xunit;

public class Test
{
    [Fact] public void Foo() {}
}

Expected behavior

Tests run (using Mono).

Actual behavior

Restore and build both work, but test fails:

dotnet-test Error: 0 : Microsoft.DotNet.Cli.Utils.CommandUnknownException: No executable found matching command "dotnet-test-xunit"  
  at Microsoft.DotNet.Cli.Utils.ProjectDependenciesCommandFactory.FindProjectDependencyCommands(String commandName, IEnumerable`1 commandArgs, String configuration, NuGetFramework framework, String outputPath, String buildBasePath, String projectDirectory)  
  at Microsoft.DotNet.Cli.Utils.ProjectDependenciesCommandFactory.Create(String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration)  
  at Microsoft.DotNet.Tools.Test.ConsoleTestRunner.DoRunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams)  
  at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.0-preview1-002702)

Product Information:
 Version:     1.0.0-preview1-002702
 Commit Sha:  6cde21225e

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  15.10
 OS Platform: Linux
 RID:         ubuntu.15.10-x6
needs triage
Source
picture jskeet
👍1

Most helpful comment

I has been some time since this issue was raised. Is the above nasty workaround the only real solution at this point? The alternative being to wait for a preview3 tooling release?

picture RehanSaeed on 10 Oct 2016
👍4

All 32 comments

As far as I can tell, one issue is that there's no specific Mono RID, so there's no runtime exe that we can create in the dotnet-test-xunit package for this.

clairernovotny picture clairernovotny on 17 May 2016

/cc @bradwilson @piotrpMSFT

clairernovotny picture clairernovotny on 17 May 2016

Smells like a tooling problem, not an xUnit.net problem.

bradwilson picture bradwilson on 17 May 2016

So I reported it in the right repo by chance? Result :)

Do let me know if there's anything I can do to help diagnose it. I realize there are a lot of variables in play here.

jskeet picture jskeet on 17 May 2016

I'm getting this error ( No executable found matching command "dotnet-test-") as well but with dotnet core on osx. I'm not finding much via googlebing so adding onto this relatively similar thread ...because it puts me in such good company. :)

julielerman picture julielerman on 18 May 2016

@julielerman can you share your project.json?

clairernovotny picture clairernovotny on 18 May 2016

I've been mucking with it but results are still the same:

{
    "version": "2.0.0-*",
    "description": "",
    "authors": [
        ""
    ],
    "dependencies": {
        "Microsoft.EntityFrameworkCore.InMemory": "1.0.0-rc2-final",
        "src": "2.0.0",
        "xunit": "2.1.0",
        "dotnet-test-xunit": "1.0.0-rc2-build10015"
    },
    "frameworks": {
        "netcoreapp1.0": {
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "type": "platform",
                    "version": "1.0.0-rc2-3002702"
                }
            },
            "imports": [
                "dnxcore50",
                "portable-net45+win8"
            ]
        }
    }
}
julielerman picture julielerman on 18 May 2016

So to be clear, the above works on Windows but not OSX?

clairernovotny picture clairernovotny on 18 May 2016

have not done any of this on windows. using vscode on a mac. Also, I have a full project that is running. I will create a 1+1=2 test and try to run that to be sure it's not a problem related to the dependency on that other project. Also, the only other ref I found to this error message was followed by "oh ..typo in project.json...sorry" but I can't see one and vscode is not reporting any issues. Thanks!

julielerman picture julielerman on 18 May 2016

dotnet-test should be dotnet test, if you're still seeing that error. If you're not typing that yourself, then it seems like you need to update your tools/SDK.

ardalis picture ardalis on 18 May 2016

You're missing the testRunner entry. This may help: http://xunit.github.io/docs/getting-started-dotnet-core.html

bradwilson picture bradwilson on 18 May 2016
👍1

@bradwilson gah! I saw that but was so focused on the dependencies section. Running up to my macbook to try it out and will let you know.

julielerman picture julielerman on 18 May 2016

got it. sorry for my little RTFM snafu, @bradwilson . Sorry @jskeet for piggy backing. You _do_ have that specification in your project.json.

julielerman picture julielerman on 18 May 2016

I have the exact same problem as @jskeet in windows with TFM net452. Maybe it's not related to linux?

project json:

{
  "version": "1.0.0-*",
  "description": "UnitTests Class Library",
  "authors": [ "albertori" ],

  "dependencies": {
    "xunit": "2.1.0",
    "dotnet-test-xunit": "1.0.0-rc2-build10015"
  },

  "frameworks": {
    "net452": {
      "dependencies": {
        "NSubstitute": "1.10.0"
      }
    }
  },
  "testRunner": "xunit"
}

dotnet --info output:

.NET Command Line Tools (1.0.0-preview1-002702)

Product Information:
 Version:     1.0.0-preview1-002702
 Commit Sha:  6cde21225e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10586
 OS Platform: Windows
 RID:         win10-x64
alberto-riggi picture alberto-riggi on 20 May 2016

@alberto-riggi: Just to check - you're having this problem on Windows? It only bit me on Linux. (My tiny sample works on Windows with both net451 and net452.)

jskeet picture jskeet on 20 May 2016

@jskeet , correct, on windows.
I saw there is another issue regarding this https://github.com/dotnet/cli/issues/1370 .
Your example in win works just targeting net452?
I've read somewhere that if you have net45x and aspnet core as TFMs it works.

alberto-riggi picture alberto-riggi on 20 May 2016

@alberto-riggi Yes, it works with just net452 - see the code/config in my first post, but with net452. It might be worth you trying that too.

jskeet picture jskeet on 20 May 2016

Thanks john, I had to use "dotnet-test-xunit": "1.0.0-rc3-build10019" though.
As pointed here
https://github.com/dotnet/cli/issues/1370#issuecomment-220723436

alberto-riggi picture alberto-riggi on 20 May 2016

Got it works with work around on Ubuntu 14.04
Project:

{
  "version": "1.0.0-*",

  "testRunner": "xunit",

  "dependencies": {
    "dotnet-test-xunit": "1.0.0-rc3-*",
    "Project": "*",
    "xunit": "2.1.0"
  },

  "frameworks": {
    "net451": {
      "dependencies": {
        "Microsoft.NETCore.Platforms": "1.0.1-rc2-*",
        "Moq": "4.2.1510.2205"
      }
    }
  }
}

Failed:
dotnet test test/ProjectFolderTest

Success:
mono test/ProjectFolderTest/bin/Debug/net451/ubuntu.14.04-x64/dotnet-test-xunit.exe test/ProjectFolderTest/bin/Debug/net451/ubuntu.14.04-x64/ProjectFolderTest.dll

thohng picture thohng on 22 May 2016
👍1

I confirm that dotnet test does NOT work on linux when targetting net451.

Setup:

  • Debian 8 (x64)
  • Mono 4.2.3
  • Dotnet 1.0.0-preview1-002702

project.json:

{
  "version": "1.0.0-*",

  "testRunner": "xunit",

  "dependencies": {
    "xunit": "2.1.0",
    "dotnet-test-xunit": "1.0.0-rc3-build10019"
  },

  "frameworks": {
    "net451": { }
  }
}

Output of dotnet test:

sebok$ dotnet test
Project Atelier.Tests (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
dotnet-test Error: 0 : Microsoft.DotNet.Cli.Utils.CommandUnknownException: No executable found matching command "dotnet-test-xunit"
  at Microsoft.DotNet.Cli.Utils.ProjectDependenciesCommandFactory.FindProjectDependencyCommands(String commandName, IEnumerable`1 commandArgs, String configuration, NuGetFramework framework, String outputPath, String buildBasePath, String projectDirectory)
  at Microsoft.DotNet.Cli.Utils.ProjectDependenciesCommandFactory.Create(String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration)
  at Microsoft.DotNet.Tools.Test.ConsoleTestRunner.DoRunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams)
  at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

However, the build created directory bin/Debug/net451/debian.8-x64, from which I can manually run dotnet-test-xunit.exe:

sebok$ cd bin/Debug/net451/debian.8-x64/
sebok$ mono dotnet-test-xunit.exe Atelier.Tests.dll
xUnit.net .NET CLI test runner (64-bit debian.8-x64)
  Discovering: Atelier.Tests
  Discovered:  Atelier.Tests
  Starting:    Atelier.Tests
  Finished:    Atelier.Tests
=== TEST EXECUTION SUMMARY ===
  Atelier.Tests  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 0,063s
ghost picture ghost on 26 May 2016
👍1

Same behaviour as @sebok - dotnet test works on windows but no osx.10.11 using net451

  • osx.10.11
  • Mono 4.2.3
  • dotnet 1.0.0-preview1-002702

project.json:

{
    "version": "1.0.0-*",
    "dependencies": {
        "dotnet-test-xunit": "1.0.0-rc2-*",
        "xunit": "2.1.0"
    },
    "testRunner": "xunit",
    "frameworks": {
        "netcoreapp1.0": {
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "version": "1.0.0-rc2-3002702",
                    "type": "platform"
                }
            },
            "imports": [
                "dnxcore50",
                "portable-net451+win8"
            ]
        },
        "net451": {}
    }
}

Running dotnet test fails with the following output:

dotnet-test Error: 0 : Microsoft.DotNet.Cli.Utils.CommandUnknownException: No executable found matching command "dotnet-test-xunit"
   at Microsoft.DotNet.Cli.Utils.ProjectDependenciesCommandFactory.FindProjectDependencyCommands(String commandName, IEnumerable`1 commandArgs, String configuration, NuGetFramework framework, String outputPath, String buildBasePath, String projectDirectory)
   at Microsoft.DotNet.Cli.Utils.ProjectDependenciesCommandFactory.Create(String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration)
   at Microsoft.DotNet.Tools.Test.ConsoleTestRunner.DoRunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

Running dotnet-test-xunit.exe from bin/Debug/net451/osx.10.11-x64 succeeds: 

xUnit.net .NET CLI test runner (32-bit Desktop .NET osx.10.11-x86)
  Discovering: NetEscapades.AspNetCore.SecurityHeaders.Test
  Discovered:  NetEscapades.AspNetCore.SecurityHeaders.Test
  Starting:    NetEscapades.AspNetCore.SecurityHeaders.Test
  Finished:    NetEscapades.AspNetCore.SecurityHeaders.Test
=== TEST EXECUTION SUMMARY ===
   NetEscapades.AspNetCore.SecurityHeaders.Test  Total: 12, Errors: 0, Failed: 0, Skipped: 0, Time: 0.284s
andrewlock picture andrewlock on 29 May 2016

If a net451 exe exists in the package, you will have to do a mono bootstrap script.

#!/usr/bin/env bash

SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
  DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
  SOURCE="$(readlink "$SOURCE")"
  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done
DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"

exec mono "$DIR/dotnet-test-xunit.exe" "$@"
borgdylan picture borgdylan on 3 Jul 2016
👍3

@onovotny @julielerman @andrewlock The script above may help you.

borgdylan picture borgdylan on 3 Jul 2016

So just putting the script where the exe is will not help. I will have to fork the repo and add a makefile to install this on *nix machines.

borgdylan picture borgdylan on 3 Jul 2016

@borgdylan Thanks for your script. I'm using, with the following improvements:

  • Including your script at my test projects root directory
  • Putting the following configuration on projector.json:

"buildOptions": { "copyToOutput": "dotnet-test-xunit" }

  • Adding the following line to .gitattributes:

dotnet-test-xunit -text

This way, your script always gets copied to output directory, and I'm able to run test as I would normally would: dotnet test

ac-esousa picture ac-esousa on 12 Jul 2016
👍2

This is a more elegant solution than the one I used.

borgdylan picture borgdylan on 12 Jul 2016

We were stumbling over the same problem recently. I'm referencing dotnet-text-xunit version 2.2.0-preview3-build1047 in a net461 project; on Ubuntu 14.04 LTS dotnet test fails with 

No executable found matching command "dotnet-test-xunit"

However, when I copy ~/.nuget/packages/dotnet-test-xunit/.../runtimes/unix-x64/lib/net451/dotnet-test-xunit.exe to the output directory of the test project (e.g. runtimes/unix-x64/lib/net451/dotnet-test-xunit.exe) and then add @borgdylan's script to the _same_ directory, dotnet test works.
That's an awful lot of work, however.

project.json looks like:

{
  "buildOptions": {
    "debugType": "portable",
    "preserveCompilationContext": true
  },

  "dependencies": {
    "dotnet-test-xunit": "2.2.0-preview3-build1047",
    "xunit": "2.2.0-beta3-build3401",
    "FluentAssertions": "4.14.0",
    "MyProject": { "target": "project" }
  },

  "testRunner": "xunit",

  "frameworks": {
    "net461": { }
  }
}
sunsided picture sunsided on 29 Sep 2016

I has been some time since this issue was raised. Is the above nasty workaround the only real solution at this point? The alternative being to wait for a preview3 tooling release?

RehanSaeed picture RehanSaeed on 10 Oct 2016
👍4

The bash script workaround no longer works when upgrading to the latest .NET SDK 1.0.1. The tests do run when using mono xunit.console.exe test.dll, but dotnet test fails to find any "again", specifying "'TestAdapterPath' does not contain any test adapters", continues with "permission denied", then exits.

bravecobra picture bravecobra on 19 Mar 2017
😕1

@bravecobra - I have the exact same problem. Do you know of any workaround?

AdamPD picture AdamPD on 29 Jun 2017

@AdamPD if you can upgrade your xunit project up to 2.3.0-beta3 you can use the dotnet-xunit tool which I added support for mono https://www.nuget.org/packages/dotnet-xunit/ https://github.com/xunit/xunit/pull/1213

TheAngryByrd picture TheAngryByrd on 23 Jul 2017
👍1

dotnet xunit is removed starting from xunit 2.4 Ref: Release Notes 2.4
Excerpt from the Release Notes:

Unfortunately, this release also removes the dotnet xunit runner, as the stability of the runner was never perfect, and it suffered from many assembly-loading related issues. Users from .NET Core can continue to use VSTest (either inside Visual Studio, or via dotnet test).

So, for xunit framework test use the command

dotnet test

See Getting Started with xUnit.net Multi-targeting with non-Windows OSes

moh-hassan picture moh-hassan on 30 Jun 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ghost picture ghost  路  210Comments
idavis picture idavis  路  57Comments
ivanz picture ivanz  路  57Comments
redowl3 picture redowl3  路  79Comments
danmosemsft picture danmosemsft  路  137Comments