Vstest: 'dotnet test --no-build' not working on Linux when binaries were compiled on Windows

[livarcocc]

_From @ChristophB125 on March 14, 2017 14:1_

Steps to reproduce

1.) Create a new MSTest or XUnit project (e.g. in VS2017) targeting NetCore (version does not matter) and optionally add the Nuget package Microsoft.TestPlatform.TestHost.
2.) Compile on Windows using the latest SDK: dotnet clean; dotnet restore; dotnet build; dotnet test
3.) Copy the whole project to Linux and run the test using the latest SDK for Linux from here:
dotnet test --no-build

Expected behavior

Test runs as usual using the binaries the were compiled on Windows

Actual behavior

Test hangs with the following error message irrespective if I have or have not the Microsoft.TestPlatform.TestHost NuGet package.:
Could not find testhost.dll for source '/home/user1/UnitTestProject3/UnitTestProject3/bin/Debug/netcoreapp1.1/UnitTestProject3.dll'. Make sure test project has a nuget reference of package "microsoft.testplatform.testhost".

Note that the test project is not referencing another project, which means it is not a duplicate of #6007 :

Environment data

dotnet --info output on the Windows machine:
.NET Command Line Tools (1.0.1)

Product Information:
Version: 1.0.1
Commit SHA-1 hash: 005db40cd1

Runtime Environment:
OS Name: Windows
OS Version: 6.1.7601
OS Platform: Windows
RID: win7-x64
Base Path: C:\Program Files\dotnet\sdk1.0.1

dotnet --info output on the Linux machine:
.NET Command Line Tools (1.0.1)

Product Information:
Version: 1.0.1
Commit SHA-1 hash: 005db40cd1

Runtime Environment:
OS Name: rhel
OS Version: 7.2
OS Platform: Linux
RID: rhel.7.2-x64
Base Path: /home/user1/Downloads/latestdotnetrelease/sdk/1.0.1

_Copied from original issue: dotnet/cli#6035_

[livarcocc]

_From @blackdwarf on March 14, 2017 17:33_

/cc @codito

[livarcocc]

I don't believe dotnet test is supposed to work across machines like that.

If you want to run a test dll from one machine into another, you can use dotnet vstest .

Can you try that?

[livarcocc]

_From @ChristophB125 on March 15, 2017 0:43_

@livarcocc I get the exact same error and hanging behaviour when using vstest and pointing it to the dll of the test project.
The reason why I want to use it this way is: Only the Windows CI machines are building and publishing the assemblies and then the tests should be run on Linux using those assemblies. This way we can be sure that the tests test a system that behaves the same as if it was in production. This allows us to avoid false positive tests (consider e.g. the case whereby assemblies compiled on Linux behave differently)

[livarcocc]

_From @ChristophB125 on March 15, 2017 15:26_

Note that not only MSTest but also XUnit projects suffers from the same issue as well.

[livarcocc]

Are you running restore after you copy the project over?

You need to either run restore on the project again to download the app dependencies to the nuget cache so that they can be found and used during run.

Or, you can try publishing your test app, copying it to the machine and using dotnet vstest. In the case of publish, it should the app dependencies to the publish folder, which you can then copy around and use without running restore.

[livarcocc]

_From @ChristophB125 on March 20, 2017 12:6_

Ok. Thanks for the information about the need for publish and restore.
After a dotnet restore on the whole solution, executing the test project as dotnet test --no-build -o bin/Debug/netcoreapp1.1/publish seems to work.
I also tried vstest and it seems that a 'dotnet restore' is not needed in this case when it is executed as:
dotnet vstest bin/Debug/netcoreapp1.1/publish/UnitTestProject3.dll

[livarcocc]

_From @ChristophB125 on March 20, 2017 13:29_

Can you maybe elaborate a bit more about the difference between dotnet test and dotnet vstest? The official CLI documentation here does not mention vstest. I'll leave the bug open in case you want to improve the error message or documentation but feel free to close it otherwise.

[livarcocc]

Moving the bug to VSTest to see about better error and/or documentation.

[Faizan2304]

@livarcocc
The dotnet test and dotnet vstest are wrapper on vstest.console.dll. The main difference between them is that dotnet vstest need test-file from which it directly run the test. On the other side dotnet test need csproj file on which it will do build (if --no-build is not specified) for you and pass all the necessary argument (like framework, testadapter path etc if needed) to vstest.console.dll.

Ideally there shouldn't be any change in behavior if you are running test with either dotnet vstest or dotnet test. Behind the scene both are same.

We can also say
dotnet vstest = dotnet test --no-build