Vstest: `dotnet test` fails to find .NET Core 3.0 SDK when installed from script to custom location

[martincostello]

Description

dotnet test fails to find .NET Core SDK 3.0 when installed from https://dot.net/v1/dotnet-install.ps1 into a custom install location with -NoPath in Azure DevOps on Windows.

This does not appear to be an issue with AppVeyor, Travis CI or Azure DevOps on Linux and macOS.

This worked with various versions of the .NET Core SDK up to and including 3.0.0 RC1, but appears to have been broken in 3.0.0 with 16.3.0.

Looking at the diff between 16.3.0 and 16.3.0-preview-20190828-03, this might be a regression caused by the changes in https://github.com/microsoft/vstest/pull/2161.

Steps to reproduce

Issue can be observed in this Pull Request from commit 612e825179e4b3435613926b513a7b9a07132753: https://github.com/justeat/httpclient-interception/pull/127

Expected behavior

Tests run.

Actual behavior

dotnet test fails with:

Test run for d:\a\1\a\JustEat.HttpClientInterception.Tests.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
Testhost process exited with error: It was not possible to find any compatible framework version
The specified framework 'Microsoft.NETCore.App', version '3.0.0' was not found.
  - Check application dependencies and target a framework version installed at:
      C:\Program Files\dotnet
  - Installing .NET Core prerequisites might help resolve this problem:
      https://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409
  - The .NET Core framework and SDK can be installed from:
      https://aka.ms/dotnet-download
  - The following versions are installed:
      1.0.4 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.5 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.7 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.8 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.9 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.10 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.11 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.12 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.13 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.14 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.15 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.0.16 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.1 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.2 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.4 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.5 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.6 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.7 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.8 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.9 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.10 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.11 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.12 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      1.1.13 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.0.0 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.0.3 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.0.5 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.0.6 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.0.7 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.0.9 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.0 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.1 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.2 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.3 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.4 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.5 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.6 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.7 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.8 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.9 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.1.12 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.2.0 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.2.1 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.2.2 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
      2.2.3 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
. Please check the diagnostic logs for more information.
Results File: D:\a\1\s\tests\HttpClientInterception.Tests\TestResults\VssAdministrator_fv-az635_2019-09-21_15_13_42.trx
Test Run Aborted.

Diagnostic logs

See here.

Environment

Azure DevOps Visual Studio 2019 image.

[martincostello]

Looks like this issue is related to x86 vs. x64, as it seems that a 32-bit PowerShell script will install the x64 version of .NET (because it looks at the machine's architecture), then the SDK will try and use 32-bit version of the SDK (which isn't installed) and then fail.

At least, this is behaviour I'm seeing using TeamCity.

[tannergooding]

I'm seeing this as well even with the x86 SDK being explicitly pulled down and installed by the script (by explicitly passing in -architecture).

An example of this failure (with public CI logs) can be found here: https://github.com/terrafx/terrafx.interop.vulkan/pull/10

You may note that the only change is the upgrade from the last Microsoft.NET.Test.Sdk preview to the 16.3.0 "stable" release and that only the x86 queues are failing. You may also note that the CI machines are pulling down the x86 .NET SDK.

[vagisha-nidhi]

@martincostello Can you please provide more information on what is the exact scenario and how to repro this?

1. Install latest dotnet sdk 3.0.100 from https://dot.net/v1/dotnet-install.ps1 into a custom location with -NoPath: .\dotnet-install.ps1 -InstallDir e:\custom_dotnet -NoPath -Version 3.0.100
2. Is this happening when the sdk installed is x86? (Install the x86 sdk by specifying -Architecture x86 as @tannergooding suggested)
3. What is the target platform of the unit test project(AnyCPU, x86, x64)?
4. Update the Microsoft.NET.Test.Sdk to 16.3.0 stable

Let me know if I am missing anything.

[tannergooding]

@vagisha-nidhi, I've simplified a repro to:

1. Ensure you have a clean machine without .NET Core already installed
   A. the Azure DevOps 'Microsoft-hosted' agents available for OSS seem to fit the bill
2. Download the 32-bit .NET Core SDK for 3.0.100
   A. This could be downloaded manually from get.dot.net
   B. This could be downloaded using the dotnet-install.ps1 script and explicitly specifying -Architecture x86
3. Create a unit test project targeting netcoreapp3.0 with a simple test
   A. This defaults to AnyCPU. I did not check if explicit x86 reproduces, as I need AnyCPU
4. Reference the Microsoft.NET.Test.Sdk, Version=16.3.0
5. Run dotnet test

You can optionally change the Microsoft.NET.Test.Sdk version to the last public preview (16.3.0-preview-20190828-03) and see that running dotnet test works as expected.

[jabberhams]

I have also encountered this error but through Azure Devops Server (actually, TFS on-prem). I used .NET Core Installer Tool Task which downloads dotnet-sdk-3.0.100-win-x64.zip and installs (sets PATH) to the build agent folder (_work_tool\dncs\3.0.100\x64). The subsequent dotnet build task builds all projects - including the test projects - and completes successfully, but the dotnet test task produces the same error at the top of this thread. No other special configuration of the tests projects themselves but my projects all build to x64.

[sfalik]

@jabberhams I have the exact same issue

[shaynevanasperen]

@tannergooding I'm having the same issue. But even when I downgrade to 16.3.0-preview-20190828-03 it fails. This only happens when I run it on TeamCity. When I run the exact same build script on my developer machine, it works fine.

[unaizorrilla]

I have the same issue on AppVeyor

[unaizorrilla]

I fix the issue downgrading Microsoft.Net.Test.Sdk 16.3.0 to 16.2.0, if this can help!

[FeodorFitsner]

Would it be reproducible if .NET Core 3.0 installed with MSI?

[snavarropino]

I fix the issue downgrading Microsoft.Net.Test.Sdk 16.3.0 to 16.2.0, if this can help!

Same issue here after installing visual studio 16.4 (that comes with .Net core 3.1). Downgrading Microsoft.Net.Test.Sdk solved the issue

[rockfordlhotka]

This SO answer might be relevant

https://stackoverflow.com/questions/59246822/after-updated-visual-studio-2019-to-16-4-0-i-cant-run-tests-with-target-framewo/59282608#59282608

[drognanar]

Tagging @cvpoienaru

[nohwnd]

I think we are talking about multiple different issues. First is the inability to find dotnet because it is not in path. And the other is inability to find dotnet of another bitness than the executor because we don't have it installed.

Both seem to be broken for 16.4.0 but partially fixed in 16.5.0

---

The original issue fails to locate the dotnet 3.0 because it was installed with -noPath, and then likely run as this C:\dotnet3.0\dotnet.exe test. This would run the vstestconsole.exe which is netcoreapp2.1 and so it would run just fine. Console would testhost.exe passing on runtime config to use netcoreapp3.0. The test host would try to execute and would run the standard dotnet resolve to find the host -> look into DOTNET_ROOT (probably not defined), look into PATH, look into ProgramFiles. This ended up finding the dotnet in ProgramFiles where only versions up to 2.2 are installed and failed to run.

---

The other issue (the simple repro above), uses one bitness of the console to execute a different bitness of the test host.

I tried this on the latest and I think it was "fixed" in the meantime by taking the path to dotnet.exe from the parent process if it is dotnet.exe as well. This runs nicely, but does not do what is advertised because the test host ends up running as 32-bitwhile it should run as 64-bit. Or the other way around (64bit console and 32-bit host).

[tannergooding]

This runs nicely, but does not do what is advertised because the test host ends up running as 32-bitwhile it should run as 64-bit. Or the other way around (64bit console and 32-bit host).

By console do you mean cmd.exe or vstestconsole.exe? In the case of the latter, does that matter? The intent is for the unit tests to be run under .NET as the targeted architecture, regardless of what the test host (which manages launching the tests and collecting metrics) runs as.

The following should all be valid/invalid:
| OS Bitness | Test Host Bitness | Unit Test Bitness | Valid |
| ---------- | ----------------- | ----------------- | ----- |
| 32-bit | 32-bit | 32-bit | Yes |
| 32-bit | 32-bit | 64-bit | No |
| 32-bit | 64-bit | 32-bit | No |
| 32-bit | 64-bit | 64-bit | No |
| 64-bit | 32-bit | 32-bit | Yes |
| 64-bit | 32-bit | 64-bit | Yes |
| 64-bit | 64-bit | 32-bit | Yes |
| 64-bit | 64-bit | 64-bit | Yes |

Running 64-bits from a 32-bit OS, even if the underlying hardware could support 64-bit is invalid (hardware will be in 32-bit mode, etc).

Running a 32-bit app from a 64-bit OS is supported; largely via SysWOW64 (Windows on Windows).

Having a 64-bit unit tests launched from a 32-bit host on a 64-bit OS is also fine. The unit tests would still be running in 64-bit mode and a 32-bit and 64-bit app can communicate with eachother over IPC or similar. Many of the Visual Studio Out of Process components work this way because VS itself is 32-bits, but compilers and other tools may need to be 64-bits.

[nohwnd]

By console do you mean cmd.exe or vstestconsole.exe?

I meant bitness of the vstest.console / dotnet test. Invoking a 64-bit test project should start a 64-bit test host even if we start it via 32-bit dotnet test. I am not 100% sure what is happening in this case, because I did not analyze it deeper. But it should be pretty easy to see once you take time to create a test project for it.

Having a 64-bit unit tests launched from a 32-bit host on a 64-bit OS is also fine. The unit tests would still be running in 64-bit mode and a 32-bit and 64-bit app can communicate with eachother over IPC or similar. Many of the Visual Studio Out of Process components work this way because VS itself is 32-bits, but compilers and other tools may need to be 64-bits.

Yup that is exactly how it works for vstest console as well. A 32-bit VS runs 64-bit TestWindowStoreHost that runs 32-bit vstest.console that then runs 32-bit or 64-bit test host and they all communicate via sockets 🙂