Azure-pipelines-agent: vstest capability not detected when installed without Visual Studio

[robinhultman]

Agent version and platform

Version of your agent? v2.114.0


OS of the machine running the agent? Windows

I have a custom build server and I have installed the test agent from https://www.microsoft.com/en-us/download/details.aspx?id=48152 without installing Visual Studio.

As far as I can see the vsts capability is added by checking (for VS2015) for the existence of the registry value of InstallDir or ShellFolder in
Software\Microsoft\VisualStudio\14.0

But that entry does not exist if the agent is installed separately.
Instead it would seem that there is an entry under InstallDir in Software\Microsoft\VisualStudio\14.0\EnterpriseTools\QualityTools

Would it be a good solution to add that registry entry to the powershell script that scans for this capability?

It is possible to add the capability manually.

[PBoraMSFT]

@robinhultman - being able to run tests when Test platform is obtained through the Test Agent is on the backlog.

[Elyseum]

In case you want to hotfix this yourself, add a file called _Add-VisualStudioTestCapabilities.ps1_ with code (check for your path)

[CmdletBinding()]
param()

if (Test-Path "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow") {
    Write-Capability -Name 'VSTest' -Value "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow"
}

to _C:vstsagentA1binpowershell_.
Restart the agent service to get the capabilities updated.

[ericsciple]

i would recommend add the user capability (in the admin UI) since the bin folder will be replaced when the updates.

[Elyseum]

I tried that, but my Deployment Groups agent appears offline in the Agent Queues screen. I've added the capability there, but it didn't show up when I tried to release it.

[ericsciple]

@RoopeshNair @kmkumaran for deployment group issue - i'll send an email to them offline as well

[ericsciple]

@Elyseum sorry i realize now your issue sounds like a separate issue, you may want to open a separate issue

[ericsciple]

@cltshivash for the VSTest capability issue

[InteXX]

@ericsciple — How to add this capability manually? Also: is this an expected fix that might have a target date for release? If it's something coming up fairly soon, I don't mind waiting for it.

I'm running the latest (as of today) release v2.120.2. Installation & config just now added visualstudio, but not vstest.

[ericsciple]

https://www.visualstudio.com/en-us/docs/build/concepts/agents/agents#capabilities

[InteXX]

@ericsciple — I saw that earlier, thank you, but it doesn't appear to provide guidance on how to manually add a capability (particularly the vstest capability). Unless I'm missing it somehow?

[ericsciple]

You can either set an env var and restart the agent, or from the admin web UI, select an agent, select the Capabilities tab (right-most pane), and you should see an Add capability button near the top in the right-most pane.

[InteXX]

@ericsciple — What should the variable's value be?

[InteXX]

@ericsciple — I think I found it. Is this it? C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow

[ericsciple]

It doesn't matter, build/release job routing only checks existence of the variable. The task will attempt to find the tool (does not consider capability value).

[ericsciple]

...just make sure the value is not empty

[InteXX]

@ericsciple — So I'll change it to "Eric's Bar & Grill" instead. That ought to do it.

Any chance of getting this into System Capabilities?

[ericsciple]

Ha, yes anything should work.

I'll defer to @nigurr @RanjanarMS to answer the System Capabilities question.

[InteXX]

@ericsciple — OK, thanks.

[nigurr]

@InteXX
Can you please give this info?

1. Visual Studio versions installed in the machine
2. Is VisualStudio capability added manually or automatically added?

[InteXX]

@nigurr:

1. VS 2017 Build Tools (only, no IDE)
2. Automatically

Also, if it helps, I'm finding that vstest.console.exe is not present on my system; apparently it's required as seems to be indicated here.

I haven't yet upgraded to TFS 2017.2, but I plan to do so sometime during the week of the 11^th. Presently I'm on 2017.1.

[InteXX]

@nigurr — I've installed the 2017.2 update, but I'm still not getting a VSTest System Capability with a newly created agent.

I also still don't have a file vstest.console.exe on my server. I don't know whether this is relevant.

[nigurr]

@InteXX you need to install VS IDE. VS build tools doesn't contain the "test platform". In order to run the tests, you require to install full IDE.

[InteXX]

@nigurr — I'd rather not if I don't have to. In fact, I've been under the impression that removing the IDE requirement was a high priority for TFS2017. Would you comment on @ericsciple's earlier advice, that I should be able to run a Test step with a Custom Capability?

[InteXX]

@nigurr — If I add a Custom Capability VSTest and copy this from my dev machine:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow\*.*

...to this on my server:

C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow

Will it work? Will I be able to run a Test Step without the IDE?

[InteXX]

@nigurr — Would you consider reopening the ticket... there's still relevant discussion to be made on the issue.

[InteXX]

@PBoraMSFT — _"...being able to run tests when Test platform is obtained through the Test Agent is on the backlog."_ Is there any status update on this? Will we be able to know when it's released?

[nigurr]

@InteXX we understand your concern and will be addressing it soon (it's on our backlog). Currently to run the VsTest task, the task has dependency on Visual Studio IDE.
We recommend to install VS IDE to run the tests.

[InteXX]

@nigurr — That's that then :-) Thanks, Nitin.

[PBoraMSFT]

@Intexx - we are picking up this work now. It is expected to be available on VSTS in a few sprints. We have decided to go with a 'tools installer' which can be used to get the test platform instead of needing the IDE. If you have the IDE, great. If you don't, the tools installer will get the test platform on the agent for you.

[InteXX]

@PBoraMSFT — Thanks for the update. It'll be a bit longer for those of us on TFS. So... IDE installation it is. Appreciate all that you do.

[AndersSahlin]

@PBoraMSFT any estimate on when the tools installer will be available?

[PBoraMSFT]

@AndersSahlin - we are working on it this sprint (126) and the first preview of the tools installer task should be available when Sprint 126 deployment on VSTS rolls out.
Will be announced in the release notes at https://docs.microsoft.com/en-us/vsts/release-notes/

[Lexcess]

@PBoraMSFT Will there be any alternative for private agents, so that vstest can be added as a capability rather than installed on demand? i.e. by installing Test Agent.

[SympaHannuPiki]

@PBoraMSFT: Could you say if this is related to following VSTS release note item: "Make test platform bits xcopyable to remove VS dependency in CI/CD workflows"? Just trying to figure what item to follow.

[PBoraMSFT]

@Lexcess, @SympaHannuPiki - this has already shipped with Sprint 126.
https://docs.microsoft.com/en-us/vsts/release-notes/2017/nov-28-vsts#test