Azure-pipelines-tasks: Custom agent with only VS Build Tools (not full Visual Studio 2017) - not found

[aarondandy]

Background

We have some custom agents with the VS build tools for 2017 but _without_ full VS2017 installed. These build agents have been working for a while now but we noticed that in the past couple days they no longer are able to build our 2017 projects. We get an error when using the VS 2017 build task of "Visual Studio 2017 was not found." When we attempt to use the MSBuild Task instead we also get an error of "MSBuild version '15.0' was not found for architecture 'x86'. Verify the architecture input value is correct and verify Visual Studio 2017 is installed. MSBuild version '15.0' is included when Visual Studio 2017 is installed." Note that we are able to see the capabilities for 2017 in the agent properties and this has previously worked with no changes to our agent host VMs on our part. I am assuming this is related to a recent update of the agent software.

Reproduction

1. Create a build that uses VS 2017
2. Select an agent pool with the 2017 SDK and MSBuild 15 but _not_ VS 2017
3. Run a build for a VS 2017 project

Expected

The build should execute VS or MSBuild.

Actual

Visual Studio 2017 or MSBuild was not found.

[bryanmacfarlane]

We have some custom agents with the VS build tools for 2017

The task has an option for you to specify the msbuild location for scenarios like this. Look in advanced.

[aarondandy]

I don't see that option. If I did though, are you saying that this is a change in behavior for the task and that our setup is no longer supported unless we install full VS2017? Here is what I see for the task:

[ericsciple]

i'm working on a PR. the workaround is use the msbuild task - it has the advanced option

[aarondandy]

@ericsciple thanks for the reply, I think that's what I was looking for!

[aarondandy]

@ericsciple since you are working this, here is some more context on the bug: Our agent VM has full VS 2015 on it and when I choose "Latest" it will try to build the project using the 2015/14.0 tooling.

[ericsciple]

@aarondandy this oversight on my part. the problem is vswhere only searches for full VS by default. what i'm going to do is fallback to searching for the BuildTools product if full VS isn't found. If you want to pre-validate the fix on your machine, you can verify the following vswhere commandline will find your buildtools install (works on my machine):

vswhere.exe -version "[15.0,16.0)" -products Microsoft.VisualStudio.Product.BuildTools -latest -format json

You can find a copy of vswhere in the task cache in the agent (agent\_work\_tasks\VSBuild_<guid>\<latestver>\vswhere.exe)

[np83]

Running
vswhere.exe -version "[15.0,16.0)" -products Microsoft.VisualStudio.Product.BuildTools -latest -format json
Does yield the MSBuild tools 2017 installed, where as
vswhere.exe -version "[15.0,16.0)
yields none.

A fix would be appriceate because most builds are now failing on our custom build agent.

[BerendWouters]

I encounter the same issue. Running using "Latest" works fine too, but is not ideal.

[np83]

I temporay adjusted the following:
ps_modules\MSBuildHelpers\PathFunctions.ps1
and altered the line where the arguments are passed to
Invoke-VstsTool -FileName "$PSScriptRoot\vswhere.exe" -Arguments "-version [15.0,15.1) -products Microsoft.VisualStudio.Product.BuildTools -latest -format json" -RequireExitCodeZero 2>&1 |
So it includes '-products ..' like @ericsciple said.

That allows us to run builds again, hope this is a usefull temporary fix for others.

@BerendWouters 'Latest' is not working for us, that switches back to VS2015, which is missing some C# 6 features we are using; causing compile errors.

[cornillemichiel]

We encountered exactly the same issue here, latest gets rid of the error, but probably uses VS2015.

[BerendWouters]

@np83 VS2015 should also work with C# 6. We are using among other String Interpolation and it compiles just fine on VS2015 :)

[ericsciple]

a fix is in progress. until the hotfix rolls to all rings, the workaround is:

for agent routing: add user capability on pool admin page.

for task: use msbuild task and advanced option to specify the location of msbuild

[codymullins]

My build failed using VS 2017 as the settings; I switched it to "latest" and it succeeded.

[bryanmacfarlane]

@codymullins - that's a different issue. This one is about private agent without VS sku installed (only build tools). But, to answer your question, on the queue tab you probably have "Hosted Pool" selected which only has VS2013, 2015. So if you selected to use VS2017, it would say not found correctly. If you select Hosted VS2017 queue then you will land on a machine with VS2017 and it would be found. We are aware of the confusion and discussing changes.

[codymullins]

@bryanmacfarlane looking at the settings, it appears you are correct. Thank you for the clarification, I did not realize I needed to select a different hosted pool.

Is there some kind of technical limitation to requiring a separate pool for VS2017 being a valid option in the build's Visual Studio Version drop down? Just out of curiosity.

[bryanmacfarlane]

@codymullins - don't want to get too far off topic for the original issue but to answer the question quickly - yes, the previous approach of stacking all versions on a single VM image is intractable. The "hosted pool" points to a set of machines with VS2010,12,13,15. 17 wouldn't fit so we're moving to a model of era of VS. So we will have VS2017, VS2015, pools and current hosted pool will be deprecated. The queue determines what pool of machines you're using. The dropdown in the msbuild task says find the version of msbuild associated with VSxxxx I chose. So, if you choose hosted pool (vs2015, etc..., no vs2017) and the msbuild task says to find use vs2017's msbuild, it's not found. Big usability problem. We're going to make next major versions of task default to PATH/Latest, have a queue/pool per VS version, and then put the queue choice more up front when creating def instead of hiding it in options (used to make sense hiding as advanced option of old world with an image that contains all tools).

[aarondandy]

~I could be wrong but this may also be impacting the VSTest tasks as well.~ Nope, the test issue was something else with the environment and our fault.

[jawn]

Hey @aarondandy , can we fix the typo in the issue title: Cusom -> Custom ?

[drub0y]

I hate to be negative, but we're going on a week here guys and still no fix? 😞

Do you realize how many builds we would have to update with these proposed workarounds because of this? Longer term I don't want them to be defined that way, I just want the build to be defined the way it is using the regular build task with no customizations so then we'd have to spend time rolling back the definitions. Therefore it's actually just easier for us to retarget to a build agent that has full VS2017 installed, but even that cost us all time to stand up, then all the teams have to retarget all of their builds (that require 2017... which is quite a few) _and_ we're effectively funneling everything down to one agent as opposed to the pool of _N_ we had set up already. I have to imagine our company pales in size compared to some of your bigger customers too. Are you really not hearing this from more people??

So I have to ask: how was this not an immediate stop and fix (or rollback)? 😢

[bryanmacfarlane]

The tasks have been fixed and they are rolling out scale unit by scale unit in a safe deployment manner. It should hit yours within the next couple of days. Our patches and deployments go through rings.

how was this not an immediate stop and fix (or rollback)?

It was an immediate fix but we still roll it out over days so we don't make it worse and break mainline (VS skus installed). The vast majority of folks have full VS installed and/or use the hosted pools. There's also a workaround.

If it was blocking (no workaround) and/or affected a large swath of users we would disregard deploying by rings and blast out everywhere.

[ericsciple]

@drub0y i've now added a patch available here you can apply without waiting for the rollout through the rings

[drub0y]

@bryanmacfarlane Well thank you very much for that explanation. That helps us understand the process and decision making for this kind of issue in the future.

I guess what this really brings to light for me is something I've worried about for a while now which probably warrants a separate discussion (if there's not one already going on somewhere) which centers around the ability to _only_ peg yourself to a major version of a task (e.g. 1.*) instead of being able to peg yourself to an exact version of a task. While I'm a big fan of this semver style of opting in to auto versioning around a major, mistakes like this can happen, so in a scenario like this if we were able to have gone back and lock into the more exact, previous version when a bug like this arises certainly seems like it might be a good feature. Still, you wouldn't want to have to go do this on every build in a scenario like this, so you'd have to be able to manage build task versions at the project level at the very least... obviously account wide would be great too. Food for thought! :smile:

[drub0y]

@ericsciple Appreciate that! We'll give that a look. Can you tell me tho, if we upload this version now, will future versions still continue to automatically update?

[bryanmacfarlane]

@drub0y - yes, you will get future updates. tasks you upload are overlayed and unioned with the in-the-box ones and sorted by version decending. you can also delete the task using tfx and it will delete the overlay.

regarding pinning to exact versions - yes, lots of discussion and considerations (too long to go into here). The good news is we will offer yaml preview in a few sprints (in dev now) and it will allow you to lock down not only the process but check-in tasks to source control and completely lock. You will also be able to specify major lock from server, point to github ref etc... We will build our tasks and have a repo with post built tasks ready to reference.

https://youtu.be/lkPnv5H2ANI

[discorev]

@bryanmacfarlane - Using tfx I've verified that our VSO service has VSBuild 1.118.0 however the new build server with agent version 2.116.1 is still not reporting MSBuild 15.0 under system capabilities.

I have tried adding this under user capabilities but queued build jobs are still not being assigned to this agent. Is there anything I can do to get jobs being assigned to this agent?

[ericsciple]

@discorev you may be missing a different capability. what i would do is add a queue-time demand for the specific agent name. then if it fails to find an agent that satisfies all of the demands, you will get an error that lists all of the demands

[ericsciple]

also i believe the agent fix is included in the 117.1 release: https://github.com/Microsoft/vsts-agent/releases/tag/v2.117.1

[ericsciple]

The task hotfix has rolled everywhere. And the agent update is available now on every scale unit, or nearly every scale unit I believe. To update the agent, click to update the agent from the context menu for the agent on the agent pool admin page.

[JiriZidek]

Upgrading to TFS 2017 U2 solved this for me, I needed to manually reinstall build agent from 2.212 to 2.217.2

[ohadschn]

@bryanmacfarlane is there a way to pin exact task versions today (either designer or YAML)? If so, please point me to some docs/samples...