Azure-pipelines-tasks: Nuget Installer looking for msbuild.exe on Mac

[jsauvexamarin]

I have a Xamarin.Android build that was working fine for many days. Then it suddenly started failing on the Nuget Restore (Installer) task with the following:

https://gist.github.com/jsauvexamarin/1bb9c3242b40560023e3588ab7c1e2d0#file-gistfile1-txt-L134

Indeed MSBuild.exe does not exist at that location because this is a Mac build agent. xbuild.exe, however, does indeed live at that location and is what I would think is the correct executable when running on a Mac.

I couldn't find the change in the Nuget Installer task that may have caused this.

I've had changes to tasks break my builds other times in past. Have the VSTS agent / task teams considered implementing task versioning? As in: when you add a task to a build def, it is automatically given the latest version of a task. And the referenced version of the task would stay locked, unless manually adjusted. That way, when the VSTS agent team updates a task in a way that breaks builds for customers, that break will not affect CI processes that have been otherwise working great. You could even include a setting in the task to allow auto-updating of the version, so that if someone WANTS to accept possibly-breaking changes (and improvements) to their build process, they can.

[ericsciple]

Task version locking is planned for next sprint (locked on major version - minor/patch version will still get latest).

Which task is busted the Xamarin.Android task or the NuGet Install task? I can help route to the appropriate team, but I'm not sure which task.

[ericsciple]

Sorry I see now that it's the NuGet Installer task.

[jsauvexamarin]

The build task sequence never makes it to the Xamarin.Android task. It fails on the NuGet Installer task. The output I linked is the detailed verbose output.

[DanRigby]

For reference the error in the build log appears as follows:

2016-10-06T01:51:43.6203630Z ##[section]Starting: NuGet restore
2016-10-06T01:51:44.0784240Z Detected NuGet version 3.3.0.212 / 3.3.0
2016-10-06T01:51:44.0820360Z SYSTEMVSSCONNECTION exists true
2016-10-06T01:51:44.0891890Z ##[warning]To connect to NuGet feeds hosted in your Team Services account/TFS project collection with NuGet 3.1 or below, edit your build definition to specify a path to a NuGet.config containing the package sources you wish to use.
2016-10-06T01:51:44.0900150Z [command]/usr/local/bin/mono /Users/drigby/VstsAgent/_work/_tasks/NuGetInstaller_333b11bd-d341-40d9-afcf-b32d5ce6f23b/0.2.17/node_modules/nuget-task-common/NuGet/3.3.0/NuGet.exe restore -NonInteractive /Users/drigby/VstsAgent/_work/2/s/App/Acquaint.XForms.sln
2016-10-06T01:51:44.5289250Z MSBuild auto-detection: using msbuild version '4.0' from '/Library/Frameworks/Mono.framework/Versions/4.6.1/lib/mono/4.5'.
2016-10-06T01:51:44.5710630Z MsBuild.exe does not exist at '/Library/Frameworks/Mono.framework/Versions/4.6.1/lib/mono/4.5/msbuild.exe'.
2016-10-06T01:51:44.5978520Z ##[error]Error: /usr/local/bin/mono failed with return code: 1
2016-10-06T01:51:44.5988780Z ##[error]Packages failed to install
2016-10-06T01:51:44.6748540Z ##[section]Finishing: NuGet restore

[jsauvexamarin]

I'm thinking _maybe_ my problem could be related to using an older version of the Mac build agent. Running some tests now to check.

...because the peculiar thing is that one Android project is seeing this problem, but another is not. Both projects are part of the same solution. One is a Xamarin native project. The other is a Xamarin Forms project. The Forms one is failing.

But I maintain that I started seeing these build failures start on my master branch nightlies, even though I hadn't committed to that branch for a couple weeks.

[madhurig]

I believe this is a known bug in NuGet that will be fixed in the final NuGet 3.5 release. What version of Xamarin do you have on your Mac? You can try specifying the path to NuGet.exe and the task should run it under Mono.

[jsauvexamarin]

Okay, yeah, super weird. Just created two brand new build defs that do nothing more than restore Nugets and build Android. One is for my Xamarin native project. The other is for my Forms project. The Xamarin native project succeeds, the Xamarin Forms one fails.

This is on a brand-new build agent machine running version 2.107.0 of the Mac build agent (https://github.com/Microsoft/vsts-agent/releases/tag/v2.107.1).

[jsauvexamarin]

@madhurig, this particular build machine is brand new and has all the newest versions of Xamarin stuff on it.

Specifically,

Xamarin Studio Community
Version 6.1.1 (build 15)
Installation UUID: 10e5c1c9-2e06-4f5f-ba71-714e249981a1
Runtime:
    Mono 4.6.1 (mono-4.6.0-branch-c8sr0/abb06f1) (64-bit)
    GTK+ 2.24.23 (Raleigh theme)

    Package version: 406010003

NuGet
Version: 3.4.3.0

Xamarin.Profiler
Not Installed

Xamarin.Android
Version: 7.0.1.2 (Xamarin Studio Community)
Android SDK: /Users/admin/Library/Developer/Xamarin/android-sdk-macosx
    Supported Android versions:
        6.0 (API level 23)

SDK Tools Version: 25.1.2
SDK Platform Tools Version: 24.0.0
SDK Build Tools Version: 23.0.2

Java SDK: /usr
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

Xamarin Android Player
Not Installed

Apple Developer Tools
Xcode 8.0 (11246)
Build 8A218a

Xamarin.iOS
Version: 10.0.1.8 (Xamarin Studio Community)
Hash: 3983064
Branch: cycle8-sr0-xi
Build date: 2016-09-23 15:34:54-0400

Xamarin.Mac
Version: 2.10.0.103 (Xamarin Studio Community)

Build Information
Release ID: 601010015
Git revision: fa52f02641726146e2589ed86ec4097fbe101888
Build date: 2016-09-22 08:03:02-04
Xamarin addins: 75d65712af93d54dc39ae4c42b21dfa574859fd6
Build lane: monodevelop-lion-cycle8-sr0

Operating System
Mac OS X 10.11.6
Darwin DXU615 15.6.0 Darwin Kernel Version 15.6.0
    Mon Aug 29 20:21:34 PDT 2016
    root:xnu-3248.60.11~1/RELEASE_X86_64 x86_64

[jsauvexamarin]

@madhurig, this line seems to indicate that Nuget.exe is found successfully, unless I'm reading it wrong: https://gist.github.com/jsauvexamarin/1bb9c3242b40560023e3588ab7c1e2d0#file-gistfile1-txt-L132

It's two lines down on 134 that's looking for MSBuild.exe for some reason.

[madhurig]

The bug was in NuGet when it looks for MSBuild: https://github.com/NuGet/Home/issues/3085. I see NuGet task allows passing in a NuGet version or a custom path. It looks like you are using 3.3.0, can you see if using any other options works.

[jsauvexamarin]

Aaah! So, perhaps when I updated my old build agent machine to the latest version of Xamarin, it also updated NuGet, which in turn introduced this issue. I'm thinking that the failure start date may coincide with the date on which I updated my build agent.

It doesn't quite explain why my Xamarin.Android Forms project in particular won't build, but it's a step in the right direction towards finding a solution.

I'll see if I can manually specify an older NuGet version.

[madhurig]

Ah it looks like the NuGet task is using a version of Nuget.exe they package with the task.. You can try using the version that ships with Xamarin and might have better success - it should under /Library/Frameworks/Mono.framework/Versions/4.6.1/lib/mono/nuget/NuGet.exe

[jsauvexamarin]

That was most definitely the correct workaround, provided that Mono 4.6.1.x is installed. Thanks @madhurig!

[madhurig]

@zjrunner: FYI .. this has come up several times recently with Xamarin builds on Mac. Not sure what the right solution for the Mac case.

[jsauvexamarin]

Note that I was seeing this issue in Xamarin.Forms-based Xamarin.Android projects, but not in Xamarin-native Xamarin.Android projects. I'm not sure if that's because of a dependency of some kind? Like maybe Xamarin.Forms requires a specific version of Nuget? Or it might be something else in the project. Not sure. But it's worth investigating.

[zjrunner]

@jsauvexamarin Do you have previous working logs that tell you what version of nuget was getting called? The problem here seems to be agent inconsistencies. The windows agents all shipped with nuget, and the version that was used in the tasks was 3.3 because of that. We started shipping in-task because of race conditions on upgrade and so kept 3.3 as the default (with the 3.5 option added).

It seems the Mac agents were setup differently and used custom in-box versions _without_ the user specifying a custom version (in the Windows agent, even if someone updated the version the agent deploys, it would be overwritten the next time the agent updated).

While I'm happy to be consistent across agents now in the defaults, it is unfortunate it caused a break here. We really want to go to 3.5 as the baseline but can't until we get the major version rev'ing that @ericsciple mentioned earlier so people can opt-in and not break on auto-upgrade in that transition.

At this point, adding the custom nuget path is the right answer. If we had caught this up front we might have been able to give a warning in the build for 3.3 on mac. At this point we should be well through the deployment impact and I'd be worried about that warning just causing more confusion if 3.3 does in fact work in some cases. Would like to wait for the 3.5 migration unless this is still causing lots of issues.

[haydenhancock]

I seem to be having a similar issue with a Xamarin.Forms-based project but in Windows.

Error: C:Agent_work_tasks\NuGetInstaller_333b11bd-d341-40d9-afcf-b32d5ce6f23b\0.2.17\node_modules\nuget-task-common\NuGet\3.3.0\NuGet.exe failed with return code: 1
Packages failed to install

There are no other details in the build log.

[JimBlizzard]

I had not changed a thing with my VSTS build definition, the Mac VSTS build agent, or the source code. I last ran the build a couple of weeks ago and it was successful every time. When I tried to run a build this morning I received these errors during the NuGet package restore step

In the _diag log file on the mac
"MsBuild.exe does not exist at '/Library/Frameworks/Mono.framework/Versions/4.4.2/lib/mono/4.5/msbuild.exe'."

and in the build summary in VSTS
"To connect to NuGet feeds hosted in your Team Services account/TFS project collection with NuGet 3.1 or below, edit your build definition to specify a path to a NuGet.config containing the package sources you wish to use."

I don't have Mono.framework/version 4..6.1 on my Mac, so I set the customer NuGet config in the build step to this
"/Library/Frameworks/Mono.framework/Versions/4.4.2/lib/mono/nuget/NuGet.exe"

This worked. I've never had to use a custom path before.

What happened?

Thanks,
bliz

[jsauve]

You can also use:
"/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/nuget/NuGet.exe"

Note the "Current" instead of a specific version number. There should be a symlinked "Current" folder in there that always points to the most recent version.

[jferlazzo]

Is anyone else having an issue with the version of NuGet being used?

When specifying the paths mentioned above:

• "/Library/Frameworks/Mono.framework/Versions/4.6.1/lib/mono/nuget/NuGet.exe"
• "/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/nuget/NuGet.exe"

It uses "Detected NuGet version 2.12.0.0 / undefined". This is regardless of if I choose the NuGet version in the build step. The path provided overrides version selection.

Unfortunately as I have a PCL project targeting a .NET Standard in my solution I need NuGet 3.5 to restore the project.json dependencies.

My Android project uses "Detected NuGet version 3.5.0.1829 / 3.5.0-rtm-1829" and seems to restore both the Android packages.config and PCL project.json.

I need to be doing the same for the iOS solution but it only seems to be doing the packages.config for iOS project due to what I assume is the NuGet version being used.

Just wondering if anyone else has come across this issue and found a work around? Or if I am doing anything wrong?

Thanks.

[zjrunner]

The detection there is internal to the task to determine 'quirks' to apply. It lets us know if the version of nuget we found supports a credentialProvider, support OnPrem(TFS) auth, or needs to fall back to nuget.config management for VSTS Auth.

Are the files you are referencing actual nuget.exe files with the same version resources, or are they shell scripts that redirect to mono+nuget.exe executions? If the latter, we won't be able to read a version resource from it and will default to the base-support you see, which means nuget.config IF we need to do auth. Note that this shouldn't alter the version of nuget we're using -- you should still see us call the version of nuget.exe you've given us so you may want to check what is actually on this machine (vs what you're calling on the Android builds)

[jferlazzo]

It seems there is also some issues with the mac installer for dotnet core sdk not adding the tools to the path.

In case anyone is having both the Nuget Restore issue above and also has dependencies to restore in a project.json file. I solved the problem by adding two build steps below:

1. Command Line - NuGet Restore
   Added the NuGet.exe file with the version I needed to my respository.
   Set Tool as "nuget".
   Set Arguments as "restore {Path to Solution}".
   Set Working Folder as path to Nuget.exe.
2. Shell Script - Dotnet Restore
   Added this script to my repository.
   Set the Script Path to location of the shell script.
   Script: /usr/local/share/dotnet/dotnet restore {Path to Project Folder}/project.json

[eorengo]

Hi there,
Any workaround for this issue?
I found no way to let nuget know that msbuild.exe does not exist at /Library/Frameworks/Mono.framework/Versions/Current/lib/mono/msbuild/15.0/bin/xbuild.exe
Thanks!

2017-12-29T20:17:10.9006990Z ==============================================================================
2017-12-29T20:17:10.9019100Z Task : NuGet Installer
2017-12-29T20:17:10.9030840Z Description : Installs or restores missing NuGet packages
2017-12-29T20:17:10.9042510Z Version : 0.2.29
2017-12-29T20:17:10.9054090Z Author : Microsoft Corporation
2017-12-29T20:17:10.9065910Z Help : More Information
2017-12-29T20:17:10.9077780Z ==============================================================================
2017-12-29T20:17:11.3786380Z SYSTEMVSSCONNECTION exists true
2017-12-29T20:17:11.3819060Z [command]/Library/Frameworks/Mono.framework/Versions/Current/bin/nuget restore -NonInteractive /Users/tecnweb/myagent/_work/1/s/HelloXamarinFormsWorld.sln -NoCache -Verbosity Detailed
2017-12-29T20:17:11.9372600Z NuGet Version: 4.3.0.4406
2017-12-29T20:17:11.9510210Z MSBuild auto-detection: using msbuild version '15.0' from '/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/msbuild/15.0/bin/'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
2017-12-29T20:17:12.2669060Z MSBuild P2P timeout [ms]: 120000
2017-12-29T20:17:12.2728510Z NuGet.CommandLine.CommandLineException: MsBuild.exe does not exist at '/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/msbuild/15.0/bin/xbuild.exe'.
2017-12-29T20:17:12.2741350Z at NuGet.CommandLine.MsBuildUtility+d__6.MoveNext () [0x00048] in :0