Xamarin-macios: Application with multiple architectures (ARMv7 and ARM64) successfully builds by using real Xcode, but it is failed by using symlink to real Xcode on macOS Catalina.

[vsafonkin]

Steps to Reproduce

1. Create symlink to Xcode: ln -s "path/to/xcode" "/Applications/Xcode.app"
2. Switch Xcode to symlink: sudo xcode-select -s /Applications/Xcode.app/Contents/Developer"
3. Create an iOS project, makes sure it has multiple architectures selected (and device-specific builds disabled), and build it.

Expected Behavior

Successfully build.

Actual Behavior

Build is failed:

lipo : error : sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find lipo 2> /dev/null' failed with exit code 16384: (null) (errno=No such file or directory) 
xcode-select: Failed to locate 'lipo', requesting installation of command line developer tools.

MTOUCH : error MT5306: Failed to create the a fat library. Please review the build log.

The command line tools is installed: sudo xcode-select --install

xcode-select: error: command line tools are already installed, use "Software Update" to install updates

Environment

macOS 10.15.3 Catalina
Xamarin.iOS 13.10
Mono 6.6
Visual Studio for Mac 8.4.5.19

Workarounds:

• set environment variable export SDKROOT=macosx
  or
• once run lipo before build /usr/bin/lipo 2>null>

[rolfbjarne]

What's the output of running the following two commands in a terminal?

xcode-select -p 
cat ~/Library/Preferences/Xamarin/Settings.plist

[vsafonkin]

Hey @rolfbjarne ,

xcode-select -p
/Applications/Xcode.app/Contents/Developer

cat ~/Library/Preferences/Xamarin/Settings.plist
No such file or directory

I use Azure Devops Xamarin.iOS task for building, probably, it cause Settings.plist file is absent.

[rolfbjarne]

Can you get a diagnostic build log (pass /verbosity:diagnostic to msbuild)? That might give us some ideas.

[vsafonkin]

build_log.txt

[rolfbjarne]

Unfortunately that didn't help much :/

• Does it also work to execute xcrun -k before building (as a workaround)?
• Can you reproduce the failure without involving building by executing lipo manually just after setting up Xcode? Something like this:

$ ln -s "path/to/xcode" "/Applications/Xcode.app"
$ sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
$ /usr/bin/lipo -detailed_info `which echo`
# Does this last command fail?

[vsafonkin]

xcrun -k before building doesn't work, build is failed with the same error.

$ ln -s "path/to/xcode" "/Applications/Xcode.app"
$ sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
$ /usr/bin/lipo -detailed_info `which echo`

input file /bin/echo is not a fat file
Non-fat file: /bin/echo is architecture: x86_64

[rolfbjarne]

I'm running out of ideas. @spouliot @chamons do have any thoughts?

[spouliot]

Nothing obvious reading the logs :(

Maybe we (mtouch) depends on some stuff that VSfM would provide ? (environment variables) Unlikely since we build from command-line all the time.

Maybe some other config on the bots are conflicting with our logic ? again I don't see how the workaround helps

[rolfbjarne]

The fact that running lipo once before the build works around the problem makes me think this is something in macOS that's not quite working as expected.

@vsafonkin would you be able to install custom debugging versions of Xamarin.iOS to figure this out?

[vsafonkin]

@rolfbjarne , yes, sure.

[rolfbjarne]

@vsafonkin can you try this package to see if it still fails: xamarin.ios-13.19.0.136.pkg?

[vsafonkin]

Hey @rolfbjarne , sorry for my delay, I've tried this package and build was successful.
Full build log (with /verbosity:diagnostic option):
xamarin.ios-13.19.0.136-build_log.txt

[rolfbjarne]

That's great news @vsafonkin! I'll clean up that branch and try to get the fix in.

[IainS1986]

Sorry if this is actually off topic, but I've just run into this same error on Azure Devops hosted mac agents.

Targeting Xamarin SDK 6_6_0.

Is this actually an issue in Xamarin SDK that needs to eventually make its way into the Azure VMs? Azure seems to be on 13.10.0.21 https://github.com/microsoft/azure-pipelines-image-generation/blob/master/images/macos/macos-10.15-Readme.md

Should I roll back to 6_4_0 or something in Azure script to work around for now, or is this similar but actually unrelated? (I assume Azure does a Symlink under the hood...)

EDIT - Tried targeting 6_4_0 but got the same issue with failed to locate 'lipo'

EDIT - Setup a task in my job on azure to use boots to download+install the package linked above so it builds with 'Xamarin.iOS 13.19.0.136 (lipo-where-are-you)' and the build succeeded 👍

[chandu612]

@rolfbjarne , I have the same issue and waiting for the fix to come through. If I understand correctly, the fix is in Xamarin.ios for the azure hosted macs. I am using macOS-10.15 hosted mac for the Azure dev ops pipeline. I still see the same issue. Am I supposed to use the latest pipeline task or something?
Thanks

[rolfbjarne]

@chandu612 the current plan is to release this fix as a stable release early next week, after that you might have to wait for the hosted macs to be updated, which might take a while longer. Alternatively you can manually install a package with the fix as the previous commenter did.

[IainS1986]

@chandu612 If you don't have the ability to add boots to your pipeline, use this task (I'm executing it as the first task of the build job)

```

• task: Bash@3
  displayName: 'Lipo Workaround'
  inputs:
  targetType: 'inline'
  script: |
  dotnet tool install --global boots
  boots https://bosstoragemirror.blob.core.windows.net/wrench/jenkins/lipo-where-are-you/b6543fb26def367908b7cea43e1ca33ee8c99451/1/package/notarized/xamarin.ios-13.19.0.136.pkg
  ```

Only adding around 1 minute to the build.

[chandu612]

@rolfbjarne and @IainS1986
That worked. thank you guys

[fraxedas]

We just tried and it worked too.
In which release is it going out?
Is there a way to track if this fix is in the hosted agent for Azure Devops?

[chamons]

You need to contact your hosting provider and ask them, I believe the bits in question are already in our stable channel.

Until then, you can install a package manually, as noted via boots above.