Source-build: macOS PR validation smoke-test failure: libicu not installed/detected in smoke-test

[dagood]

https://dev.azure.com/dnceng/public/_build/results?buildId=742764&view=logs&j=dd0961f5-eb75-5263-969b-8fed4b0393f9&t=e3c3f94f-154c-5989-8736-91280ba7f044&l=75

  SDK under test is:
  Process terminated. Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
     at System.Environment.FailFast(System.String)
     at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()
     at System.Globalization.GlobalizationMode..cctor()
     at System.Globalization.CultureData.CreateCultureWithInvariantData()
     at System.Globalization.CultureData.get_Invariant()
     at System.Globalization.CultureInfo..cctor()
     at System.Version.TryParseComponent(System.ReadOnlySpan`1<Char>, System.String, Boolean, Int32 ByRef)
     at System.Version.ParseVersion(System.ReadOnlySpan`1<Char>, Boolean)
     at System.Version.TryParse(System.String, System.Version ByRef)
     at Microsoft.DotNet.PlatformAbstractions.Native.PlatformApis.GetDarwinVersion()
     at Microsoft.DotNet.PlatformAbstractions.Native.PlatformApis.GetOSVersion()
     at Microsoft.DotNet.PlatformAbstractions.RuntimeEnvironment..cctor()
     at Microsoft.DotNet.PlatformAbstractions.RuntimeEnvironment.GetRuntimeIdentifier()
     at Microsoft.DotNet.Cli.MulticoreJitProfilePathCalculator.CalculateProfileRootPath()
     at Microsoft.DotNet.Cli.MulticoreJitActivator.StartCliProfileOptimization()
     at Microsoft.DotNet.Cli.MulticoreJitActivator.TryActivateMulticoreJit()
     at Microsoft.DotNet.Cli.Program.Main(System.String[])
  ./smoke-test.sh: line 399: 86759 Abort trap: 6           "$dotnetDir/dotnet" --info
/Users/runner/work/1/s/build.proj(230,5): error MSB3073: The command "./smoke-test.sh --minimal --projectOutput --configuration Release --archiveRestoredPackages --excludeWebHttpsTests" exited with code 134.

This seems to be flakiness: the PR that hit this didn't change anything about this job, and the "Initialize job" step appears identical to an earlier PR validation run that failed. Time will tell.

[dagood]

This may not be flakiness... happened 3 times in a row on https://dev.azure.com/dnceng/public/_build/results?buildId=742764&view=results. Pool configuration change maybe? FYI @dseefeld @crummel

[dseefeld]

@dotnet/dnceng Is this a known issue or configuration issue for OSX images?

[MattGal]

Taking a peek

[MattGal]

Couple things:

• Checking out https://github.com/actions/virtual-environments, it doesn't look like they've ever consciously been installing libICU on the hosted macOS machines.
• There is a movement currently to move from 3rd to 1st-party hosted mac machines; the agents running on them would be the same but dependencies like this may have been missed.
• The hosted macOS builds do run on VMs, so if you can feasibly bootstrap the version you want, party on.

I need to talk to folks in AzDO about their timeline of moving to the "new" macs; this was supposed to be completed last week but is still ongoing. This is likely your problem as I see the same PR building on different days hitting the problem and not.

[MattGal]

I chatted with @safern and @tarekgh who may want to weigh in here. There has been code change in this area that you may be missing.

Please see https://github.com/dotnet/runtime/pull/39900, specifically this commit.

[safern]

Just to add to what @MattGal mentioned, the extended version of the fix in order to support building runtime in macOS Big Sur is: dotnet/runtime#39900

[dagood]

@safern will that be ported to 3.1 where we hit this error? Is there a tracking issue?

[safern]

@safern will that be ported to 3.1 where we hit this error? Is there a tracking issue?

I believe @ViktorHofer was porting that to downstream branches.

[dagood]

A few more bits to connect the dots between this and https://github.com/dotnet/runtime/pull/39900: it looks like originally this problem came up in https://github.com/dotnet/core-eng/issues/10304, which shows up exactly the same way it does for us. The initial fix was https://github.com/dotnet/runtime/pull/39833:

To sum up the findings, this is because of the recent cmake upgrade to 3.18 that got pushed to brew with https://github.com/Homebrew/homebrew-core/commit/7927380833f30b0db83eb6de2e7587b82cdf25d0.

A build that ran 8hours ago (presumably with cmake 3.17.2) still got the expected /usr/lib/libicucore.dylib, a build from 4hours ago got the Xcode path.

Filed an issue with CMake: https://gitlab.kitware.com/cmake/cmake/-/issues/21007

Then https://github.com/dotnet/runtime/pull/39900 looks like an improved fix in line with new macOS changes.

[safern]

Then dotnet/runtime#39900 looks like an improved fix in line with new macOS changes.

@ViktorHofer ported it to coreclr3.1 which is the one that builds S.Globalization.Native in 3.1. https://github.com/dotnet/coreclr/commit/d3a7344cd4b5c1479b0ab325a1b17eaf7ce9be7a

[safern]

Note that only the simple fix was ported. If we need to build on Big Sur for 3.1 we can also port the complete fix.

[ViktorHofer]

The fix was backported into the 2.1, 2.2 and 3.1 branches. Here's the change for 3.1: https://github.com/dotnet/coreclr/commit/d3a7344cd4b5c1479b0ab325a1b17eaf7ce9be7a.

[dagood]

Ah, expected some backlink to show up for that at least. 😄 For future reference:
2.1 https://github.com/dotnet/coreclr/pull/28067
3.1 https://github.com/dotnet/coreclr/pull/28066

Thanks, we should uptake those changes eventually. When push comes to shove, source-build doesn't currently have a critical reason to build on macOS at all, so no worries about building on Big Sur as far as we're concerned.

[dagood]

Responding to @asbjornu from https://github.com/dotnet/sdk/issues/11795#issuecomment-675158663:

But before dotnet/source-build#1685 is resolved, it's hard to tell. Is it possible to have that fast-tracked somehow, @dagood?

Not in the "correct fix" way, because there isn't a build of .NET Core yet that incorporates that fix. Any workaround we add now would have to be reverted once we get a build. (It's not feasible to rush a build.)

You could lower your CMake version to emulate how we managed to build the last release on macOS before the CMake breaking change (from our project's PoV).

Or you should be able to fairly easily incorporate the fix (https://github.com/dotnet/coreclr/pull/28066) as a patch file in your own process. This might end up being a patch file (https://github.com/Homebrew/formula-patches) that in turn adds a patch file into source-build. (Straight up patch files would work for the source-build tarballs we produce, but IIRC you're using the the GitHub source archives, which don't include the CoreCLR source initially.)

[asbjornu]

Thanks, @dagood. While I can indeed provide a patch in the formula, I feel patching the source of code that is not within the main repository is going to be difficult. What's the timeframe on a new release incorporating dotnet/coreclr#28066?

[omajid]

I feel patching the source of code that is not within the main repository is going to be difficult.

One trick that I have used is to copy my own patch files to the patches/coreclr directory. Source-build applies all patches in that directory at build time, so you don't have to worry about patching something that is not within the main repo.

[dagood]

What's the timeframe on a new release incorporating dotnet/coreclr#28066?

I'd expect it at the earliest between 2-3 weeks from now.

[asbjornu]

Ok, then I'll wait.

[dagood]

I believe this is fixed now. The 3.1.108 release PR didn't hit this--its macOS job was green: https://github.com/dotnet/source-build/pull/1732.

[asbjornu]

Indeed, the build successfully completes locally now! With that, I'll wrap up the Homebrew formula and submit a PR as soon as possible. Thanks!

[asbjornu]

@dagood, just a little note on the following:

When push comes to shove, source-build doesn't currently have a critical reason to build on macOS at all, so no worries about building on Big Sur as far as we're concerned.

I hope this view changes as soon as Homebrew/homebrew-core#60929 is merged. 😅🙏

[dagood]

Yeah, I think that being able to point at that high profile usage probably would have justified making our own patch to fix up our upstream (rather than waiting for its next release).