Bazel: wrapped_clang is not built with the requested Xcode versions

[keith]

Description of the problem / feature request:

Currently wrapped_clang is built as part of the local_config_cc setup:

https://github.com/bazelbuild/bazel/blob/0b5f341700f6de633b72d8d6c0370b5e40630c53/tools/cpp/osx_cc_configure.bzl#L111-L121

Currently it doesn't take into account what you pass for --xcode_version. Because of this if you change your xcode-select on macOS and build with a different version of Xcode than is requested with this flag, you produce a different wrapped_clang. This ends up breaking remote cache compatibility with other versions even if the build would otherwise be done with the correct Xcode version.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• sudo xcode-select -s /Applications/Xcode-10.2.1.app
• bazel build some_ios_target --xcode_version=10.2.1 --remote_http_cache=...
• bazel clean --expunge
• sudo xcode-select -s /Applications/Xcode-11beta3.app
• bazel build some_ios_target --xcode_version=10.2.1 --remote_http_cache=...

With the second build, you will get nearly no remote cache hits because wrapped_clang differs as an input to every action, even though you requested the same Xcode version from the command line.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 0.28.0

[keith]

This would be less of an issue if changing Xcode versions globally with xcode-select invalidated wrapped_clang, but this requires a clean --expunge to recover from

[keith]

@scentini do you have a suggestion for how to fix this? Having to clean --expunge whenever switching versions here is a pain for our large team

[scentini]

cc/ @hlopko

[keith]

@hlopko do you have any suggestion for how to fix this?

[brentleyjones]

This should also potentially consider the DEVELOPER_DIR environment variable, per my duped issue: #11716.

[keith]

Would passing that through cause invalidation when the user's Xcode version changed? Maybe if we correctly accessed it in the workspace rule it could help?

[brentleyjones]

If the user set their Xcode version that way, then yes. But if all they did was change xcode-select -s, I don't believe so.

[keith]

Yea, I was at least thinking that in your bazel wrapper you could always specify that var if it would fix this

[brentleyjones]

Ohh, yeah, it could read the set value and then export it. The workspace rule can state that it should invalidate based on the environment variable. I think that would work nicely.

It would differ from --xcode_version though.

[oquenchil]

If the PR above didn't fix this issue, please re-open.

[brentleyjones]

We can specify the DEVELOPER_DIR in a bazel wrapper now after #11719, but I'm not sure if @keith's original issue was fixed.

[keith]

Using DEVELOPER_DIR in a wrapper around bazel definitely works around the original issue, but it would be nice if it was solved for everyone without having to manually do that.

[keith]

One thing I realized with the DEVELOPER_DIR workaround is that if the user overwrites their Xcode.app with a newer version, the DEVELOPER_DIR will not change, but the version of Xcode will. So that's a case where this workaround isn't enough 😞

[jerrymarino]

@keith one way to sidestep this is to just use a custom local_config_cc. e.g.

bazel fetch :*  && ditto $(bazel info execution_root)/../../external/local_config_cc tools/local_config_cc

# WORKSPACE
local_repository(name="local_config_cc", path="tools/local_config_cc")

YMMV for different host arch than x86_64 🤔

It might be possible to factor out the binaries into their own input to the cc rules rather than compiling in the repo rule, but I don't know the long term plans for this / overlap with rules_cc off the top of my head!

[keith]

copying that in is definitely an interesting idea, we also run some things in our build on Linux so I think we'd have to do that in a cross platform way though. Definitely handling the arch problem (although you could check fat binaries in there instead if you only supported macOS) would be necessary too

I think doing the input route is easier said than done because it'd be a bit of a circular dependency

[jerrymarino]

@keith in order to make the binary an input to the toolchain w/o a circle you'll need another rule to compile it: e.g. a rule that consumes only the relevant xcode config bits and not the custom toolchain.

As the toolchain will be specific to macOS, you might need to override local_config_cc conditionally when on macOS to use the hardcoded one. You might be able to get away with switching the entire generated toolchain with --override_repository instead of having to create a cross platform one.

[jerrymarino]

For what it's worth, it seems like making wrapped_clang binaries an input to the toolchain seems like good way to fix this outside of forking the generated one