Bazel: Too many unrelated "xcode version must be specified" errors

[ittaiz]

Description of the problem / feature request:

I've been working the past few days on rules_scala and its toolchain and I got several times the following message:

ERROR: /private/var/tmp/.../.../external/local_config_cc/BUILD:50:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL. If your Xcode version has changed recently, try: "bazel clean --expunge" to re-run Xcode configuration

rules_scala has no c/cpp code of itself but maybe the toolchain triggers something.
Additionally I did not change anything about my Xcode version and in fact I'm pretty certain it's not fully installed since when I open the app it says something about needing to install additional components.

If I run clean expunge it fixes the problem but I don't think this is a valid solution.

What operating system are you running Bazel on?

OS X Sierra

What's the output of bazel info release?

release 0.15.0-homebrew

[tetromino]

We have also encountered this problem internally on some users' macs, and there seem to be several mechanisms that can trigger it. We don't have a good solution yet.

If you see the error again, could you please upload your java.log?

Also, which version(s) of XCode command line tools do you have installed, and where? (see xcode-select -p, which clang, and clang --version from the command line)

@rupertks - pinging you since you might have context on this issue.

[ittaiz]

/Applications/Xcode.app/Contents/Developer
/usr/bin/clang

Apple LLVM version 8.0.0 (clang-800.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

where is the java.log?

[ittaiz]

also, may I ask why you think this is P3? Having to do bazel clean expunge periodically sounds like it should be P1

[sergiocampama]

can you provide the output of this command?

cat $(bazel info execution_root)/external/local_config_xcode/BUILD

EDIT:

and cat $(bazel info execution_root)/external/local_config_cc/BUILD

[ittaiz]

Happened again after "fiddling" with toolchains (specifically java_toolchain/host_java_toolchain/javabase/host_javabase)

package(default_visibility = ['//visibility:public'])

xcode_config(name = 'host_xcodes')
# Error: Running xcodebuild -version failed, return code 256, stderr: Process terminated by signal 15, stdout:

package(default_visibility = ["//visibility:public"])

load(":osx_archs.bzl", "OSX_TOOLS_ARCHS")

CC_TOOLCHAINS = [(
cpu + "|compiler",
":cc-compiler-" + cpu,
) for cpu in OSX_TOOLS_ARCHS]

cc_library(
name = "malloc",
)

cc_library(
name = "stl",
)

filegroup(
name = "empty",
srcs = [],
)

filegroup(
name = "cc_wrapper",
srcs = ["cc_wrapper.sh"],
)

cc_toolchain_suite(
name = "toolchain",
toolchains = dict(CC_TOOLCHAINS),
)

[
filegroup(
name = "osx_tools_" + arch,
srcs = [
":cc_wrapper",
":libtool",
":make_hashed_objlist.py",
":wrapped_clang",
":wrapped_clang_pp",
":wrapped_ar",
":xcrunwrapper.sh",
],
)
for arch in OSX_TOOLS_ARCHS
]

[
apple_cc_toolchain(
name = "cc-compiler-" + arch,
all_files = ":osx_tools_" + arch,
compiler_files = ":osx_tools_" + arch,
cpu = arch,
dwp_files = ":empty",
dynamic_runtime_libs = [":empty"],
linker_files = ":osx_tools_" + arch,
objcopy_files = ":empty",
static_runtime_libs = [":empty"],
strip_files = ":osx_tools_" + arch,
supports_param_files = 0,
)
for arch in OSX_TOOLS_ARCHS
]

[
toolchain(
name = "cc-toolchain-" + arch,
exec_compatible_with = [
# This toolchain will only work with the local autoconfigured
# platforms.
"@bazel_tools//platforms:autoconfigured",
# TODO(katre): add autodiscovered constraints for host CPU and OS.
],
target_compatible_with = [
# TODO(katre): add autodiscovered constraints for host CPU and OS.
],
toolchain = ":cc-compiler-" + arch,
toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)
for arch in OSX_TOOLS_ARCHS
]
```

[ittaiz]

Also I'd really appreciate understanding why you think this is P3.

[helenalt]

@sergiocampama could you elevate the priority of this? It's not urgent but would be good to get done within a quarter.

[phb]

This happens to me pretty much weekly and is really annoying. We don't have any C/C++ code in our project so it's quite unclear why this is happening at all.

[ittaiz]

Just happened to me yesterday. Obviously no CPP code.
Bazel itself recommending clean expunge is awful IMHO
On Tue, 20 Nov 2018 at 3:47 Johan Björk [e-mail] wrote:

This happens to me pretty much weekly and is really annoying. We don't
have any C/C++ code in our project so it's quite unclear why this is
happening at all.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/6056#issuecomment-440106703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIFxdNxxwX-a_lqHjWZr6jqj9PQkEdks5uw18rgaJpZM4WWV3G
.

[dslomov]

@sergiocampama ping?

[sergiocampama]

I'm sorry, but I don't have the bandwidth to look into this issue. AFAIA, this is part of the bazel infrastructure setup, so the bazel team should have way more insight into the details on how this works.

[dslomov]

@aragos @jmmv who is the right person to look at this?

[jmmv]

I can take a look at why Xcode selection fails (not at why rules_scala needs C++), but I'm afraid this doesn't have enough data for me to reproduce the issue.

The fact that xcodebuild -version was terminated by a signal is very suspicious though.

Is there a deterministic reproduction process for this bug?

Are you running Santa and/or VSCode by any chance?

The next time this happens, could you check if a simple bazel shutdown is sufficient to correct the issue? There were some changes back in April (3a292efa55075e020d60daeeb7566b7e2ea721c2) to remove an on-disk cache of Xcode locations. The suggestion to run clean --expunge may be stale but I'm not sure.

Lastly it looks like you use Bazel 0.15. Is there any chance you could upgrade to at least 0.18? There have been various fixes to the xcode_locator tool since then to make it more reliable and they may actually fix this problem. (Or in any case, some of those fixes was to add extra logging in case of problems, which could be very helpful here.)

[ittaiz]

This happened with 0.19.2 (first reported when I used 15 but upgraded a few
versions since).
I’ll try shutdown.
Btw,
I think that rules_scala isn’t relevant here and that any starlark rule
with toolchains can cause this but I’m not really sure
On Wed, 28 Nov 2018 at 19:39 Julio Merino [e-mail] wrote:

I can take a look at why Xcode selection fails (not at why rules_scala
needs C++), but I'm afraid this doesn't have enough data for me to
reproduce the issue.

The fact that xcodebuild -version was terminated by a signal is very
suspicious though.

Is there a deterministic reproduction process for this bug?

Are you running Santa and/or VSCode by any chance?

The next time this happens, could you check if a simple bazel shutdown is
sufficient to correct the issue? There were some changes back in April (
3a292ef
https://github.com/bazelbuild/bazel/commit/3a292efa55075e020d60daeeb7566b7e2ea721c2)
to remove an on-disk cache of Xcode locations. The suggestion to run clean
--expunge may be stale but I'm not sure.

Lastly it looks like you use Bazel 0.15. Is there any chance you could
upgrade to at least 0.18? There have been various fixes to the
xcode_locator tool since then to make it more reliable and they may
actually fix this problem. (Or in any case, some of those fixes was to add
extra logging in case of problems, which could be very helpful here.)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/6056#issuecomment-442537455,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF2t3h0DXw4keGqH3qdTN6C9AcpTpks5uzspdgaJpZM4WWV3G
.

[phb]

Just had this happen again with 0.18.1.
$ bazel build //... --nobuild
DEBUG: /private/var/tmp/_bazel_me/60a181658afb41895a4dba3b821290b1/external/bazel_tools/tools/osx/xcode_configure.bzl:87:9: Invoking xcodebuild failed, developer dir: /Applications/Xcode.app/Contents/Developer ,return code 256, stderr: Process terminated by signal 15, stdout:

• update, I then tried to run a go target
  $ bazel run @xx_xxyz//:buildifier
  INFO: SHA256 (https://codeload.github.com/golang/tools/zip/5d2fd3ccab986d52112bf301d47a819783339d0e) = 31c4b514df1b40d7ade2e6f824dca35a36d9454983e7955074ac6f47f1eb3292
  DEBUG: /private/var/tmp/_bazel_me/60a181658afb41895a4dba3b821290b1/external/bazel_tools/tools/osx/xcode_configure.bzl:87:9: Invoking xcodebuild failed, developer dir: /Applications/Xcode.app/Contents/Developer ,return code 256, stderr: Process terminated by signal 15, stdout:
  INFO: Analysed target @xx_xxyz//:buildifier (13 packages loaded).
  INFO: Found 1 target...
  ERROR: /private/var/tmp/_bazel_me/60a181658afb41895a4dba3b821290b1/external/io_bazel_rules_go/BUILD.bazel:8:1: GoStdlib external/io_bazel_rules_go/darwin_amd64_stripped/stdlib~/pkg failed: I/O exception during sandboxed execution: xcrun failed with code 1.
  This most likely indicates that SDK version [10.10] for platform [MacOSX] is unsupported for the target version of xcode.
  Process exited with status 1
  stdout: stderr: xcodebuild: error: SDK "macosx10.10" cannot be located.
  xcodebuild: error: SDK "macosx10.10" cannot be located.
  xcrun: error: unable to lookup item 'Path' in SDK 'macosx10.10'
  Target @xx_xxyz//:buildifier failed to build
  Use --verbose_failures to see the command lines of failed build steps.
  INFO: Elapsed time: 17.261s, Critical Path: 5.13s
  INFO: 8 processes: 8 darwin-sandbox.
  FAILED: Build did NOT complete successfully
  FAILED: Build did NOT complete successfully

--
I can then confirm that bazel shutdown fixes the issue.

[ittaiz]

Did shutdown help? Also are you doing changes related to toolchains?
On Thu, 29 Nov 2018 at 0:42 Johan Björk [e-mail] wrote:

Just had this happen again with 0.18.1.
$ bazel build //... --nobuild
DEBUG:
/private/var/tmp/_bazel_me/60a181658afb41895a4dba3b821290b1/external/bazel_tools/tools/osx/xcode_configure.bzl:87:9:
Invoking xcodebuild failed, developer dir:
/Applications/Xcode.app/Contents/Developer ,return code 256, stderr:
Process terminated by signal 15, stdout:

I still don't have a clear idea when it's happening, possibly something
fairly unique with my workflow is that I'm using a lot of different bazel
workspaces often at the same time.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/6056#issuecomment-442633556,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF0Hqv0QA4h-h3gn65oEa3MYpV_5fks5uzxFlgaJpZM4WWV3G
.

[phb]

Yes shutdown helped.

We do register a custom toolchain (internal compiler).

It looks vaguely like this

toolchain_type(name = "compiler")

toolchain(
    name = "darwin_compiler",
    exec_compatible_with = [
        "@bazel_tools//platforms:osx",
        "@bazel_tools//platforms:x86_64",
    ],
    toolchain = ":darwin_compiler_impl",
    toolchain_type = ":compiler",
)

toolchain(
    name = "linux_compiler",
    exec_compatible_with = [
        "@bazel_tools//platforms:linux",
        "@bazel_tools//platforms:x86_64",
    ],
    toolchain = ":linux_compiler_impl",
    toolchain_type = ":compiler",
)

compiler_toolchain(
    name = "darwin_compiler_impl",
    compiler = select({
        ":local_build": "//:local_compiler_jar_macos",
        ":s3_build": ":compiler_jar_macos",
        "//conditions:default": ":compiler_jar_macos",
    }),
    service = select({
        ":local_build": "//:local_compiler_service_jar_macos",
        ":s3_build": ":compiler_service_jar_macos",
        "//conditions:default": ":compiler_service_jar_macos",
    }),
)
# provider details for the various jars skipped

[pcj]

If this happens to me it's often temporally-related to running gazelle (only observed it on mac, never linux): bazel run //:gazelle (I don't know why, just leaving a breadcrumb in case someone is investigating this). Perhaps a race condition between another tool's write and bazel's own reading of BUILD.bazel files? Completely guessing here. For @jmmv I also do use vscode.

bazel shutdown always solves the problem.

[s50600822]

redisliu's solution from https://github.com/bazelbuild/bazel/issues/4314 works for me
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer; sudo xcodebuild -license; bazel clean --expunge;
Reference: https://stackoverflow.com/questions/45276830/xcode-version-must-be-specified-to-use-an-apple-crosstool

[phb]

Still happens for us with bazel 0.22. Please let me know if there is any other information that would be helpful as it reproduces pretty regularly for me.

[jmmv]

The reason I asked about VSCode is because of: https://github.com/bazelbuild/bazel/issues/4603. We troubleshot similar problems there but eventually traced them to a bug in Santa. I wonder if something similar is going on here.

Again, the fact that xcodebuild seems to have crashed is strange and is likely the trigger here. You could try using the startup --unlimit_coredumps option in all your builds so, the next time this hits, you might get a useful coredump under /cores/.

[phb]

Ah, no VSCode or Santa here. I'll enable coredumps and will let you know if something appears

[keith]

One way to easily reproduce the original issue here is by ctrl-c'ing while bazel is Fetching @local_config_xcode; fetching:

% ./bazelw build --config=debug Modules/PassengerApp:Lyft
INFO: Invocation ID: 4d6704d8-a396-48a0-98db-16537ed62162
Analyzing: target //Modules/PassengerApp:Lyft (19 packages loaded, 23 targets configured)
    Fetching @local_config_xcode; fetching
^C
ERROR: build interrupted
INFO: Elapsed time: 2.878s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (19 packages loaded, 23 targets configured)
% ./bazelw build --config=debug Modules/PassengerApp:Lyft
INFO: Invocation ID: 6e7d8c6f-c3c1-46b3-a19b-a89450214c33
ERROR: /private/var/tmp/_bazel_ksmiley/c8513c989333be3c89713e16206869b2/external/local_config_cc/BUILD:58:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-watchos_x86_64: Xcode version must be specified to use an Apple CROSSTOOL. If your Xcode version has changed recently, try: "bazel clean --expunge" to re-run Xcode configuration
ERROR: Analysis of target '//Modules/PassengerApp:Lyft' failed; build aborted: Analysis of target '@local_config_cc//:cc-compiler-watchos_x86_64' failed; build aborted
INFO: Elapsed time: 1.510s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (9 packages loaded, 106 targets configured)
    Fetching @go_sdk; fetching

The contents of local_config_xcode/BUILD in this state end up having error messages about how xcodebuild or xcode_locator failed to run (depending on when you cancelled)

[keith]

We've worked around this currently by checking in our own Xcode config and passing --xcode_version_config=//:host_xcodes so that bazel never fetches this. Although it can still happen if you ctrl-c out of local_config_cc as well. The possible downside is you need to know your users are using the same Xcode version

[keith]

This PR https://github.com/bazelbuild/bazel/pull/7519 fixes the ctrl-c issue (and likely other issues where this comes up naturally)

[keith]

Another repro case that @ob pointed out:

I think if bazel fails in the loading phase (before analysis) you get this error

[keith]

The analysis repro case isn't fixed by my PR. The easiest way for me to repro is by adding an invalid label to the extensions of an ios_application rule, and then:

bazel clean --expunge
bazel build //:ios-app
# Fix the invalid extension label
bazel build //:ios-app

And you'll end up with this error. For some reason I can't easily reproduce this by breaking a dep.

[keith]

Because of this known issue https://github.com/bazelbuild/bazel/issues/6056#issuecomment-467142752 we might want to leave this open, up to you all

[keith]

FYI https://github.com/bazelbuild/bazel/issues/7592

[Unknown9527]

When you meet this problem, enter " xcode-select". Then follow the "reset" step, this problem will never happen.

sudo xcode-select --reset