Bazel: Bazel 0.5.0 build fails on Mac OS X El Capitan

[zanes2016]

Description of the problem / feature request / question:

Bazel build fails on Mac OS X El Capitan (Version 10.11.6). The same build completes without any issue on Ubuntu 14.04. It looks like this is a mac specific issue.

Mac (build error):

$ bazel version
Build label: 0.5.0-homebrew
Build target: bazel-out/darwin_x86_64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri May 26 14:50:47 2017 (1495810247)
Build timestamp: 1495810247
Build timestamp as int: 1495810247
$
$
$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1510
$
$
$ bazel build --verbose_failures :all
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
    File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 829
        _get_escaped_darwin_cxx_inc_directories(repository_ctx, cc)
    File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 726, in _get_escaped_darwin_cxx_inc_directories
        for toolchain in toolchains: ...
type 'NoneType' is not iterable.
INFO: Elapsed time: 3.445s

Ubuntu (no error):

$ bazel version
Build label: 0.5.0
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri May 26 12:11:50 2017 (1495800710)
Build timestamp: 1495800710
Build timestamp as int: 1495800710
$
$
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty
$
$
$ # I already ran the build, it succeeded without any issues
$ bazel build --verbose_failures :all
INFO: Found 1 target...
Target //:env up-to-date (nothing to build)
INFO: Elapsed time: 0.128s, Critical Path: 0.00s

If possible, provide a minimal example to reproduce the problem:

On a mac, the following produces the error

$ bazel build --verbose_failures :all

Environment info

• Operating System:
  ProductName: Mac OS X
  ProductVersion: 10.11.6
  BuildVersion: 15G1510

• Bazel version (output of bazel info release):
  release 0.5.0-homebrew

Anything else, information or logs or outputs that would be helpful?

I'm trying to build the same project on both mac and ubuntu which is version controlled with git. So i'm confident that the two build sources are identical.

[damienmg]

Can you try to rebuild after bazel clean --expunge?

[damienmg]

(I use macOS El Capitan without issue, btw)

[zanes2016]

I still have the same issue:

$ bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
$
$ bazel build :all
..............................................
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
    File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 829
        _get_escaped_darwin_cxx_inc_directories(repository_ctx, cc)
    File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 726, in _get_escaped_darwin_cxx_inc_directories
        for toolchain in toolchains: ...
type 'NoneType' is not iterable.
INFO: Elapsed time: 4.959s

[damienmg]

/cc @mhlopko undetected regression of 0.5.0 :(

@c-parsons this is happening because xcode_locator does not returns any toolchain, I don't know how it works but if we do not find a user side workaround, we need to do a patch release.

[damienmg]

@zanes2016: as much information about your xcode installation would help. gcc --version also

[zanes2016]

$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[damienmg]

/cc @c-parsons now Bazel is failing to build C++ code if you don't have xcode installed.

@zanes2016 seems like my only fix right now is to install xcode :( sorry for that.

[c-parsons]

I've thrown together a fix for this, sending it to you right now, Damien.
Sorry for the breakage -- we apparently don't have a good CI environment to exercise this situation.

[damienmg]

Well if installing xcode fix the issue this might be an acceptable workaround. 0.5.1 should be in less than a month.

Yes having all the use case in a CI is hard.

[MarkusTeufelberger]

Looks like #3060 is another instance of this?

[damienmg]

Indeed it is, we will start seeing a lot of bug report, good thing I am in the US right now, I'll might cut a patch release if that start to have a lot of feedback.

[damienmg]

(but first I would like to confirm @c-parsons fix is working, which I believe should be ok)

[damienmg]

Any idea how to test it?
@zanes2016 or @zhexuany, would any of you willing to help in an interactive session (talking on IRC or hangout)?

[zanes2016]

@damienmg I should be available for a hangout in about 20 minutes

[zhexuany]

@damienmg I am available right now.

[damienmg]

Best plan of action we have right now:

1. Test @c-parsons change on @zanes2016 or @zhexuany machine (or anyone hitting the issue)
2. Roll-back update to 0.5.0 for bazel (homebrew/homebrew-core#13967, /cc @ilovezfs)
3. Create a 0.5.1rc1 that just include @c-parsons change today
4. Have 0.5.1rc1 released early next week, we would have to have an exception to our release policy.
5. Bump to 0.5.1 for the release in homebrew.

Up to 4 there is no need for further agreement, I let the rest of the Munich team decide on 3 on Monday while the US are on holidays.

@zanes2016 @zhexuany thanks! mail me at [email protected] so we can chat on hangout (or ping me on irc freenode / dmarting). I need 10-20 min to prepare the rc branch

[zanes2016]

For completeness, I've also verified that this issue exists on MacOS Sierra:

$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.12.5
BuildVersion:   16F73
$
$
$ bazel build :all
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
    File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 829
        _get_escaped_darwin_cxx_inc_directories(repository_ctx, cc)
    File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 726, in _get_escaped_darwin_cxx_inc_directories
        for toolchain in toolchains: ...
type 'NoneType' is not iterable.
INFO: Elapsed time: 0.203s

[damienmg]

@zhexuany tested it with Xcode, still failing, might need more than the default install. With definitely needs a patch release.

[zhexuany]

ERROR: /private/var/tmp/_bazel_zhexuany/6bfb8836d8a7c642713005d68a02df63/external/local_config_cc/BUILD:57:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL.
ERROR: Analysis of target '//src/test/java/com/pingcap/tikv:AllTests' failed; build aborted.
INFO: Elapsed time: 33.385s
ERROR: Couldn't start the build. Unable to run tests.

hope this can help in some senses.

[damienmg]

FTR the test rc failed to build the release notes, I'll can extract the binary for test but a patch release seems like it has a hard time or it is because I tweaked the cherry-picking.

[damienmg]

The darwin package has been suceesfully built so we can at least test the fix itself.

[zhexuany]

Already try the latest built binary, error is still present.

[damienmg]

Ok rolling back 4b24a61e0 is too complex because of merge conflicts :( I won't be able to do it today probably. I'll concentrate on reverting the homebrew change. Since @c-parsons is probably already off and @mhlopko knows cc_configure.bzl and the release process, handing the bug over to marcel so he can resume operation on monday.

[damienmg]

@ilovezfs: brew audit fail because I am trying to reverting the stable version, is it not a use case planned by homebrew?

Handing off to @mhlopko, there is nothing more I can do right now and I would need more time to prepare a proper fix.

[damienmg]

For people hitting that bug: the solution should be to install Xcode and the ios tooling.

[zhexuany]

just for future people. sudo xcode-select -s /Applications/Xcode.app/Contents/Developer could do the the trick if you install Xcode and bazel still failing.

[damienmg]

Thanks Yang!

On Fri, May 26, 2017, 6:18 PM Yang Zhexuan [e-mail] wrote:

just fro future people. sudo xcode-select -s
/Applications/Xcode.app/Contents/Developer could do the the trick if you
install Xcode and bazel still failing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3063#issuecomment-304416921,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHf32qFtbXKaknasRMzLM0rxeATK3dks5r93nWgaJpZM4NoEvI
.

[appsforartists]

FWIW, the installer on the Releases page has the same issue. @zhexuany's workaround resolves it.

[ilovezfs]

@damienmg I've reverted the upgrade in brew. However, I did not push a version_scheme bump to force downgrades, otherwise we will have to carry version_scheme 1 in the formula forever.

[damienmg]

Thank you very much, no need for force downgrade that's perfect thanks!

There is a workaround we should publish on the release note and keep the plan for a path release. Delegating to people currently in Munich to follow-up.

[hlopko]

Hello everybody,
thank you all for your prompt reports and investigations! It's very unfortunate we didn't catch this regression before. If we decide that Bazel shouldn't depend on Xcode (read below), I'd suggest having a mac slave that only has command line tools installed, not the full Xcode.

Please correct me if I'm wrong:
The fix by @c-parsons will make the error go away, but people will still be blocked on the next-in-line error (Xcode version must be specified to use an Apple CROSSTOOL). And we have no other fix than installing the Xcode. To me (and @steren) replacing one very bad error message with one only a little bit bad doesn't warrant a patch release. I'll discuss with @c-parsons what a principled solution would be and evaluate possibilities of:

• not relying on Xcode, with command line tools being enough (like they were up until now)
• if relying on Xcode, having better, very explicit error message telling the user you need Xcode.

@damienmg Do you think Bazel relying on Xcode is acceptable?

I'll update the release notes, blog post, and the documentation to clearly state you need to have Xcode installed in order to use Bazel 0.5.0.

Regarding homebrew, let's wait for @c-parsons decision and then either create 0.5.1 with a principled fix and update the recipe to use 0.5.1 directly, or if the principled fix cannot be implemented in O(days), let's serve 0.5.0, and redirect all affected users to the updated documentation.

I'd like to add that the sudo xcode-select -s /Applications/Xcode.app/Contents/Developer is only needed if you already had command line tools installed. This command will replace /usr/bin/clang and friends with those coming from Xcode.app. If you didn't have command line tools installed, installing Xcode is enough.

[ilovezfs]

If Bazel isn't actually using anything in Xcode that's not available in the CLT, requiring Xcode would be a mistake, I think.

[hlopko]

I fully agree, let's wait for @c-parsons to tell us what parts of Xcode Bazel needs.

[damienmg]

US are on holiday today. The principle fix will takes several patch to do,
I was leaning toward a rollback but the rollback has conflict with 2 of
your changes, it was already late so I just handover the bug to you.
Homebrew has reverted the update so no new client will install it, avoiding
more people coming here. I added a message to the release notes page, maybe
it is not really visible.

Bazel does not requires Xcode to build C++ code, only to bootstrap. We
should keep it that way.

On Mon, May 29, 2017 at 4:32 AM Marcel Hlopko [e-mail]
wrote:

I fully agree, let's wait for @c-parsons https://github.com/c-parsons
to tell us what parts of Xcode Bazel needs.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3063#issuecomment-304639648,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHf9yyLElnoXeI0UmHiiETD9vqxi0yks5r-qzhgaJpZM4NoEvI
.

[c-parsons]

Hello folks.
Indeed rolling back the apple crosstool changes would be a difficult task given the number of dependent changes that are in the release.
I will quickly try to draft a patch change which will turn off apple crosstool if xcode is not present.

Sorry for the inconvenience.

[hlopko]

Hello everybody,
Chris was super fast with the fix, and we have a 0.5.1RC1 ready. Would you mind testing it if it fixes the problem for you? You can get it from

https://storage.googleapis.com/bazel/0.5.1/rc1/index.html

If yes, I'll close this issue. Thanks!

[ittaiz]

Marcel,
0.5.1 will only contain this fix, right?
On Wed, 31 May 2017 at 18:17 Marcel Hlopko [e-mail] wrote:

Hello everybody,
Chris was super fast with the fix, and we have a 0.5.1RC1 ready. Would you
mind testing it if it fixes the problem for you? You can get it from

https://storage.googleapis.com/bazel/0.5.1/rc1/index.html

If yes, I'll close this issue. Thanks!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3063#issuecomment-305220147,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF-v-in6R55kyWhFB9grTJ6iJ4wUUks5r_YSPgaJpZM4NoEvI
.

[evanj]

0.5.1RC1 seems to fix the problem for me on OS X 10.11.6. Thanks!

[hlopko]

@ittaiz Yes, only that fix. No new behavior, no need to upgrade bazel if you don't need the fix.
@evanj Awesome! I'm letting it sit for 2 days and then I'll release it and upgrade the homebrew recipe. Thank you all!

[ilovezfs]

I've now shipped 0.5.1 in Homebrew.

[zhexuany]

@c-parsons
Still failing.
ERROR: /private/var/tmp/_bazel_zhexuany/d90cfb1d83b5fac0d2a4fea11702e03d/external/local_config_cc/BUILD:57:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL.

[hlopko]

Just to be sure, also after bazel clean --expunge?

[zhexuany]

My bad. Now error is gone. Thanks.

[philipposslicher]

Error is still present in 10.13 High Sierra

[varming]

Ran into this error today after upgrading to 0.5.2 in homebrew.

[damienmg]

@philipposslicher @varming: reopening a new bug.

[phynero]

ERROR: /private/var/tmp/_bazel_phynero/759a3c362e95012bc330dc146849fc13/external/local_config_cc/BUILD:49:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL.
ERROR: Analysis of target '//:add_person_java_lite' failed; build aborted.

T-T WHAT CAN IDO

[phynero]

xcode_config(name = 'host_xcodes')

Error: Running xcodebuild -version failed, return code 72, stderr: xcrun: error: unable to find utility "xcodebuild", not a developer tool or in PATH , stdout:

[c-parsons]

What type of target are you trying to build?

[nornagon]

I fixed this by running sudo xcodebuild -license and agreeing to the TOS.

(Sadly, I then encountered clang: error: unknown argument: '-fno-canonical-system-headers')

[dpursehouse]

I just had this problem on OSX 10.12.6 with bazel 0.5.4, and it was fixed by running bazel clean --expunge

[c-parsons]

dpursehouse: Can you provide more information? Did you have XCode installed beforehand? What kind of project were you trying to build?

[dpursehouse]

@c-parsons yes, I have XCode (version 9.0) installed. I was building the gerrit project.

The error appeared suddenly, having not appeared before. I had not upgraded bazel or XCode.

The only thing that might be related is that I regularly switch between and build different branches.