Bazel: Immediate "couldn't connect to server at..."

[semistrict]

Every time I start Bazel, I get an immediate "Error: couldn't connect to server at .. after 60 seconds". This happens immediately, despite saying "after 60 seconds". Every subsequent bazel command works fine after that.

OS: macOS Sierra
Bazel version: 180d1b56b8aea8fab974cfa57e9182598990710c (from source)

(This reproduces with a version I installed from Homebrew as well)

[kchodorow]

Is there a way for us to try to repro on Sierra before 9/20?

[semistrict]

You could run the beta: https://beta.apple.com/sp/betaprogram/
Alternatively, is there a log file I can send you?

[ilovezfs]

The Homebrew Sierra bottling is hitting the same problem.

Build successful! Binary is here: /private/tmp/bazel-20160915-86731-gz7zl7/bazel-0.3.1/output/bazel
==> ./output/bazel --output_user_root /private/tmp/bazel-20160915-86731-gz7zl7/bazel-0.3.1/output_user_root build scripts:bash_completion
Extracting Bazel installation...
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Error: couldn't connect to server at '/private/tmp/bazel-20160915-86731-gz7zl7/bazel-0.3.1/output_user_root/5ec46ad567f71bb474882be0925ebe14/server/server.socket' after 60 seconds.

Full log
https://gist.github.com/ilovezfs/22a501e7e4a3bb7e88a27fd29b8a76e6
(https://bot.brew.sh/job/Homebrew%20Sierra%20Testing/754/version=sierra_2/consoleFull)

[damienmg]

Sierra is getting released today...

[ilovezfs]

it's already shipped. https://itunes.apple.com/us/app/macos-sierra/id1127487414?mt=12

[damienmg]

:/ we have no machine with it. We need to investigate though.

Is there some change in the network setting in Sierra?

[ilovezfs]

@damienmg I think it's an uncaught exception because the process exits with the couldn't connect error seemingly immediately but then it works the second time because the socket is already created.

[ilovezfs]

Josephs-Mac:Homebrew joe$ brew test -vd bazel
git config --file=/usr/local/Homebrew/.git/config --replace-all homebrew.devcmdrun true
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/bazel.rb
Testing bazel
==> Using the sandbox
/usr/bin/sandbox-exec -f /tmp/homebrew20160920-3906-l41isl.sb /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/test.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/bazel.rb -vd
/usr/local/Homebrew/Library/Homebrew/test.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/bazel.rb
==> /usr/local/Cellar/bazel/0.3.1/bin/bazel build //:bazel-test
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Error: couldn't connect to server at '/private/var/tmp/_bazel_joe/5bb467a61a612689569eed257b2a0133/server/server.socket' after 60 seconds.
/usr/local/Homebrew/Library/Homebrew/debrew.rb:11:in `raise'
BuildError: Failed executing: /usr/local/Cellar/bazel/0.3.1/bin/bazel build //:bazel-test
1. raise
2. ignore
3. backtrace
4. irb
5. shell
Choose an action: 5
When you exit this shell, you will return to the menu.
bash-3.2$ ls -al
total 16
drwx------  6 joe   staff  204 Sep 20 11:19 .
drwxrwxrwt  5 root  wheel  170 Sep 20 11:19 ..
-rw-r--r--  1 joe   staff   97 Sep 20 11:19 BUILD
drwxr-xr-x  3 joe   staff  102 Sep 20 11:19 Library
-rw-r--r--  1 joe   staff  109 Sep 20 11:19 ProjectRunner.java
-rw-r--r--  1 joe   staff    0 Sep 20 11:19 WORKSPACE
bash-3.2$ pwd
/private/tmp/bazel-test-20160920-3907-dh07nu
bash-3.2$ ls -l /private/var/tmp/_bazel_joe/5bb467a61a612689569eed257b2a0133/server/server.socket
srw-------  1 joe  wheel  0 Sep 20 11:19 /private/var/tmp/_bazel_joe/5bb467a61a612689569eed257b2a0133/server/server.socket
bash-3.2$ /usr/local/Cellar/bazel/0.3.1/bin/bazel build //:bazel-test
____Loading package: 
____Loading...
____Loading package: @bazel_tools//tools/jdk
____Loading package: @bazel_tools//tools/android
____Loading package: @local_jdk//
____Loading package: @local_config_cc//
____Loading complete.  Analyzing...
____Loading package: @bazel_tools//third_party/java/jdk/langtools
____Found 1 target...
____Building...
____[0 / 5] BazelWorkspaceStatusAction stable-status.txt
Target //:bazel-test up-to-date:
  bazel-bin/bazel-test.jar
  bazel-bin/bazel-test
____Elapsed time: 7.263s, Critical Path: 2.12s
bash-3.2$ /usr/local/Cellar/bazel/0.3.1/bin/bazel build //:bazel-test
____Loading...
____Found 1 target...
Target //:bazel-test up-to-date:
  bazel-bin/bazel-test.jar
  bazel-bin/bazel-test
____Elapsed time: 0.174s, Critical Path: 0.01s
bash-3.2$ rm /private/var/tmp/_bazel_joe/5bb467a61a612689569eed257b2a0133/server/server.socket
bash-3.2$ /usr/local/Cellar/bazel/0.3.1/bin/bazel build //:bazel-test                         
Killed non-responsive server process (pid=3925)
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Error: couldn't connect to server at '/private/var/tmp/_bazel_joe/5bb467a61a612689569eed257b2a0133/server/server.socket' after 60 seconds.

[damienmg]

It's is probably a client/server communication that's why I suspect network issue (even though the communication goes through localloop). I try to get my hand on a Sierra machine but that won't be before tomorrow.

[ilovezfs]

@damienmg this seems sufficient to work around it

diff --git a/Formula/bazel.rb b/Formula/bazel.rb
index b652d3f..1de37d1 100644
--- a/Formula/bazel.rb
+++ b/Formula/bazel.rb
@@ -15,6 +15,9 @@ class Bazel < Formula
   depends_on macos: :yosemite

   def install
+    inreplace "src/main/cpp/blaze.cc", "for (int ii = 0; ii < 600; ++ii) {",
+                                       "for (int ii = 0; ii < 60000; ++ii) {"
+
     ENV["EMBED_LABEL"] = "#{version}-homebrew"
     # Force Bazel ./compile.sh to put its temporary files in the buildpath
     ENV["BAZEL_WRKDIR"] = buildpath/"work"

so maybe the units are different than expected?

[ilovezfs]

@damienmg https://github.com/Homebrew/homebrew-core/pull/5041

[damienmg]

Why not replace to 120? Isn't this enough?

[ilovezfs]

@damienmg any chance for a new release tag? or does this need to be applied as a patch ...

[petemounce]

We're affected by this, too.

[jtbr]

Running Linux mint 17.3 (based on ubuntu 14.04), with bazel 0.3.1 and oracle JDK8 newly installed, I'm getting apparently similar results. I'm trying to build tensorflow 0.10.0. Their configure calls

bazel clean --expunge
bazel fetch //...

I get this:

Extracting Bazel installation...
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Error: couldn't connect to server at '/data/.caches/bazel/_bazel_username/e04a85f773e4b9562f802da07a4ccbf3/server/server.socket' after 60 seconds.
Killed non-responsive server process (pid=3965)
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Error: couldn't connect to server at '/data/.caches/bazel/_bazel_username/e04a85f773e4b9562f802da07a4ccbf3/server/server.socket' after 60 seconds.

The dots take maybe 10 minutes on a very fast machine, and nothing appears to happen. I don't find any useful logs anywhere. Is this the same issue, or can you point to other issues or ways to debug? Thanks.