Bazel: CI, TF serving: failing with bazel 0.9.0 rc2

[laszlocsomor]

Blocking: https://github.com/bazelbuild/bazel/issues/3773

https://ci.bazel.io/job/Global/job/TensorFlow_Serving/311/consoleFull

Ran as part of Bazel 0.9.0 rc2 test: https://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/bazel/job/release/533/Downstream_projects/

Binaries: https://releases.bazel.build/0.9.0/rc2/index.html

Example log:

[variation=,node=ubuntu_16.04-x86_64] ERROR: /home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/local_config_sycl/sycl/BUILD:4:1: First argument of 'load' must be a label and start with either '//', ':', or '@'. Use --incompatible_load_argument_is_label=false to temporarily disable this check.
[variation=,node=ubuntu_16.04-x86_64] ERROR: /home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/local_config_sycl/sycl/BUILD:6:1: First argument of 'load' must be a label and start with either '//', ':', or '@'. Use --incompatible_load_argument_is_label=false to temporarily disable this check.
[variation=,node=ubuntu_16.04-x86_64] ERROR: /home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/local_config_sycl/sycl/BUILD:41:9: Traceback (most recent call last):
[variation=,node=ubuntu_16.04-x86_64]   File "/home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/local_config_sycl/sycl/BUILD", line 38
[variation=,node=ubuntu_16.04-x86_64]       cc_library(name = "syclrt", srcs = [sycl_libr...")], <3 more arguments>)
[variation=,node=ubuntu_16.04-x86_64]   File "/home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/local_config_sycl/sycl/BUILD", line 41, in cc_library
[variation=,node=ubuntu_16.04-x86_64]       sycl_library_path
[variation=,node=ubuntu_16.04-x86_64] name 'sycl_library_path' is not defined
[variation=,node=ubuntu_16.04-x86_64] ERROR: /home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/com_googlesource_code_re2/BUILD:96:1: First argument of 'load' must be a label and start with either '//', ':', or '@'. Use --incompatible_load_argument_is_label=false to temporarily disable this check.
[variation=,node=ubuntu_16.04-x86_64] ERROR: /home/ci/.cache/bazel/_bazel_ci/f5b9cd05622a75c4b59e41a2d2460c19/external/com_googlesource_code_re2/BUILD:98:1: name 're2_test' is not defined (did you mean 'ios_test'?)

[laszlocsomor]

TODO:

• repro the failure with Bazel 0.9.0 rc2
• try to repro with Bazel 0.8.1, see if it's failing with that too
  
  
  
  • if failing, then mark as non-blocker for #3773
  
  
  • if not failing, bisect commits to find culprit
  
  

[laszlocsomor]

This is still failing with Bazel 0.9.0 rc3.
I suppose the fix of TF Serving is not yet submitted.

[laszlocsomor]

@laurentlb : do you know where can I track the bugfix to TF Serving?

[laszlocsomor]

I don't see this issue anymore with RC6's post-submit tests (https://ci.bazel.io/job/bazel/job/release/617/), so I'm closing the bug.

[psyhtest]

Same issue on aarch64 with just released 0.9.0. No problem with 8.1.0 though.

[laszlocsomor]

Looks a lot like https://github.com/bazelbuild/bazel/issues/4289

[laszlocsomor]

@psyhtest : Do you know how I could repro this locally?

[psyhtest]

@laszlocsomor I've just reproduced using Collective Knowledge (CK) on an Ubuntu 16.04 x86_64 machine as follows:

Install CK and main repositories

$ sudo pip install ck
$ ck pull repo:ck-env
$ ck pull repo:ck-tensorflow

Install Bazel 0.9.0

$ ck install package:tool-bazel-0.9.0-linux

If you already have a Bazel 0.9.0 executable somewhere (or if you are on Windows), you can skip its installation via CK and simply register it with CK:

$ ck detect soft:tool.bazel --full_path=<absolute path to the executable>

for example:

$ ck detect soft:tool.bazel --full_path=/usr/local/bin/bazel

Install TensorFlow 1.4.0 from sources

$ ck install package:lib-tensorflow-1.4.0-src-cpu
...
--------------------------------
Build with bazel

........
ERROR: /home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/local_config_sycl/sycl/BUILD:4:1: First argument of 'load' must be a label and start with either '//', ':', or '@'. Use --incompatible_load_argument_is_label=false to temporarily disable this check.
ERROR: /home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/local_config_sycl/sycl/BUILD:6:1: First argument of 'load' must be a label and start with either '//', ':', or '@'. Use --incompatible_load_argument_is_label=false to temporarily disable this check.
ERROR: /home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/local_config_sycl/sycl/BUILD:30:9: Traceback (most recent call last):
        File "/home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/local_config_sycl/sycl/BUILD", line 27
                cc_library(name = "syclrt", srcs = [sycl_libr...")], <3 more arguments>)
        File "/home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/local_config_sycl/sycl/BUILD", line 30, in cc_library
                sycl_library_path
name 'sycl_library_path' is not defined
ERROR: /home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/com_googlesource_code_re2/BUILD:96:1: First argument of 'load' must be a label and start with either '//', ':', or '@'. Use --incompatible_load_argument_is_label=false to temporarily disable this ch$
ck.
ERROR: /home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/com_googlesource_code_re2/BUILD:98:1: name 're2_test' is not defined (did you mean 'ios_test'?)
ERROR: /home/anton/.cache/bazel/_bazel_anton/d465f618840594936e5e9d3b94e6edb2/external/com_googlesource_code_re2/BUILD:100:1: name 're2_test' is not defined (did you mean 'ios_test'?)

[laszlocsomor]

@psyhtest : Great, thank you for the detailed instructions! I see similar errors.

Apparently CK pulls an older version of TensorFlow, namely https://github.com/tensorflow/tensorflow/commit/d752244fbaad5e4268243355046d30990f59418f, which I guess must be the baseline for 1.4.0, from November 1:

  $ cd ~/CK-TOOLS/lib-tensorflow-src-cpu-1.4-linux-64/src && git rev-parse HEAD
d752244fbaad5e4268243355046d30990f59418f

The load() errors ("First argument of 'load' must be a label ...") are definitely caused by Bazel 0.9.0, see release notes:

• First argument of 'load' must be a label. Path syntax is removed.
  (label should start with '//' or ':').

[psyhtest]

@laszlocsomor Thanks for confirming! As I said, a workaround is to keep using Bazel 0.8.1.

TensorFlow 1.4.0 is less than 2 months old and is practically the latest stable release (1.4.1 only contains a few bug fixes). Either they need to speed up or you guys to slow down :))

[laszlocsomor]

Fortunately TF is fixed at HEAD, so the next release will have the bugfix.

you guys [need] to slow down :))

Haha, I don't remember anyone saying that before ;)