Pipelines: bazel test fails at org_tensorflow/third_party/gpus/cuda_configure.bzl

[Ark-kun]

On a clean master checkout:

$ bazel test --action_env=PATH --define=grpc_no_ares=true //backend/...
Starting local Bazel server and connecting to it...
ERROR: /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/17d56a732ef8eb95c69f60d2038f8235/external/org_tensorflow/third_party/gpus/cuda_configure.bzl:115:1: load() statements must be called before any other statement. First non-load() statement appears at /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/17d56a732ef8eb95c69f60d2038f8235/external/org_tensorflow/third_party/gpus/cuda_configure.bzl:26:1. Use --incompatible_bzl_disallow_load_after_statement=false to temporarily disable this check.
ERROR: error loading package '': in /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/17d56a732ef8eb95c69f60d2038f8235/external/org_tensorflow/tensorflow/workspace.bzl: Label '@org_tensorflow//third_party:nccl/nccl_configure.bzl' crosses boundary of subpackage '@org_tensorflow//third_party/nccl' (perhaps you meant to put the colon here: '@org_tensorflow//third_party/nccl:nccl_configure.bzl'?)
ERROR: error loading package '': in /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/17d56a732ef8eb95c69f60d2038f8235/external/org_tensorflow/tensorflow/workspace.bzl: Label '@org_tensorflow//third_party:nccl/nccl_configure.bzl' crosses boundary of subpackage '@org_tensorflow//third_party/nccl' (perhaps you meant to put the colon here: '@org_tensorflow//third_party/nccl:nccl_configure.bzl'?)
INFO: Elapsed time: 7.180s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

https://github.com/tensorflow/tensorboard/issues/2181

P.S. Why are we building tensorflow as part of our backend?

[Ark-kun]

The problems are with Tensorflow bazel files.

Fixes are in master version (no release yet):
https://github.com/tensorflow/tensorflow/commit/494d2cc5cfd04c3c376f10b7d7c417e388b08282
https://github.com/tensorflow/tensorflow/commit/2243bd6ba9b36d43dbd5c0ede313853f187f5dce

[Ark-kun]

Workaround: downgrade bazel binary to .23 version.

[IronPan]

Update - It's now using bazel 0.24
https://github.com/kubeflow/pipelines/blob/f477a4feb490a796d2da611f0952f03456b25036/backend/Dockerfile#L1

[yanniszark]

@IronPan @Ark-kun bumping this issue since tensorflow 1.14 is out now and seems to fix this issue.

[Ark-kun]

With TF 1.14.0 I have the following build error on latest Bazel:

$ /usr/bin/bazel run //:gazelle
INFO: Writing tracer profile to '/usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/command.profile.gz'
INFO: SHA256 (https://github.com/tensorflow/tensorflow/archive/v1.14.0.tar.gz) = aa2a6a1daafa3af66807cfe0bc77bfe1144a9a53df9a96bab52e3e575b3047ed
DEBUG: Rule 'org_tensorflow' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "aa2a6a1daafa3af66807cfe0bc77bfe1144a9a53df9a96bab52e3e575b3047ed"
DEBUG: Call stack for the definition of repository 'org_tensorflow' which is a http_archive (rule definition at /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/external/bazel_tools/tools/build_defs/repo/http.b
zl:262:16):
 - /usr/local/google/home/avolkov/go/src/github.com/kubeflow/pipelines/WORKSPACE:39:1
ERROR: /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/external/org_tensorflow/tensorflow/workspace.bzl:18:1: file '@io_bazel_rules_closure//closure:defs.bzl' does not contain symbol 'filegroup_external'
ERROR: error loading package '': Extension file 'tensorflow/workspace.bzl' has errors

[Ark-kun]

@neuromage Does the backend still depend on tensorflow?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.