Bazel: Mysterious "cannot find .bzl file" error

[lberki]

I was trying to learn a bit of TensorFlow, so I started building it from source. Up came a nice error message:

ERROR: /home/lberki/tf/tensorflow/tensorflow/python/BUILD:1: Extension file not found: 'google/protobuf/protobuf.bzl'.
ERROR: /home/lberki/tf/tensorflow/tensorflow/tensorboard/BUILD:43:1: error loading package 'tensorflow/python': Extension file not found: 'google/protobuf/protobuf.bzl' and referenced by '//tensorflow/tensorboard:tensorboard'.

Turns out, protobuf.bzl is not loaded from tensorflow/python/BUILD, but from tensorflow/core/platform/default/build_config.bzl, which _is_ referenced from the BUILD file.

It would be nice to have some indication of this (instead of me tracking down which files are loaded recursively)

[davidzchen]

@lberki - Just to confirm, did you pull in all the git submodules in the tensorflow repo? I got the same error when I forgot to pull in the protobuf submodule.

[lberki]

The problem was, of course, a missing --recurse-submodules command line option, but the error message is highly non-intuitive. I work with Bazel since a looong time, and if I had problems tracking it down, people with less experience will have problems, too.

[davidzchen]

Agreed. A number of other TensorFlow users have also been confused by the same error message caused by this exact problem.

[aliabbasjp]

@davidzchen

I get following error:

$bazel run -c opt //tensorflow/models/rnn/translate:translate -- --data_dir ./data_dir --train_dir ./checkpoints_directory --en_vocab_size=40000 --fr_vocab_size=40000

ERROR: /home/temp_user/tensorflow/tensorflow/python/BUILD:1: Extension file not found: 'google/protobuf/protobuf.bzl'.
ERROR: /home/temp_user/tensorflow/tensorflow/BUILD:40:1: error loading package 'tensorflow/python': Extension file not found: 'google/protobuf/protobuf.bzl' and referenced by '//tensorflow:tensorflow_py'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 0.119s
ERROR: Build failed. Not running target.
temp_user@versionx:~/tensorflow$ bazel run -c opt //tensorflow/models/rnn/translate:translate -- --data_dir ./data_dir --train_dir ./checkpoints_directory --en_vocab_size=40000 --fr_vocab_size=40000 google/protobuf/protobuf.bzl^C
temp_user@versionx:~/tensorflow$ bazel run -c opt //tensorflow/models/rnn/translate:translate -- --data_dir ./data_dir --train_dir ./checkpoints_directory --en_vocab_size=40000 --fr_vocab_size=40000 --recurse-submodules
ERROR: /home/temp_user/tensorflow/tensorflow/python/BUILD:1: Extension file not found: 'google/protobuf/protobuf.bzl'.
ERROR: /home/temp_user/tensorflow/tensorflow/BUILD:40:1: error loading package 'tensorflow/python': Extension file not found: 'google/protobuf/protobuf.bzl' and referenced by '//tensorflow:tensorflow_py'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 0.101s
ERROR: Build failed. Not running target.

[davidzchen]

@aliabbasjp Did you clone the tensorflow repository with the --recurse-submodules flag as described in the Installing from Sources docs?

git clone --recurse-submodules https://github.com/tensorflow/tensorflow

Try running git submodule --init update in your repository and building again.

[edwinter]

FYI, this is the correct command on a MAC to fix: git submodule update --init (git version 2.0.2)

[davidzchen]

Sorry, my comment had a typo. As @edwinter said, the correct command is git submodule update --init.

[vuw-ecs-kevin]

The problem was, of course, a missing --recurse-submodules command line option,
but the error message is highly non-intuitive. I work with Bazel since a looong time,
and if I had problems tracking it down, people with less experience will have problems, too.

I am such a person. I had such problems. Only a web search that turned up this post
enabled me to track down what's actually going on: so a massive thank-you.

This has been my first experience of Bazel and I wanted to say that it's even less intuitive
than you suggest if you don't want to be on the cutting edge and so download a TensorFlow
"release", because their most recent "release" tarball doesn't contain the required submodules
and nor do those tarball contents suggest that you might need something in the empty directory
that you do get.

You first get told that that directory needs a BUILD file, but that it doesn't have to do anything,
however, if you "touch" the file, which is a pretty standard way to create a file that doesn't do
anything, you then get the "missing bzl file" messages.

Anyroad, once again, many thanks for documenting this here.

[damienmg]

This should be closed the message is reasonable. We should push TensorFlow to use a git_repository instead of git submodules.