Serving: GPU support is not working

[nhe150]

git clone --recurse-submodules https://github.com/tensorflow/serving

configure tensorflow with gpu support

cd tensorflow
./configure

Got error:
ERROR: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '': Encountered error while reading extension file 'cuda/build_defs.bzl': no such package '@local_config_cuda//cuda': Traceback (most recent call last):
File "/mydockerhome/tf/serving/tensorflow/third_party/gpus/cuda_configure.bzl", line 473
_create_cuda_repository(repository_ctx)
File "/mydockerhome/tf/serving/tensorflow/third_party/gpus/cuda_configure.bzl", line 427, in _create_cuda_repository
repository_ctx.symlink(cudnn_lib_path, "cuda/" + symlink_...)
java.io.IOException: Could not create symlink from /mydockerhome/cudnn5_download/lib64/libcudnn.so to /root/.c
cd ..

build TF serving with cuda support

bazel build -c opt --config=cuda tensorflow_serving/...

Error:
Caused by: java.lang.IllegalStateException: com.google.devtools.build.lib.packages.NoSuchTargetException: no such target '@org_tensorflow//third_party/gpus/crosstool:crosstool': target 'crosstool' not declared in package 'third_party/gpus/crosstool' defined by /root/.cache/bazel/_bazel_root/5eb2099d3f39bf59c987d9a6eaa2e0cb/external/org_tensorflow/third_party/gpus/crosstool/BUILD

[kirilg]

@davidzchen I was also getting issues with GPU after autoconfig. Have you encountered this issue? Do you know what's causing it?

[promiseofcake]

I was able to get past this error when pinning to the latest HEAD of tensorflow locally as well as explicitly exporting all of my TF variables ahead of ./configure

Based upon having CUDA 7.5 && cuDNN 5.1.3 on my host machine.

export TF_NEED_CUDA=1
export TF_CUDA_VERSION=7.5
export TF_CUDNN_VERSION=5.1.3
export CUDA_TOOLKIT_PATH=/usr/local/cuda-7.5
export CUDNN_INSTALL_PATH=/usr/lib/x86_64-linux-gnu/ # based upon your cuDNN path
export TF_CUDA_COMPUTE_CAPABILITIES=3.0 #AWS compute
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-7.5/targets/x86_64-linux/lib:/usr/lib/x86_64-linux-gnu/

Running into additional errors after that however.

[davidzchen]

@nhe150 @kirilg Can you try using the latest tensorflow HEAD? I just submitted tensorflow/tensorflow#4285, which should fix this and which also prints more useful error messages if @local_config_cuda is not correctly configured to build with GPU support when building with --config=cuda, such as in this case that you ran into.

[kirilg]

@davidzchen using the latest TF @ head, I was able to run ./configure (previously was failing), but actual compilation of tensorflow_serving/... still failed with the same error mentioned by another user here.

[davidzchen]

I see. In that case, let's close this bug and continue on #186?