Mediapipe: Object Detection tensorflow example with (Nvidia) GPU on Linux

[ksachdeva]

Hi,

On linux (Ubuntu 18.04) [that nvidia 1080 GPU] I have managed to run object_detection example with tensorflow calculators which I compiled using below command

bazel build -c opt \
        --define no_aws_support=true \
        --copt -DMESA_EGL_NO_X11_HEADERS 
    mediapipe/examples/desktop/object_detection:object_detection_tensorflow

Above compilation goes through and takes time because it is now compiling tensorflow as well.

However, I am not sure if it compiled tensorflow with CUDA support. I am familiar with compiling tensorflow separately with CUDA support and it seems that as part of mediapipe it is compiled only in CPU mode.

Please guide on how to ensure that I am using tensorflow with GPU support.

Regards & thanks
Kapil

[jiuqiant]

Yes, MediaPipe is with TensorFlow CPU by default because we need to support macOS, WSL, and also the Linux without NVIDIA GPUs.

I think the way to use TensorFlow GPU is

1. Clone the TensorFlow repo and config it to have the GPU support by specifying the path to CUDA and CuDNN .
2. Modify the org_tensorflow rule in the WORKSPACE file to be a new_local_repository that can point to your local TensorFlow directory that has the GPU configs.

I haven't tested this approach yet. But, it should work in theory.

[ksachdeva]

Thanks @jiuqiant

I found another approach; just finished testing and it seems to work.

Before issuing the command

bazel build -c opt \
        --define no_aws_support=true \
        --copt -DMESA_EGL_NO_X11_HEADERS 
    mediapipe/examples/desktop/object_detection:object_detection_tensorflow

I set following environment variables -

export TF_NEED_CUDA=1
export TF_CUDA_VERSION=10.0
export CUDA_TOOLKIT_PATH=/usr/local/cuda-10.0

and then in the console I can see compilation of various cuda files happening.

However, what I am still not sure of is if indeed GPU was used when session was executed. Would appreciate if you could confirm or deny that.

Regards
Kapil

[wolterlw]

@ksachdeva why don't you run nvidia-smi and check weather the process is using GPU resources?

[mgyong]

@ksachdeva Pls let us know if your method for setting environment variables works by checking nvidia-smi to see if GPU is used.

[ksachdeva]

Setting the above environment variables definitely compile with cuda as I see from .cu.cc files being compiled.

However, I do not see any change in gpu status usage with nvidia-smi.

I was reading the code for tensorflow session and inference calculator and see this comment in the protobuf

// [DEPRECATED] If true, this calculator will try to initialize local Tensor
// Processing Unit (TPU) hardware so that the Tensorflow session loaded from
// this saved model may benefit from TPU speedups. If you want to use this
// feature, you need to make sure that the calculator runs on a machine that
// has TPU hardware installed. The saved model should have correct device
// placements in the graph (have the ops already placed on TPU), typically if
// the saved model was exported through TPUEstimator then device placement is
// automatically taken care of.
optional bool use_tpu = 5 [deprecated = true];

https://github.com/google/mediapipe/blob/137867d0888a7e5a02089829360b3d32857cdad9/mediapipe/calculators/tensorflow/tensorflow_session_from_saved_model_generator.proto#L45

Based on above comment it seems like that it is expected that the saved_model should be constructed such that graphs (or nodes of graphs) are assigned to devices (GPU or TPU).

[d61h6k4]

Hi,
I follow this instruction to run Mediapipe's TensorFlow examples with CUDA.

1. Add build instructions to .bazelrc file (took them from official tensorflow repo)

# This config refers to building with CUDA available. It does not necessarily
# mean that we build CUDA op kernels.
build:using_cuda --define=using_cuda=true
build:using_cuda --action_env TF_NEED_CUDA=1
build:using_cuda --crosstool_top=@local_config_cuda//crosstool:toolchain

# This config refers to building CUDA op kernels with nvcc.
build:cuda --config=using_cuda
build:cuda --define=using_cuda_nvcc=true

1. Declare where is CUDA

export TF_CUDA_PATHS=/usr/local/cuda-10.1,/usr/lib/x86_64-linux-gnu,/usr/include

Latest Mediapipe depends on Tensorflow which requires cuda 10.1, that why I added /usr/local/cuda-10.1, but from 10.1 cuda bring cudablas as a separate library and store not in cuda folder, that why I added /usr/lib/x86_64-linux-gnu - here cudablas.so ,/usr/include - here cublas.h

Now, to build with cuda you only need to add --config cuda:

bazel build -c opt --config cuda \
        --define no_aws_support=true \
        --copt -DMESA_EGL_NO_X11_HEADERS 
    mediapipe/examples/desktop/object_detection:object_detection_tensorflow

[jiuqiant]

@d61h6k4, thanks for sharing your solution!

I also managed to run MediaPipe with TensorFlow GPU inference on a Google Cloud Ubuntu instance. Everything except the last step (the build command) works very well! I ran into some missing file issue during the build process. Then, I figured out that the trick is to append --spawn_strategy=local to the bazel command. The following command works for me.

bazel build -c opt --config cuda \
        --define no_aws_support=true \
        --copt -DMESA_EGL_NO_X11_HEADERS  \
        --spawn_strategy=local \
    mediapipe/examples/desktop/object_detection:object_detection_tensorflow

We are going to add the TensorFlow GPU inference setup instructions into our official documentation. Thanks again for your contribution.

[jiuqiant]

The documentation update is in https://github.com/google/mediapipe/commit/7bad8fce626ab3b0f41d818afac4df1e757c0969. Please find the detailed cuda setup steps at https://github.com/google/mediapipe/blob/master/mediapipe/docs/gpu.md#tensorflow-cuda-support-and-setup-on-linux-desktop