Serving: HOW CAN I limit the GPU's MEMORY .

[goodluck4s]

I want to deploy a model by tensorflowServing+nvidia-docker on GPU .
HOW CAN I limit the GPU's MEMORY .
cause it's tend to use all memory of GPU .
I want to limit the GPU memory used to below 5G (10G in total) .

[gowthamkpr]

@goodluck4s This is not related to serving, its related to tensorflow itself. You have to limit the the tensorflow models memory growth itself. You can go through the following issue and the following article to understand how to limit GPU usage by tensorflow.

[goodluck4s]

Thanks for your reply, ANd I know how to solve this problem

[goodluck4s]

if i use the following way to run my model serve , Is there a suitable environment variable to control GPU's memory usage

docker run --runtime=nvidia -p 8501:8501 \
--mount type=bind, \
source=/tmp/tfserving/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu, \
target=/models/half_plus_two \
-e NVIDIA_VISIBLE_DEVICES=3 \
-e MODEL_NAME=half_plus_two -t tensorflow/serving:latest-gpu &

[gowthamkpr]

@goodluck4s You can take a look at this comment where your question has been answered. Thanks!

[goodluck4s]

thanks for you reply and i had solve this problem

[gowthamkpr]

@goodluck4s Thank you for the reply. I am closing this issue as it has been resolved.

[junneyang]

@gowthamkpr how to solve this problem, thanks