Carla: How to set a specific GPU device to use when running multiple thread?

[nina124]

I am implementing A3C algorithm with CARLA, but by default, it always use GPU0.
How set a specific GPU device to use?

[mvpcom]

Dear @nina124, Which framework are you using? That should not be depended on CARLA for training your network. You can easily configure the GPU using these commands for various frameworks:

Tensorflow:

with tf.device('/gpu:1')

Keras:

from keras import backend as K
import tensorflow as tf
with K.tf.device('/gpu:1'):
    config = tf.ConfigProto(intra_op_parallelism_threads=4,\
           inter_op_parallelism_threads=4, allow_soft_placement=True,\
           device_count = {'CPU' : 1, 'GPU' : 1})
    session = tf.Session(config=config)
    K.set_session(session)

Theano:
THEANO_FLAGS='floatX=float32,device=cuda0,gpuarray.preallocate=1' python <yourScript>.py
or even change the Theano rc file ($HOME/.theanorc):

[global]
floatX = float32
device = cuda0

[gpuarray]
preallocate = 1

Other Methods:
The other way is using an environmental variable, only works when used before starting your script:
export CUDA_VISIBLE_DEVICES=1

You can even set that environmental variable in your python code before importing framework:

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" 
os.environ["CUDA_VISIBLE_DEVICES"] = ""

[nina124]

Hi, @mvpcom
Thanks a lot for your reply! However, I do not mean to specify the GPU card for training the algorithm :)
CARLA utilizes the default GPU device 0 to render frames. With A3C, multiple threads of CARLA need to be run at the same. Even when not using neural network, running these multiple CARLAs with the same GPU0 to render is impractical, so I need to specify which GPU device to render for each CARLA thread.
By the way, I tried CUDA_VISIBLE_DEVICES=2 ./CarlaUE4.sh /Game/Maps/Town01 -carla-server -benchmark -fps=15 -windowed -ResX=800 -ResY=600 and CUDA_VISIBLE_DEVICES=2 ./client_example.py -a, it still uses the default GPU0...

[felipecode]

Hey @nina124
We have a solution for that.
Our solution uses vnc and virtualgl for running headless and selecting GPU. I will try to organize and post it in a while. It needs a few steps to make it work.

[nina124]

@felipecode Thank you very much!
Really look forward to your post. I have been searching the solution for several days!
And I also suggest add it into the documentation. Many researchers may need it :)

[nina124]

Hi, @felipecode @nsubiron
How are things going?
I am still waiting for the solution. Hope to try the CARLA environment soon.

[felipecode]

Hey @nina124,
Sorry for the delay to answer. Right now I have an experimental tutorial that we used it
to select GPU in our servers. I have to check a few things before making a pull request for
the 0.72. But, If you could give it a try and send me feedback it would be very useful !
Here is the link:
https://github.com/carla-simulator/carla/blob/tutorial_for_running_headless/Docs/carla_on_server.md

Cheers

[nina124]

Hey @felipecode
I just tried the tutorial. Simply following the tutorial, it runs successfully!

Further information:
Ubuntu 16.04, NVIDIA driver 387.26, CARLA 0.7.0

I ran a total of 12 CARLA servers simultaneously (3 GTX 1080 cards, 4 instances each), each with

DISPLAY=:8 vglrun -d :7.<gpu_number> ./CarlaUE4.sh /Game/Maps/Town02 -carla-server -benchmark -fps=15 -carla-world-port=xxxx
python PythonClient/client_example.py -a -p xxxx

But I havent tried training A3C with multiple CARLA environments.

[laksgreen]

@mvpcom

Thanks, It's working to me also, I would like to use 3 GPUs and 8 GPUs for a single docker container, Is it possible to implement in Turicreate and Tensorflow?

Thanks,

[pranavAL]

Hi @felipecode @nsubiron is there any specific reason for Carla server to have extremely low fps on a remote gpu server with multiple GPU but works fine on a laptop with single GPU and lower configuration in comparison to the GPU server