Tensorrt: The TensorRT support multi-gpus inference?

[SyGoing]

Description

hi,when there is multiple gpus on one computer, is there any operation to set which gpu to run the model. For example ,there are 2 gpus, I want the gpu 0 to run the model A and the gpu 1 to run the model B？

[jkjung-avt]

Set environment variable 'CUDA_VISIBLE_DEVICES=0' for process A, and 'CUDA_VISIBLE_DEVICES=1' for process B.

[SyGoing]

OK， thanks for your reply, I will try. And I find the caffe sets gpu id by the function--- static void SetDevice(const int device_id); So ,is there any function in the TensorRT's API

[rmccorm4]

From the FAQ: https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#faq

Q: How do I use TensorRT on multiple GPUs?

A: Each ICudaEngine object is bound to a specific GPU when it is instantiated, either by the builder or on deserialization. To select the GPU, use cudaSetDevice() before calling the builder or deserializing the engine. Each IExecutionContext is bound to the same GPU as the engine from which it was created. When calling execute() or enqueue(), ensure that the thread is associated with the correct device by calling cudaSetDevice() if necessary.

---

There is also some more details on this issue here: https://github.com/NVIDIA/TensorRT/issues/219#issuecomment-559249117

[SyGoing]

Sincerely thanks for your help @rmccorm4 @jkjung-avt, It works for me ~~~