Opennmt-py: How do I train on multiple GPUs? Is it supported yet?

[justinas-kazanavicius]

I see these lines in train.py

if len(opt.gpuid) > 1:
   sys.stderr.write("Sorry, multigpu isn't supported yet, coming soon!\n")

But the repo description says:

Beta Features (committed):

• multi-GPU

So, is it supported now? If so, how do I do multi GPU training?

[srush]

We're working on it now. It's top priority.

[bezigon]

How about Horovod? https://github.com/uber/horovod#pytorch

[ghost]

@bezigon Do you make it works with the combination of Horovod and OpenNMT-pytorch ? Is it as simple as the usage case in Horovod's ReadME ?

[ghost]

Hi @srush , do you have a rough schedule that the multiple-GPU features is available ? Thx

[sajastu]

@srush I have the same problem. Has anyone solved this issue? Does OpenNMT-py support multi-gpu processing?

Thanks,

[vince62s]

Read the faq.md

[sajastu]

@vince62s Thanks for the guideline. I actually used the world_size and gpu_ranks but it didn't work with the following error:

RuntimeError: the distributed NCCL backend is not available; try to recompile the THD package with CUDA and NCCL 2+ support at /opt/conda/conda-bld/pytorch_1533672544752/work/torch/lib/THD/process_group/General.cpp:17

The code I was running:
python train.py -save_model models/radModel -data data/radModel -copy_attn -layers 1 -reuse_copy_attn -world_size 2 -gpu_ranks 0 1 -batch_size 20

I also exprted CUDA_VISIBLE_DEVICES=0,1

[shahbazsyed]

@sajastu Did you try installing NCCL?
You can find the instructions here