Opennmt-py: Option -gpuid not working as it should

[goncalomcorreia]

When I use -gpuid option in current master of OpenNMT, regardless of what gpu I choose, the training script always chooses gpu 0.
If I use multigpu, for example, -gpuid 2 3, the training goes to gpus 0 and 1.

Is this a known issue?

[goncalomcorreia]

For anyone who's having this problem, if you want to train on gpus that are not gpu 0, you can change the visible devices with CUDA_VISIBLE_DEVICES. For example, doing export CUDA_VISIBLE_DEVICES=1,2 will make your experiments go to gpu 1 and then gpu 2 in the current master.

[vince62s]

AFAIK, this has always been the behavior of OpenNMT-py.

[rbommasanimozilla]

As far as I understand, this is really a pytorch+CUDA "inconsistency"

[goncalomcorreia]

@vince62s, I'm pretty sure this was not the behavior of OpenNMT-py as I used that option all the time to train on gpus other than 0, without changing the CUDA_VISIBLE_DEVICES

[vince62s]

well you know that multi gpu was not working before the recent changes, right ?

But I'll look into it.

[vince62s]

ok here is the thing.
BEFORE I introduced multi-gpu, the behavior was to take the GPU id of the command line.
But it worked only with single gpu.

WHEN I introduced the multi-gpu, I replicated the behavior of Fairseq, which clearly says we need to specify the CUDA_VISIBLE_DEVICES env var.

The main reason is that, it is ultimately capable of multi node training, and it is way easier to handle it this way.

Nevertheless, I take the point into account, but a PR would be welcome.
Cheers.

[goncalomcorreia]

Yes, I know multigpu wasn't working before and I was very excited to know that it was working now - it was actually the main reason I pulled the latest changes.

I think that since, as you said, it may be hard to have the previous behavior with multi gpu, the -gpuid option may be a bit misleading in that regard. Some possible solutions:

1) Not sure if this is possible, but we could change the CUDA_VISIBLE_DEVICES after parsing the gpuid option (not a fan of this one since it changes an env var without the user knowing)

2) Another possibility is to have something like -n_gpus instead, where you choose the number of gpus you want to use and then specify in the documentation that you need to change your CUDA_VISIBLE_DEVICES with the gpu ids that you are able to use in that moment

3) Or OpenNMT could just replicate Fairseq in that regard and just use all the gpus available as default and, as in the option above, specifying in the documentation that the env var CUDA_VISIBLE_DEVICES needs to be changed with the gpus the user wants to train on

Not sure which of these options is the better one. Personally, I would go for number 2.

I would be happy to do a pull request with the preferred behavior for this.

[vince62s]

1. would work in the short term, BUT not if we want to support multi node, it may require a bit more work (see gpu_rank option)

[goncalomcorreia]

Maybe just a heads up in the README then (in step 2's description)?

[alphadl]

although I have assigned the gpuid by using the CUDA_VISIBLE_DEVICES , the gpu 0 still be automatically used as a priority....

[sbmaruf]

@goncalomcorreia What is the problem of Solution number 1. we can do it by os.environ['CUDA_VISIBLE_DEVICES']. When you change the env var, you could put that into the log that you are changing the environment variable, and also the env change is only going to effect in that python run. According to me it's the easiest way to solve this issue.

@vince62s
Regarding the current solution, I also think it's not that so eyecatching but a clever way. But this is not properly documented in the documentation. Currently, it is written as,

-gpuid [] Use CUDA on the listed devices.

I think this is not that so clear at the first sight.

[vince62s]

guys,
a PR would be welcome, in the meantime I updated the FAQ to be clearer.

https://github.com/OpenNMT/OpenNMT-py/blob/master/docs/source/FAQ.md

[vince62s]

can you guys have a look at #962 and tell me if you're comfortable with these new options ?

[goncalomcorreia]

Just to be sure I got it right, does the model run in all gpus available by default so you can then choose which gpus you want to use with CUDA_VISIBLE_DEVICES?

[vince62s]

Not exactly.
First you need to set which GPU you want to use with CUDA_VISIBLE_DEVICES

Then case (1) on a single node, you need to set:
world_size = 3 (if you want to run an 3 GPU, given you need to have made 3 visible at least)
gpu_ranks = 0 1 2
I know, this looks a little cumbersome because you need to set 2 parameters.

But you will get it when looking at case (2), on 2 nodes:
Node 1:
master_ip=ip_address_node1
master_port=portnumber_node1
world_size = 7
gpu_ranks = 0 1 2
Node2:
masterip=ipaddress_node1
master_port=portnumber_node1
world_size = 7
gpuranks = 3 4 5 6

for node 1 you need to have set at least 3 cuda visible devices
for node 2 at least 4

[Henry-E]

I would recommend adding this to the opts section since it's not super clear the first time around what to do

[noe]

For me, it is not working on a single node, when trying to use nodes other than the first ones: my node has 8 GPUs and I want to use GPUs 2 and 3. I am setting --world_size 8 --gpu_ranks 2 3 and it gets stuck indefinitely. Setting --world_size 2 is of no use either. CUDA_VISIBLE_DEVICES is properly set to 2,3.

This happens with the current master: https://github.com/OpenNMT/OpenNMT-py/commit/ee810e509579d0a7d6f46ecefb2c9c7a646956c8

[vince62s]

Single node you need
gpu_ranks 0 1
world_size 2
And set cuda visible device as you need

[vince62s]

@Henry-E can you PR a doc / opts amendment ?

[Henry-E]

Sure

On Thu, Nov 8, 2018, 6:21 PM Vincent Nguyen [e-mail]
wrote:

@Henry-E https://github.com/Henry-E can you PR a doc / opts amendment ?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/OpenNMT/OpenNMT-py/issues/840#issuecomment-437084324,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMB2GCCp0vAaYUVR_o2PZRDFNE6iQc60ks5utGgtgaJpZM4VSzel
.

[noe]

Setting arguments as @vince62s suggested (-world_size 2 -gpu_ranks 0 1, with CUDA_VISIBLE_DEVICES=2,3) gets me this error:

Traceback (most recent call last):
  File "../../OpenNMT-py/train.py", line 118, in <module>
    main(opt)
  File "../../OpenNMT-py/train.py", line 48, in main
    p.join()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 124, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 50, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "../../OpenNMT-py/train.py", line 105, in signal_handler
    raise Exception(msg)
Exception:

-- Tracebacks above this line can probably
                 be ignored --

Traceback (most recent call last):
  File "/home/usuaris/veu/noe.casas/experiments2/eneu_lemma/OpenNMT-py/train.py", line 59, in run
    gpu_rank = onmt.utils.distributed.multi_init(opt, device_id)
  File "/home/usuaris/veu/noe.casas/experiments2/eneu_lemma/OpenNMT-py/onmt/utils/distributed.py", line 27, in multi_init
    world_size=dist_world_size, rank=opt.gpu_ranks[device_id])
  File "/home/usuaris/veu/noe.casas/experiments2/eneu_lemma/env/lib/python3.6/site-packages/torch/distributed/__init__.py", line 94, in init_process_group
    group_name, rank)
RuntimeError: Address already in use at /pytorch/torch/lib/THD/process_group/General.cpp:17

At that time, GPUs 0 and 1 were taken by other processes.

[vince62s]

"At that time, GPUs 0 and 1 were taken by other processes."

by a another onmt-py run or something else ?
monitor what you're doing with watch nvidia-smi

[noe]

GPUs 0 and 1 were taken by other non-onmt-py processes owned by other users of the machine. Target GPUs were 2 and 3, which were free, yet OpenNMT-py said "Address already in use". Consistently reproducible in my environment.

[vince62s]

As I said, please open a terminal and run "watch nvidia-smi"
run it again and see if your python processes take GPU 2 and 3

[noe]

The process never appears in nvidia-smi . The exception happens just after starting OpenNMT-py, so I ran OpenNMT-py several times (in a loop) while I was watching nvidia-smi and OpenNMT-py never appeared in the output.

However, I was mistaken: the problem is not 100% reproducible. In 2 different nodes I can reproduce it systematically but in a third node I can not reproduce it, so there are other factors going on here. I am running in a slurm cluster; in theory it should not affect how stuff works, but I am no expert in slurm, so not totally sure.

[BinWone]

The process never appears in nvidia-smi . The exception happens just after starting OpenNMT-py, so I ran OpenNMT-py several times (in a loop) while I was watching nvidia-smi and OpenNMT-py never appeared in the output.

However, I was mistaken: the problem is not 100% reproducible. In 2 different nodes I can reproduce it systematically but in a third node I can not reproduce it, so there are other factors going on here. I am running in a slurm cluster; in theory it should not affect how stuff works, but I am no expert in slurm, so not totally sure.

Yes, I encounter the same problem, have you solve it at last?

[noe]

Nope. I still think this is a problem related to OpenNMT-py, as I have never got anything alike with tensor2tensor or fairseq in the same environment, but I can't provide further info to properly diagnose it.