Tacotron2: multi-gpu error

[Jiff-Zhang]

hello, everyone, I am trying to run the tacotron2 with multi-gpu, my command is
python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True

the output as following:
FP16 Run: True
Dynamic Loss Scaling: True
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing distributed

and it hangs and never go to the training step. If there any advice for this problem? Thanks for your reply sincerely.

[gsoul]

Please take a look here: https://github.com/NVIDIA/tacotron2/issues/56

[Jiff-Zhang]

@gsoul yeah, thanks for your suggestion, I have viewed #56, but I am not familiar with docker, so I wonder if there is another solution for this problem.

[gsoul]

do you have "distributed.dpt" file in the root of the repo folder and did you try to manually delete it?

[Jiff-Zhang]

yeah, the "distribted.dpt" is in my repo folder, Does it work to delete it?

[rafaelvalle]

Yes. Delete the file and please confirm that it works.

[Jiff-Zhang]

@rafaelvalle @gsoul yeah, thanks for your advice. But there still a problem, finally I replace distributed.py with APEX distributed.py file and it works. Hope to be helpful to someone newly here.

[begeekmyfriend]

It also works for me. @rafaelvalle why not replace distributed.py with this one on your repo?