Tacotron2: Distributed setup bug

[IwraStudios]

I got this error while executing the distributed setup described in README:

"./distributed.py", line 95, in allreduce_hook
AttributeError: 'Parameter' object has no attribute '_execution_engine'

With a bit of searching it seems to be related to: https://github.com/pytorch/pytorch/commit/6b3a4637d6e87a543b3df6edb825c07d7dabe611#diff-28369e7c07a82d014949812f10d9a188

(And it also seems the same problem is/was in https://github.com/NVIDIA/sentiment-discovery/blob/master/model/distributed.py)

[rafaelvalle]

@IwraStudios thanks for sharing.
@raulpuric any thoughts on this given that we had a similar problem in our sentiment-discovery repo?

[IwraStudios]

@rafaelvalle Found the issue from sentiment-discovery on the topic https://github.com/NVIDIA/sentiment-discovery/issues/16
https://github.com/NVIDIA/sentiment-discovery/pull/17/commits/919d4c86d5c66e29a9d3fc256e1a90659d67902a
I'll test it out

[IwraStudios]

@rafaelvalle https://github.com/NVIDIA/sentiment-discovery/commit/919d4c86d5c66e29a9d3fc256e1a90659d67902a Fixes the issue for tacotron2.

But now there is a new error (for fp16_run=True only, normal works fine):
RuntimeError: Expected a Tensor of type torch.cuda.HalfTensor but found a type torch.cuda.FloatTensor for sequence element 3 in sequence argument at position #1 'tensors'

[rafaelvalle]

@IwraStudios can you share the full error trace?

[IwraStudios]

Ah right sorry about that

Traceback (most recent call last):
  File "train.py", line 291, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 222, in train
    optimizer.backward(loss)
  File "nv-wavenet/pytorch/tacotron2/fp16_optimizer.py", line 362, in backward
    self.loss_scaler.backward(loss.float())
  File "nv-wavenet/pytorch/tacotron2/loss_scaler.py", line 80, in backward
    scaled_loss.backward()
  File "anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 89, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "nv-wavenet/pytorch/tacotron2/distributed.py", line 88, in allreduce_params
    coalesced = _flatten_dense_tensors(grads)
  File "nv-wavenet/pytorch/tacotron2/distributed.py", line 19, in _flatten_dense_tensors
    flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)
RuntimeError: Expected a Tensor of type torch.cuda.HalfTensor but found a type torch.cuda.FloatTensor for sequence element 3  in sequence argument at position #1 'tensors'

[rafaelvalle]

@IwraStudios thank you. we currently do not have the cycles to work on fixing this bug and need to migrate the current code to NVIDIA's Automatic Mixed Precicion (AMP) library.
If you're ok with using DistributedDataParallel without FP16, just disable FP16 by setting fp16_run=False.

[IwraStudios]

Alright, I'll disable it for now.
Shall I close the issue or do you want to keep it open until fp16 gets fixed?

[rafaelvalle]

Let's keep it open as a record that the issue has to be solved!

[yhgon]

@IwraStudios

One solution for distributed FP16 training for Tacotron2 would be use NGC:18.03-py3 and simple patch.
I already build docker images you would just use docker pull hryu/pytorch:t2-2 it is base on NGC 18.03 which based on pytorch 0.3 and APEX patch. see detail Dockerbuild file. after that, you need to implement simple patch for source codengc-patch for pytorch 0.3 this patch don't support pytorch 0.4. so it is just temporal solution.

detail instruction are in readme file

[yhgon]

@IwraStudios close this bug.

I check FP16 training work well in NGC 18.06-py3 with below method.

replace current distributed.py with new apex distributed.py

python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True
['train.py', '--output_directory=outdir', '--log_directory=logdir', '--hparams=distributed_run=True,fp16_run=True', '--n_gpus=2', '--group_name=group_2018_07_04-043620', '--rank=0']
['train.py', '--output_directory=outdir', '--log_directory=logdir', '--hparams=distributed_run=True,fp16_run=True', '--n_gpus=2', '--group_name=group_2018_07_04-043620', '--rank=1']
FP16 Run: True
Dynamic Loss Scaling True
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing distributed
Done initializing distributed


Epoch:     0
train.py:200: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
  if hparams.distributed_run else loss.data[0]
train.py:200: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
  if hparams.distributed_run else loss.data[0]
OVERFLOW! Skipping step. Attempted loss scale: 4294967296
OVERFLOW! Skipping step. Attempted loss scale: 2147483648.0
OVERFLOW! Skipping step. Attempted loss scale: 1073741824.0
OVERFLOW! Skipping step. Attempted loss scale: 536870912.0
OVERFLOW! Skipping step. Attempted loss scale: 268435456.0
OVERFLOW! Skipping step. Attempted loss scale: 134217728.0
OVERFLOW! Skipping step. Attempted loss scale: 67108864.0
OVERFLOW! Skipping step. Attempted loss scale: 33554432.0
OVERFLOW! Skipping step. Attempted loss scale: 16777216.0
OVERFLOW! Skipping step. Attempted loss scale: 8388608.0
OVERFLOW! Skipping step. Attempted loss scale: 4194304.0
OVERFLOW! Skipping step. Attempted loss scale: 2097152.0
OVERFLOW! Skipping step. Attempted loss scale: 1048576.0
OVERFLOW! Skipping step. Attempted loss scale: 524288.0
OVERFLOW! Skipping step. Attempted loss scale: 262144.0
OVERFLOW! Skipping step. Attempted loss scale: 131072.0
/scratch/github/tacotron2/fp16_optimizer.py:173: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.
  return torch.nn.utils.clip_grad_norm(fp32_params, clip)
Train iter       16 loss 49.316841 Grad Norm 10.800460 3.22s/it
/scratch/github/tacotron2/fp16_optimizer.py:173: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.
  return torch.nn.utils.clip_grad_norm(fp32_params, clip)
OVERFLOW! Skipping step. Attempted loss scale: 65536.0
Train iter       18 loss 40.572620 Grad Norm 24.962288 3.18s/it
Train iter       19 loss 23.865871 Grad Norm 18.794416 3.18s/it
Train iter       20 loss 13.441618 Grad Norm 15.939060 3.23s/it

[jferrer21]

@yhgon

Do you know (estimate) how long would take to train nv-wavenet with LJSpeech dataset and
1 Tesla V100 GPU?

[hdmjdp]

@IwraStudios @yhgon I think you can just " flat = torch.cat([t.half().contiguous().view(-1) for t in tensors], dim=0)" to fix this.