Tacotron2: train.py fails to train

[nikitos9000]

Hi @rafaelvalle , thanks for the great implementation!

But I can't start the training right as stated in your README (with pytorch-0.4.0):

$ python train.py -o outdir -l logdir
FP16 Run: False
Dynamic Loss Scaling True
Distributed Run: False
cuDNN Enabled: True
cuDNN Benchmark: False
/root/tacotron2/layers.py:35: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  self.conv.weight, gain=torch.nn.init.calculate_gain(w_init_gain))
/root/tacotron2/layers.py:15: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  gain=torch.nn.init.calculate_gain(w_init_gain))
Epoch: 0
Traceback (most recent call last):
  File "train.py", line 272, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 197, in train
    y_pred = model(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 110, in forward
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 121, in scatter
    return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/scatter_gather.py", line 36, in scatter_kwargs
    inputs = scatter(inputs, target_gpus, dim) if inputs else []
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/scatter_gather.py", line 29, in scatter
    return scatter_map(inputs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/scatter_gather.py", line 16, in scatter_map
    return list(zip(*map(scatter_map, obj)))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/scatter_gather.py", line 16, in scatter_map
    return list(zip(*map(scatter_map, obj)))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/scatter_gather.py", line 14, in scatter_map
    return Scatter.apply(target_gpus, None, dim, obj)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/_functions.py", line 74, in forward
    outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams)
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/comm.py", line 143, in scatter
    chunks = tensor.chunk(len(devices), dim)
RuntimeError: chunk expects at least a 1-dimensional tensor

Thanks!

[raulpuric]

Think I have a working version for 0.4 will submit a PR in an hour

[rafaelvalle]

@nsmetanin thanks for sharing the issue.
@raulpuric thanks for submitting the PR.

[raulpuric]

let me know if this works https://github.com/NVIDIA/tacotron2/tree/single-gpu-and-0.4

and I'll merge it in

[nikitos9000]

Unfortunately, still doesn't work:

Traceback (most recent call last):
  File "train.py", line 286, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 211, in train
    y_pred = model(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/tacotron2/model.py", line 511, in forward
    encoder_outputs, targets, memory_lengths=input_lengths)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/tacotron2/model.py", line 398, in forward
    memory, mask=~get_mask_from_lengths(memory_lengths))
  File "/root/tacotron2/utils.py", line 8, in get_mask_from_lengths
    ids = torch.arange(0, max_len, out=torch.LongTensor(max_len)).cuda()
TypeError: expected torch.LongTensor (got torch.cuda.LongTensor)

(and also a typo at line 197 in train.py if distributed_run and ... but should be if hparams.distributed_run and ...)

[rafaelvalle]

@nsmetanin please give it a try from master and let us know if it works for you.

[nikitos9000]

Still the same error from master:

FP16 Run: False
Dynamic Loss Scaling True
Distributed Run: False
cuDNN Enabled: True
cuDNN Benchmark: False
/root/tacotron2/layers.py:35: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  self.conv.weight, gain=torch.nn.init.calculate_gain(w_init_gain))
/root/tacotron2/layers.py:15: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  gain=torch.nn.init.calculate_gain(w_init_gain))
Epoch: 0
Traceback (most recent call last):
  File "train.py", line 285, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 210, in train
    y_pred = model(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/tacotron2/model.py", line 511, in forward
    encoder_outputs, targets, memory_lengths=input_lengths)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/tacotron2/model.py", line 398, in forward
    memory, mask=~get_mask_from_lengths(memory_lengths))
  File "/root/tacotron2/utils.py", line 8, in get_mask_from_lengths
    ids = torch.arange(0, max_len, out=torch.LongTensor(max_len)).cuda()
TypeError: expected torch.LongTensor (got torch.cuda.LongTensor)

[rafaelvalle]

@nsmetanin Are you trying to run on the CPU?

[nikitos9000]

No, I'm running on a GPU server inside the Docker container — it's based on nvidia/9.1-cudnn7-runtime-ubuntu16.04. Another PyTorch-based repos work fine. I've installed PyTorch version 0.4 with pip3 install http://download.pytorch.org/whl/cu91/torch-0.4.0-cp35-cp35m-linux_x86_64.whl (it seems that it's appropriate CUDA-9.1-compiled version)

BTW, if I add '.cpu()' to max_len = torch.max(lengths).cpu() on the line above, it fails with RuntimeError: slice() cannot be applied to a 0-dim tensor. on the same line

[rafaelvalle]

@nsmetanin in utils.py, please try replacing
ids = torch.arange(0, max_len, out=torch.LongTensor(max_len)).cuda() with
ids = torch.arange(0, max_len).long().cuda().

[raulpuric]

see https://github.com/NVIDIA/tacotron2/pull/4 too

[nikitos9000]

Great, thanks, now these lines work. But another problem arises:

Traceback (most recent call last):
  File "train.py", line 285, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 210, in train
    y_pred = model(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/tacotron2/model.py", line 522, in forward
    "constant", 0)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 1929, in pad
    return ConstantPadNd.apply(input, pad, value)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/padding.py", line 27, in forward
    output = input.new(input.size()[:(ctx.l_diff)] + new_dim).fill_(ctx.value)
TypeError: torch.Size() takes an iterable of 'int' (item 3 is 'Tensor')

(NB: I changed batch_size to = 24 because it didn't fit the 8 Gb of GPU memory, idk if it could caused this problem or not)

[raulpuric]

Are u on the official 0.4 wheel or top of tree?

[nikitos9000]

@raulpuric yes, official, right as I stated above — http://download.pytorch.org/whl/cu91/torch-0.4.0-cp35-cp35m-linux_x86_64.whl (just from pytorch.org)

[nikitos9000]

So the workaround is to use max_len.long().item() instead of max_len on line 521 in 'model.py' — but I'm not sure if it's the correct solution.

[raulpuric]

yep. That's one way to do it and should work through to v0.5

the version up in https://github.com/NVIDIA/tacotron2/tree/padding-patch-0.4 should work now for versions 0.3 and 0.4

will merge in an hour or so when others are convince as well :)

[rafaelvalle]

@nsmetanin The current code on master should work with your setup as well. Let us know

[nikitos9000]

It works now, great, thanks!

[gabrer]

@nsmetanin in utils.py, please try replacing
ids = torch.arange(0, max_len, out=torch.LongTensor(max_len)).cuda() with
ids = torch.arange(0, max_len).long().cuda().

I have got a similar issue with Pytorch 0.4.1 and have just fixed it changing torch.LongTensor() with .long().