Tacotron2: Can't Figure Out How to Take Mel-Spectrogram from nvidia/tacotron2 to nv-wavenet/pytorch (I'm new to Pytorch)

[MXGray]

Got this error when I tried Tacotron2 inference.py (converted from ipynb) and took the mel-spectrogram to nv-wavenet/pytorch/inference.py:

RuntimeError: Expected 4-dimensional weight for 4-dimensional input [1, 1000, 513, 1], but got weight of size [80, 80, 1024] instead

I don't think I'm doing it right.
In tacotron2/inference.py, how do I use this?

mel = torch.load(spec_from_mel) #Or mel_outputs_postnet?
import audio_processing
from audio_processing import dynamic_range_decompression
mel = dynamic_range_decompression(mel)
mel = mel.cpu().numpy()
mel = mel.transpose()

And, how do I save that to a .pt tensor for nv-wavenet/pytorch/inference.py?
I'm doing this (and failing with the error above when I run the .pt file through nv-wavenet/pytorch/inference.py):

filename = '/opt/AITTSv3/nv-wavenet/pytorch/newdemopts/x.pt'
mel = torch.tensor(mel)
mel = torch.save(mel, filename)

Please help. Thanks!

[rafaelvalle]

nv-wavenet's inference.py takes a torch.FloatTensor with 2 dimensions, i.e. 80 by time.
In Tacotron2 you'll need to do this modifications to mel_outputs_postnet before saving:
remove the batch dimension, i.e. [1, 80. Time] -> [80, Time]
unwrap it from the Variable, i.e. mel_outputs_postnet = mel_outputs_postnet.data

You can also add something along these lines (not tested) to the ipython notebook and use nv-wavenet from there. With this snippet there's no need to modify mel_outputs_postnet

```

NV-Wavenet Synthesis on the notebook

import nv_wavenet
wavenet_path = '/path_to_wavenet/checkpoint'
wavenet = torch.load(wavenet_path)['model']
wavenet.eval()
nv_wavenet = nv_wavenet.NVWaveNet(**(wavenet.export_weights()))
nv_wavenet.add_cond_input(mel_outputs_postnet[:, :, :-1])
nv_wavenet_audio = nv_wavenet.infer()[0]
ipd.Audio(nv_wavenet_audio, rate=sampling_rate)
```

If you get it to work on the jupyter notebook, please send us a pull request such that we can add it to the repo!

[MXGray]

@rafaelvalle
Hi. I tried adding the lines you recommended to the ipython notebook (since I don't know how to do the other thing you said because I'm new to Pytorch), but:
I got this error:

File "inference.py", line 101, in
nv_wavenet.add_cond_input(mel_outputs_postnet[:, :, :-1])
AttributeError: 'NVWaveNet' object has no attribute 'add_cond_input'

So, I tried this:

cond_input = mel_outputs_postnet[:, :, :-1]
nv_wavenet_audio = nv_wavenet.infer(cond_input, 3) #3 is persistent in nv_wavenet.py

But, I got this:

cond_input.size()[0:3:2])
AssertionError: Inputs are channels x batch x num_layers x samples.
Channels and num_layers should be sizes: (128, 16)
But input is: torch.Size([1, 999])

Please help. Thanks!

[MXGray]

@rafaelvalle
Solved! Now to train Tacotron2 and NV-WaveNet more. :)
Here's what I did in tacotron2/inference.py:

mel = torch.autograd.Variable(mel_outputs_postnet)
mel = mel.reshape(80,1000)
mel = mel.data

filename = '/path-to-nv-wavenet/pytorch-directory/file.pt'
mel = torch.save(mel, filename)

Then, I just used subprocess to run nv-wavenet/pytorch/inference.py to process the newly saved .pt file.
I'll share predicted WAV samples after I've trained both Tacotron2 and NV-WaveNet more. Thanks! :)

[MXGray]

Solved with rafaelvalle's valuable guidance. :)

[dehanlazuardi]

@MXGray can you elaborate

Then, I just used subprocess to run nv-wavenet/pytorch/inference.py to process the newly saved .pt file

i try to make wav from mel_outputs_postnet
all i do is

1. convert mel_outputs_postnet to cond_input

with open('/home/dehan_l/project/speech_generation/nv-wavenet/pytorch/config-1.json') as f:
        data = f.read()
config = json.loads(data)
train_config = config["train_config"]
global data_config
data_config = config["data_config"]
global dist_config
dist_config = config["dist_config"]
global wavenet_config 
wavenet_config = config["wavenet_config"]

model_wavenet = wavenet.WaveNet(**wavenet_config).cuda()
cond_input = model_wavenet.get_cond_input(mel_outputs_postnet)

2 inference with pretrained model wavenet provided in README.MD

model = torch.load('/home/dehan_l/project/speech_generation/nv-wavenet/pytorch/downloaded/model.pt' )
wavenet = nv_wavenet.NVWaveNet(**model)

samples = wavenet.infer(cond_input, nv_wavenet.Impl.PERSISTENT)[0]

audio = utilswave.mu_law_decode_numpy(samples.cpu().numpy(), 256)
audio = utilswave.MAX_WAV_VALUE * audio
wavdata = audio.astype('int16')
write('/home/dehan_l/project/speech_generation/nv-wavenet/pytorch/downloaded/audio_tacoo.wav',16000, wavdata)

all i get noise output

i suspect there is different configuration in tacotron-2 and wavenet, (i use default configuration from both of them)

1. could someone point out where is my problem and what is the solution?
2. should i train my own wavenet model?, what configuration on nv-wavenet and tacotron-2 that has to be macthed?
3. how long is wavenet training?

@rafaelvalle