Tacotron-2: WaveNet different sample_rate / separate training question

[gloriouskilka]

First of all, thank you for the great implementation of Taco2 with WaveNet.

I'd like to run WaveNet training while I'm still implementing/fixing my text data preprocessing, and I'm planning to maybe retrain that WaveNet model later after training Taco using its generated mel output.

Also, I'd like to have one WaveNet model to be able to generate several voices, if it'll improve overall quality or training time. I will train it to speak other voices later.

To start training the first voice right now, as I understood, I need to:

1. set gin_channels=16 https://github.com/Rayhane-mamah/Tacotron-2/blob/350d5e123532383c22f8e96258a306b2bdb7f6eb/hparams.py#L123
2. change speaker_id = '<no_g>' to speaker_id=0 removing Exception above https://github.com/Rayhane-mamah/Tacotron-2/blob/350d5e123532383c22f8e96258a306b2bdb7f6eb/datasets/wavenet_preprocessor.py#L130 https://github.com/Rayhane-mamah/Tacotron-2/blob/350d5e123532383c22f8e96258a306b2bdb7f6eb/tacotron/synthesizer.py#L112
3. set train_with_GTA = False here https://github.com/Rayhane-mamah/Tacotron-2/blob/350d5e123532383c22f8e96258a306b2bdb7f6eb/hparams.py#L201
4. Run wavenet_preprocess.py
5. Run train.py --model="WaveNet"

Am I doing it right? When I run WaveNet training, I got this assertion fall
https://github.com/Rayhane-mamah/Tacotron-2/blob/350d5e123532383c22f8e96258a306b2bdb7f6eb/wavenet_vocoder/models/wavenet.py#L478-L479

InvalidArgumentError (see above for traceback): assertion failed: [] [Condition x == y did not hold element-wise:] [x (model/inference/strided_slice_7:0) = ] [12000] [y (model/inference/strided_slice_8:0) = ] [8000]

If I set gin_channels=-1 back in hparams.py, training runs ok.

UPD: Nope, gin_channels=-1 and gin_channels=16 works the same.

I don't know why it didn't work previous times when I changed that and this, maybe I forgot to remove preprocessed files.

This error happens definitely when I change default hop_size and win_size to

    hop_size = 200,
    win_size = 800,

I chose these values because I changed sample_rate to 16000, I have audio in 16k.

Should I lower hop_size and win_size? How?

[Rayhane-mamah]

Hi, yes it seems to be most likely related to timesteps more than global conditioning. So is it working now?

For 16kHz, to have 50ms window size and 12.5ms frame shift, you ideally want to have something like the following:

• num_freq = 513, # (= n_fft / 2 + 1) only used when adding linear spectrograms post processing network
• n_fft = 1024, #Extra window size is filled with 0 paddings to match this parameter
• hop_size = 200, #For 22050Hz, 275 ~= 12.5 ms
• win_size = 800, #For 22050Hz, 1100 ~= 50 ms (If None, win_size = n_fft)
• sample_rate = 16000, #22050 Hz (corresponding to ljspeech dataset)

The rule for this is that win_size / sample_rate = time in seconds (ideally 0.05). hop_size should always be 1/4 of win_size to have 12.5ms shift. n_fft, I usually pick it to be the closest power of 2 that is bigger than win_size (in 16kHz it's 1024). num_freq is (n_fft /2 + 1). That's the whole logic in there. You start by picking the sample rate and build the parameters around it.

Please let me know how things go for you :)

[gloriouskilka]

Thanks for detailed reply! I didn't change num_freq and n_fft previous time.

Well, it worked when I used default parameters. But when I change them all to values you gave, I get same assertion.

Exiting due to exception: assertion failed: [] [Condition x == y did not hold element-wise:] [x (model/inference/strided_slice_7:0) = ] [12000] [y (model/inference/strided_slice_8:0) = ] [8000]

Interesting. Maybe I should just switch my audio to 22050Hz and use default parameters, as they just work, and forget about this problem. But probably my whole training time would increase significantly. Hm.

[Rayhane-mamah]

Oh there's a detail I forgot to mention: when working with Wavenet you also need to change this parameter:
https://github.com/Rayhane-mamah/Tacotron-2/blob/ffeca51f9196c8a013bed017276cd16c1a9c2956/hparams.py#L119

upsample_scales elements product needs to be equal to hop_size. i.e: hop_size = prod(upsample_scales). In default parameters, there is 300 = 15 * 20 so the property is verified.

When switching to 16kHz, normally hop size would be 200, so the upsamples scales should be something like [10, 20] (200 = 10 * 20). Please also make sure you have that change made. If the problem still persists afterwards, please post the entire crash log (In order for me to efficiently pinpoint the bug location). thanks :)

[gloriouskilka]

That fixed the problem, thanks :)

When I was trying to figure out why that assertion should be there, I noticed that some upsampling is happening.

Also searched every usage of parameters I changed but didn't find this comment about "hop size" because there is a space symbol in it. Probably it'll be better for future generations to change that comment to prod(scales) should be equal to hop_size with variable name in it?

[Rayhane-mamah]

Yes you're probably right, I am setting this modification for next documentation update. thanks for your contributions :)

Feel free to reopen the issue if you find that there's still something wrong related to upsampling with different audio parameters ;)

[hiyassat]

thank you rayhane-mamah
the folowng params work for me
num_freq = 513, # (= n_fft / 2 + 1) only used when adding linear spectrograms post processing network
n_fft = 1024, #Extra window size is filled with 0 paddings to match this parameter
hop_size = 200, #For 22050Hz, 275 ~= 12.5 ms
win_size = 800, #For 22050Hz, 1100 ~= 50 ms (If None, win_size = n_fft)
sample_rate = 16000, #22050 Hz (corresponding to ljspeech dataset)
upsample_scales = [10, 20]