Tensorflowtts: synthesize test set

[Zak-SA]

Any tutorial on how to synthesize a test set or demo server using specific checkpoint while the training is ongoing?

Training is ongoing for Tacotron 2, I Decoded mel-spectrogram from folder ids for a checkpoint and I would like to listen to a synthesized sentences (wav) generated by the checkpoint, because TensorFlowTTS is not synthesizing a sentence with every checkpoint and only generating prediction graphs unlike Tacotron-2 (Rayhane-mamah) or Tacotron (Keithito).

I would appreciate the help.

[Zak-SA]

I tried the tacotron2_inference but I am getting the following error:

ValueError: Unable to load weights saved in HDF5 format into a subclassed Model which has not created its variables yet. Call the Model first, then load the weights.

tacotron2_inference saying not to build a model if we want to save it to PB

and how to synthesize the validation set to wav?

Thanks

[dathudeptrai]

@Zak-SA if you run the notebook, you should use tensorflow-gpu==2.2.0

[Zak-SA]

@Zak-SA if you run the notebook, you should use tensorflow-gpu==2.2.0

Thanks for your reply.

What other way to test the model and synthesize test sentences? And how to generate audio for the dev set?

[dathudeptrai]

@Zak-SA you can use GL (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/notebooks/griffin_lim_tensorflow.ipynb) here to convert mel predicted to audio. This is just for check because GL always noise, you should training vocoder. You can also try inference in colab https://colab.research.google.com/drive/1akxtrLZHKuMiQup00tzO2olCaN-y3KiD?usp=sharing as reference

[Zak-SA]

@Zak-SA you can use GL (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/notebooks/griffin_lim_tensorflow.ipynb) here to convert mel predicted to audio. This is just for check because GL always noise, you should training vocoder. You can also try inference in colab https://colab.research.google.com/drive/1akxtrLZHKuMiQup00tzO2olCaN-y3KiD?usp=sharing as reference

Thanks,

which vocoder you recommend to train and use?
Can you post the link to the vocoder you recommend?

Thanks

[dathudeptrai]

https://github.com/TensorSpeech/TensorFlowTTS/tree/master/examples/multiband_melgan

[Zak-SA]

https://github.com/TensorSpeech/TensorFlowTTS/tree/master/examples/multiband_melgan

Thanks. Highly appreciated

[Zak-SA]

what version of TensorFlow required for training Multi-band MelGAN? I am using TensorFlow-GPU==2.3 and I am getting this error when trying to train

File "examples/multiband_melgan/train_multiband_melgan.py", line 492, in
main()
File "examples/multiband_melgan/train_multiband_melgan.py", line 366, in main
] + 2 * config["multiband_melgan_generator"].get("aux_context_window", 0)
KeyError: 'multiband_melgan_generator'

[dathudeptrai]

@Zak-SA the current master :D

[Zak-SA]

sorry it was typo, I meant tensorflow

[Zak-SA]

@Zak-SA the current master :D

I updated the question, I am using Tensorflow-gpu==2.3 and getting am error when start training Multi-band MelGAN

[dathudeptrai]

@Zak-SA can you replace all multiband_melgan_generator to multiband_melgan_generator_params ?

[Zak-SA]

I did and it's working :)

one question, I got this message and i believe it's related to my gpu
2020-08-09 04:38:05.647319: W tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1972] No (suitable) GPUs detected, skipping auto_mixed_precision_cuda graph optimizer

would disabling the auto_mixed_precision affect the training?

Thanks again, I really appreciate how quick you respond to questions and issues

[dathudeptrai]

@Zak-SA what is ur GPU ? 2080TI or 1080Ti ?. This is jusst warning so everything should be ok :)).

[Zak-SA]

thanks, I am currently using old GPU, it's Titan black, I will order 2080TI to replace my current gpu

thanks again

[Zak-SA]

so it will be training generator with only stft loss for 400k step then resume and start training generator + discriminator starting from checkpoint 200k?

[Zak-SA]

actually it's training limit is 4m not 400k

is that normal?

[dathudeptrai]

@Zak-SA that normal, after 200k steps, the training progress will be stop :)) the nyou need resume and continue training. 1M steps is enough to get the best performance. 4M is just default setting, you can stop whenever you want :D.

[Zak-SA]

Thanks, I just got confused because the upper limit showed 4m. Thanks again

[Zak-SA]

training stuck and the last message is:
2020-08-09 05:16:44.399937: F tensorflow/stream_executor/cuda/cuda_fft.cc:435] failed to initialize batched cufft plan with customized allocator:

[dathudeptrai]

@Zak-SA this is a problem about CUDA version. Let try Cuda 10.1 and CuDNN 7.6.5. It should fix -_-.

[Zak-SA]

That's what i have, I have Cuda 10.1 and CuDNN 7.6.5

[Zak-SA]

I am re installing Ubuntu with Cuda 10.1 and CuDNN 7.6.5 and Tensorflow-gpu==2.3 then will try to train again

[Zak-SA]

I re installed Ubuntu, Cuda 10.1 , CuDNN 7.6.5 and tensorflow-gpu==2.3 and still getting the same error
Any ideas? Could it be gpu issue?

[dathudeptrai]

@Zak-SA https://github.com/keithito/tacotron/issues/300. CUDA 10.0/CuDNN 7.5 is the solution ?

[Zak-SA]

I will try with Cuda 10.0 and CuDNN 7.5

[Zak-SA]

I decided to get new GPU becauseI keep having issues, do you think RTX 2080 super will work fine for this task? And how different it's from RTX 2080TI ?
Thanks

[dathudeptrai]

@Zak-SA what do you mean? I use 2080Ti and everything is fine

[Zak-SA]

@dathudeptrai yes I know you using 2080TI, but I plan to get 2080 super not TI which have 8Gb memory instead of 11 for the TI. So i was asking if you think the 2080 super will be good for the task same as the one you are using (2080TI)

[Zak-SA]

@dathudeptrai if I managed to train the vocoder using my current gpu then I would wait for few more months but if I couldn't then I will probably go with the 2080 super. It should be much better than the 1080.
If I am not mistaken Cuda 10.0 and CuDNN 7.5 won't work with tensorflow-gpu==2.3 or even 2.2.
So I am still trying to debug the issues using Cuda 10.1 and CuDNN 7.6.5

[dathudeptrai]

@Zak-SA i do not know why this problem occur since it's not related with the code or model itself. 2080 is enough to training fastspeech/melgan/mb-melgan.

[Zak-SA]

@dathudeptrai it could be my gpu, old gpu may have problems with new updates and packages and when it comes to cuda dn tensorflow,. I will keep trying to troubleshoot from my side and I don't beleive it's the code but rather it's something either with my hardware or software. But I really appreciate your continuous help and quick response