Tensorflowtts: Are there any plans to train VCTK Multiband MelGAN model?

[abylouw]

I unfortunately do not have the computing resources to train a MB MelGAN model with VCTK.

My other option is to use the one from https://github.com/kan-bayashi/ParallelWaveGAN, and it is working nicely with my FastSpeech2 models from this repository trained on my languages, I would just prefer to have only tensorflow models as this will make the usage a lot simpler.

I have tried the conversion notebook https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/notebooks/convert_melgan_from_pytorch_to_tensorflow.ipynb, and it works for MelGAN, but not for MB MelGAN, as per my message here https://github.com/kan-bayashi/ParallelWaveGAN/issues/174

So my question is if there are any plans to train and release VCTK Multiband MelGAN models?

[dathudeptrai]

@abylouw i will help you convert weight from kan’s repo to my implementation :)) just be patient

[abylouw]

@dathudeptrai Thank you very much

[abylouw]

@dathudeptrai I have looked at the https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/notebooks/convert_melgan_from_pytorch_to_tensorflow.ipynb notebook again but I am not having any success.

I have adapted the notebook to use this repository's TFMelGANGenerator class, instead of the ParallelWaveGAN's one (there are small differences, ParallelWaveGAN uses conv2d whilst TensorflowTTS uses conv1d). I loaded the VCTK Multiband MelGAN model into the pytorch class (parallel_wavegan.models.MelGANGenerator) and then converted to TFMelGANGenerator. Both the ParallelWaveGAN's and TensorflowTTS give exactly the same output for the test example:

https://github.com/kan-bayashi/ParallelWaveGAN/blob/88a829b5ef923657b4fb590e0614dec5e106a971/notebooks/convert_melgan_from_pytorch_to_tensorflow.ipynb#L345-L348

fake_mels = np.random.sample((1, 80, 250)).astype(np.float32)
with torch.no_grad():
    y_pytorch = pytorch_melgan(torch.Tensor(fake_mels))
y_tensorflow = tf_melgan.predict(np.transpose(fake_mels, (0, 2, 1)))

But it does not correspond to the y_pytorch output. I do know that the shapes are different, (1, 4, 18750) for y_pytorch and (1, 18750, 4) for y_tensorflow in this example,
and I do a np.transpose(y_pytorch.numpy(), (0, 2, 1)) in order to compare the outputs, i.e. before the input to the PQMF. I would expect the outputs to be the same (as is the case when one runs the notebook example as is, i.e. for ljspeech and normal MelGAN), but they differ significantly.

Do you maybe have any ideas on why the conversion does not work?

[dathudeptrai]

@abylouw wait me a bit :)), i'm reviewing a repo then release the first version today then i will make all melgan/mb-melgan on kan's repo convertable to this framework, it should work since melgan already worked :D.

[dathudeptrai]

@abylouw can you make the fork of this repo then update the notebook u are using to convert melgan from kan’s repo to this framework? So i can diẻctly modify from that notebook.

[dathudeptrai]

@abylouw just want to remind you -_-.

[abylouw]

@dathudeptrai I got sidetracked, I will do it today.

[abylouw]

@dathudeptrai Here is my notebook: https://github.com/abylouw/TensorflowTTS/blob/master/notebooks/multiband_melgan_from_parallelwavegan_to_tensorflowtts.ipynb

[dathudeptrai]

@abylouw i will take a look this weekend :D

[ZDisket]

@abylouw How well does it work? Finetune or use as universal vocoder? Can you share audio samples?

[abylouw]

@ZDisket I want to compare fine-tuning to universal vocoder. I am not at my computer now, will share samples later

[dathudeptrai]

@abylouw give me the email then i will send you the notebook

[abylouw]

@dathudeptrai Thank you very much

• Edited to remove email

[dathudeptrai]

@abylouw i want to public the colab here but i found there is a man, who also do TTS, just copy and pass some core module here but didn't copyright and add reference, i don't want he continue to copy :)).

Pls confirm when it really work for you.

[ZDisket]

@dathudeptrai I got the link in my email and I'm requesting access cause I wanna look at it too since the link is set to private, you already know my email. Thanks

[abylouw]

@dathudeptrai Thank you very much, it is working. I need to study it a bit more to understand it.