Tacotron2: How to combine these models with gst-tacotron,

[shwetagargade216]

Hello, @rafaelvalle Really nice work.

I would like to combine this work with global style tokens.. please let me know

[rafaelvalle]

That's is a great idea, @shwetagargade216.
Please let us know and add a pull request if you decide to implement the global style tokens!

[Yeongtae]

I have done to implement prosody modeling in https://arxiv.org/abs/1803.09047 without GST.
https://github.com/Yeongtae/tacotron2/tree/prosody_speaker_embedding_test
This branch can learn a model which is f(ref audio, speaker index, text) = target audio
Global style tokens will be added soon.

[rafaelvalle]

@Yeongtae what do you mean by GST? The term is not explicitly on the paper.
Can you submit a PR with audio samples?

[Yeongtae]

@Yeongtae what do you mean by GST? The term is not explicitly on the paper.
Can you submit a PR with audio samples?

@rafaelvalle The paper of global style token is the extension of https://arxiv.org/abs/1803.09047.
If someone implements attention between randomly initialized embeddings and reference encoder in my branch, it makes work easily.

Because there is so many side job in my branches, such as Korean char embedding, audio preprocessing, melspectrogram nomalization, GTA synthesis, etc,... , it makes original repository complicate if i submit a PR. In addition, I saw an your reply that you would like to make original repository simple.

If I do some engineering to make a branch simple, I will consider submit a PR.
Thanks for your attention.

[pravn]

GST works here:
https://google.github.io/tacotron/publications/global_style_tokens/
https://arxiv.org/abs/1803.09017
https://arxiv.org/abs/1803.09047

We need to implement various kinds of style encoders (@yongtae which repo
is this?)
See also the new work which is an GAN improvement on style tokens:
https://openreview.net/forum?id=ByzcS3AcYX

pravn.wordpress.com

On Wed, Feb 13, 2019 at 5:46 PM Yeongtae [e-mail] wrote:

@Yeongtae https://github.com/Yeongtae what do you mean by GST? The term
is not explicitly on the paper.
Can you submit a PR with audio samples?

@rafaelvalle https://github.com/rafaelvalle The paper of global style
token is the extension of https://arxiv.org/abs/1803.09047.
If someone implements attention between randomly initialized embeddings
and reference encoder in my branch, it makes work easily.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/134#issuecomment-463451484,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhnVfagGbmOUBQzFfkV9c-RxE5rc3WTks5vNL_ZgaJpZM4abvBC
.

[Yeongtae]

@pravn
https://github.com/Yeongtae/tacotron2/tree/prosody_encoder_test
It works for only one speaker.
https://github.com/Yeongtae/tacotron2/tree/prosody_speaker_embedding_test
It works for multiple speakers.

There is an another extension of GST.
The main problem of GST is loosing time specific information.
This paper overcomes the limitation.
https://arxiv.org/abs/1811.02122

[shwetagargade216]

@Yeongtae Could you please share the samples of gst .

[shwetagargade216]

And also @Yeongtae have you did multi - speaker gst ?
Means Is it possible to combine

1. @rafaelvalle (tacotron2 + waveglow model) which is in pytorch with the
2. gst (syang1919) but mainly this work is in tensorflow
3. also multi-speaker ( for voice cloning) just like biadu voice cloning with few sample

I trying to combine all the above 3 work together.
please give me any comments on this or guidance?
Thanks in advance

[Yeongtae]

• I can't share samples because of private issue in my Korean data.
• In addition, I'm fail to converge English model using M-AILAB dataset. There is a difference which uses phone embedding or char embedding between Korean and English. char embedding makes it hard to converge.

1. There is need to few modification for combining waveglow, such as inference function
   -waveglow(f(ref audio,speaker id, text) = target audio.
2. As mentioned earlier, you can easily implement it to combine attention between randomly initialized embeddings and reference encoder in my branch.
3. In my opinion, There is no too much implementation for speaker adaptation such as https://google.github.io/tacotron/publications/speaker_adaptation/.
   The main problem is computation.
   Since the dataset volumes in speaker adaptation papers are huge, you must have distributed learning system.
   In more detail, You need n v100 GPUs to learn model.
   In my case, I'm using 8 v100 GPUs in GCP VM. It is so so so~~~ expensive.

[shwetagargade216]

Thanks for the quick reply. Your comments are really appreciated.
For now will just combine @rafaelvalle work with your multi-head attention for reference audio just like gst.
One more question currently I'm also working on M-ALIBA data, but I gone through other issues, which has mentioned that, corresponding data set has some issues, even though after cleaning results are not good like ljspeech.
Should I try Ljspeech data or M-ALIAB. Ow I need to some extra pre-processing and need to clear it before the training.

[Yeongtae]

Phone embedding and melspectrogram normalization mainly improve converging model.

If I take some progress, I will report it.

[rafaelvalle]

@Yeongtae you can use ARPAbet for english if you think a phone-like embedding will be easier than char embedding.

[jaywantmahajan]

@Yeongtae , nice work.
1 - Is your "https://github.com/Yeongtae/tacotron2/tree/prosody_speaker_embedding_test" is complete or still in progress?
2 - Which dataset you have successfully used to train your implementation?

[yxt132]

@Yeongtae you can use ARPAbet for english if you think a phone-like embedding will be easier than char embedding.

@rafaelvalle Can you elaborate on how to use ARPAbet? I am curious how it will improve the convergence and give it a try. Thanks.

[yxt132]

3. https://google.github.io/tacotron/publications/speaker_adaptation/

@Yeongtae Can you tell a little bit the Korean training corpus you used? Roughly how many speakers and how many hours of data in total? Thanks. In your opinion, what is the advantage of prosody speaker embedding (or speaker verification) versus GST? If we can successfully implement GST, why would we still need prosody speaker embedding model?

[Yeongtae]

1. https://google.github.io/tacotron/publications/speaker_adaptation/

@Yeongtae Can you tell a little bit the Korean training corpus you used? Roughly how many speakers and how many hours of data in total? Thanks. In your opinion, what is the advantage of prosody speaker embedding (or speaker verification) versus GST? If we can successfully implement GST, why would we still need prosody speaker embedding model?

1. My private Korean dataset consist of 5 speaker, around 50 hour, unbalanced data volumes for each.
   
   
   
   • In my test, only a model for a speaker with a lot of data and various styles makes different results, such as difference pitch, accent.
   
   
2. There is no advantage. It's just previous work of GST.

[shwetagargade216]

@Yeongtae @rafaelvalle
deepvoice3 and neural voice cloning with few samples both have implemented speaker adaption on single speaker instead of multi-speaker.
As per my TTS research Tacotron2+wavenet has interesting sound quality,
For the speaker adaption need to just fine tune pre- trained model on few samples.

any comments ?

[rafaelvalle]

@Yeongtae
Does this branch https://github.com/Yeongtae/tacotron2/tree/prosody_speaker_embedding_test
have attention between randomly initialized embeddings and reference encoder ?

[Yeongtae]

@rafaelvalle
No.
But There is it in our private repository.

[rafaelvalle]

Folks, stay tuned: we have an implementation of multi-speaker global style tokens to be released.

[freds0]

Great! counting the minutes to test !!!
@rafaelvalle , I need some help: even with a reasonably sized dataset (about 40 hours) but home recording, I still haven't gotten good results. Some sentences have good speech, others not. I have already tested the repositories:

https://github.com/NVIDIA/tacotron2
https://github.com/NVIDIA/OpenSeq2Seq

Can you give me any tips on which variables we can change for best results?

[yxt132]

Folks, stay tuned: we have an implementation of multi-speaker global style tokens to be released.

Cannot wait!

[pannous]

Will GSTs allow Speaker Adaptation in real time?

[rafaelvalle]

With the multi-speaker model you can still perform inference faster than real-time.
We'll possibly release the code next week.

[rafaelvalle]

https://github.com/NVIDIA/mellotron
Our mellotron repo uses a Multispeaker Tacotron with GST.