Real-time-voice-cloning: Speaker Embedding Question

[fazlekarim]

When synthesizing the text to speech on the audio of your choice (where you recommend using three audio files of the same speaker), do you average the speaker embeddings (from the three audio files) and input that into the trained model? If you don't average the speaker embeddings, what do you do? If its too much work to explain, you can point to me which line of code deals with this. I can figure the rest out.

[CorentinJ]

You can read my thesis or the GE2E paper for a formal explanation, but what happens is that the mel spectrogram of an utterance is split in chunks of constant duration and an embedding is computed for each. The normalized average of these embeddings is the utterance embedding. You can average multiple utterance embeddings to form a speaker embedding (note that this speaker embedding is not normalized as per the GE2E paper, but if I ever implement them I would normalize them). A speaker embedding from a single utterance is exactly equal to that utterance's embedding, so I sometimes do not make the distinction. The code that handles this is encoder/inference.py. It is fairly well documented, so give it a look.

In the toolbox, the embedding is only from a single utterance (the currently selected one, which is usually the last one you loaded).

[fazlekarim]

"Embeddings are L2-normalized (this isn't important here, but if you want to make your own # embeddings it will be)."

So if I have a new speaker where I want to use its embedding, do I L2 normalize it or not?

[CorentinJ]

I'd recommend so. The difference isn't going to be big unless you have a lot of utterances anyway.