Tensorflowtts: FastSpeech2 training with MFA and Phoneme-based

[ZDisket]

When training FastSpeech2 (fastspeech2_v2) with phonetic alignments extracted from MFA I get the error described:

/content/TensorflowTTS/tensorflow_tts/trainers/base_trainer.py in run(self)
     65         )
     66         while True:
---> 67             self._train_epoch()
     68 
     69             if self.finish_train:

/content/TensorflowTTS/tensorflow_tts/trainers/base_trainer.py in _train_epoch(self)
     87         for train_steps_per_epoch, batch in enumerate(self.train_data_loader, 1):
     88             # one step training
---> 89             self._train_step(batch)
     90 
     91             # check interval

<ipython-input-39-dd452e77975e> in _train_step(self, batch)
     75         """Train model one step."""
     76         charactor, duration, f0, energy, mel = batch
---> 77         self._one_step_fastspeech2(charactor, duration, f0, energy, mel)
     78 
     79         # update counts

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    578         xla_context.Exit()
    579     else:
--> 580       result = self._call(*args, **kwds)
    581 
    582     if tracing_count == self._get_tracing_count():

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    642         # Lifting succeeded, so variables are initialized and we can run the
    643         # stateless function.
--> 644         return self._stateless_fn(*args, **kwds)
    645     else:
    646       canon_args, canon_kwds = \

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
   2418     with self._lock:
   2419       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 2420     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   2421 
   2422   @property

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _filtered_call(self, args, kwargs)
   1663          if isinstance(t, (ops.Tensor,
   1664                            resource_variable_ops.BaseResourceVariable))),
-> 1665         self.captured_inputs)
   1666 
   1667   def _call_flat(self, args, captured_inputs, cancellation_manager=None):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1744       # No tape is watching; skip to running the function.
   1745       return self._build_call_outputs(self._inference_function.call(
-> 1746           ctx, args, cancellation_manager=cancellation_manager))
   1747     forward_backward = self._select_forward_and_backward_functions(
   1748         args,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    596               inputs=args,
    597               attrs=attrs,
--> 598               ctx=ctx)
    599         else:
    600           outputs = execute.execute_with_cancellation(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

InvalidArgumentError:  Incompatible shapes: [16,823,80] vs. [16,867,80]
     [[node mean_absolute_error/sub (defined at <ipython-input-39-dd452e77975e>:115) ]] [Op:__inference__one_step_fastspeech2_341496]

Errors may have originated from an input operation.
Input Source operations connected to node mean_absolute_error/sub:
 mel (defined at <ipython-input-39-dd452e77975e>:77)    
 tf_fast_speech2_2/mel_before/BiasAdd (defined at /content/TensorflowTTS/tensorflow_tts/models/fastspeech2.py:196)

Function call stack:
_one_step_fastspeech2

I did everything I could think of to rule out my durations as the problem including verification that length is the same, so I don't know what happened.
Interestingly enough, when training with mixed_precision off the same error happens but with different values:

InvalidArgumentError:  Incompatible shapes: [16,763,80] vs. [16,806,80]
     [[node mean_absolute_error/sub (defined at <ipython-input-39-dd452e77975e>:115) ]] [Op:__inference__one_step_fastspeech2_449871]

Errors may have originated from an input operation.
Input Source operations connected to node mean_absolute_error/sub:
 tf_fast_speech2_3/mel_before/BiasAdd (defined at /content/TensorflowTTS/tensorflow_tts/models/fastspeech2.py:196)  
 mel (defined at <ipython-input-39-dd452e77975e>:77)

Function call stack:
_one_step_fastspeech2

Am I missing something?

[dathudeptrai]

@ZDisket see here https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/fastspeech2/train_fastspeech2.py#L116-L117. Loss f0/duration/energy is fine but loss mel-spectrogram missmatch length. I pretty sure that ur sum(duration) != len(mel), maybe just some samples or maybe all samples :D. Pls ref this comment (https://github.com/TensorSpeech/TensorflowTTS/issues/46#issuecomment-656120059)

[l4zyf9x]

which duration value you use when training ground truth or predict? If you use gt duration then the only reason comes in my mind is that some sample has sum of duration different with length of Melspectrogram. You could use the following code to ignore outlined sample while loading data.

difference = []
for i in tqdm(range(len(self.characters))):
        len_mel = len(np.load(self.mel_files[i]))
        duration = np.load(self.duration_files[i])
        total_duration = np.sum(duration)
    if len_mel != total_duration:
        difference.append(i)

Then filtering difference ids

[ZDisket]

@l4zyf9x @dathudeptrai Thanks, I'll see how deep the problem goes.

[ZDisket]

It seems that the length of the normalized mels is slightly higher than that of the sum of durations.
match(1).txt
Plotting one mel, it looks like the difference is in silence. So I have to go through all the mels and cut them off where the duration stops.
In this image for example, the mel has len 586 while the duration sum is 554

Or I could pad the duration by adding a SIL token (see here) at the end.
Which approach is more optimal?

[dathudeptrai]

@abylouw do you have any thought ? .@ZDisket it seems the length missmatch too much, in my preprocess for F0/energy, i also padding or drop some last elements to make the length is equal but the mismatch is just 1 frame. cc: @azraelkuan

[ZDisket]

@dathudeptrai I don't think it's too much when considering that these are phonetic durations. I'm considering just adding the difference to the last duration for every utterance, but I want to hear what you guys think about it first.

[dathudeptrai]

hmm, i don't have any experience in this case @ZDisket. maybe try to just adding to the last duration and see what happend when training :D. I think @abylouw succeed to train FastSpeech2 with the duration extract from MFA (or something like this) so i will wait his comment in this case.

[ZDisket]

@dathudeptrai I've gone ahead and it's training fine, the very first loss values look fine and predictions at 3000 steps.zip (very early) look normal. ming024 stated in his README.md that it only takes an hour of training on a GTX 1080 to start producing decent samples.

[dathudeptrai]

@ZDisket let see, i don't know what is the proper way to solve the length missmatch is this case, hope the model don't have any problem at the end of the mel :D

[l4zyf9x]

@ZDisket One question. Do you train Tacotron and extract duration with the same Mel that you use to train FastSpeech2?

[ZDisket]

@l4zyf9x Those are extracted durations on ground truth LJSpeech audio from Montreal Forced Aligner. I didn't use a teacher model.

[l4zyf9x]

As @dathudeptrai mention before. I think you should reference this comment https://github.com/TensorSpeech/TensorflowTTS/issues/46#issuecomment-656120059. I think the solution is that you config MFA have the same time resolution. For example: frame_rate=16000, hop_length=256. So you need set rate of MFA = 1000 * 256 / 16000 = 16(ms). Finally, it will be possible to mismatch 1 to 2 frame, you just need pad or drop the last frame

[ZDisket]

@l4zyf9x If that was the problem then it'd be much worse. I already took that into account for my second to frame conversion, taking it from here.

[ZDisket]

Although with some problems, the model at 30k is capable of generating speech. Here's a notebook.
@dathudeptrai you like?

[dathudeptrai]

@ZDisket the mel looks good at 30k steps, let training it around 150k steps and compared the performance. note that fastspeech2_v2 is small version

[ZDisket]

@dathudeptrai Does the small version lower quality, or is it supposed to be the same or higher?

[dathudeptrai]

@ZDisket need to tune :)). I think we can tune and find the smaller version without worse quality. Let see the performance after 150k steps.

[l4zyf9x]

@ZDisket I just have look at your code and MFA documentation. Following your code, when config MFA, frame_shift should be 1000*256/22050~11.61. What is your MFA frame_shift

[ZDisket]

@l4zyf9x My relevant code is:

tg = textgrid.TextGrid.fromFile("./TextGrids/" + tgp)
  pha = tg[1]
  durations = []
  phs = "{"
  for interval in pha.intervals:
    mark = interval.mark
    if mark in sil_phones:
      mark = "SIL"
    dur = interval.duration()*(sarate/hopsz)
    durations.append(int(dur))

I didn't read that part of the documentation and since both ming024 and I are getting results, it's not necessary.

[abylouw]

Is trim_silence true in your config? I did not use librosa to trim the silences, but rather trimmed them from the labels as aligned by our HTK implementation. We have a SIL start and end phone for all utterances, and in the pre-processing script instead of trimming with librosa as here:

https://github.com/TensorSpeech/TensorflowTTS/blob/84c004172d604ecb966e1a905b81d6e1db2fc2e0/tensorflow_tts/bin/preprocess.py#L206-L212

we basically trim the first and last labels of the phones:

def trim_silences(audio, durs, samplerate):
    audio_start = durs[0] * samplerate
    audio_end = (np.sum(durs) - durs[-1]) * samplerate
    trimmed_audio = audio[int(round(audio_start)):int(round(audio_end))]
    return trimmed_audio

where the durs list contains the durations of the phones in seconds.

[ZDisket]

@dathudeptrai After 110k steps, the performance is pretty bad, you can see samples in the notebook. I think I'll instead cut the mels at the time of data loading and train fastspeech2_v1. I also added training instructions here

[dathudeptrai]

@ZDisket thanks, i will take a look, phoneme based should better than charactor fastspeech2 v1 here. I am review everything now then release first version. After that i will train phoneme, support multigpu/tpu ...

[ZDisket]

@dathudeptrai Are you going to train the current version which adds the difference to the last duration? There are a few problems in the ends of the audio samples, although I suspect it could be just fastspeech2 v2 being bad.

[dathudeptrai]

@ZDisket not yet, i need to review and release first

[ZDisket]

As can be seen in the notebook with a 55k fastspeech2_v1 model ,the solution to the ending problem is to append SIL at inference time.

[dathudeptrai]

@ZDisket let me take a look and tuning with you, phoneme should be better than charactor-based, we should found the way to solve the problem about MFA on silence part. This bellow image visualize the mel of the text : Despite its shortness, it is the most transited motorway in Catalonia and one of the most transited highways in Spain It is also the first highway that has implemented on charactor-based and ur phoneme-based now:

• charactor-based
  

• phoneme-based:
  

Look at these images, we can see there is a large gap between phoneme + MFA with charactor + tacotron-2 alignment. Seems the true problem is on silence part.

[ZDisket]

@dathudeptrai The model at 110k presents those problems I suspect because I thought I could get away with using SIL both as regular silence and padding, so now I have to train with a dedicated PAD one. I'm also implementing a function where the preprocessor will cut off the audio based on the sum of durations.

[dathudeptrai]

@ZDisket yeah. It seems that the phoneme-based right now is not good to predict stop/silence part except "," token :D

[ZDisket]

@dathudeptrai Do you think this could help?

[dathudeptrai]

@ZDisket maybe, let try it. I will follow ur instruction and read the code carefully.

[ZDisket]

@dathudeptrai If you want to train it then you can still follow the instructions here but also add --trimlist trimlist.npy as an argument to tensorflow-tts-preprocess.

[ZDisket]

I think a dedicated pad character did the trick, it's got no ending issues although this depends on what is inserted after the text. Using the same text: _Despite its shortness, it is the most transited motorway in Catalonia and one of the most transited highways in Spain It is also the first highway that has implemented_, the model at 130k gets a different result depending on what is appended to the input at inference time
SIL + END , worst performing (stretches out the saying)

END, (ends abrutply)

SIL (best performing)

Also training curves for the 130k model (ignore the weird one around 60ki, that's just a product of saving and resuming later)

@dathudeptrai What do you think? (You can test it out in the Colab notebook)

[dathudeptrai]

@ZDisket hmm, i am wondering about f0 and energy loss, in all my experiments with ljspeech, it never overfit like above. there is a confict between duration loss and f0/energy loss. Example at the same position, the network need learn how to predict duration = 0 but f0/energy > 0.

[ZDisket]

@dathudeptrai Only thing I can think of is mixed precision, did you train with it on? (I do). In all my experiments the f0 started overfitting at 10k.

[dathudeptrai]

@ZDisket i always use mixed_precision for training fastspeech. is there any position which have duration = 0 but f0/enery is large ? (i mean duration = 0 but f0/energy tell that position is voice ?). BTW, training with charactor-based never overfit.

[ZDisket]

@dathudeptrai

is there any position which have duration = 0 but f0/enery is large ? (i mean duration = 0 but f0/energy tell that position is voice ?)

Haven't checked yet. But I think ming024's model also had overfitted f0, so maybe it is a thing of MFA.

[dathudeptrai]

@ZDisket we should re-think about punctuation on MFA, maybe we need sp insertion ?. BTW, i think fastspeech2 author's use another tool or some postprocess to make it works, hmm.

[ZDisket]

@dathudeptrai sp insertion where, at the end?

[dathudeptrai]

at punctuation, i think. BTW, can you see this https://github.com/ivanvovk/DurIAN/tree/master/filelists. He provide a duration and phoneme already for ljspeech. I'm reading his repo.

[dathudeptrai]

cc: @azraelkuan , need ur help now :))

[ZDisket]

@dathudeptrai I think the aligner already inserts sps at punctuation otherwise they wouldn't be there at all.

[dathudeptrai]

@ZDisket can you just download this file https://github.com/ivanvovk/DurIAN/tree/master/filelists and training :)). He compute already, i'm reading his repo to see what he did.

[ZDisket]

@dathudeptrai His transcriptions aren't different from mine. Here's his phonetic transcription of _LJ025-0076_ (pau is his silent token)

M EH1 N IY0 AE1 N AH0 M AH0 L Z pau AH1 V pau IY1 V IH0 N K AH0 M P L EH1 K S S T R AH1 K CH EU0 R W IH1 CH L IH1 V P EH1 R AH0 S IH1 T IH0 K L IY0 W IH0 DH IH1 N pau AH1 DH EU0 R Z pau AA1 R HH OW1 L IY0 D IH0 V OY1 D AH1 V AH0 N AE1 L AH0 M EH1 N T EU0 R IY0 K AE1 V AH0 T IY0 pau

Here is mine

M EH1 N IY0 AE1 N AH0 M AH0 L Z SIL AH0 V SIL IY1 V IH0 N K AH0 M P L EH1 K S S T R AH1 K CH ER0 W IH1 CH L IH1 V P EH2 R AH0 S IH1 T IH0 K L IY0 W IH0 DH IH1 N SIL AH1 DH ER0 Z SIL AA1 R HH OW1 L IY0 D IH0 V OY1 D AH1 V AH0 N AE2 L AH0 M EH1 N T ER0 IY0 K AE1 V AH0 T IY0 SIL END

In the zip file there are my phonetic transcriptions
metadata (4).zip

[dathudeptrai]

@ZDisket can you check his sum durations and the length of the mel ?

[ZDisket]

@dathudeptrai That'd require me to grab the LJ dataset and run preprocessing again just to grab the lengths of some mels. But since he uses the exact same aligner (as I can tell because the transcriptions are matching), then the duration sums should be the same before my script that compensates for the shortness. We'd have to check how he deals with that issue, but it shouldn't make that much of a difference.

[dathudeptrai]

@ZDisket see this https://github.com/ivanvovk/DurIAN/issues/3

[ZDisket]

@dathudeptrai Interesting, so I've been training for a long while with duration mismatches. Although you'll have to explain to me what using floor means since I'm not good at math, then I'll modify it and train yet another model.
Although I think it may not matter because I get the durations in seconds and then multiply them by sampling rate/hop length

[dathudeptrai]

let me check after lunch :D. @azraelkuan where are you now :v

[dathudeptrai]

@ZDisket why don't u use this dictionary ? (http://mlmlab.org/mfa/dictionaries/english.dict) instead of librispeech ?

[dathudeptrai]

@ZDisket https://github.com/ZDisket/TensorflowTTS/blob/master/examples/fastspeech2/mfa/postmfa.py#L93 ->> change int() to round() :))

[dathudeptrai]

Is trim_silence true in your config? I did not use librosa to trim the silences, but rather trimmed them from the labels as aligned by our HTK implementation. We have a SIL start and end phone for all utterances, and in the pre-processing script instead of trimming with librosa as here:

https://github.com/TensorSpeech/TensorflowTTS/blob/84c004172d604ecb966e1a905b81d6e1db2fc2e0/tensorflow_tts/bin/preprocess.py#L206-L212

we basically trim the first and last labels of the phones:

def trim_silences(audio, durs, samplerate):
    audio_start = durs[0] * samplerate
    audio_end = (np.sum(durs) - durs[-1]) * samplerate
    trimmed_audio = audio[int(round(audio_start)):int(round(audio_end))]
    return trimmed_audio

where the durs list contains the durations of the phones in seconds.

@ZDisket after you replace int to round, you can follow this comment. You can add SIL at the begin and ending of phoneme then trimed audio based on its duration rather than using librosa. I think all issues are solved :D. BTW, i see you fix this line https://github.com/ZDisket/TensorflowTTS/blob/master/tensorflow_tts/processor/ljspeech.py#L177, seems this is my bug ?, is this line important ?

[ZDisket]

@dathudeptrai Before fixing that line I was getting weird errors with ARPA input.
I don't think artificially adding phonemes that matter like SIL artificially will end well.

Also, my latest model (which I now have at 160k iter) was trained with duration-based trimming, see here. I consider the ending SIL a vital part since without it the audio cuts off suddenly at inference time. I'm satisfied with the model I've got now and its performance (as always, you can test it out in the notebook). If you really want those changes and can't navigate my code, I can implement them but you'll have to train it.

[dathudeptrai]

@ZDisket https://github.com/ZDisket/TensorflowTTS/blob/master/examples/fastspeech2/mfa/postmfa.py#L93 ->> change int() to round() :))

@ZDisket :)) this line should fix :)) i think it will improve significantly. After changed, the sum(durs) and len(mel) mismatch around 2-3 frames (which i think very small)

[ZDisket]

@dathudeptrai Done. I'll begin training in a bit because the model I've got has unsatisfactory performance.

[ZDisket]

@dathudeptrai Turning int() into round() reduced the mismatch but now some are greater than the mel and even with compensation code I added I get this:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  ConcatOp : Dimensions of inputs should match: shape[0] = [13,772,384] vs. shape[1] = [1,773,384]
     [[{{node tf_fast_speech2/length_regulator/while/body/_1/concat}}]]
     [[tf_fast_speech2/mul/_154]]
  (1) Invalid argument:  ConcatOp : Dimensions of inputs should match: shape[0] = [13,772,384] vs. shape[1] = [1,773,384]
     [[{{node tf_fast_speech2/length_regulator/while/body/_1/concat}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference__one_step_fastspeech2_117054]

Function call stack:
_one_step_fastspeech2 -> _one_step_fastspeech2

I think int() durations plus trimming is good enough.

[dathudeptrai]

@ZDisket https://github.com/ZDisket/TensorflowTTS/blob/master/examples/fastspeech2/mfa/postproc.py#L66 it not correct, it may make durations negative.

I think int() durations plus trimming is good enough.

well, the problem here is that we try to make it better, make it get the best performance that it can obtain. Just like in the future, we will have FastSpeech3, FastSpeech4 .... :)).

[ZDisket]

@dathudeptrai I also thought about that. Do you have any idea if not subtracting from the last silence, should I distribute the difference over all the durations? It could get messy to artificially manipulate durations like that which is why it's better to have them all be greater than the length of the mel.
I think int() durations and trimming is the best we'll get right now.

[dathudeptrai]

@ZDisket you can random over all position and minus only 1 until you get sum(dur) = len mel. Do not forget to check all durations are positive. You can also just minus 1 at positions that represent the punctuation. The len mel and sum dur is differeny but smaller should be better. Round make more sensen than int because 3.9 is 4 rather than 3

[ZDisket]

@dathudeptrai I haven't checked the average difference in int() dur sum and mel len with trimming enabled which should have helped. By the way, if I implement it, would you train it?
The current model I've got performs well but destabilizes on long text (greater than 8 seconds), do you know why?

[dathudeptrai]

@ZDisket i can train it and implement it also. Just want let you know an idea :D. The reason of destabilizes cause by the duration mismatch, when you take int, 3.1 = 3.9. The self attention very sensitive with this kind of problem, when duration more accurate, it should be better than my charactor-based pretrained.

[ZDisket]

@dathudeptrai I don't really see the value in using round() since all the manipulation required to fit them will offset the more accurate durations instead of int() where just the last phoneme was affected so making it worse so, by all means, do modify and train it yourself.

[dathudeptrai]

@ZDisket hi, can i ask if you can make a pull request for phoneme preprocessing, i mean we will have another folder on train/valid and it's phoneme_ids. I want training tacotron-2 for phoneme-based also :D

[ZDisket]

@dathudeptrai

can i ask if you can make a pull request for phoneme preprocessing

I don't exactly understand what you're asking, but I can make a PR. Also, ming024's FS2 also has int() durations but his model works fine on longer prompts, it could be that my model needs more training (his is up to 300k iters)

[dathudeptrai]

@ZDisket let me train phoneme and see, it should better than charactor-based here. I'm almost finished multi-gpu implementation for trainer.

[ZDisket]

@dathudeptrai You training phoneme with int or round durations?

[dathudeptrai]

i will training with round since it make more sense and accurate

[ZDisket]

@dathudeptrai Good luck then.

[ZDisket]

@dathudeptrai It seems you were very right. I applied a patch and trained a model with round() durations, and now it's way more stable on longer prompts.

[machineko]

@ZDisket are u training multi-speaker or multiple single speaker models?
From my experience with taco2 and flowtron training multi-speaker are a lot better on custom not long speakers dataset and help to reduce overfitting.

[dathudeptrai]

@ZDisket i just heard ur samples, it looks better than the previous version, i think i will look at it again, it can be improve more :D

[dathudeptrai]

@machineko i tried to training with multi-speaker on my private dataset, it very good.

[ZDisket]

@dathudeptrai That model's only trained up to 40k steps, it has a lot to improve.

[dathudeptrai]

@ZDisket oh, that good, the model normally convergence at 150k steps for ljspeech. i need to add some readme for multispeaker then i will work with you to optimize fastspeech2 + MFA + phoneme :D. @ZDisket will you try my multi-gpu trainer, hehe.

[machineko]

@dathudeptrai Where u using taco2 trained on the same dataset to extract duration for FS2? Cause i heard its possible to train with just libri/lj pretrianed taco2 extraction but never try it.
Also if u have some code for multi-speaker ill love to have a look :P

[ZDisket]

@dathudeptrai I only have one GPU. I'm also interested in multispeaker since I have a lot of small datasets.

[dathudeptrai]

@machineko all model here already supported multi-speaker. As you can see, tacotron2 and fastspeech/fastspeech2 has speaker_ids parameters, you just need modify the data-loader to return the speaker_ids :)) that is all.

[dathudeptrai]

@ZDisket that still ok, the new base trainer class can be train for both single and multigpu. BTW, i also have some modification to make mb-melgan/melgan.stft and tacotron more stable, hope you can train mb-melgan with the new version of this repo :)).

[ZDisket]

@dathudeptrai I'll take a look at mb-melgan once I'm done training this model. I'm also impressed with the current model's ability as a universal vocoder, except for some metallic noise it does pretty well, do you think we could have a good mb-melgan universal vocoder by training it on a diverse dataset like LibriSpeech downsampled to 22KHz?

[machineko]

@dathudeptrai Ye i saw this, and what about duration extraction did u use taco2 pretrained on ljspeech or u train tacotron2 on new dataset first?

[dathudeptrai]

@ZDisket sure :)) mb-melgan can be consider as universal vocoder, i want do this but i can't, i don't have GPU enough + disk space :D.

@machineko ofc i used tacotron-2 pretrained on ljspeech to extract duration for ljspeech and training fastspeech later. You can also use MFA to extract duration @ZDisket will make PR for that later i think :D.

[machineko]

@dathudeptrai Sorry i wasn't precise, i was asking about "i tried to training with multi-speaker on my private dataset".
Cause i hear u can extract duration from taco2 pretrained on other datasets and then you can try Fastspeech on ur own dataset even without pretrained taco2

@ZDisket Ill train your version with phonemes on my multi-speaker dataset and post result here later :)

[Dicksonchin93]

just some ideas to bounce off here to get some extra oomph from LJSpeech with phoneme durations using MFA, you could round the top k values below 0.5 to the ceiling and round the bottom k values above 0.5 to the floor, values above or below 0.5 are already rounded off to the bottom and ceiling respectively and you just need to tweak them a little bit. You do this only after the initial rounding and only on k number of durations according to the difference between actual total duration and total phoneme duration. SPN also should be retained as it means unknown words.

[janson91]

hi,
I have another question:
when I train fastspeech2 on my own datasets with mfa and phoneme-based, I got an error like:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [2,189,80] vs. [2,443,80]
[[node mean_absolute_error/sub (defined at examples/fastspeech2/train_fastspeech2.py:121) ]] [Op:__inference__one_step_fastspeech2_33441]

which seems that shape of my output mel (mel_before) doesn't match the target mel.
What should I do? Maybe some parameters were neglected by me.

[ZDisket]

@janson91 That's weird. Did you run postproc.py? It should have solved all the conflicts, are you following the guide in my README?

[machineko]

@ZDisket I have same problem and yes postproc didn't resolve issues with my dataset on your fork (ofc i change everything to fit my dataset), thats why i'm focusing on rebuilding base FS2 for multi-speaker first and later debug problems with your fork.

[ZDisket]

The model converged so fast I overfit it a bit, although it seems that later models have better audio quality but a bit more trouble on longer prompts. Audio samples: 20k 30k 50k 60k


So which model do you guys think is best?
@machineko That's weird, I trained a model on one of my custom datasets and it worked well, although what I do is change my datasets to adapt to the dataloader (in this case, LJSpeech), not the other way around.

[dathudeptrai]

@ZDisket how about continue to training, i think everything is ok, the quality is good but i still think further training can improve quality. Here is my tensorboard for FastSpeech2 V1 (charactor-based) and as you know, the audio samples on the github.io is 150k checkpoints.

Seem using MFA got better durations loss but worse f0/energy loss compared with charactor-based :D, but it's ok :D.

FYI, i think f0/energy loss can solve with more and more dataset because in my private datasets, it's larger compared with ljspeech the f0/energy loss never overfit like that. Also, let see the tensorboard here #https://github.com/TensorSpeech/TensorflowTTS/issues/46#issuecomment-652261380, it is very good.

[ZDisket]

@dathudeptrai My model at 140k becomes too unstable at longer prompts, that's why I limited to 60k or less.

[dathudeptrai]

@ZDisket maybe phoneme overfit faster than charactor and i think the quality at 60k is good enough. But in fact, i think all TTS system use streaming mode so each chunk maybe just 5->10s so the problem about longer prompts is not a real problem in that case.

[ZDisket]

@dathudeptrai So we're going to compensate for that problem by splitting up the prompts into chunks?

[dathudeptrai]

@ZDisket yes, in the real production, i don't think there is a TTS system that inference whole text. They normally split the text into multiple chunk then when u are playing first chunk, you also inferencing second chunk. When you playing second chunk, you also inferencing third chunk, ... This is a basic, both ASR and TTS should do that.

[ZDisket]

@dathudeptrai Alrighty, I'll publish the model at 140k and 60k.

[dathudeptrai]

@ZDisket how can we transform alignments by words to charactor ?

[machineko]

@ZDisket i just change dataloader to return speaker id not many other changes outside some path fixes etc. to other things but my samples can be max (~14s len) so longer then LJ as I said ill probably debug in next few days, right now I'm training multi speaker on pure FS2 with taco2 extraction.

[dathudeptrai]

@machineko in case you can't train a good FS2, pls re-train Tacotron-2 on ur dataset or use MFA to extract durations.

[ZDisket]

@dathudeptrai https://github.com/ZDisket/TensorflowTTS/blob/master/examples/fastspeech2/mfa/postmfa.py#L94
According to TextGrid documentation on pip, [1] from the return is phonemes, [0] is words. So just change tg[1] to tg[0], although I'll probably implement the option to use word alignments some time. You'd also have to remove the {} because those are for wrapping ARPA input.

[machineko]

@dathudeptrai Yes its probably true I'm just testing it (as i write earlier i saw in few places that ljspeech taco2 should be good enough for that :P)

Also, If duration, energy and f0 loss are not valid in later stages of training maybe we should change their weights in loss function later on so it won't affect loss function as much and can lead to more stable training later on.

[dathudeptrai]

@ZDisket it's word aligment not charactor alignment ?

[ZDisket]

@dathudeptrai Oops, I got the two mixed up.

[Dicksonchin93]

these are my graphs for LJSpeech with MFA

[ZDisket]

@Dicksonchin93 Cool. Got audio samples?

[Dicksonchin93]

35k 30k and 20k

[Dicksonchin93]

it got nan losses for the mels at 37kth epoch so no examples after that, I'll be experimenting with more robust optimizers

[ZDisket]

@Dicksonchin93 Those links are set to private. Set them to public so anyone who has it can see it.

[Dicksonchin93]

Done

[ZDisket]

@Dicksonchin93 I like your 35k sample, it was the sweet spot according to my tensorboard but I missed it since I set it to save every 10k steps. Mind sharing the model? Also, are you training with mixed precision?

[Dicksonchin93]

I can share it temporarily here , yeah with mixed precision

[dathudeptrai]

@Dicksonchin93 disble mixed precision so you won't get nan :)). This is a rare case, the mixed_precision module is still experimental so it's not always work

[Dicksonchin93]

makes sense, trying now @dathudeptrai thanks!

[ZDisket]

@dathudeptrai What do you think, 140k or 35k better? Also, I think there's not much to add at this moment. I also fine-tuned one of my small datasets on my 140k model and it doesn't present the same problem.

[manmay-nakhashi]

@ZDisket yes, in the real production, i don't think there is a TTS system that inference whole text. They normally split the text into multiple chunk then when u are playing first chunk, you also inferencing second chunk. When you playing second chunk, you also inferencing third chunk, ... This is a basic, both ASR and TTS should do that.

@dathudeptrai i tried this streaming approach but it is affecting a naturalness of audio output , do you have any method where i can keep audio naturalness and also do a streaming?

[dathudeptrai]

@manmay-nakhashi that is the thing you should take a time to solve :)). I masked this question as out-of-scope :v

[manmay-nakhashi]

@dathudeptrai i have spend some time and achieved streaming and still able to keep a quality with punctuation split , but i was thinking if there is better way :P , i can push that script to repo for better inference example

[machineko]

@manmay-nakhashi You can always train additional network to normalize split parts / or just normalize mels via python scripts.

[manmay-nakhashi]

@machineko i don't think it's a normalization problem if i do a windowing , speech output is not natural means it speaks as if it is reading two or three words one after another there is no natural flow. will normalization handle this thing ?

[machineko]

@manmay-nakhashi Send examples i'll check it later, I've done streaming for flowtron and it works fine (I've added a bit of silence to end of the first mel and then join both, normalize and push thru waveglow you can try this idea here)

[manmay-nakhashi]

@machineko i'll create a new issue and send examples over there

[manmay-nakhashi]

@dathudeptrai @ZDisket how do i get corrospoding valid_symbols , _arpabet and _letters for MFA for non-english language ? any idea ?

[dathudeptrai]

@Dicksonchin93 how about ur training progress ?

@manmay-nakhashi each language has it own symbols :)), i think i can't help in this case :v

[sjainlucky]

@ZDisket Can you provide some link or resources through which we can get information about MFA and extract durations from the wav file using MFA.

[machineko]

@sjainlucky check his fork everything is there

[Dicksonchin93]

blue is without mixed precision, orange is with

[dathudeptrai]

@Dicksonchin93 the result quite good i think, can you share what did you do ? what is the difference between ur implementation and @ZDisket implementation, we use the same MFA, is it right ?

[Dicksonchin93]

yeah same MFA, I added this

transcr = transcr.replace('-',' ')
transcr = transcr.replace(':', ' ')
transcr = transcr.replace(',', ' ')
transcr = transcr.replace('.', ' ')
transcr = transcr.replace('  ', ' ')

to get more matches to the dictionary for MFA. Additionally, I used rounding with a dynamic threshold if the sum of the durations doesn't match the Mel lengths otherwise it's 0.5. Also added in SPN as its own identifier as it is not silent but represents unknown words.

[dathudeptrai]

@Dicksonchin93 what do you mean dynamic threshold ?, can you share a snippet code for this procedure ?

[Dicksonchin93]

sure, I explained it in my first comment here. If you don't mind to scroll above :D

def adjust_durations_with_dynamic_threshold(
    raw_float_durations, difference_in_frames, durations_in_frames
):
    """

    Parameters
    -----------
    raw_float_durations: list
        list of float durations
    difference_in_frames: int
        number of difference in frames of computed durations and actual wav file number of frames
        considering hop size
    durations_in_frames: list
        list of rounded durations

    Returns
    -------
    durations_in_frames: list
        corrected int durations
    """

    is_not_integer_durations = []
    is_not_integer_duration_indexes = []
    for idx, raw_float_duration in enumerate(raw_float_durations):
        if not raw_float_duration.is_integer():
            only_floating_points = raw_float_duration - int(raw_float_duration)
            if difference_in_frames > 0:
                # more than 0.5 is already rounded up
                if only_floating_points < 0.5:
                    is_not_integer_durations.append(only_floating_points)
                    is_not_integer_duration_indexes.append(idx)
            elif difference_in_frames < 0:
                # less than 0.5 is already rounded down
                if only_floating_points > 0.5:
                    is_not_integer_durations.append(only_floating_points)
                    is_not_integer_duration_indexes.append(idx)

    if difference_in_frames > 0:
        top_k_highest_indices = np.argpartition(
            is_not_integer_durations, -int(difference_in_frames)
        )[-int(difference_in_frames):]
        chosen_indexes = np.array(is_not_integer_duration_indexes)[top_k_highest_indices]
        for idx, chosen_index in enumerate(chosen_indexes):
            durations_in_frames[chosen_index] += 1
    elif difference_in_frames < 0:
        difference_in_frames = np.abs(difference_in_frames)
        top_k_lowest_indices = np.argpartition(
            is_not_integer_durations, int(difference_in_frames)
        )[:int(difference_in_frames)]
        chosen_indexes = np.array(is_not_integer_duration_indexes)[top_k_lowest_indices]
        for idx, chosen_index in enumerate(chosen_indexes):
            durations_in_frames[chosen_index] -= 1
    return durations_in_frames

[dathudeptrai]

@Dicksonchin93 thanks, somehow i missed it :v. BTW, how about the quality of ur MFA + Fastspeech2 above, is it good ?

[dathudeptrai]

sure, I explained it in my first comment here. If you don't mind to scroll above :D

def adjust_durations_with_dynamic_threshold(
    raw_float_durations, difference_in_frames, durations_in_frames
):
    """

    Parameters
    -----------
    raw_float_durations: list
        list of float durations
    difference_in_frames: int
        number of difference in frames of computed durations and actual wav file number of frames
        considering hop size
    durations_in_frames: list
        list of rounded durations

    Returns
    -------
    durations_in_frames: list
        corrected int durations
    """

    is_not_integer_durations = []
    is_not_integer_duration_indexes = []
    for idx, raw_float_duration in enumerate(raw_float_durations):
        if not raw_float_duration.is_integer():
            only_floating_points = raw_float_duration - int(raw_float_duration)
            if difference_in_frames > 0:
                # more than 0.5 is already rounded up
                if only_floating_points < 0.5:
                    is_not_integer_durations.append(only_floating_points)
                    is_not_integer_duration_indexes.append(idx)
            elif difference_in_frames < 0:
                # less than 0.5 is already rounded down
                if only_floating_points > 0.5:
                    is_not_integer_durations.append(only_floating_points)
                    is_not_integer_duration_indexes.append(idx)

    if difference_in_frames > 0:
        top_k_highest_indices = np.argpartition(
            is_not_integer_durations, -int(difference_in_frames)
        )[-int(difference_in_frames):]
        chosen_indexes = np.array(is_not_integer_duration_indexes)[top_k_highest_indices]
        for idx, chosen_index in enumerate(chosen_indexes):
            durations_in_frames[chosen_index] += 1
    elif difference_in_frames < 0:
        difference_in_frames = np.abs(difference_in_frames)
        top_k_lowest_indices = np.argpartition(
            is_not_integer_durations, int(difference_in_frames)
        )[:int(difference_in_frames)]
        chosen_indexes = np.array(is_not_integer_duration_indexes)[top_k_lowest_indices]
        for idx, chosen_index in enumerate(chosen_indexes):
            durations_in_frames[chosen_index] -= 1
    return durations_in_frames

@ZDisket what is ur thought ?.
mfa_dynamic_round.zip

[Dicksonchin93]

here is the 200k audio

[ZDisket]

@Dicksonchin93 I like it, do this so that we may see its performance on longer prompts:

As the virus has spread around the world, countries have reached different stages of their outbreaks at different times, In some places the number of excess deaths may still increase in the coming weeks and months, especially as figures are revised, while in others the number of deaths is beginning to return to normal levels.

I somehow haven't received email notifications from your earlier posts, this function looks very nice. Also, since SPN means unknown, wouldn't it be better to just skip it?

[Dicksonchin93]

it's pretty unstable too here There is no easy way to skip it if you'd like you can manually remove those unknown word periods in the audio and modify the textgrid accordingly. But adding to the vocabulary will be a much easier work lol

[ZDisket]

@Dicksonchin93 I was thinking of skipping it like this in the postmfa.py script:

nono_phones = ["spn"]
 for interval in pha.intervals:
        mark = interval.mark
        if mark in nono_phones:
          continue
        if mark in sil_phones:
          mark = "SIL"

Also, while your code is innovative, I don't see much improvement especially in what I'm looking for (long prompt stability) to justify the added complexity.

[Dicksonchin93]

yeah you could but you should also remove it from the actual audio

[machineko]

OK guys one think if u really want to get better audio quality on longer sentences just retrain model on longer sentences its working pretty well with other models so fastspeech should be also good :)

LJspeech stats:
Mean Clip Duration | 6.57 sec
Min Clip Duration | 1.11 sec
Max Clip Duration | 10.10 sec

It'll be hard to get a lot better results on longer sentences without tuning model on some longer sentences and maybe add some postnet for joining mels.

I have almost the same discussion in (nvidia taco2 repo*), and solution is to just find better dataset for your use case.

[Dicksonchin93]

Or just concat samples in LJSpeech and train

[dathudeptrai]

I do not know but the charactor model can infer stable long sentences, you can hear at the demo page https://tensorspeech.github.io/TensorflowTTS/ or play with our colab

[ZDisket]

@machineko What he said, the model should be able to infer long sentences like the character one. Also @dathudeptrai, you have a better understanding of FastSpeech2 than me, can we ignore tokens without cutting them from the audio?

[Dicksonchin93]

I finetuned using my private data where the max length of all the audio is just 7 secs and didn't have the long sentence instability here

[ZDisket]

@Dicksonchin93 I noticed that too with one of my datasets. Also what vocoder and model you using for that sample? There's a pretty good VCTK mb-melgan universal vocoder.

[Dicksonchin93]

I just finetuned both fastspeech2 and mbmelgan, where can I get that VCTK trained model?

[dathudeptrai]

@ZDisket

can we ignore tokens without cutting them from the audio?

we should check how many unknow word in the dataset first, i think add SPN is much easier to do and it's ok. You can skip but i think the self-attention model is not good to solve this case, CNN or LSTM can be easily adapt.

There's a pretty good VCTK mb-melgan universal vocoder.

Did you try the pretrained VCTK and do inference? is that ok.

[dathudeptrai]

@Dicksonchin93 i have a one version VCTK MB-melgan convert from pytorch to tensorflow. You can use it as universal vocoder or just load and do fintune :v.

[Dicksonchin93]

sounds perfect, do you mind sharing it to me? I still wonder why nobody combines VCTK, Librispeech, and LJSpeech together and train a model though

[ZDisket]

@dathudeptrai
Yes, I saved the weights from that notebook you gave me into my Google Drive (you should publish the model by the way). The only flaw is that the audio is a little bit high on the low frequencies but this is nothing for some bass and treble manipulation. @Dicksonchin93 You can grab the link and see it in action with my notebook: https://colab.research.google.com/drive/1wXdeTQQdMdhkpvto7hDVgEfCanreNoE9?usp=sharing

[dathudeptrai]

@Dicksonchin93 you can combine :)), i can't since i don't have enough disk space to train VCTK :v

[Dicksonchin93]

@dathudeptrai @ZDisket Do you guys have any idea on how to reduce the metallic effect to make it more natural? I'm looking in some manual audio processing to remove some of these metallic sounds

[ZDisket]

@Dicksonchin93 If you can isolate the metallic noise to use as a noise profile then a conventional noise reduction filter (like the one found in Audacity) might be enough. Otherwise, try to train it more or with a better dataset, or train v1 which is heavier and higher quality if you're doing v2.

[dathudeptrai]

can anyone help me write a correct code for phoneme_to_sequence id in https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/processor/ljspeech.py :(. I just want to train tacotron-2 with phoneme now :(. Here is my code :(

    def phoneme_to_sequence(self, text):
        sequence = []
        clean_text = _clean_text(text, [self.cleaner_names])
        phoneme = G2P(clean_text)
        for p in phoneme:
            if p in valid_symbols:
                p = "@" + p
            if p in _symbol_to_id.keys():
                sequence.append(_symbol_to_id[p])
        return sequence

[ZDisket]

@dathudeptrai I just take the phonetic equivalent of the transcription (I do it from the MFA output, you'll have to run G2P on it) and modify metadata.csv to have those wrapped between curly braces {AA1, ...} before doing any preprocessing steps. The LJSpeech preprocessor is capable of handling the rest.

[dathudeptrai]

@ZDisket ok, i'm training with phoneme based on G2P now :D

[manmay-nakhashi]

@dathudeptrai i have espeak based phoneme to sequence will it work ?

[dathudeptrai]

@manmay-nakhashi i think G2P is faster than espeak :v.

[manmay-nakhashi]

@dathudeptrai but less language support , i couldn't find anyway to load custom g2p model into python

[dathudeptrai]

@manmay-nakhashi As you can see, every languages/datasets should have its own preprocessor.py. The important is not that G2p less language support, this is divide and conquer strategy. Almost each language have its own framework to convert charactor to phoneme, we don't need use general framework for this :v. In my point, interms of model implementation, more general is good. But interms of preprocessing code, more general -> less flexible :v, also less readable.

[manmay-nakhashi]

@dathudeptrai you are right in term of flexibility but if we can make a system as a custom model for G2P based on user defined phoneme output then we can make both flexible and generic.

[dathudeptrai]

@manmay-nakhashi yes, that is on my plan, make preprocessing stage as class based so user can inherit base_preprocessing class to implement their preprocessing class. As base_trainer here :D

[manmay-nakhashi]

@dathudeptrai exactly that would make this framework more robust, and scalable

[janson91]

@dathudeptrai
hi, I have trained Phoneme-based fastspeech2 and I get result like this:
It seemed to be over fitting. And the audio results generate by griffin lim sound bad. Can you give me some suggestion?

thx

[dathudeptrai]

@janson91 everything ok., the mel looks good. GL is just to check the output is correct or not, it always generates bad audios. You should train vocoder such as mb-melgan

[janson91]

@dathudeptrai
ok， mb-melgan is training, but it is very slow. It will take hundreds of hours to train on 8 gpus(1080ti)

[dathudeptrai]

@janson91 the batch_size in the config is for 1 gpu and it's 64. If you training with 8gpus, i suggest you training with batch_size = 8 or 16 :)). If you use batch_size = 64 that mean global_batch_size is 64 * 8 = 512 :))). Mb -Melgan need to be train around 1M steps with batch_size 64 :D.

[janson91]

Ok thx @dathudeptrai

[janson91]

@dathudeptrai hi,
I just adjusted sampling_rate: 48000 in config yaml for sampling rate of my raw data is 48000
however, I get the synthesis audio half length of original audio.
Do I leave out other parameters?

thx

[dathudeptrai]

@janson91 the sampling_rate in processing config should be the same with the sampling_rate in training config :D.

[janson91]

@dathudeptrai sampling_rate of two config is same. So are there other possible parameters that lead to incorrect result?

[dathudeptrai]

@janson91 could you create a new issue ? and ofc, please give us ur preprocessing config and training config :D .

[dathudeptrai]

fastspeech + MFA is supported in (https://github.com/TensorSpeech/TensorFlowTTS/tree/master/examples/fastspeech2_multispeaker). I will close this issue, thanks for all ur help :D