Tacotron-2: Evaluation on Chinese mandarin

[begeekmyfriend]

eval-30000.zip
Here are the evaluation results of my training on 12-hour Chinese mandarin corpus. The voice sounds natural but still somewhat rough.
The modification has been opened on my own repo with mandarin branch.
Thanks a lot for this work!

[Rayhane-mamah]

Hello @begeekmyfriend, awesome results, thank you very much for sharing them!

Oh okey.. I didn't expect to find a 1min long audio! I'm not Mandarin expert but the overall results sound okay.

About the audio quality, as I'm used to say, inverting Mel spectrograms directly can be challenging and will always have that "robotic" noise in it, I am currently working on integrating Wavenet as Vocoder which will give the human like speech quality so stay tuned!

Being the perfectionist I am, I will try figuring out a way to improve the predicted mels even more however, I won't settle for less than a perfect match with real spectros :) (even though more training should improve quality a bit)

Awesome work, feel free to share any further results, trained models, or even merge the branches to add mandarin support if you like!

[begeekmyfriend]

The audio has been concatenate with 12 separated wav files from 12 sentences by FFmpeg...

The point I appreciate is that there need to be only less dataset to be feed to train out an effective model on your project. You know it is usually tough for many people to make out a long time corpus and sanity check is valuable as well. I am going to train more steps for evaluation and I will also post my results at that time.

I would mark this result as a baseline for all the tacotron projects I have ever seen. And I believe your work will help other authors to improve their jobs. Thanks again!

[Rayhane-mamah]

Yeah I noticed tge concatenation :)

Thank you sir for you support, I'm glad this work suits your needs.

Hopefully we can achieve better results in the upcoming days!

Looking forward for your results!

[begeekmyfriend]

Terminal_train_log.zip
It seems there is vibration for loss value during the training at 100+K steps. Is there any hyper parameter to be adjusted such as decay learning rate?
...
By the way, the loss value dropped to 0.3 at 107K steps after the vibration.

[Rayhane-mamah]

Hello again @begeekmyfriend, thanks for reporting that.

I noticed these vibrations as well. After having a second look at the learning rate decay function I noticed the horrible mistake I was making in the computation of this decay.. Anyway learning rates were bigger than what they are supposed to be which explains those vibrations..

Crazy how a typo can make your life a hell.. :) it's fixed now, and learning rate decay is set to start after 50k steps and reach its minimal value (1e-5) at 150k steps. (d28cfa9a77afc87902100bd5b2113fbb8541227e)

Here's what the new learning rate evolution looks like (lr vs training_steps):

• Phase 1: Constant learning rate (1e-3)
• Phase 2: Exponential decay from 1e-3 to 1e-5
• Phase 3: Constant learning rate (1e-5)

Since in the T2 paper, decay parameters were not specified, I tried optimizing those params for our case, so they might need some extra tweaking.

You will also notice a faster learning of attention and a faster loss drop.. From these two observations one could guess that model quality will improve, we'll find out once training is finished.

Note:
i messed around with the preprocessing a bit. You maybe want to keep that in mind in case it affects your preprocessed data. (Made changes in the indices of the feeder and metadata).
I also took off lowercasing in the cleaners because I noticed it wasn't useful for english cleaners. If you however need that lowercasing for mandarin you can put it back in tacotron/utils/cleaners.py.

[butterl]

@begeekmyfriend Hi Leo ,Thanks for your work. which dataset are you using, thchs30？

[begeekmyfriend]

No, I am using my private dataset. THCHS30 is feasible as well.

[Rayhane-mamah]

Okey you tell me how it goes then, I'll be running few tests on my own too.
Hopefully we find the optimal scheme.

On Mon, 16 Apr 2018, 03:04 Leo Ma, [e-mail] wrote:

@Rayhane-mamah https://github.com/Rayhane-mamah I am trying Noam Scheme
as Keith Ito's tacotron does. It is also mentioned in tensor2tensor
tensorflow/tensor2tensor#280
https://github.com/tensorflow/tensor2tensor/issues/280

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-381459677,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhFSwD5wZHF3jVaGeXSBJ-Zh1DfHDe1Oks5to_wlgaJpZM4TSyum
.

[begeekmyfriend]

I have changed the hyper parameter on learning rate decay as follows since there is vibration on loss value after 10K steps.

tacotron_start_decay = 10000, #Step at which learning decay starts
tacotron_decay_steps = 10000, #starting point for learning rate decay (and determines the decay slope) (UNDER TEST)
tacotron_decay_rate = 0.33, #learning rate decay rate (UNDER TEST)

[begeekmyfriend]

train.log.zip
I think the loss value decreases very slow in your tacotron according to that in Keith Ito's implementation. In train.log we can see the avg_loss decreasing smoothly and in Terminal_train_log it often freezes for a long time.

[Rayhane-mamah]

@begeekmyfriend, you provided my logs of a previous repo version, did you upgrade to use latest one? I am seeing loss "explosions" in the log, I hope you're note referring to those when you say loss is vibrating..

Just to be sure we're talking about the same thing, here's what loss vibrations look like to me:

After a smoothing of 0.93:

The overall loss is still decreasing slowly.
If by vibrations you're still referring to these explosions, you can set tacotron_scale_regularization = True in hparams.py, should take care of it.

I also want to point out that T1 and T2 use different loss functions and we do not rescale our mel outputs the same way. (I normalize my mels to [-4, 4]).

You can change this option back to [0, 1] by setting in hparams.py:

symmetric_mels = False,
max_abs_value = 1.,

[ghost]

@begeekmyfriend
Thanks for you contribution. :)
I am applying THCHS30 as the speech corpus here, however I find out an "interesting" phenomenon:

Step    2500 [1.775 sec/step, loss=0.84299, avg_loss=0.84627]
Writing summary at step: 2500
Saving checkpoint to: logs-Tacotron/pretrained/model.ckpt-2500
Saving alignment, Mel-Spectrograms and griffin-lim inverted waveform..
Input at step 2500: tantwo huaone yitwo xianfour defive wangtwo shuone wentwo gaithree getwo youtwo yutwo faone shengone zaifour tangtwo shunfour zongone yongthree zhenone niantwo jianone suotwo yithree youfour beifour chengone weitwo yongthree zhenone getwo xinone~________________________________________________________________________________________

The tones of pinyin (presenting as 1,2,3,4,5) will be automatically converted into one, two, three, four, five. :(
The original input at step 2500 should be:

Input at step 2500: tan2 hua1 yi2 xian4...
How could I correct this fatal input? :(

[begeekmyfriend]

@DavidAksnes It never happens to me...

[Rayhane-mamah]

Hello david,

Use basic cleaners instead of english cleaners in hparams.py

For english numbers are converted to normalized words.

On Wed, 18 Apr 2018, 10:58 Leo Ma, [e-mail] wrote:

@DavidAksnes https://github.com/DavidAksnes It never happens to me...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-382333191,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhFSwI6S0PLjJaw0fHiOdBVh360P4O4jks5tpw4tgaJpZM4TSyum
.

[begeekmyfriend]

train.log.zip
Above all are the tensorboard and the training log with tacotron model of yours and Keith Ito's. I have run your latest version with learning rate as follows:

tacotron_start_decay = 50000, #Step at which learning decay starts
tacotron_decay_steps = 25000, #starting point for learning rate decay (and determines the decay slope) (UNDER TEST)
tacotron_decay_rate = 0.33, #learning rate decay rate (UNDER TEST)
tacotron_initial_learning_rate = 1e-3, #starting learning rate
tacotron_final_learning_rate = 1e-5, #minimal learning rate

I used the same dataset as THCHS-30 with 34h long. We can see that the loss value is still very high. In my experience, loss less than 0.1 is the acceptable result as Ito's tacotron achieved.

[butterl]

I use patch of @begeekmyfriend for CHS and My machine is slow only goes to 20K step now(run from yesterday), the loss stays at 0.6x, and same thing happens to the original repo of @Rayhane-mamah(0.5x up to 60k+)

[Rayhane-mamah]

@begeekmyfriend and @butterl, About the loss values, it would be preferable if you don't try comparing between my work and keithito's. Let me explain why such comparison isn't useful:

Tacotron Feature prediction network outputs are mel-spectrograms, we can consider them as matrices of shape [mel_frames, 80]. Let me take the examples of [100, 80] -> the model will have to predict 8000 values, one vector of [1, 80] at a time (or matrix of [5, 80] at a time if reduction factor r=5).

Here, I took the liberty to plot two random distributions of 8000 points, with different means and std values.

What I want to bring your attention to here is the fact that when preprocessing the data, I tend to create bigger distances between data points (in our case, distance between brown and red). You might wonder why I do that, here's the reason.

If you want to scale your data to [0, 1] like in keithito's work please make these changes in hparams.py:

symmetric_mels = False, max_abs_value = 1.,

Loss will drop to 3e-3.. the thing is, loss value itself is absolute and is not normalized by data norm or anything, so comparing losses from two differently distributed outputs is a little extreme..

I also want to point out that by widening the output range to [-4, 4], we were able to reduce blurriness in outputs probably due to the added penalization on L2 loss.

With this said, I also experienced better output quality with outputs_per_step=1 (i.e: no reduction factor, predict outputs one frame at a time). It's actually predictable since it's easier for the model to predict next mel frame only instead of trying to predict 5 frames at a time.

Attention is also working:

I however noticed that the model is moving forward too fast, leading to a fast reading when synthesizing without teacher forcing, I'm currently trying to determine the cause and set it as a user choice (despite the reading speed, everything is audible and less robotic).

Please note that using r=1 will cause a training and synthesis slowdown of about x2.5~x3. and batch size will probably need to be dropped down to 32.

I wonder if dropping the learning rate this fast is beneficial.. I'll do some experiments on that and tell you how it goes.

Note:
Mel spectros will always be blurry when trained with a L2 loss, but it isn't a problem for the wavenet. paper reference: "However, when trained on ground truth features and made to synthesize from predicted features, the result is worse than the opposite. This is due to the tendency of the predicted spectrograms to be oversmoothed and less detailed than the ground truth – a consequence of the squared error loss optimized by the feature prediction network. When trained on ground truth spectrograms, the network does not learn to generate high quality speech waveforms from oversmoothed features."

So if you happen to have a pretrained Wavenet model on ground truth labels, it will most likely not give human like speech quality. Retrain the vocoder instead on the GTA synthesized outputs of the Frame Prediction Network.

[begeekmyfriend]

I see. Thank you for your explain. I am afraid I have to look closely into it and make myself understand your meaning:-)

[Rayhane-mamah]

I know i'm not the best at giving lectures or explain stuff to people..

If there is anything I need to develop or explain further please let me
know.

On Thu, 19 Apr 2018, 10:42 Leo Ma, [e-mail] wrote:

I see. Thank you for your explain. I am afraid I have to look closely into
it and make myself understand your meaning:-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-382674958,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhFSwI5MjwsmzCurSZbwGSkjvbQ9jfQDks5tqFv1gaJpZM4TSyum
.

[butterl]

Thanks Rayhane-mamah , tried both thchs30 and LJspeech datasets, the loss still in 0.5x even goes to 60k+ steps and for begeekmyfriend's plot 4.x even to 10k+, while in your 46000 step plot the loss is 0.23x.
It's there anything we're missing to reach that? set outputs_per_step=1 ?

[Rayhane-mamah]

In this last plot I provided I'm not using a reduction factor. i.e: set
outputs_per_step=1

Training will be 2~3 times slower however and you will probably need to
drop batch size to 32. (Everything is in hparams.py)

On Fri, 20 Apr 2018, 03:08 butterl, [e-mail] wrote:

Thanks Rayhane-mamah , tried both thchs30 and LJspeech datasets, the loss
still in 0.5x even goes to 60k+ steps and for begeekmyfriend's plot 4.x
even to 10k+, while in your 46000 step plot the loss is 0.23x.
It's there anything we're missing to reach that?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-382943742,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhFSwKIgD7hgiu70vqiuwT3fbd4RyO-oks5tqUMagaJpZM4TSyum
.

[dsmiller]

There's a bug related to this... if you set '--hparams "outputs_per_step=1"' on the command line (but don't change hparams.py, there will be a mis-matched shape error later on. I fixed it by changing tacotron.py from
stop_projection = StopProjection(is_training, scope='stop_token_projection')
to
stop_projection = StopProjection(is_training, hp.outputs_per_step, scope='stop_token_projection')
because the hparams from the command line will not be seen in the StopProjection constructor. This doesn't fix the actual problem, though; I've had other issues where I need to change hparams.py or the command-line hparams have no effect.

[Rayhane-mamah]

@dsmiller, thank you for reporting that, I will look into it and fix it for next commit.

In the meantime, I updated the repo to work my exact actual params I'm using for training. (0b26fa19ceaf9465e8fa62982730a2b42829a8dd)

And still decreasing: (Let's hope we don't overfit :) )

I will upload the english pretrained model as soon as it converges.

[begeekmyfriend]

Specially, as for Chinese, there won't be over-fitting in my opinion. The Chinese characters are enormous though, the combinations of Chinese Pinyin are finite. Moreover， unlike English, the pronunciation of vowels of Chinese Pinyin are nearly unique. That says we can make some deterministic relation between Chinese pronunciation and Latin characters. Once the machine remembers the relation, it almost will not predict the pronunciation wrong. I would like to appreciate Youguang Zhou, Father of Chinese Pinyin here that his approach has brought huge advantages on Chinese mandarin TTS;-P

[begeekmyfriend]

eval-112000.zip
Terminal_train_log.zip
Here are the latest evaluation at 112K steps and training log. I have to say the effects are amazing!

[begeekmyfriend]

@Rayhane-mamah I think the hyper parameter predict_linear important and necessary for the quality of audio synthesis. Below is the reference from Tacotron paper:

Figures 4(a) and 4(b) demonstrate the benefit of using the post-processing net. We trained a model
without the post-processing net while keeping all the other components untouched (except that the
decoder RNN predicts linear-scale spectrogram). With more contextual information, the prediction
from the post-processing net contains better resolved harmonics (e.g. higher harmonics between
bins 100 and 400) and high frequency formant structure, which reduces synthesis artifacts.

[Rayhane-mamah]

@begeekmyfriend, I thought so too, but only if you're not willing to use wavenet as a vocoder.

In fact wavenet can take care of the small noise in the predicted mels. If however you want to invert the mel outputs directly then using the post processing network to predict linear spectros is the way to go.

Also, if you do use the post processing net, please make sure to invert the linear spectros when evaluating, audio quality will be much better than mels inversion.

The only downside is the big slowdown that comes with the post processing net.. for the moment I only took the same architecture as keithito but I saw some alternatives in some other google works, I might try them out in the future.

But yeah to get a clean quality with just tacotron and griffin lim you should set predict_linear to True and invert wav from the linear spectros.

[begeekmyfriend]

An exception has been thrown when I want to synthesize with linear prediction:

Traceback (most recent call last):
  File "synthesize.py", line 33, in <module>
    main()
  File "synthesize.py", line 27, in main
    tacotron_synthesize(args)
  File "/home/leoma/Tacotron-2/tacotron/synthesize.py", line 76, in tacotron_synthesize
    run_eval(args, checkpoint_path, output_dir)
  File "/home/leoma/Tacotron-2/tacotron/synthesize.py", line 14, in run_eval
    synth.load(checkpoint_path)
  File "/home/leoma/Tacotron-2/tacotron/synthesizer.py", line 24, in load
    self.model.initialize(inputs, input_lengths)
  File "/home/leoma/Tacotron-2/tacotron/models/tacotron.py", line 40, in initialize
    raise ValueError('Model is set to use post processing to predict linear spectrograms in training but no linear targets given!')
ValueError: Model is set to use post processing to predict linear spectrograms in training but no linear targets given!

[dsmiller]

Any results on the problem of the model moving too fast when r == 1? I see the same results (better prosody and cleaner speech, but too fast) and can run some experiments if you have ideas.

[Rayhane-mamah]

I am not sure but it seemed like further training reduced that problem. It
seems like attention want to move forward fast.

You can give me your opinion on the pretrained model I provided in the
latest issue ( I will set the link to it later)

All in all, it seems okey.. I tried however taking off the attention wights
cumulation and only feed previous weights for attention computation. This
only slowed down the attention learning process ( alignments take too long
to get learned and don't reach a good quality). This also does not reduce
the speech speed, only creates failure cases where the model gets stuck on
some characters.

If you suspect something else to be the reason please let me know, I'll
check it out.

On Wed, 25 Apr 2018, 13:26 dsmiller, [e-mail] wrote:

Any results on the problem of the model moving too fast when r == 1? I see
the same results (better prosody and cleaner speech, but too fast) and can
run some experiments if you have ideas.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-384268287,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhFSwORHe4jZC5u_j3qbC8hniGvnCG4Kks5tsGt_gaJpZM4TSyum
.

[candlewill]

Thanks @Rayhane-mamah . Nice work. Based on your code, I could also get some good Chinese synthesized samples. I can share some of them here: https://goo.gl/DiVNNz

I tried to predict linear. The new samples are here: https://goo.gl/8XhcsF I think it's better than mel.

[Rayhane-mamah]

@begeekmyfriend sorry for this super late answer!

You probably already took down the condition by now :) but in case you didn't here's a quick fix (it will be added to the repo next commit). In tacotron.py:

if not gta and self._hparams.predict_linear==True and linear_targets is None and mel_targets is not None:
raise ValueError('Model is set to use post processing to predict linear spectrograms in training but no linear targets given!')

@candlewill Thank you so much for sharing your results. I don't know Chinese so I'm gonna trust you when you say samples are good :)

It would be great if you could also accompany these samples with their input sequences. Thanks again for sharing!

[begeekmyfriend]

@candlewill Your samples seem the voice of each word is too short for speaking compared with mine...

linear-eval-189000.zip
Terminal_train_log_linear.zip

@Rayhane-mamah I have synthesized out the evaluation by linear spectrograms. The effect seems no good to the one by mel spectrograms. I am expecting your implementation on wavenet:-)
By the way, if we want to predict linear spectrograms, we need to add some code below this line as follows:

if hparams.predict_linear:
        self.linear_outputs = self.model.linear_outputs

[candlewill]

Yes, I think so. @begeekmyfriend Do you have any idea why this phenomenon occurs.

[begeekmyfriend]

I think it is the quality of dataset since the code we share is the same. My private 12-hour recording is from a professional news male anchor whose voice is quite clear and charming. Despite of this it seems the evaluation reads a bit faster than normal. In my opinion there is still something to be adjusted for the tacotron model.

[begeekmyfriend]

@Rayhane-mamah I think in hparam.py the num_freq should be set as 1025 for the shape of linear_outputs and linear_targets and the fft_size be set as 2048 for STFT in audio utilities according to Keith Ito's implementation.

[Rayhane-mamah]

@begeekmyfriend, sorry for the late answer!

if you want the model to be faithful to keithito's work, you should make the changes you listed. I however picked those hparams to have same audio preprocessing with Wavenet..

I also don't like how mels look when fft_size=2048.. Also, it would be better to use frame_shift_ms and set hop_size to None if you make these changes. :)

[JK1532]

I trained the model with sample_rate=48000 on chinese mandarin dataset.And I have get good samples,But the speed of speech is a bit fast.Have you met this problem? @begeekmyfriend @candlewill

[butterl]

@JK1532 you could share some Audio samples to see how "the speed of speech is a bit fast"

[sayyoume]

@begeekmyfriend 你好我想问下，拼音标注是怎么标注的？格式是怎样的?有没有例子呢？

[begeekmyfriend]

@sayyoume 样式就是THCHS-30里的拼音标注，你可以使用python-pinyin

[begeekmyfriend]

@candlewill I saw there was no blur in the high frequency of your samples. Would you please to tell me that how long was your dataset and each sentence fed to train?

[candlewill]

@begeekmyfriend My corpus have about 10, 000 sentences, almost 12 hours.

[begeekmyfriend]

@candlewill How many steps did you take to train out such results? It is strange that there is blur in the high frequncey of my spectrograms. The below one is your sample.
By the way, is the transcript of Chinese mandarin like as follows?

据北京青年报报道，春运抢票高峰频现一票难求，“曲线”回家受追捧
ju4 bei3 jing1 qing1 nian2 bao4 bao4 dao4 , chun1 yun4 qiang3 piao4 gao1 feng1 pin2 xian4 yi1 piao4 nan2 qiu2 , qu1 xian4 hui2 jia1 shou4 zhui1 peng3

[begeekmyfriend]

@candlewill Would do you like to share me the hyper parameters? Like outputs_per_step, tacotron_initial_learning_rate, tacotron_start_decay, tacotron_decay_steps, tacotron_decay_rate? Thank you! Below are mine:

outputs_per_step = 2,
...
tacotron_start_decay = 50000, #Step at which learning decay starts
tacotron_decay_steps = 40000, #Determines the learning rate decay slope (UNDER TEST)
tacotron_decay_rate = 0.2, #learning rate decay rate (UNDER TEST)
tacotron_initial_learning_rate = 1e-3, #starting learning rate

[candlewill]

@begeekmyfriend Here is my hyper parameters

outputs_per_step = 1,
...
tacotron_start_decay = 50000,
tacotron_decay_steps = 50000,
tacotron_decay_rate = 0.4,
tacotron_initial_learning_rate = 1e-3, 

[begeekmyfriend]

@candlewill What was the sample rate of your dataset? Have you tried 16KHz?

[DaisyHH]

Hi, @begeekmyfriend, 我用的thchs30的语料做的实验，为什么跑了200000步的结果，还是远不如你的30000步的结果（可以读出来，但是分词不太正确，不那么连贯，而且每一次读的声音也不是固定的，这一次是男声，下一次可能就是女声了，而且女声也不是一个女声），是因为语料thchs30的原因吗？

[begeekmyfriend]

@DaisyHH 是的，这个项目不支持multi-speaker，而且16KHz的音频也不行，得用22050Hz（起码对这个项目而言）

[begeekmyfriend]

@v-yunbin 如果你有Audition的话，可以对比一下 @candlewill 的样本，下图你的样本频谱在高频部分是模糊的，我也被这个问题困扰。

[v-yunbin]

其实这个我发现了，我觉得这个问题可能是由于一下几个原因导致，还未验证：

1. 由于原始语音部分中高频部分就比较少，而我们使用的采样率是16khz。
2. 可能是在前期数据处理的时候，损失掉了部分高频信号。
3. 还有可能是使用的数据量不够，导致高频部分较少。

a. 30hr data on step 2k

b. 2hr data on step 2k

[begeekmyfriend]

@v-yunbin 你说的有道理，可惜我的是男声，本身训练语音的高频成份较少，所以更容易模糊，个人认为女声比男声好训练一些

[v-yunbin]

@begeekmyfriend 男声数据多一点也是可以的吧 --voice='male'要修改，默认是‘female'

[butterl]

@v-yunbin @begeekmyfriend I found that with code from head of repo， the high freq power will decrease with the train step grow. The older code do not have this effect， 300K step also have good high freq power (loss all code and data with harddisk fail , and do not remember the exactly commit id)

23K

And 140K seems loss a lot in high freq

And 240K has even lesser power in high freq

@candlewill are you using a order repo ,do you remember commit id？ seem with outputs_per_step = 1, I could not run with batch >8 with P100 from google cloud

[begeekmyfriend]

@butterl Would you like to provide the samples both in 140K steps and 240K steps? By the way, are the results mel or linear outputs? And how long is your total dataset?

[butterl]

@begeekmyfriend the dataset is thchs30 , and this is mel output， do not tried linear yet , seems linear is much better from mel. also with the new repo，feed eval mel file to wavenet trained with real wav ,the output is just noise ，and the older tacotron code（lost one） is good

for audio sample , because of firewall rule, I could not transfer file from the borrowed vm. tried talked with IT guy, but failed :(

[begeekmyfriend]

@butterl It seems you only used the male voices of THCHS-30 for training, right? You may drag your wav files (zipped) into the dialog box of this issue to upload your samples to the github warehouse. For the old tacotron version, you may try https://github.com/r9y9/Tacotron-2

[v-yunbin]

I notice some changes about Griffin Lim parameter :
original repo: power = 1.55,
new repo: power = 1.2,

[butterl]

@begeekmyfriend , wav and all env are on remote vm and could not get it to internet connected PC :(
trained with nearly all THCHS-30, the wav some time turn out to be male and some time female and even strange voice (muti-speaker will have this )

[v-yunbin]

voice set is important, if the data you used is female, you set it as female, the train will converge faster.

[begeekmyfriend]

@v-yunbin The --voice option is only for the path of M-AILABS dataset. It does nothing with other dataset.

[candlewill]

@begeekmyfriend Sorry for late reply. I used the old version repo to get the result without blur. And the sample rate is 22050. I also tried with 16k, but high freq was blur.

[v-yunbin]

@begeekmyfriend 请问你跑成功wavenet train 了么

[begeekmyfriend]

@v-yunbin I ran r9y9's wavenet vocoder, not this one.

[DaisyHH]

corpus THCHS-30 has no punctuation, With punctuation, the separation of sentences is clearer
Using phoneme instead of characters in /tacotron/utils/symbols.py will get better performance

@begeekmyfriend
now I can get a not bad wave file, but the sound still go along with a electrical like noise at 30k+ step . a little like your result in the beginning of eval-30000.zip.

this is because of training step is not enough , or some parameter in hparams.py?

[begeekmyfriend]

@DaisyHH If you just need vocoder like griffin-lim, you need ~200K steps.

[v-yunbin]

@begeekmyfriend 请问tacotron你跑到了多少步？我的跑到了50k，合成的语音里有“嗤嗤”d的声音，是还没有完全收敛导致的么
step-50k-eval.zip

[begeekmyfriend]

@v-yunbin 你早就收敛了，你可以参考 @butterl 的训练https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-400577802

[Rayhane-mamah]

I don't understand Chinese much but I'm assuming you're saying "this issue is a peace of cake and it is fixed" :)

[v-yunbin]

@begeekmyfriend 合成的语音停顿混乱可能是什么原因啊？我听了你的样本好像没有这个问题，https://github.com/Rayhane-mamah/Tacotron-2/issues/122
text:
yu2 jian4 jun1 wei4 mei3 ge4, you3 cai2 neng2 de5 ren2, ti2 gong1 ping2 tai2,ta1 shi4 yin1 pin2 ling3 yu4 de5, tao2 bao3 tian1 mao1, zai4 zhe4 ge4 ping2 tai2 shang4, mei3 ge4 nei4 rong2 sheng1 chan3 zhe3, dou1 ke3 yi3 hen3 fang1 bian4 de5,shi1 xian4 zi4 wo3 jia4 zhi2, geng4 duo1 de5 ren2, yong1 you3 wei1 chuang4 ye4 de5 ji1 hui4, bu4 guo4 ta1 men5 zhi3 shi4 da1 dang4, bu2 shi4 chang2 jian4 de5 fu1 qi1 dang4 mo2 shi4, yong4 yu2 jian4 jun1 de5 hua4 lai2 shuo1, zhe4 ge4 mo2 shi4 ye3 bu4 chang2 jian4,
wave:
speech-wav-00001-linear.zip

[begeekmyfriend]

@v-yunbin There is no punctuation in THCHS-30 dataset, you need to train your own.

[kunguang]

@begeekmyfriend 我这个也是THCHS-30的训练集。为啥Loss这么低了，图像缺是这样的

[begeekmyfriend]

@kunguang 你用的最新版本么，要到40K步左右才对齐。

[kunguang]

@begeekmyfriend 嗯嗯，是的。整体上用的你的分支上的最新的master代码，做了一些修改。
1。 preprocess.py,symbols.py,以及datasets文件夹下的代码用的mandarin分支下的代码，

1. 另外，我把cleaner中的text = expand_numbers(text) 去掉了，取消了数字编码。要不然把音调都用One，two编码了。。
   我的机器，一天也就才跑1800步。。40k_得20天啊。。你跑了多少天得到的40k的结果。。

[begeekmyfriend]

@kunguang GTX 1080Ti about 3~4 days. By the way, you can set outputs_per_step = 5 for acceleration.
You should use my mandarin-new branch.

[kunguang]

多谢，我试试

[logicxin]

@begeekmyfriend begeekmyfriend 您好，

1. mandarin-new分支是基于中文普通话语料在Tacotron-2 上做的改动吗？训练基于中文普通话的模型话， 可以使用mandarin-new分支吗？
2. mandarin-new分支支持单机多GPU吗？
3. 依次训练tacotron2 与 wavenet 两个模型总共使用3~4天吗？

[begeekmyfriend]

@logicxin
mandarin-new branch is for G&L and the evaluation is satisfactory. WaveNet is also available but G&L is recommended by me.
This branch is only for single GPU.
If you want wavenet. you might close predict_linear option and train it about 120K steps and then train with wavenet for about 1M steps. It would take you about 2 weeks for training for all. But in wavenet mode it would takes you half an hour to synthesize a clip of wav file with duration of only serveral seconds. That is why I choose G&L as the synthesizer.

[logicxin]

@begeekmyfriend 合成的语音停顿混乱可能是什么原因啊？我听了你的样本好像没有这个问题，#122
text:
yu2 jian4 jun1 wei4 mei3 ge4, you3 cai2 neng2 de5 ren2, ti2 gong1 ping2 tai2,ta1 shi4 yin1 pin2 ling3 yu4 de5, tao2 bao3 tian1 mao1, zai4 zhe4 ge4 ping2 tai2 shang4, mei3 ge4 nei4 rong2 sheng1 chan3 zhe3, dou1 ke3 yi3 hen3 fang1 bian4 de5,shi1 xian4 zi4 wo3 jia4 zhi2, geng4 duo1 de5 ren2, yong1 you3 wei1 chuang4 ye4 de5 ji1 hui4, bu4 guo4 ta1 men5 zhi3 shi4 da1 dang4, bu2 shi4 chang2 jian4 de5 fu1 qi1 dang4 mo2 shi4, yong4 yu2 jian4 jun1 de5 hua4 lai2 shuo1, zhe4 ge4 mo2 shi4 ye3 bu4 chang2 jian4,
wave:
speech-wav-00001-linear.zip

@v-yunbin ，
THCHS-30 训练结果怎么样了呢， 我的当第二个模型wavenet的step=30w时， 结果完全是混乱的声音。

[begeekmyfriend]

@logicxin You need to check your own training corpus such as whether the puntuations were marked correctly or not...

[logicxin]

You need to check your own training corpus such as whether the puntuations were marked correctly or not

现在正使用THCHS-30 作为训练语料，原因有可能是以下原因吗？

1. 因为该语料的原因吗？比如其是多个男女speaker混合语料
2. 训练的step太少（现在是step=30w）？

如果使用纯男声语料进行训练，并使用您的mandarin-new 分支。有一个问题：
mandarin-new 分支与现有Rayhane-mamah/Tacotron-2 的master分支相比，做了哪些优化调整呢？

[begeekmyfriend]

@logicxin 我的分支主要在汉化上，其它地方基本不变，训练结果你也听到了，不过我没有用THCHS-30，一个是多人语音，另一个就是没标点。

[QueenKeys]

@begeekmyfriend 合成的语音停顿混乱可能是什么原因啊？我听了你的样本好像没有这个问题，#122
text:
yu2 jian4 jun1 wei4 mei3 ge4, you3 cai2 neng2 de5 ren2, ti2 gong1 ping2 tai2,ta1 shi4 yin1 pin2 ling3 yu4 de5, tao2 bao3 tian1 mao1, zai4 zhe4 ge4 ping2 tai2 shang4, mei3 ge4 nei4 rong2 sheng1 chan3 zhe3, dou1 ke3 yi3 hen3 fang1 bian4 de5,shi1 xian4 zi4 wo3 jia4 zhi2, geng4 duo1 de5 ren2, yong1 you3 wei1 chuang4 ye4 de5 ji1 hui4, bu4 guo4 ta1 men5 zhi3 shi4 da1 dang4, bu2 shi4 chang2 jian4 de5 fu1 qi1 dang4 mo2 shi4, yong4 yu2 jian4 jun1 de5 hua4 lai2 shuo1, zhe4 ge4 mo2 shi4 ye3 bu4 chang2 jian4,
wave:
speech-wav-00001-linear.zip

@v-yunbin ，
THCHS-30 训练结果怎么样了呢， 我的当第二个模型wavenet的step=30w时， 结果完全是混乱的声音。

你好，我在运行wavenet的train.py的时候报错：
Traceback (most recent call last):
File "/home/queen/document/Experiment/Tacotron-2-master/wavenet_vocoder/train.py", line 296, in
wavenet_train(args, log_dir, hparams, input_path)
File "/home/queen/document/Experiment/Tacotron-2-master/wavenet_vocoder/train.py", line 268, in wavenet_train
return train(log_dir, args, hparams, input_path)
File "/home/queen/document/Experiment/Tacotron-2-master/wavenet_vocoder/train.py", line 175, in train
model, stats = model_train_mode(args, feeder, hparams, global_step)
File "/home/queen/document/Experiment/Tacotron-2-master/wavenet_vocoder/train.py", line 123, in model_train_mode
feeder.input_lengths, x=feeder.inputs)
File "/home/queen/document/Experiment/Tacotron-2-master/wavenet_vocoder/models/wavenet.py", line 219, in initialize
y_hat = self.step(x, c, g, softmax=False) #softmax is automatically computed inside softmax_cross_entropy if needed
File "/home/queen/document/Experiment/Tacotron-2-master/wavenet_vocoder/models/wavenet.py", line 565, in step
x, h = conv(x, c, g_bct)
TypeError: __call__() takes 2 positional arguments but 4 were given
请问怎么解决？

[begeekmyfriend]

No need THCHS-30 any more

wget https://weixinxcxdb.oss-cn-beijing.aliyuncs.com/gwYinPinKu/BZNSYP.rar

[antontc]

Hello. I use your hyper parameters. Tacotron is 200k. I use THCHS-30. Now I run synthesize and get audio like this, and it is so bad. May be you can help me?

wavs.zip

[JamesZHANGatTJU]

@begeekmyfriend 你好，我在解压THCHS-30之后尝试进行预处理，出现了如下错误，请问如何解决？多谢！
initializing preprocessing..
Selecting data folders..
0it [00:00, ?it/s]
Write 0 utterances, 0 mel frames, 0 audio timesteps, (0.00 hours)
Traceback (most recent call last):
File "preprocess.py", line 112, in
main()
File "preprocess.py", line 108, in main
run_preprocess(args, modified_hp)
File "preprocess.py", line 85, in run_preprocess
preprocess(args, input_folders, output_folder, hparams)
File "preprocess.py", line 18, in preprocess
write_metadata(metadata, out_dir)
File "preprocess.py", line 30, in write_metadata
print('Max input length (text chars): {}'.format(max(len(m[5]) for m in metadata)))
ValueError: max() arg is an empty sequence

[begeekmyfriend]

No THCHS-30 any more.

wget https://weixinxcxdb.oss-cn-beijing.aliyuncs.com/gwYinPinKu/BZNSYP.rar

[wuzhonglijz]

@begeekmyfriend 你好，我想请教一下为什么我的模型只能合成大约14s的语音，之后的就发不出来声音了，你的模型似乎可以合成一大段文字。

[begeekmyfriend]

printf "file %s\n" *.wav > list.txt
ffmpeg -f concat -i list.txt -c copy demo.wav

[wuzhonglijz]

printf "file %s\n" *.wav > list.txt
ffmpeg -f concat -i list.txt -c copy demo.wav

Thanks for your reply, this script very useful to me. 👍

[lydhr]

@begeekmyfriend Could you please share your pretrained model based on your private dataset?

[wuqi930907]

您好，我在您的https://github.com/begeekmyfriend/Tacotron-2/tree/mandarin-griffin-lim分支下训练了一个中文语音合成模型，但是发现一个问题，当我在同一块显卡上挂载多个模型进行并发合成的时候，发现每多挂载一个模型，合成时间就会线性增加。比如只挂载一个模型合成一句话需要0.7s，当挂载两个模型同时合成同一句话的时候，每个模型合成时间都会变成1.4s左右，请问您也是这样的情况吗？如果是这样的话实在是不利于实时合成和实际生产。 @begeekmyfriend

[canyanolGitHub]

eval-30000.zip
Here are the evaluation results of my training on 12-hour Chinese mandarin corpus. The voice sounds natural but still somewhat rough.
The modification has been opened on my own repo with mandarin branch.
Thanks a lot for this work!
can you write a blog for us to show we how to train tts mandarin use Tacotron 2 ?

[coderLong]

THCHS30 语料中，train data中有30个人讲话，能够区分出来哪个wav 文件是哪个人说的吗？A2_2.wav 这个文件A2表示speaker_id吗？

[begeekmyfriend]

Please use biaobei open mandarin corpus. Do not mention THCHS30 any more.

[zhuangzhuangxie]

我用标贝数据集训练了7万步（batch_size=4）,但是alignment图怎么还是这个样子呢？

[begeekmyfriend]

Batch size has to be no less than 32.

[zhuangzhuangxie]

@begeekmyfriend
batch_size设成32 我的GPU会out of memory, 我是否可以设成4？
上图是不是因为训练还不够所以alignment图没有对齐？

[yiyouls]

eval-112000.zip
Terminal_train_log.zip
Here are the latest evaluation at 112K steps and training log. I have to say the effects are amazing!

amazing results!

[zyyuan0915]

eval-112000.zip
Terminal_train_log.zip
Here are the latest evaluation at 112K steps and training log. I have to say the effects are amazing!

amazing results!

What data set is used?Does the data use rhythm annotation information?

[ccl-private]

@begeekmyfriend Hi, 我用标贝数据集训练了7万多step，batch_size设置为32，alignment正常：

但是，在合成这句话时停顿还是有问题：“在英国支持华为的问题上，英美双方分歧早在特朗普访问前已经明确化，从一个侧面反映出在贸易战中，世界第一和第二大经济体美国和中国，在不列颠展开的政治经济角力。”（选自BBC News）

我用逗号隔开：“在英国支持华为的问题上，英美双方分歧，早在特朗普访问前，已经明确化，从一个侧面反映出，在贸易战中，世界第一和第二大经济体，美国和中国，在不列颠展开的，政治经济角力。”停顿的地方终于正常，但停顿的时间过长。
eval结果：2.zip

现在想请教一下您是否有好的解决方法。

[begeekmyfriend]

这个alignment叫正常？你改了我的fork代码了吧

[ccl-private]

我修改了preprocessor.py的代码，另外还将batch_size设置为32，其他地方没动。

关于alignment我不是很懂哈～之前主要做机器视觉相关研究，最近要加入机器语音才开始研究语音合成。能否稍微解释一下哈，不胜感激。

[begeekmyfriend]

这种alignment比较糟糕，建议用我原版分支的代码，另外，你的train.txt但愿没搞错

[ccl-private]

我下载了您的原版分支代码，preprocessor.py里根据.trn文件来处理数据读入，而我在标贝下载的数据集没有.trn文件，似乎是您有对数据集进行了额外的处理。所以我只好修改了preprocessor.py的部分代码。至于batch_size设置为32，是由于GPU显存的限制。
关于train.txt文件，下面是其前几行的截图：

另外请教一下alignment什么情况下才是正常的呀～

[begeekmyfriend]

当然是这种的，句子越短，像素点越大，但是不应该断断续续的，我的代码不会变成你这样：https://github.com/Rayhane-mamah/Tacotron-2/issues/18#issuecomment-382637788
您的train.txt似乎没有问题，Batch size不小于32也不会影响到。原因不明……
还有，标贝的语料都是短句，你用它一次性生成长句会有问题的，可以考虑断句。

[ccl-private]

我打算完全用您的代码重新训练一遍看看。但是由于我下载的标贝数据集没有.trn文件，preprocessor.py部分代码没法执行。为了和您的实验保持一致，可否提供您用于生成.trn文件的代码～
另外补上我的mel_spectrogtam图，希望有助于找到alignment异常的原因～

[Bobb10]

@begeekmyfriend

I used your preprocess.py for the Mandarin Chinese language (https://github.com/begeekmyfriend/Tacotron-2/blob/mandarin-griffin-lim/preprocess.py) to reproduce it to Dutch language.

I have a dataset folder which looks likes the LJSpeech Dataset containing all the wav files and the metadata.csv file. However, when I run the preprocess.py script (adjusted to my own dataset ofcrouse):

python preprocess.py --data="HPSpeech-1.1" --language="nl-NL" --voice="male" -- reader="jan_meng" --merge_books=True --book="northandsouth"

I get the following error:

preprocess.py: error: unrecognized arguments: dataset=HPSpeech-1.1.

My dataset folder is named HPSpeech-1.1 and I modified the defaults and settings preprocess.py to my own requirements.

Any idea how to solve this?

[begeekmyfriend]

@Bobb10 You might use grffin-lim branch and set linear_prediction as False to synthesize mel spectrograms. And then train with WaveRNN.

python synthesize.py --mode 'synthesis'

[yuyq96]

我用 @begeekmyfriend 的代码训练45k之后也是断开的，不过感觉合成的结果没有太大的问题。
eval.zip（没有手动添加逗号）

@begeekmyfriend Hi, 我用标贝数据集训练了7万多step，batch_size设置为32，alignment正常：

但是，在合成这句话时停顿还是有问题：“在英国支持华为的问题上，英美双方分歧早在特朗普访问前已经明确化，从一个侧面反映出在贸易战中，世界第一和第二大经济体美国和中国，在不列颠展开的政治经济角力。”（选自BBC News）

我用逗号隔开：“在英国支持华为的问题上，英美双方分歧，早在特朗普访问前，已经明确化，从一个侧面反映出，在贸易战中，世界第一和第二大经济体，美国和中国，在不列颠展开的，政治经济角力。”停顿的地方终于正常，但停顿的时间过长。
eval结果：2.zip

现在想请教一下您是否有好的解决方法。

[coderLong]

我也在做中文合成，附件是我合成的demo， 能否听一下你合成的demo , 加个qq 或者 微信 交流一下
我也想优化停顿

[email protected]

From: Ya-Qi Yu
Date: 2019-06-26 14:09
To: Rayhane-mamah/Tacotron-2
CC: Malong; Comment
Subject: Re: [Rayhane-mamah/Tacotron-2] Evaluation on Chinese mandarin (#18)
我用 @begeekmyfriend 的代码训练45k之后也是断开的，不过感觉合成的结果没有太大的问题。
eval.zip（没有手动添加逗号）
@begeekmyfriend Hi, 我用标贝数据集训练了7万多step，batch_size设置为32，alignment正常：

但是，在合成这句话时停顿还是有问题：“在英国支持华为的问题上，英美双方分歧早在特朗普访问前已经明确化，从一个侧面反映出在贸易战中，世界第一和第二大经济体美国和中国，在不列颠展开的政治经济角力。”（选自BBC News）
我用逗号隔开：“在英国支持华为的问题上，英美双方分歧，早在特朗普访问前，已经明确化，从一个侧面反映出，在贸易战中，世界第一和第二大经济体，美国和中国，在不列颠展开的，政治经济角力。”停顿的地方终于正常，但停顿的时间过长。
eval结果：2.zip
现在想请教一下您是否有好的解决方法。
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[yuyq96]

@coderLong 我的demo已经贴在上一条回复里了，没有看到你的附件，建议用网页端查看。

[ccl-private]

@yuyq96 我也把我45k轮的模型拿来测试了一下，停顿错误减少了，但效果还是不如你的好。

45k.zip

训练过程我直接用python train.py --model='Tacotron-2'
不知道分开来训练有没有区别。
你是分开练Tacotron和WaveNet吗，还是直接Tacotron-2

[1105060120]

@JamesZHANGatTJU ，哥们，我也遇到那个问题，请问你如何解决的。

[maocaixia]

@begeekmyfriend
Hello, your mandarin branch is missing?

[begeekmyfriend]

[maocaixia]

@begeekmyfriend
Thanks for your reply!
I'm training the tacotron model and the wavenet model separately, but the loss shown is negative when I'm training the wavenet model. And the tacotron training is good.
Do you know what the reason is?

[JasonWei512]

@begeekmyfriend
Thanks for your reply!
I'm training the tacotron model and the wavenet model separately, but the loss shown is negative when I'm training the wavenet model. And the tacotron training is good.
Do you know what the reason is?

It's normal if your WaveNet's "input_type" in hparams.py is set to "raw". Keep training.

[begeekmyfriend]

@maocaixia In fact what I continue to train with mel GTA is WaveRNN vocoder with simpler architecture. The performance of evaluation wav form is good enough for listening.

[maocaixia]

@begeekmyfriend @JasonWei512
OK, thanks so much! I will have a try!

[maocaixia]

Hello, I find the training time is too long for wavenet, maybe <100k for one day. and the convert time is slow. I want to do the almost real time converting (from text to voice) with tacotron. any suggestion about it ? is your branch mandarin-mel-prediction good enough? thanks sincerely!
@begeekmyfriend @JasonWei512

[maocaixia]

And the output vioce is not good as expected, this is the result with tacotron 100000 step and wavenet 360000 step.
wavenet-audio-mel-batch_0_sentence_0(5).zip

[JasonWei512]

@maocaixia
My results at 900K steps: Eval.zip
It's better than 200K but still very noisy. Now the loss stucks at -5.0 and doesn't decrease.

If you want fast synthesis, you can try begeekmyfriend's fork (which uses WaveRNN instead of WaveNet). Non-parallel WaveNet synthesis is SUPER slow.
Or you can just use the "live" synthesize mode (which uses Griffin-Lim instead of WaveNet, fastest but lower audio quality):

python synthesize.py --mode='live'

If you want the best audio quality, you can look into https://github.com/r9y9/wavenet_vocoder

[maocaixia]

@begeekmyfriend @JasonWei512
Hey, thanks you so much!!
I use the tacotron+wavernn and i'm training the model, the intermediate result is not bad.
I still have two questions and apprecisite it if you have any suggestion.
(1) the wavernn training can use multiple gpus? and the training time is still so long. maybe 10days
(2) the detection time is about 30sec for a word like "苟利国家生死以", any suggestion to accelerate the process?
Anyway, thank you!

[begeekmyfriend]

The WaveRNN inference could be used with model < 500K (5 days might be)
And you can convert WaveRNN generate code into TRT

[KuangDD]

训练好模型后，在合成推理的时候发现：句子越长语速越快，这个问题怎么解决呢？

[maocaixia]

@begeekmyfriend @JasonWei512
Thank's your advice! I use the tacotron+wavernn to generate voice in file sentences.txt, there are multiple sentences in sentences.txt, when generating about three sentences, there is a bug. I find the tacotron time is about 1~3s, but when the bug occurs, the tacotron inference time is only about 200ms, I think it's the tacotron problem and don't know why. do you know any reason? here are the pictures of normal and bug time below.

[maocaixia]

And the numpy version is 1.16.2, librosa version is 0.6.3

[maocaixia]

The same data sometimes goes wrong sometimes without error. It's really strange!