Tensorflowtts: 🇨🇳 Chinese TTS now available 😘

Created on 11 Aug 2020 · 32Comments · Source: TensorSpeech/TensorFlowTTS

Chinese TTS now available, thank @azraelkuan for his support :D. The model used Baker dataset here (https://www.data-baker.com/open_source.htmlt). The pretrained model licensed under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/) since the dataset is non-commercial :D

Pls check out the colab bellow and enjoy :D.

https://colab.research.google.com/drive/1YpSHRBRPBI7cnTkQn1UcVTWEQVbsUm1S?usp=sharing

Note: this is just init results, there are more things can be done to make the model better.

cc: @candlewill @l4zyf9x @machineko

Feature Request 🤗 enhancement 🚀 good first issue 🤔

Source

dathudeptrai

❤8 👍6 😄2

Most helpful comment

@jinfagang jsut download biaobei data and extract it to baker

tensorflow-tts-preprocess --dataset baker --rootdir ~/Data/baker --outdir dump --config ./preprocess/baker_preprocess.yaml

and train it using baker's yaml.

azraelkuan on 30 Aug 2020

👍4

All 32 comments

Hello. Thanks for your great work! I'm new to the TTS area and this notebook could be a good start.
However, I gave it a try and found out the Chinese model at present doesn't make pauses. I wonder has this been one of your potential improvements yet?

wyp1996 on 27 Aug 2020

cc: @azraelkuan (person in charge)

dathudeptrai on 27 Aug 2020

@wyp1996 for now, we do not have a frontend model, but we have place #1,2,3,sil in the training

azraelkuan on 28 Aug 2020

@dathudeptrai Does it already in master branch for support?

jinfagang on 29 Aug 2020

@jinfagang everything is on master branch. (updated content :D.)

dathudeptrai on 29 Aug 2020

@dathudeptrai Any readme on how to train on Biaobei data?

jinfagang on 30 Aug 2020

@jinfagang jsut download biaobei data and extract it to baker

tensorflow-tts-preprocess --dataset baker --rootdir ~/Data/baker --outdir dump --config ./preprocess/baker_preprocess.yaml

and train it using baker's yaml.

azraelkuan on 30 Aug 2020

👍4

@azraelkuan As I know the sampling rate of the audio in the Biaobei dataset is 48k, but in the baker_preprocess.yaml the sampling rate is set as 24k. I didn't try the preprocess. Is that a mistake or I misunderstand the code?

IreneZhou2018 on 4 Sep 2020

@IreneZhou2018 the sampling rate in config is target sampling rate, if the dataset's sample rate is 48k so we re-sample it (see code here https://github.com/TensorSpeech/TensorFlowTTS/blob/master/tensorflow_tts/bin/preprocess.py#L194-L196)

dathudeptrai on 4 Sep 2020

@dathudeptrai ok, thanks for your reply and the work is amazing!

IreneZhou2018 on 4 Sep 2020

TensorFlowTTS Serving：
https://github.com/MachineLP/QDServing/tree/master/model_serving
https://github.com/MachineLP/QDServing

MachineLP on 8 Sep 2020

TensorflowTTS训练数据生成：拉取文本数据、将文本专为拼音、基于阿里云TTS生成TensorflowTTS训练音频、训练前的preprocess/normalize：https://github.com/MachineLP/TensorFlowTTS_chinese/tree/master/generate_tts_data

MachineLP on 8 Sep 2020

TensorflowTTS训练数据生成：拉取文本数据、将文本专为拼音、基于阿里云TTS生成TensorflowTTS训练音频、训练前的preprocess/normalize：https://github.com/MachineLP/TensorFlowTTS_chinese/tree/master/generate_tts_data

Hi, do you have a more specific Readme? It seems promising and I'd like to have to try :)

wyp1996 on 8 Sep 2020

@jinfagang jsut download biaobei data and extract it to baker
tensorflow-tts-preprocess --dataset baker --rootdir ~/Data/baker --outdir dump --config ./preprocess/baker_preprocess.yaml
and train it using baker's yaml.

Hongpeng1992 on 8 Sep 2020

it seems that fastspeech2 model do not work properly when sentence is long ? like 君不见黄河之水天上来奔流到海不复回君不见高堂明镜悲白发朝如青丝暮成雪人生得意须尽欢莫使金樽空对月

Hongpeng1992 on 8 Sep 2020

it seems that fastspeech2 model do not work properly when sentence is long ? like 君不见黄河之水天上来奔流到海不复回君不见高堂明镜悲白发朝如青丝暮成雪人生得意须尽欢莫使金樽空对月

https://github.com/TensorSpeech/TensorFlowTTS/issues/208#issuecomment-688356211

dathudeptrai on 8 Sep 2020

Thank you . I am still evaluating the model .

Hongpeng1992 on 8 Sep 2020

Chinese TTS欢迎加微信:lp9628，进入微信群讨论训练测试细节问题。

MachineLP on 8 Sep 2020

@dathudeptrai I try to train get this error:

2020-10-16 22:28:20.499294: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:906] Skipping loop optimization for Merge node with control input: cond/branch_executed/_8
Traceback (most recent call last):
  File "examples/tacotron2/train_tacotron2.py", line 488, in <module>
    main()
  File "examples/tacotron2/train_tacotron2.py", line 476, in main
    trainer.fit(
  File "/media/jintian/samsung/source/ai/swarm/exp/TensorFlowTTS/tensorflow_tts/trainers/base_trainer.py", line 870, in fit
    self.run()
  File "/media/jintian/samsung/source/ai/swarm/exp/TensorFlowTTS/tensorflow_tts/trainers/base_trainer.py", line 101, in run
    self._train_epoch()
  File "/media/jintian/samsung/source/ai/swarm/exp/TensorFlowTTS/tensorflow_tts/trainers/base_trainer.py", line 123, in _train_epoch
    self._train_step(batch)
  File "examples/tacotron2/train_tacotron2.py", line 109, in _train_step
    self.one_step_forward(batch)
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 545, in call
    outputs = execute.execute(
  File "/home/jintian/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:    Trying to access element 62 in a list with 62 elements.
     [[{{node while_19/body/_1/while/TensorArrayV2Read_1/TensorListGetItem}}]]
     [[tacotron2/encoder/bilstm/forward_lstm/PartitionedCall]] [Op:__inference__one_step_forward_23575]

Function call stack:
_one_step_forward -> _one_step_forward -> _one_step_forward

My command:

python examples/tacotron2/train_tacotron2.py \                                                                                                               ⎇  master ✘ !?|73cac7f
  --train-dir ./dump/train/ \
  --dev-dir ./dump/valid/ \
  --outdir ./examples/tacotron2/exp/train.tacotron2.baker.v1/ \
  --config ./examples/tacotron2/conf/tacotron2.baker.v1.yaml \
  --use-norm 1 \
  --mixed_precision 0 \
  --resume ""

jinfagang on 16 Oct 2020

The punctuation pause does not seem to be handled.
Arabic numbers cannot be directly predicted.
Hope this can be mixed in Chinese and English.

leijue222 on 21 Oct 2020

Chinese TTS now available, thank @azraelkuan for his support :D. The model used Baker dataset here (https://www.data-baker.com/open_source.htmlt). The pretrained model licensed under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/) since the dataset is non-commercial :D

Pls check out the colab bellow and enjoy :D.

https://colab.research.google.com/drive/1YpSHRBRPBI7cnTkQn1UcVTWEQVbsUm1S?usp=sharing

Note: this is just init results, there are more things can be done to make the model better.

cc: @candlewill @l4zyf9x @machineko

@jinfagang everything is on master branch. (updated content :D.)

hello, the link to baker dataset was expired , and the officical website show that have no right to access the dataset , i hate to say that ,but can you provide annother way to get the dataset?

jucaowei on 22 Oct 2020

@jucaowei The link is here. The data only has a female voice.

leijue222 on 22 Oct 2020

@jucaowei The link is here. The data only has a female voice.

thank you !!

jucaowei on 22 Oct 2020

@jucaowei The link is here. The data only has a female voice.

404 error ,you can acess the website ? i got 404 not found
HTTP Status 404 – Not Found
Type Status Report
Message /open_source.htmlt
Description The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.

jucaowei on 22 Oct 2020

@jucaowei
I can access the website normally. Which country are you in now? Maybe a VPN is needed for network problems?

leijue222 on 22 Oct 2020

@jucaowei
I can access the website normally. Which country are you in now? Maybe a VPN is needed for network problems?

i already use VPN with HK node,not working, but my friend access the website right now, really appreciate for you reply

jucaowei on 22 Oct 2020

@azraelkuan Hi! Thanks for your work. Compared with some other reproduction projects, your reproduced tacotron2 can synthesize very long sentences without stress or omission.
I have tried your job to achieve a maximum of about 90 seconds. To be reasonable, the Biaobei dataset is relatively short sentences, and the model trained with Biaobei should not be able to synthesize such long sentences.
Have you done any special treatment of long sentences?

leijue222 on 7 Nov 2020

no, this tacotron is implemented by the author of this project, i just use
cn dataset to train it.
may be i use the phoneme as the input, and each phoneme has a token #0
after it.

yuze notifications@github.com 于2020年11月7日周六下午2:32写道：

@azraelkuan https://github.com/azraelkuan Hi! Thanks for your work.
Compared with some other reproduction projects, your reproduced tacotron2
can synthesize very long sentences without stress or omission. Have you
done any special treatment of long sentences?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TensorSpeech/TensorFlowTTS/issues/201#issuecomment-723400006,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFSMBODUYYMWSRSPLUYJQ4LSOTSZJANCNFSM4P3B4TPA
.

--
Kuan Chen （陈宽）
Speech Lab, Shanghai Jiao Tong University
Tel: +86 17621207116

azraelkuan on 9 Nov 2020

用声码器模型把mel频谱转为语音信号需要注意什么？
我的采样率是22050 hop size =100

c1a1o1 on 15 Dec 2020

TTS交流群，VX:WorldSeal，欢迎相互交流和讨论~

xiaoyangnihao on 17 Dec 2020

@azraelkuan It seems there is a mismatch between the hopsize 300 in preprocess and in tacotron/fastspeech2 and melgan: 256

https://github.com/TensorSpeech/TensorFlowTTS/blob/master/preprocess/baker_preprocess.yaml

hop_size: 300            # Hop size. (fixed value, don't change)

https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/tacotron2/conf/tacotron2.baker.v1.yaml

hop_size: 256            # Hop size.

https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/fastspeech2/conf/fastspeech2.baker.v2.yaml

hop_size: 256            # Hop size.

https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/melgan/conf/melgan.v1.yaml

hop_size: 256            # Hop size.

ronggong on 1 Jan 2021

@azraelkuan May I know how many gpus and batch_size you have used to train the tacotron2/fastspeech2 and melgan model in the colab? https://colab.research.google.com/drive/1YpSHRBRPBI7cnTkQn1UcVTWEQVbsUm1S?usp=sharing

ronggong on 1 Jan 2021

Was this page helpful?

0 / 5 - 0 ratings