Tensorflowtts: training FastSpeech2 on different language

[MODAK27]

I am training FS2 on non English Dataset and i have transformed cleaners to "transcilerated_cleaners" and have also changed the valid letters used .
while Training I am incurring error , 2 possible reasons could be
1.Do i need to train my nw dataset on TAc02 for extracting durations that gets fit in FS2?
2.I need to change the valid symbols in Cleaners module?
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 243, in __call__
ret = func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(args, *kwargs)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 785, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))

File "/usr/local/lib/python3.6/dist-packages/examples_tts/fastspeech2/fastspeech2_dataset.py", line 205, in generator
duration = self.duration_load_fn(duration_file)

File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 457, in load
raise ValueError("Cannot load file containing pickled data "

ValueError: Cannot load file containing pickled data when allow_pickle=False

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 1986, in execution_mode
yield
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 655, in _next_internal
output_shapes=self._flat_output_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2363, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: Cannot load file containing pickled data when allow_pickle=False
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 243, in __call__
ret = func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(args, *kwargs)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 785, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))

File "/usr/local/lib/python3.6/dist-packages/examples_tts/fastspeech2/fastspeech2_dataset.py", line 205, in generator
duration = self.duration_load_fn(duration_file)

File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 457, in load
raise ValueError("Cannot load file containing pickled data "

ValueError: Cannot load file containing pickled data when allow_pickle=False

 [[{{node PyFunc}}]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/tts-replica/examples_tts/fastspeech2/train_fastspeech2.py", line 513, in
main()
File "/content/tts-replica/examples_tts/fastspeech2/train_fastspeech2.py", line 505, in main
resume=args.resume,
File "/usr/local/lib/python3.6/dist-packages/examples_tts/fastspeech/train_fastspeech.py", line 303, in fit
self.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 67, in run
self._train_epoch()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 87, in _train_epoch
for train_steps_per_epoch, batch in enumerate(self.train_data_loader, 1):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 631, in __next__
return self.next()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 670, in next
return self._next_internal()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 661, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 1989, in execution_mode
executor_new.wait()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: Cannot load file containing pickled data when allow_pickle=False
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 243, in __call__
ret = func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(args, *kwargs)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 785, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))

File "/usr/local/lib/python3.6/dist-packages/examples_tts/fastspeech2/fastspeech2_dataset.py", line 205, in generator
duration = self.duration_load_fn(duration_file)

File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 457, in load
raise ValueError("Cannot load file containing pickled data "

ValueError: Cannot load file containing pickled data when allow_pickle=False

 [[{{node PyFunc}}]]

[dathudeptrai]

@MODAK27 can you try set allow_pickle=True when using np.load ?. for example, you can replace https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/fastspeech2/fastspeech2_dataset.py#L203 to

self.mel_load_fn(mel_file, allow_pickle=True)

In our preprocessing, for example here https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/bin/preprocess.py#L306, we save the file with allow_pickle=False so everything is ok, it seems that ur file save with allow_pickle=True

[MODAK27]

@MODAK27 can you try set allow_pickle=True when using np.load ?. for example, you can replace https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/fastspeech2/fastspeech2_dataset.py#L203 to

self.mel_load_fn(mel_file, allow_pickle=True)

In our preprocessing, for example here https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/bin/preprocess.py#L306, we save the file with allow_pickle=False so everything is ok, it seems that ur file save with allow_pickle=True

Sir, I tried it.But it gives another error
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2020-07-18 21:10:57.684122: W tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1908] No (suitable) GPUs detected, skipping auto_mixed_precision graph optimizer
Traceback (most recent call last):
File "/content/tts-replica/examples_tts/fastspeech2/train_fastspeech2.py", line 513, in
main()
File "/content/tts-replica/examples_tts/fastspeech2/train_fastspeech2.py", line 505, in main
resume=args.resume,
File "/usr/local/lib/python3.6/dist-packages/examples_tts/fastspeech/train_fastspeech.py", line 303, in fit
self.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 67, in run
self._train_epoch()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 89, in _train_epoch
self._train_step(batch)
File "/content/tts-replica/examples_tts/fastspeech2/train_fastspeech2.py", line 78, in _train_step
self._one_step_fastspeech2(charactor, duration, f0, energy, mel)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 644, in _call
return self._stateless_fn(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2420, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
self.captured_inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 598, in call
ctx=ctx)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [16,21,384] vs. [16,158,384]
[[node gradient_tape/tf_fast_speech2/add_1/BroadcastGradientArgs (defined at content/tts-replica/examples_tts/fastspeech2/train_fastspeech2.py:126) ]] [Op:__inference__one_step_fastspeech2_44644]

Function call stack:
_one_step_fastspeech2

[dathudeptrai]

@MODAK27 here

Incompatible shapes: [16,21,384] vs. [16,158,384]

That means the bug tell there is a sample that the sum(durations) != len(mel). I see many people got this bug, maybe i need to add some note on README to tell the user check the sum(duration) and len(mel) by theirself before training.

[MODAK27]

@MODAK27 here

Incompatible shapes: [16,21,384] vs. [16,158,384]

That means the bug tell there is a sample that the sum(durations) != len(mel). I see many people got this bug, maybe i need to add some note on README to tell the user check the sum(duration) and len(mel) by theirself before training.

Thanx sir. Your work on improving opensource TTS is highly appreciated and I am learning a lot from you. 💯 As I have mentioned earlier,
I am training FS2 on non English Dataset and i have transformed cleaners to "transcilerated_cleaners" and have also changed the valid letters used .
while Training I am incurring error , 2 possible SOLUTIONS could be
1 I need to train my nw dataset on TAc02 for extracting durations that gets fit in FS2?
2.Do I need to change the valid symbols(ARPAbet symbols) in LJspeech module, I have already changed letters in there?

[MODAK27]

And I having one more concern, SInce I am running this on COlab (GPU/TPU) but stilll while training it's showing
No (suitable) GPUs detected, skipping auto_mixed_precision graph optimizer

[dathudeptrai]

@MODAK27

I need to train my nw dataset on TAc02 for extracting durations that gets fit in FS2?

Yes, you need re-train tacotron-2 in ur dataset to get proper alignment or you can use MFA to extract durations.

Do I need to change the valid symbols(ARPAbet symbols) in LJspeech module, I have already changed letters in there?

Yes, if your dataset is non-english so you need to change the symbols by urself to match with ur target languages.

And I having one more concern, SInce I am running this on COlab (GPU/TPU) but stilll while training it's showing
No (suitable) GPUs detected, skipping auto_mixed_precision graph optimizer

That mean the GPU of colab is not support mixed_precision, that fine.

BTW, can you share ur colab training notebook ?

[MODAK27]

@MODAK27

I need to train my nw dataset on TAc02 for extracting durations that gets fit in FS2?

Yes, you need re-train tacotron-2 in ur dataset to get proper alignment or you can use MFA to extract durations.

Do I need to change the valid symbols(ARPAbet symbols) in LJspeech module, I have already changed letters in there?

Yes, if your dataset is non-english so you need to change the symbols by urself to match with ur target languages.

And I having one more concern, SInce I am running this on COlab (GPU/TPU) but stilll while training it's showing
No (suitable) GPUs detected, skipping auto_mixed_precision graph optimizer

That mean the GPU of colab is not support mixed_precision, that fine.

BTW, can you share ur colab training notebook ?

yes sir, Thanx for your quick reply
https://colab.research.google.com/drive/1A4MBtu6jcHnnhY9oM-ERbEirc2RJKbjC?usp=sharing

[manmay-nakhashi]

@MODAK27 you have to make your own language processor like LJSpeechProcessor containing all the symbols and dont use english_cleaner make your own cleaner or use non_english cleaner you have to change your embedding size according to your max sentence length or you can dump some sentences to reduce your max_embedding_size because it increases training time and vocab size= len(symbols) accordingly for gujrati.

[sjainlucky]

@manmay-nakhashi I have been trying to train Tacotron 2 on Non-english Dataset and I have changed cleaners to transliteration cleaner and changed valid symbol but while training I incurred following error:-

0% 0/200000 [00:00 2020-07-19 17:44:46.335075: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
Traceback (most recent call last):
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 594, in
main()
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 586, in main
resume=args.resume,
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 392, in fit
self.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 67, in run
self._train_epoch()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 89, in _train_epoch
self._train_step(batch)
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 117, in _train_step
charactor, char_length, mel, mel_length, guided_attention
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 644, in _call
return self._stateless_fn(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2420, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
self.captured_inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 598, in call
ctx=ctx)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: [_Derived_] Trying to access element 22 in a list with 18 elements.
[[{{node while_21/body/_1/TensorArrayV2Read_1/TensorListGetItem}}]]
[[tacotron2/StatefulPartitionedCall/encoder/bilstm/backward_lstm/StatefulPartitionedCall]] [Op:__inference__one_step_tacotron2_85874]

Function call stack:
_one_step_tacotron2 -> _one_step_tacotron2 -> _one_step_tacotron2

[manmay-nakhashi]

@sjainlucky what all files did you change ?

[sjainlucky]

@manmay-nakhashi I have just changed valid_symbols variable and _letters variable and I was trying to train it on Tacotron 2

[manmay-nakhashi]

@sjainlucky you have to change examples/tacotron2/tacotron_dataset.py file for symbol import

[sjainlucky]

@manmay-nakhashi Thank you sir for your quick reply.

[sjainlucky]

@manmay-nakhashi Exactly what changes we have to implement in that I am not getting it.

[manmay-nakhashi]

@sjainlucky can you upload your code somewhere so i can look into it ?

[sjainlucky]

@manmay-nakhashi This is the colab link for the implementation of the code
https://colab.research.google.com/drive/1_xSW32ivIk3itfWFsr0Yar4vCbNIv0cn?usp=sharing

[manmay-nakhashi]

@sjainlucky change use_fixed_shapes: false in config file and increase max_iteration=4000 may be according to dataset or remove longer sentence from your data set

[MODAK27]

@dathudeptrai sir can FS2 or tac02 ( ur model implementation ) be used for multi speaker corpus?

[dathudeptrai]

@MODAK27 yes, you just need return proper speaker_id in the dataloader

[janson91]

@MODAK27 here

Incompatible shapes: [16,21,384] vs. [16,158,384]

That means the bug tell there is a sample that the sum(durations) != len(mel). I see many people got this bug, maybe i need to add some note on README to tell the user check the sum(duration) and len(mel) by theirself before training.

my sum(durations) == len(mel) but I get this error:
Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [0,450,384] vs. shape[1] = [1,790,384]
[[{{node tf_fast_speech2/length_regulator/while/body/_1/concat}}]]
[[tf_fast_speech2/mul_2/_212]]
@dathudeptrai can you give me some advice that what's wrong with my code

Thx

[dathudeptrai]

@janson91 can you check again 2 things bellow:

1. all ur file have sum(durations) == len(mel)
2. all element of durations is positive ?

[janson91]

@janson91 can you check again 2 things bellow:

1. all ur file have sum(durations) == len(mel)
2. all element of durations is positive ?

ur right. one element of a duration is negative，and the code worked when I delete the wrong file.

thx

[dathudeptrai]

@janson91 glad to see i can help you, if you see this project is helpful, give our repo a star and share our work with ur friends =))).

[janson91]

@janson91 glad to see i can help you, if you see this project is helpful, give our repo a star and share our work with ur friends =))).

haha.
and i get a problem again. @dathudeptrai
Invalid argument: Incompatible shapes: [16,37,384] vs. [16,36,384]
[[node tf_fast_speech2/add (defined at ...tensorflow_tts/models/fastspeech2.py:160) ]]
[[tf_fast_speech2/mul_2/_212]]

and I check my durations. some element is zero. is that the cause of the error?

[dathudeptrai]

@janson91 pull the newest code and test again :D. zero is fine.

[janson91]

@dathudeptrai ok, thx

[MODAK27]

@dathudeptrai sir I have been trying it in different language but have been stuck. My new language has:
Sample rate: 48000
Channels: 1
Subtype: PCM_16
I have adopted basic_cleaners and have changed punctuations and letters in processor file.
I had to cmment it out coz of some problems being created

assert (

    #     rate == config["sampling_rate"]
    # ), f"{utt_id} seems to have a different sampling rate."

I am getting an error as:

Total params: 31,985,490
Trainable params: 31,975,250
Non-trainable params: 10,240

---

2020-07-22 18:39:36.169271: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 1282 of 2108
2020-07-22 18:39:46.176076: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 1917 of 2108
2020-07-22 18:39:49.119685: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
2020-07-22 18:39:49.126894: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 39659520 exceeds 10% of free system memory.
2020-07-22 18:39:49.146869: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 63950976 exceeds 10% of free system memory.
Traceback (most recent call last):
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 594, in
main()
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 586, in main
resume=args.resume,
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 392, in fit
self.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 67, in run
self._train_epoch()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 89, in _train_epoch
self._train_step(batch)
File "/content/tts-replica/examples_tts/tacotron2/train_tacotron2.py", line 117, in _train_step
charactor, char_length, mel, mel_length, guided_attention
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 644, in _call
return self._stateless_fn(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2420, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
self.captured_inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 598, in call
ctx=ctx)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.CancelledError: [_Derived_]Loop execution was cancelled.
[[{{node while_21/LoopCond/_55}}]]
[[tacotron2/StatefulPartitionedCall/encoder/bilstm/forward_lstm/StatefulPartitionedCall]] [Op:__inference__one_step_tacotron2_85876]

Function call stack:
_one_step_tacotron2 -> _one_step_tacotron2 -> _one_step_tacotron2

For reference colab:
https://colab.research.google.com/drive/1A4MBtu6jcHnnhY9oM-ERbEirc2RJKbjC?usp=sharing

[dathudeptrai]

The repo now support three languages so i think you have an answer already, closed.