Tensorflowtts: fastspeech2 trainning question

[learnerkun]

2020-07-16 15:29:27,314 (train_fastspeech2:226) INFO: (Step: 200) train_f0_loss = 1.9053.
2020-07-16 15:29:27,317 (train_fastspeech2:226) INFO: (Step: 200) train_energy_loss = 2.1922.
2020-07-16 15:29:27,321 (train_fastspeech2:226) INFO: (Step: 200) train_mel_loss_before = 0.6235.
2020-07-16 15:29:27,324 (train_fastspeech2:226) INFO: (Step: 200) train_mel_loss_after = 0.6925.
File "/data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/examples/fastspeech2/train_fastspeech2.py", line 519, in
main()
File "/data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/examples/fastspeech2/train_fastspeech2.py", line 511, in main
resume=args.resume,
File "/data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/examples/fastspeech/train_fastspeech.py", line 308, in fit
self.run()
File "/data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 67, in run
self._train_epoch()
File "/data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 89, in _train_epoch
self._train_step(batch)
File "/data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/examples/fastspeech2/train_fastspeech2.py", line 80, in _train_step
self._one_step_fastspeech2(charactor, duration, f0, energy, mel)
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(args, *kwds)
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 611, in _call
return self._stateless_fn(args, *kwds) # pylint: disable=not-callable
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
self.captured_inputs)
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 598, in call
ctx=ctx)
File "/usr/local/anaconda3/envs/tf2.2/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [8,512,80] vs. [8,620,80]
[[node mean_absolute_error/sub (defined at data2/l50013588/pycharmProjects/FastSpeech2/TensorflowTTS-master/examples/fastspeech2/train_fastspeech2.py:120) ]] [Op:__inference__one_step_fastspeech2_96220]

everytime I start train，it reports this log，and everytime the step is not same，the log reported at 100 to 700.I think the question is that the mels length in a batch is not same.

[dathudeptrai]

@learnerkun the step is not the same because we shuffe training set before training it at every epoch :D. The bug is because sum(durs) != len(mel) at some samples. You need to check by urself if there are some samples has this problem :D

[learnerkun]

I know there is one solution,set the batch 1,but I think this is not a good solution.I want know if there are some other solutions

[learnerkun]

Ok, I will try it.

[dathudeptrai]

@learnerkun the problem is not mels length in a batch is not same but the len of mel-predicted != len of mel-groundtruth

[learnerkun]

You are right,two durations files are wrong.thanks!