Tensorflowtts: Fine-tuning does not work

[sujeendran]

Hi,
I am trying to fine-tune the LJSpeech pretrained tacotron 2 model to fit custom voice dataset in English. I made the changes as mentioned for rebuilding with a new Embedding layer. But it fails to build it the second time. Since my dataset is also english but with different voice, I used the same vocab_size.
This is the change made for the finetuning code with respect to the training code.
`pretrained_config = Tacotron2Config(**config["tacotron2_params"])

tacotron2 = TFTacotron2(pretrained_config, training=True, name='tacotron2')

tacotron2._build()

tacotron2.summary()

tacotron2.load_weights(path)

pretrained_config.vocab_size = len(symbols)

new_embedding_layers = TFTacotronEmbeddings(pretrained_config, name='embeddings')

tacotron2.encoder.embeddings = new_embedding_layers

# re-build model

tacotron2._build() #BREAKS HERE

tacotron2.summary()`

Error:
`Total params: 31,966,546
Trainable params: 31,956,306
Non-trainable params: 10,240

2020-06-22 10:26:33.976375: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at resource_variable_ops.cc:637 : Not found: Resource localhost/encoder/embeddings/character_embeddings/weight_147/N10tensorflow3VarE does not exist.
Traceback (most recent call last):
File "examples/tacotron2/finetune_tacotron.py", line 518, in
main()
File "examples/tacotron2/finetune_tacotron.py", line 471, in main
tacotron2._build()
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow_tts/models/tacotron2.py", line 677, in _build
self(input_ids, input_lengths, speaker_ids, mel_outputs, mel_lengths, 10, training=True)
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in __call__
outputs = self.call(cast_inputs, args, *kwargs)
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(args, *kwds)
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 611, in _call
return self._stateless_fn(args, *kwds) # pylint: disable=not-callable
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
self.captured_inputs)
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call
ctx=ctx)
File "/anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
(0) Failed precondition: Error while reading resource variable encoder/embeddings/LayerNorm/beta_164 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/encoder/embeddings/LayerNorm/beta_164/N10tensorflow3VarE does not exist.
[[node encoder/embeddings/LayerNorm/batchnorm/ReadVariableOp (defined at /anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow_tts/models/tacotron2.py:153) ]]
[[decoder/while/body/_1/decoder_cell/assert_positive/assert_less/Assert/AssertGuard/pivot_f/_289/_53]]
(1) Failed precondition: Error while reading resource variable encoder/embeddings/LayerNorm/beta_164 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/encoder/embeddings/LayerNorm/beta_164/N10tensorflow3VarE does not exist.
[[node encoder/embeddings/LayerNorm/batchnorm/ReadVariableOp (defined at /anaconda/envs/py37_tensorflow/lib/python3.7/site-packages/tensorflow_tts/models/tacotron2.py:153) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_call_8492]

Function call stack:
call -> call`

Since the end goal for me is to just change the voice, i initially tried to train the model further from the pretrained checkpoint with my dataset since this worked for me with PyTorch Tacotron2 implementation from Nvidia. But somehow the model's voice doesn't change here while the quality is getting better. That is why i decided to redo the embedding layer fine-tune.
Should I train it from scratch?

[dathudeptrai]

@sujeendran can you do:

pretrained_config.vocab_size = NEW_VOCAB_SIZE
tacotron2 = TFTacotron2(pretrained_config, training=True, name='tacotron2')
tacotron2._build()
tacotron2.summary()
tacotron2.load_weights(path, by_name=True, skip_mismatch=True)

....

[sujeendran]

@dathudeptrai tried it. Still no success. But the continued training on the pretrained model has started slowly changing the voice of the model. So I think I will stick with this method for now.

[dathudeptrai]

@sujeendran i don't know why it happend :)) but i think everything is still ok if you keep the same vocab_size and fine-tuning :d.

[sujeendran]

@dathudeptrai ok will do that. By the way what learning rate would you suggest to continue training by loading weights from pretrained checkpoint h5 file?

[dathudeptrai]

@sujeendran the same :)).

[sujeendran]

@dathudeptrai thanks a lot!

[sujeendran]

@dathudeptrai One last question. From my experience with vocoders, they are mostly independent from the speakers voice as they can take any spectrogram and generate audio irrespective of the voice of training data. Does this apply to the MB-Melgan implementation in this repo too or requires speaker specific re-training for multiple speakers?

[dathudeptrai]

@sujeendran the mel-spectrograms itself contain speaker infomation so you don't need specific any speaker embedding for MB-melgan, it can be learn.