Tensorflowtts: Error when loading pretrained model ljspeech to training another language

[linhld0811]

When i tried to load pretrained model(model_65000.h5) like the tutorials in the examples/tacotron2/README.md to train on the other language, i got the error like this:

tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable encoder/embeddings/LayerNorm/beta_164 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/encoder/embeddings/LayerNorm/beta_164/N10tensorflow3VarE does not exist.

And when i used another pretrained(model_40000.h5), i got another error:

(0) Failed precondition: Error while reading resource variable encoder/embeddings/LayerNorm/gamma_157 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/encoder/embeddings/LayerNorm/gamma_157/N10tensorflow3VarE does not exist.
[[node encoder/embeddings/LayerNorm/batchnorm/mul/ReadVariableOp (defined at /data/linhld6/Project/TensorflowTTS-master/tensorflow_tts/models/tacotron2.py:163) ]]
[[decoder/while/LoopCond/_71/_86]]
(1) Failed precondition: Error while reading resource variable encoder/embeddings/LayerNorm/gamma_157 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/encoder/embeddings/LayerNorm/gamma_157/N10tensorflow3VarE does not exist.

Can anyone help me to fix this error?! Thanks!

[dathudeptrai]

@linhld0811 cay you try

pretrained_config.vocab_size = NEW_VOCAB_SIZE
tacotron2 = TFTacotron2(pretrained_config, training=True, name='tacotron2')
tacotron2._build()
tacotron2.summary()
tacotron2.load_weights(path, by_name=True, skip_mismatch=True)

....

btw, what is ur tensorflow version ?

[linhld0811]

Here my code:

tacotron2 = TFTacotron2(config=Tacotron2Config(**config["tacotron2_params"]), training=True, name='tacotron2')
    tacotron2._build()
    #tacotron2.summary()
    if args.pretrained_model != None:
        print("use pretrained_model ljspeech", args.pretrained_model)
        tacotron2.load_weights(args.pretrained_model)
        pretrained_config = Tacotron2Config(**config["tacotron2_params"])
        pretrained_config.set_params(len(symbols))
        print("vocab_size of new model:", pretrained_config.vocab_size)
        new_embedding_layers = TFTacotronEmbeddings(pretrained_config, name='embeddings')
        tacotron2.encoder.embeddings = new_embedding_layers
        tacotron2._build()
        tacotron2.summary()

I use TF version 2.2.0

[dathudeptrai]

@linhld0811 no i mean this code, i just test and it worked.

    pretrained_config = Tacotron2Config(**config["tacotron2_params"])
    pretrained_config.vocab_size = NEW_VOCAB_SIZE
    tacotron2 = TFTacotron2(config=pretrained_config, training=True, name='tacotron2')
    tacotron2._build()
    tacotron2.summary()
    tacotron2.load_weights("./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-120000.h5", by_name=True, skip_mismatch=True)

[linhld0811]

Thanks, i fixed this error. I loaded model but did not use the param skip_mismatch=True of the function tacotron2.load_weights

[dathudeptrai]

@sujeendran Fixed.

[linhld0811]

I have a question, can you help me to figured out:

• Can i use your pretrained tacotron2 model to extract duration of my own dataset(not english) to teach fastspeech model?! Or i have to training tacotron2 model on my dataset and use best_model to extract duration?!

[dathudeptrai]

You cannot do that :))

[sujeendran]

@dathudeptrai I tried the same. Didn't work. I'm also running Tensorflow 2.2.0. To be clear, here is the code after your fix:

    pretrained_config = Tacotron2Config(**config["tacotron2_params"])
    tacotron2 = TFTacotron2(pretrained_config, training=True, name='tacotron2')
    tacotron2._build()
    tacotron2.summary()
    tacotron2.load_weights("./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-1400.h5", by_name=True, skip_mismatch=True)
    pretrained_config.vocab_size = len(symbols)
    new_embedding_layers = TFTacotronEmbeddings(pretrained_config, name='embeddings')
    tacotron2.encoder.embeddings = new_embedding_layers
    # re-build model
    tacotron2._build()
    tacotron2.summary()

[dathudeptrai]

@sujeendran no, the code should be like this, do not need re build and redefine embedding layer

    pretrained_config = Tacotron2Config(**config["tacotron2_params"])
    pretrained_config.vocab_size = NEW_VOCAB_SIZE
    tacotron2 = TFTacotron2(config=pretrained_config, training=True, name='tacotron2')
    tacotron2._build()
    tacotron2.summary()
    tacotron2.load_weights("./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-120000.h5", by_name=True, skip_mismatch=True)

[sujeendran]

@dathudeptrai Oh, in that case my fix was doing the same. Thanks! :) Maybe you can update the documentation for fine-tuning now that this is clear so others don't fall into this issue again.

[dathudeptrai]

Yeah i will update it asap :))) maybe add some flags :3