Transformers: Help with converting fine-tuned PT model to TF checkpoint
How do I convert PT model (.bin) to TF checkpoint successfully so that I can start serving using bert-as-a-service?
Below are the steps and errors:
Huggingface v2.2.1, Pytorch 1.2, TF 2.0
- executed run_lm_finetuning.py to fine-tune an already finetuned model (clinicalBERT) on the target domain dataset. Successfully saved all the necessary files (.bin, config, vocab etc.)
- To convert PT to TF, executed convert_pytorch_checkpoint_to_tf2.py with --tf_dump_path="/tf_test/" --model_type="bert" --pytorch_checkpoint_path="../pytorch_model.bin" --config_file='../config.json'
below was the error
Traceback (most recent call last):
File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 248, in
only_convert_finetuned_models=args.only_convert_finetuned_models)
File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 194, in convert_all_pt_checkpoints_to_tf
compare_with_pt_model=compare_with_pt_model)
File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 115, in convert_pt_checkpoint_to_tf
tf_model = load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path)
File "/home/imagen/skc/environments/.virtualenvs/lstm_dev_tf2x/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 82, in load_pytorch_checkpoint_in_tf2_model
return load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys)
File "/home/imagen/skc/environments/.virtualenvs/lstm_dev_tf2x/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 145, in load_pytorch_weights_in_tf2_model
assert name in pt_state_dict, "{} not found in PyTorch model".format(name)
**AssertionError: cls.seq_relationship.weight not found in PyTorch model**
- I wanted to test PT to TF conversion, so I've pointed the script to original clinicalBERT model directory and it successfully converted. However, it was saved as .h5 model and not .ckpt
3.1 Ran below code to convert .h5 to save it as checkpoint - however, it seems not possible to save as checkpoint without creating the model architecture
ran below code for saving as .ckpt in tf2.0
import tensorflow as tf
from keras.models import load_model
saver = tf.train.Checkpoint()
model = load_model("../converted_model-tf_model.h5", compile=False)
sess = tf.compat.v1.keras.backend.get_session()
save_path = saver.save("../converted_model-tf_model.ckpt")
So, in order to successfully use a fine-tuned model in bert-as-a-service
Was there anything I am doing incorrectly when fine-tuning a model? because, somehow the PT to TF conversion goes smoothly for clinicalBERT, but not for fine-tuned version of it (AssertionError: cls.seq_relationship.weight not found in PyTorch model)
How to save as checkpoint (.ckpt) instead of .h5 model? this is for bert-as-a-service? if this is not possible, please suggest alternatives (is creating architecture a necessary step?)
2069 - fwiw - I've used cleaned up version of the script
Thanks
sivakumarch
All 7 comments
Hi @thomwolf - any suggestion would be greatly appreciated.
I am looking forward to hosting one of the fine-tuned model (pytorch) using bert-as-a-service library. However, TF conversion seems to be the way to go, and I'm stuck as the script throws above errors that I am unable to understand.
sivakumarch
on 12 Dec 2019
Hello! Indeed there seems to be a bug with the conversion script. In the meantime, here's how you can load your PyTorch checkpoint in a TF model:
from transformers import BertForMaskedLM, TFBertForMaskedLM
# The script should have already done that
model = BertForMaskedLM.from_pretrained("bert-base-cased")
model.save_pretrained("here")
# Load the PyTorch model in TensorFlow
tf_model = TFBertForMaskedLM.from_pretrained("here", from_pt=True)
# Save the TensorFlow model
tf_model.save_pretrained("tf_test")
You can then convert the generated .h5 model in a ckpt, like is described in this issue or this stackoverflow issue
LysandreJik
on 12 Dec 2019
Thanks for the suggestion @LysandreJik
I just tried this approach.
In my case, I fine tuned a model on MLM using run_lm_finetuning.py
from transformers import BertConfig, BertTokenizer, BertModel, BertForMaskedLM
import os
tokenizer = BertTokenizer.from_pretrained(ft_cbert)
model = BertModel.from_pretrained(ft_cbert)
model.save_pretrained(str(os.path.join(ft_cbert, "pt_bertmodel")))
model = BertForMaskedLM.from_pretrained(str(os.path.join(ft_cbert, "pt_bertmodel")))
model.save_pretrained(str(os.path.join(ft_cbert, "pt_maskedlm_bertmodel")))
model = TFBertModel.from_pretrained(os.path.join(ft_cbert, "pt_maskedlm_bertmodel"), from_pt=True)
model.save_pretrained(os.path.join(ft_cbert, "tf_maskedlm_bertmodel"))
Now, when loading the pytorch model, TF doesn't seem to find weights and initializing all of the layers to 0 (correct me if I am interpreting incorrectly); I see a list of weights not loaded from pytorch model at the end of the log.
I1212 16:31:52.322784 139685136627520 modeling_utils.py:334] loading weights file /home/imagen/skc/bert/data/gold-regions/gold-finetune/cb-finetune-with-eval/pt_bertmodel/pytorch_model.bin
I1212 16:31:55.378468 139685136627520 configuration_utils.py:71] Configuration saved in /home/imagen/skc/bert/data/gold-regions/gold-finetune/cb-finetune-with-eval/pt_maskedlm_bertmodel/config.json
I1212 16:31:57.219412 139685136627520 modeling_utils.py:205] Model weights saved in /home/imagen/skc/bert/data/gold-regions/gold-finetune/cb-finetune-with-eval/pt_maskedlm_bertmodel/pytorch_model.bin
I1212 16:31:57.220998 139685136627520 configuration_utils.py:148] loading configuration file /home/imagen/skc/bert/data/gold-regions/gold-finetune/cb-finetune-with-eval/pt_maskedlm_bertmodel/config.json
I1212 16:31:57.222085 139685136627520 configuration_utils.py:168] Model config {
"attention_probs_dropout_prob": 0.1,
"finetuning_task": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"num_labels": 2,
"output_attentions": false,
"output_hidden_states": false,
"output_past": true,
"pruned_heads": {},
"torchscript": false,
"type_vocab_size": 2,
"use_bfloat16": false,
"vocab_size": 28996
}I1212 16:31:57.222966 139685136627520 modeling_tf_utils.py:255] loading weights file /home/imagen/skc/bert/data/gold-regions/gold-finetune/cb-finetune-with-eval/pt_maskedlm_bertmodel/pytorch_model.bin
I1212 16:31:57.293533 139685136627520 modeling_tf_pytorch_utils.py:78] Loading PyTorch weights from /home/imagen/skc/bert/data/gold-regions/gold-finetune/cb-finetune-with-eval/pt_maskedlm_bertmodel/pytorch_model.bin
I1212 16:31:58.017100 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/embeddings/word_embeddings/weight:0
I1212 16:31:58.018263 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/embeddings/position_embeddings/embeddings:0
I1212 16:31:58.019075 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/embeddings/token_type_embeddings/embeddings:0
I1212 16:31:58.019884 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/embeddings/LayerNorm/gamma:0
I1212 16:31:58.020372 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/embeddings/LayerNorm/beta:0
I1212 16:31:58.020853 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/self/query/kernel:0
I1212 16:31:58.021338 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/self/query/bias:0
I1212 16:31:58.021814 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/self/key/kernel:0
I1212 16:31:58.022383 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/self/key/bias:0
I1212 16:31:58.022871 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/self/value/kernel:0
I1212 16:31:58.023389 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/self/value/bias:0
I1212 16:31:58.023855 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/output/dense/kernel:0
I1212 16:31:58.024335 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/output/dense/bias:0
I1212 16:31:58.024829 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/output/LayerNorm/gamma:0
I1212 16:31:58.025296 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/attention/output/LayerNorm/beta:0
I1212 16:31:58.025762 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/intermediate/dense/kernel:0
I1212 16:31:58.026222 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/intermediate/dense/bias:0
I1212 16:31:58.026710 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/output/dense/kernel:0
I1212 16:31:58.027182 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/output/dense/bias:0
I1212 16:31:58.027667 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/output/LayerNorm/gamma:0
I1212 16:31:58.028124 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._0/output/LayerNorm/beta:0
I1212 16:31:58.028624 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/self/query/kernel:0
I1212 16:31:58.029091 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/self/query/bias:0
I1212 16:31:58.029582 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/self/key/kernel:0
I1212 16:31:58.030059 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/self/key/bias:0
I1212 16:31:58.030566 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/self/value/kernel:0
I1212 16:31:58.031049 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/self/value/bias:0
I1212 16:31:58.031528 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/output/dense/kernel:0
I1212 16:31:58.032037 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/output/dense/bias:0
I1212 16:31:58.032562 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/output/LayerNorm/gamma:0
I1212 16:31:58.033143 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/attention/output/LayerNorm/beta:0
I1212 16:31:58.033643 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/intermediate/dense/kernel:0
I1212 16:31:58.034140 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/intermediate/dense/bias:0
I1212 16:31:58.034643 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/output/dense/kernel:0
I1212 16:31:58.035099 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/output/dense/bias:0
I1212 16:31:58.035623 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/output/LayerNorm/gamma:0
I1212 16:31:58.036166 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._1/output/LayerNorm/beta:0
I1212 16:31:58.036743 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/self/query/kernel:0
I1212 16:31:58.037309 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/self/query/bias:0
I1212 16:31:58.037782 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/self/key/kernel:0
I1212 16:31:58.038266 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/self/key/bias:0
I1212 16:31:58.038728 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/self/value/kernel:0
I1212 16:31:58.039192 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/self/value/bias:0
I1212 16:31:58.039664 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/output/dense/kernel:0
I1212 16:31:58.040130 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/output/dense/bias:0
I1212 16:31:58.040640 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/output/LayerNorm/gamma:0
I1212 16:31:58.041108 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/attention/output/LayerNorm/beta:0
I1212 16:31:58.041579 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/intermediate/dense/kernel:0
I1212 16:31:58.042079 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/intermediate/dense/bias:0
I1212 16:31:58.042617 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/output/dense/kernel:0
I1212 16:31:58.043088 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/output/dense/bias:0
I1212 16:31:58.043587 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/output/LayerNorm/gamma:0
I1212 16:31:58.044040 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._2/output/LayerNorm/beta:0
I1212 16:31:58.044509 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/self/query/kernel:0
I1212 16:31:58.045005 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/self/query/bias:0
I1212 16:31:58.050858 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/self/key/kernel:0
I1212 16:31:58.051367 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/self/key/bias:0
I1212 16:31:58.051822 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/self/value/kernel:0
I1212 16:31:58.052374 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/self/value/bias:0
I1212 16:31:58.052869 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/output/dense/kernel:0
I1212 16:31:58.053370 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/output/dense/bias:0
I1212 16:31:58.053862 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/output/LayerNorm/gamma:0
I1212 16:31:58.054336 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/attention/output/LayerNorm/beta:0
I1212 16:31:58.054825 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/intermediate/dense/kernel:0
I1212 16:31:58.055315 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/intermediate/dense/bias:0
I1212 16:31:58.055775 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/output/dense/kernel:0
I1212 16:31:58.056253 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/output/dense/bias:0
I1212 16:31:58.056724 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/output/LayerNorm/gamma:0
I1212 16:31:58.057177 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._3/output/LayerNorm/beta:0
I1212 16:31:58.057679 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/self/query/kernel:0
I1212 16:31:58.058135 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/self/query/bias:0
I1212 16:31:58.058606 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/self/key/kernel:0
I1212 16:31:58.059053 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/self/key/bias:0
I1212 16:31:58.059546 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/self/value/kernel:0
I1212 16:31:58.060031 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/self/value/bias:0
I1212 16:31:58.060508 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/output/dense/kernel:0
I1212 16:31:58.060971 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/output/dense/bias:0
I1212 16:31:58.061455 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/output/LayerNorm/gamma:0
I1212 16:31:58.061920 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/attention/output/LayerNorm/beta:0
I1212 16:31:58.062463 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/intermediate/dense/kernel:0
I1212 16:31:58.062933 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/intermediate/dense/bias:0
I1212 16:31:58.063439 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/output/dense/kernel:0
I1212 16:31:58.063920 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/output/dense/bias:0
I1212 16:31:58.064412 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/output/LayerNorm/gamma:0
I1212 16:31:58.064872 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._4/output/LayerNorm/beta:0
I1212 16:31:58.066597 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/self/query/kernel:0
I1212 16:31:58.068921 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/self/query/bias:0
I1212 16:31:58.069412 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/self/key/kernel:0
I1212 16:31:58.069909 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/self/key/bias:0
I1212 16:31:58.070411 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/self/value/kernel:0
I1212 16:31:58.070859 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/self/value/bias:0
I1212 16:31:58.071335 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/output/dense/kernel:0
I1212 16:31:58.071808 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/output/dense/bias:0
I1212 16:31:58.072312 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/output/LayerNorm/gamma:0
I1212 16:31:58.072788 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/attention/output/LayerNorm/beta:0
I1212 16:31:58.073315 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/intermediate/dense/kernel:0
I1212 16:31:58.073767 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/intermediate/dense/bias:0
I1212 16:31:58.074249 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/output/dense/kernel:0
I1212 16:31:58.074745 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/output/dense/bias:0
I1212 16:31:58.075211 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/output/LayerNorm/gamma:0
I1212 16:31:58.075714 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._5/output/LayerNorm/beta:0
I1212 16:31:58.076181 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/self/query/kernel:0
I1212 16:31:58.076673 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/self/query/bias:0
I1212 16:31:58.077143 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/self/key/kernel:0
I1212 16:31:58.077627 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/self/key/bias:0
I1212 16:31:58.078094 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/self/value/kernel:0
I1212 16:31:58.078586 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/self/value/bias:0
I1212 16:31:58.079055 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/output/dense/kernel:0
I1212 16:31:58.079540 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/output/dense/bias:0
I1212 16:31:58.080033 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/output/LayerNorm/gamma:0
I1212 16:31:58.080506 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/attention/output/LayerNorm/beta:0
I1212 16:31:58.080977 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/intermediate/dense/kernel:0
I1212 16:31:58.081467 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/intermediate/dense/bias:0
I1212 16:31:58.081947 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/output/dense/kernel:0
I1212 16:31:58.082474 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/output/dense/bias:0
I1212 16:31:58.082974 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/output/LayerNorm/gamma:0
I1212 16:31:58.083476 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._6/output/LayerNorm/beta:0
I1212 16:31:58.083951 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/self/query/kernel:0
I1212 16:31:58.084461 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/self/query/bias:0
I1212 16:31:58.084934 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/self/key/kernel:0
I1212 16:31:58.085417 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/self/key/bias:0
I1212 16:31:58.085875 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/self/value/kernel:0
I1212 16:31:58.086349 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/self/value/bias:0
I1212 16:31:58.086802 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/output/dense/kernel:0
I1212 16:31:58.087476 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/output/dense/bias:0
I1212 16:31:58.087949 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/output/LayerNorm/gamma:0
I1212 16:31:58.088423 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/attention/output/LayerNorm/beta:0
I1212 16:31:58.089007 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/intermediate/dense/kernel:0
I1212 16:31:58.089831 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/intermediate/dense/bias:0
I1212 16:31:58.090376 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/output/dense/kernel:0
I1212 16:31:58.090837 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/output/dense/bias:0
I1212 16:31:58.091311 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/output/LayerNorm/gamma:0
I1212 16:31:58.091777 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._7/output/LayerNorm/beta:0
I1212 16:31:58.092295 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/self/query/kernel:0
I1212 16:31:58.092808 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/self/query/bias:0
I1212 16:31:58.093313 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/self/key/kernel:0
I1212 16:31:58.093771 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/self/key/bias:0
I1212 16:31:58.094259 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/self/value/kernel:0
I1212 16:31:58.099888 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/self/value/bias:0
I1212 16:31:58.100401 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/output/dense/kernel:0
I1212 16:31:58.100865 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/output/dense/bias:0
I1212 16:31:58.101369 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/output/LayerNorm/gamma:0
I1212 16:31:58.101860 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/attention/output/LayerNorm/beta:0
I1212 16:31:58.102412 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/intermediate/dense/kernel:0
I1212 16:31:58.103574 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/intermediate/dense/bias:0
I1212 16:31:58.104034 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/output/dense/kernel:0
I1212 16:31:58.104549 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/output/dense/bias:0
I1212 16:31:58.105008 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/output/LayerNorm/gamma:0
I1212 16:31:58.105483 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._8/output/LayerNorm/beta:0
I1212 16:31:58.105949 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/self/query/kernel:0
I1212 16:31:58.106442 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/self/query/bias:0
I1212 16:31:58.106897 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/self/key/kernel:0
I1212 16:31:58.107369 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/self/key/bias:0
I1212 16:31:58.107837 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/self/value/kernel:0
I1212 16:31:58.108303 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/self/value/bias:0
I1212 16:31:58.108789 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/output/dense/kernel:0
I1212 16:31:58.109263 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/output/dense/bias:0
I1212 16:31:58.109742 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/output/LayerNorm/gamma:0
I1212 16:31:58.110190 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/attention/output/LayerNorm/beta:0
I1212 16:31:58.110669 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/intermediate/dense/kernel:0
I1212 16:31:58.111116 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/intermediate/dense/bias:0
I1212 16:31:58.111589 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/output/dense/kernel:0
I1212 16:31:58.112125 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/output/dense/bias:0
I1212 16:31:58.112630 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/output/LayerNorm/gamma:0
I1212 16:31:58.113107 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._9/output/LayerNorm/beta:0
I1212 16:31:58.113591 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/self/query/kernel:0
I1212 16:31:58.114055 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/self/query/bias:0
I1212 16:31:58.114537 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/self/key/kernel:0
I1212 16:31:58.115001 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/self/key/bias:0
I1212 16:31:58.115493 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/self/value/kernel:0
I1212 16:31:58.115964 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/self/value/bias:0
I1212 16:31:58.116458 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/output/dense/kernel:0
I1212 16:31:58.116904 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/output/dense/bias:0
I1212 16:31:58.117376 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/output/LayerNorm/gamma:0
I1212 16:31:58.117864 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/attention/output/LayerNorm/beta:0
I1212 16:31:58.118321 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/intermediate/dense/kernel:0
I1212 16:31:58.118805 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/intermediate/dense/bias:0
I1212 16:31:58.119260 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/output/dense/kernel:0
I1212 16:31:58.119747 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/output/dense/bias:0
I1212 16:31:58.120195 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/output/LayerNorm/gamma:0
I1212 16:31:58.120673 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._10/output/LayerNorm/beta:0
I1212 16:31:58.121122 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/self/query/kernel:0
I1212 16:31:58.121608 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/self/query/bias:0
I1212 16:31:58.122125 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/self/key/kernel:0
I1212 16:31:58.122639 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/self/key/bias:0
I1212 16:31:58.123139 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/self/value/kernel:0
I1212 16:31:58.127967 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/self/value/bias:0
I1212 16:31:58.128448 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/output/dense/kernel:0
I1212 16:31:58.128974 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/output/dense/bias:0
I1212 16:31:58.129623 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/output/LayerNorm/gamma:0
I1212 16:31:58.130099 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/attention/output/LayerNorm/beta:0
I1212 16:31:58.130589 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/intermediate/dense/kernel:0
I1212 16:31:58.131052 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/intermediate/dense/bias:0
I1212 16:31:58.131555 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/output/dense/kernel:0
I1212 16:31:58.132040 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/output/dense/bias:0
I1212 16:31:58.132566 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/output/LayerNorm/gamma:0
I1212 16:31:58.133050 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/encoder/layer_._11/output/LayerNorm/beta:0
I1212 16:31:58.133538 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/pooler/dense/kernel:0
I1212 16:31:58.133999 139685136627520 modeling_tf_pytorch_utils.py:159] Initialize TF weight tf_bert_model_3/bert/pooler/dense/bias:0
I1212 16:31:58.654147 139685136627520 modeling_tf_pytorch_utils.py:169] Weights or buffers not loaded from PyTorch model: {'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight'}
sivakumarch
on 12 Dec 2019
Hmm it says it's initializing all the weights from the PyTorch model, so they're not initialized to zero.
It's indeed not loading some weights from the PyTorch models, which are not needed for the TF model you're initializing (you're loading a BertForMaskedLM in a TFBertModel, so some weights are not used).
LysandreJik
on 12 Dec 2019
Thanks for the clarification @LysandreJik
This way I am able to save to the model as .h5 version. However, since this step only saves model weights, converting .h5 to .ckpt is not straightforward as it requires the suitable architecture defined (when I am loading it in non-hface libs like tf.keras). It seems the model is not saved using model.save() instead with save_weights(). One needs to define the architecture to load weights and save as .ckpt. It would be great if there is an option to save the model including the necessary architecture to be loaded in TF. Let me know if I am missing something here.
Thank you.
sivakumarch
on 15 Dec 2019
Thanks for the clarification @LysandreJik
This way I am able to save to the model as .h5 version. However, since this step only saves model weights, converting .h5 to .ckpt is not straightforward as it requires the suitable architecture defined (when I am loading it in non-hface libs like tf.keras). It seems the model is not saved using model.save() instead with save_weights(). One needs to define the architecture to load weights and save as .ckpt. It would be great if there is an option to save the model including the necessary architecture to be loaded in TF. Let me know if I am missing something here.
Thank you.
same question
ZhuoranLyu
on 12 Mar 2020
Thanks for the clarification @LysandreJik
This way I am able to save to the model as .h5 version. However, since this step only saves model weights, converting .h5 to .ckpt is not straightforward as it requires the suitable architecture defined (when I am loading it in non-hface libs like tf.keras). It seems the model is not saved using model.save() instead with save_weights(). One needs to define the architecture to load weights and save as .ckpt. It would be great if there is an option to save the model including the necessary architecture to be loaded in TF. Let me know if I am missing something here.
Thank you.
Hi, I have the same question, been stuck with this, have you solved the issue?
Thanks you.
pranavpawar3
on 5 May 2020
Related issues
HansBambel
路
3Comments
zhezhaoa
路
3Comments
lemonhu
路
3Comments
alphanlp
路
3Comments
quocnle
路
3Comments
Most helpful comment
Thanks for the clarification @LysandreJik
This way I am able to save to the model as .h5 version. However, since this step only saves model weights, converting .h5 to .ckpt is not straightforward as it requires the suitable architecture defined (when I am loading it in non-hface libs like tf.keras). It seems the model is not saved using model.save() instead with save_weights(). One needs to define the architecture to load weights and save as .ckpt. It would be great if there is an option to save the model including the necessary architecture to be loaded in TF. Let me know if I am missing something here.
Thank you.