Transformers: Problem about convert TF model and pretraining

[zhezhaoa]

First of all, Thank you for this great job. I use the official tensorflow implementation to pretrain on my corpus and then save the model. I want to convert this model to pytorch format and use it, but I got the error:
Traceback (most recent call last):
File "convert_tf_checkpoint_to_pytorch.py", line 105, in
convert()
File "convert_tf_checkpoint_to_pytorch.py", line 86, in convert
pointer = getattr(pointer, l[0])
AttributeError: 'Parameter' object has no attribute 'adam_m'

Could you give me some advice? Thank you very much.

It is great if you can release the pretrain code. I think it is useful even we cannot use TPU. Because we can fine-tune above google's pertained model.

[thomwolf]

Hi @zhezhaoa, I see, I will fix this in the next release.

For now you should be able to fix that by installing the repo from source (git clone the repo and pip install -e . and changing line 53 of convert_tf_checkpoint_to_pytorch.py from
if name[-1] in ["adam_v", "adam_m"]:
to
if any(n in ["adam_v", "adam_m"] for n in name):

[zhezhaoa]

Thank you very much! It could be great if you can provide pertaining code like the official TF implementation.

[thomwolf]

Ok this loading issue is now fixed in master and the new 0.4.0 release.