Flair: Error "Unable to load weights from pytorch checkpoint " when using custom BertEmbeddings

[abeermohamed1]

Describe the bug
Below error thrown
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

To Reproduce
use the below
BertEmbeddings(data_folder+'/config.json', data_folder+'/pytorch_model.bin' , data_folder+'/vocab.txt'),

Additional context
UnpicklingError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, model_args, *kwargs)
469 try:
--> 470 state_dict = torch.load(resolved_archive_file, map_location="cpu")
471 except Exception:

4 frames
UnpicklingError: invalid load key, '{'.

During handling of the above exception, another exception occurred:

OSError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, model_args, *kwargs)
471 except Exception:
472 raise OSError(
--> 473 "Unable to load weights from pytorch checkpoint file. "
474 "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "
475 )

OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

[stefan-it]

Hi @abeermohamed1 ,

could you please provide a longer code snippet.

Usually, the instantiation of BertEmbeddings should look like:

from flair.embeddings import BertEmbeddings

embeddings = BertEmbeddings(bert_model_or_path="SpanBERT/spanbert-base-cased")

But you can of course use a local path name, like /mnt/bert-base-custom-model. Then the following files must be located in that folder:

• config.json - BERT model configuration
• pytorch_model.bin - Hugging Face Transformers compatible weights for PyTorch
• vocab.txt - Vocab file

Then you can pass this path to BertEmbeddings:

from flair.embeddings import BertEmbeddings

embeddings = BertEmbeddings(bert_model_or_path="/mnt/bert-base-custom-model")

Please let me know, if that helps :)

[abeermohamed1]

@stefan-it Thank you for your help :) it is working fine now. I will close the issue once learning is completed.

Appreciated @stefan-it

[abeermohamed1]

Hi @stefan-it I am getting the below error

RuntimeError: CUDA out of memory. Tried to allocate 352.00 MiB (GPU 0; 15.90 GiB total capacity; 14.28 GiB already allocated; 259.88 MiB free; 14.94 GiB

[abeermohamed1]

@stefan-it solve me problem thank you