Transformers: Weights not initialized from pretrained model

[lemonhu]

Thanks for your awesome work!

When I execute the following code for a named entity recognition tasks:
model = BertForTokenClassification.from_pretrained("bert-base-uncased", num_labels=num_labels)

Output the following information:

Weights of BertForTokenClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
Weights from pretrained model not used in BertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

What puzzles me is that the parameters of the classifier are not initialized.

[rodgzilla]

Hi!

Those messages are correct, the pretrained weights that have been released by Google Brain are just the ones of the core network. They did not release task specific weights. To get a model that solves a specific classification task, you would have to train one yourself or get it from someone else.

@thomwolf There have been multiple issues about this specific behavior, maybe we should add some kind of text either as a print while loading the model or in the documentation. I would be happy to do it. What would you prefer?

[lemonhu]

Oh, I see, I will train the model with my own dataset, thank you for your answer.

[thomwolf]

Yes you are right @rodgzilla we should detail a bit the messages in modeling.py to say that These weights will be trained from scratch.