Flair: Use Custom Fine-tuned Bert Model

[adich23]

Is there a way to use Fine-tuned Bert model in the BertEmbeddings class?
As per the code, input to class is "bert_model: str = 'bert-base-uncased',".
I have fine-tuned the language model on my domain dataset and want to extract out Word/Sentence embeddings from it now.

[alanakbik]

Hi @adich23 that sounds really interesting. The current code does not support it, but I think that would be easy enough to add as a feature. In the constructor, instead of passing a string like 'bert-base-uncased' you could then pass a path to a model file. I think only the BertEmbeddings() class init method would need to be updated.

Would you be interested in adding this as a feature and doing a pull request for this?

[adich23]

@alanakbik I would like to do a PR, I think during the model initialization (in BertModel.from_pretrained) I have to pass 'state_dict' argument which will point to the path.
Similarly 'state_dict' can be added to BertEmbeddings constructor.
Your views?

[alanakbik]

Right now, I think you need to get the serialized model which is currently downloaded from AWS if not there. See the variable resolved_archive_file in the from_pretrained() method. This one I believe needs to point to the model that you give the embedding.

[stefan-it]

That's an interesting question. Once we found a good way to support own trained or fine-tuned models we could also support BioBERT:

https://arxiv.org/abs/1901.08746

Adding support for biomedical NER datasets would also be a great enhancement :)

[adich23]

Closing as PR merged.