Flair: Consider adding fine tuning BERT for text classification and sequence tagging?

[gccome]

Currently flair supports feature-based transfer learning from BERT. Any plan on adding fine-tuning based module in near future? That would be a great addition!

[mauryaland]

I allow myself to highlight the freshly available paper Parameter-Efficient Transfer Learning for NLP. It seems relevant to this topic for further disccusion.

[gccome]

@mauryaland interesting paper! Thanks for sharing!

[tc-wolf]

You could override BertEmbeddings

class TrainableBERT(BertEmbeddings):
    pass

and set self.static_embeddings = False in the constructor, and override the _add_embeddings_internal method to do the same thing as the current method but without model.eval() and the with torch.no_grad() block.

AFAICT this should allow fine-tuning on the full input embeddings. You'd need to be able to set this to eval mode (and not calculating gradients) when done training and doing predictions, however.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alanakbik]

Fine-tuning has been added to master branch (#1492) and will be part of next Flair release.