Transformers: Is there pre-train bert or xlnet from scratch code ?

Created on 29 Apr 2020  ·  3Comments  ·  Source: huggingface/transformers

❓ Questions & Help

Details


A link to original question on Stack Overflow:

LM (Pretraining)
Source
picture xealml

Most helpful comment

@xealml Curious whether you managed to train XLNet? If so, any pointers you could share?

picture jbmaxwell on 24 Sep 2020
👍2

All 3 comments

Yes, you can use the run_language_modeling.py script for pre-training e.g. BERT from scratch:

https://huggingface.co/transformers/examples.html#language-model-training

(Just leave the model_name_or_path parameter empty for pre-training from scratch)

stefan-it picture stefan-it on 29 Apr 2020
👍1

@stefan-it thank u.

xealml picture xealml on 29 Apr 2020

@xealml Curious whether you managed to train XLNet? If so, any pointers you could share?

jbmaxwell picture jbmaxwell on 24 Sep 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

delip picture delip  ·  3Comments
0x01h picture 0x01h  ·  3Comments
rsanjaykamath picture rsanjaykamath  ·  3Comments
hsajjad picture hsajjad  ·  3Comments
zhezhaoa picture zhezhaoa  ·  3Comments