Transformers: How can I generate new text after having fine-tuned BERT on a custom dataset ?

Created on 23 Mar 2019 · 4Comments · Source: huggingface/transformers

Hey,

Once I've fine-tuned the Language Model, how can I get it to generate new text ? Is there any example available ?
Thanks !

Discussion wontfix

Source

MoMe36

👍4

Most helpful comment

Hi,
It's quite difficult to use BERT to generate text as BERT is not a causal language model per se.
Here is an example: https://github.com/nyu-dl/bert-gen by @W4ngatang and @kyunghyuncho.

thomwolf on 27 Mar 2019

👍3

All 4 comments

Also interested in this!

Astraithious on 25 Mar 2019

Hi,
It's quite difficult to use BERT to generate text as BERT is not a causal language model per se.
Here is an example: https://github.com/nyu-dl/bert-gen by @W4ngatang and @kyunghyuncho.

thomwolf on 27 Mar 2019

👍3

Bert was not trained for text generation since it's not trained in the classical lm setting. However there are some new approaches that doesn't rely on next word predictions in the classical lm way. Have a look at: Insertion Transformer and Insertion-based Decoding.

mhattingpete on 8 Apr 2019

👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.