Transformers: Fine-tuning model and Generation

[antmarakis]

Hello!

I am beginner and I just wanted to run some experiments, but I've hit a road block. I am trying to generate text using run_generator.py after I fine-tune a model on my data using simple_lm_finetuning.py. I've looked around a bit, and I'm not sure how to go about this, or if this is possible at all. I don't see an option for run_generator to use BERT models, and I'm not sure how to bridge the two scripts.

Basically what I want to do is to fine-tune a model on my data and then generate text. Can this be done with run_generator and simple_lm_finetuning?

Thank you!

---

EDIT:

Forgot to add my code:

python pytorch-transformers/examples/lm_finetuning/simple_lm_finetuning.py \
    --train_corpus data.txt \
    --bert_model bert-base-uncased \
    --do_lower_case \
    --output_dir finetuned_lm/ \
    --do_train

python pytorch-transformers/examples/run_generation.py \
    --model_type=transfo-xl \
    --length=20 \
    --model_name_or_path='finetuned_lm'

[sakalouski]

Having the same question - how to use bert for generation?

[Bruno-bai]

the same problem， how to train my own data for text generation?

[thomwolf]

We'll add an example for fine-tuning this month.

[joistick11]

@thomwolf as I read in other issues, BERT model cannot be used to generate text directly (your reply https://github.com/huggingface/pytorch-transformers/issues/401#issuecomment-477111518).
What exact examples are you planning to add? Thanks.

[mvedang]

@Bruno-bai did you figure out how to train own data?

[sakalouski]

Not really. Would appreciate a tutorial:)

On Mon, Aug 19, 2019 at 5:22 AM Vedang Mandhana [e-mail]
wrote:

@Bruno-bai https://github.com/Bruno-bai did you figure out how to train
own data?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/huggingface/pytorch-transformers/issues/874?email_source=notifications&email_token=AHZ2KL5Q2RIKUTGARMX6EY3QFINYTA5CNFSM4IGHELH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4RUR4I#issuecomment-522406129,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHZ2KL3FS6CDZQN75KNXVPDQFINYTANCNFSM4IGHELHQ
.

[mukatee]

this would be very useful example to have. to finetune gpt2, xlnet, ... and run generation from the finetuned model. Don't know whether bert supports generation or not, but the ones that do..

[nikhilno1]

I am too struggling with similar problem. I want to train a non-english (hindi) language model on my custom dataset and use it for text generation. From what I understood, BERT sucks at text generation as it uses MLM for training. The ones that do well (gpt,trans-xl,xlnet) don't have a pretrained multilingual model available.

@Bruno-bai @sakalouski are you looking for training own data for language generation? Coz I have done it for classification and can help with that.

[amin-nejad]

Hi @thomwolf

We'll add an example for fine-tuning this month.

Has this example been added yet?

Thanks

[LysandreJik]

Hi @amin-nejad, the example has been added and is available here.

[amin-nejad]

Thanks @LysandreJik. Will this also work with Transformer-XL if we just modify the source code to include the Transformer-XL Config, LMHeadModel and Tokenizer as a model class? Or will it require more substantial changes?

[amin-nejad]

Using run_lm_finetuning.py seemingly works for Transformer-XL if we additionally import the Transformer-XL Config, LMHeadModel and Tokenizer and modify the MODEL_CLASSES to include them. We also need to provide the block_size as a command line parameter. Training curves look reasonable and decoding also happens without errors using run_generation.py but the model output is pretty much always just a bunch of equals signs e.g. = = = = = = = = = etc. for me at least anyway. Clearly more substantial changes are required to run_lm_finetuning.py to make it work. If anyone knows what/why, please let me know

[thomwolf]

One thing we should do (maybe when we have some bandwidth for that with @LysandreJik) is to push back a PR to PyTorch repo to add an option to have biases on all clusters of PyTorch's Adaptive Softmax so we can rely on the official Adaptive Softmax implementation instead of having our own.

That would make the job of maintaining and upgrading Transformer-XL a lot easier as it's currently the most cumbersome code base to maintain.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[plandes]

I'd like to fine tune my own Bert embeddings for my (in English) domain. Is there any update on this? Thanks in advance.