Transformers: [docs] Provide a barebones GPT-2 colab notebook

[aolko]

Please provide a barebones "pick up and go" GPT-2 colab notebook for text generation, just like gpt-2-simple does

[julien-c]

Do you mean just for inference? Or fine-tuning too, like in the one you linked?

[aolko]

Yes, fine-tuning as well.

[minimaxir]

The notebook described in #2676 is a good example of something that could work; however the current implementation is not very user friendly, which was the design goal of the gpt-2-simple notebook. (my text generating package which extends transformers will have it as a feature)

[aolko]

The notebook described in #2676 is a good example of something that could work; however the current implementation is not very user friendly, which was the design goal of the gpt-2-simple notebook. (my text generating package which extends transformers will have it as a feature)

@minimaxir your provided notebook has external dependencies (examples/run_lm_finetuning.py), which is a no-no for this case, all the source has to be laid out in the notebook's code blocks, just like in gpt-2-simple's.

[minimaxir]

Agreed. The issue is that there is no functional training interface in the library itself, which is why I'm creating one that extends it (as it's a narrow use case).

[aolko]

@minimaxir so perhaps you can make a notebook that fully satisfies this issue in this case?

[aolko]

so, guys, can you give me an approx ETA for this issue? Kinda need that fix now

[BramVanroy]

so, guys, can you give me an approx ETA for this issue? Kinda need that fix now

I don't think there are currently specific plans to create a GPT-2 notebook. If you have a look at all the pull requests (https://github.com/huggingface/transformers/pulls) you can see that the team is hard at work on a range of different features and fixes. One of those is ready-to-go docker images with notebooks (https://github.com/huggingface/transformers/pull/3051) but as far as I can see GPT-2 doesn't have a special place there.

You can always try to create this yourself or ask specific questions on Stack Overflow.

That being said, you can have a look at https://github.com/huggingface/transformers/pull/3063 which is currently implementing generation for GPT-2 and others in Tensorflow.

[aolko]

which is currently implementing generation for GPT-2 and others in Tensorflow.

that actually sucks, since i'm targeting pytorch

[minimaxir]

If you really need to generate text in PyTorch on short notice, you can finetune the GPT-2 model using gpt-2-simple, and run the TF -> PyTorch conversion scripts in transformers, then you can load that and generate it from it.

[LysandreJik]

The #3063 that Bram mentioned targets TensorFlow because it's already implemented in PyTorch.

[aolko]

If you _really_ need to generate text in PyTorch on short notice, you can finetune the GPT-2 model using gpt-2-simple, and run the TF -> PyTorch conversion scripts in transformers, then you can load that and generate it from it.

except finetuning should be done later (if at all), as for right now it's either distilgpt2 or gpt-2-large, pretrained.

[aolko]

So essentially there's nothing so far. Even after 7 14 21 28 35 42+ days, issue is still hot 🔥

[Kreijstal]

I'm interested on this

[aolko]

upd: despite not providing any feedback in this issue they've sneakily added at least something

[BramVanroy]

upd: despite not providing any feedback in this issue they've sneakily added at least something

Please be aware that this is a large open-source repository that is maintained by a company that has many other concerns, too. However, being open-source, collaboration is encouraged. Because of the huge interest in NLP and specifically this library, it is incredibly hard to monitor all new issues while also fixing bugs and taking care of all other responsibilities, i.e. a day-to-day job.

Bumping this topic by complaining does not help anyone, but the team is very open to receiving and reviewing PRs, so feel free to add your contributions to make the library better. Alternatively, you can encourage others to help you out by sharing this issue on other platforms. I have marked the issue as a "Good first issue', encouraging others to give it a go.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.