Transformers: How to contribute to “Write with transformer”?

[mauceri]

🚀 I would like to contribute to a French version of this App

I’m French, I write short stories, and I’m also a software engineer

Motivation

I’ll retire in 6 months and I wanted to build such an app before I stumbled on your demo.

Additional context

https://www.linkedin.com/in/mauceri/

[BramVanroy]

What is it that you can contribute? The only (yet impressive) thing that is going on is language modeling. Can you contribute a pre-trained French model for one of the frameworks? That's (as far as I know) the only way to contribute.

[mauceri]

Thanks Bram, I’m going to investigate what the cost could be for XLNet on clevergrid https://www.clevergrid.io/?pk_campaign=ga-gpu-1&pk_source=adwords&pk_medium=sem&pk_content=gpuasaservicefr&gclid=CjwKCAjwibzsBRAMEiwA1pHZrvm8ozRMrbcDR7YoYiKqsq6gEnPo9AecJwjKzBxa8L-4_hB6ny4uARoCwfMQAvD_BwE

Envoyé de mon iPad

Le 28 sept. 2019 à 09:44, Bram Vanroy [e-mail] a écrit :

What is it that you can contribute? The only (yet impressive) thing that is going on is language modeling. Can you contribute a pre-trained French model for one of the frameworks? That's (as far as I know) the only way to contribute.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[jqueguiner]

Hi all,
We (ovh) are open to calculate it for free.

[BramVanroy]

Not sure if a new French language model is still necessary after Camembert has been introduced.

[julien-c]

That's awesome news, @jqueguiner! Let us know if we can help.

@BramVanroy To work well with Write With Transformer, we would want more like a FR-pretrained GPT-2-like model. CamemBERT wouldn't do on generation out of the box.

See also the more specific issue: https://github.com/huggingface/transformers/issues/1356

[mauceri]

For generation CamemBERT is of no use I think...

Envoyé de mon iPad

Le 22 nov. 2019 à 14:02, Bram Vanroy [e-mail] a écrit :


Not sure if a new French language model is still necessary after Camembert has been introduced.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[mauceri]

Yes, CamemBERT is awesome, but for WWT we need a FR-Pretrained GPT-2 model!

Envoyé de mon iPad

Le 22 nov. 2019 à 14:55, Julien Chaumond [e-mail] a écrit :


That's awesome news, @jqueguiner! Let us know if we can help.

@BramVanroy To work well with Write With Transformer, we would want more like a FR-pretrained GPT-2-like model. CamemBERT wouldn't do on generation out of the box.

See also the more specific issue: #1356

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[jqueguiner]

Camembert doesnt offer generation, only syntax analysis and masking due to the nature of the network. Multiple mask generation () gives really uggly results as you cna test here : https://market-place.ai.ovh.net/#!/apis/43323c37-59e7-4092-b23c-3759e7c09288/pages/94d31892-4e64-446f-9318-924e64346f9e

IMO we should start training using OSCAR dataset
https://traces1.inria.fr/oscar/

@julien-c yes we can start with a collab GPT2 french training ipynb together then I'll prepare the env for a DGX1 or something similar. I didn't train a GPT2 before. IS it scaling over multiple GPU's ? do we need horovod adaptation ?

[BramVanroy]

Oops, sorry everyone. I thought this was a general French model question. My bad.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.