Transformers: Implement PyTorch and/or TensorFlow sequence classification architectures for causal language models

[LysandreJik]

🚀 Feature request

The architecture GPT2ForSequenceClassification was added in #7501 in PyTorch. It would be great to have it in TensorFlow (cf. issues #7622), but it would also be great to have it for other causal models: ~OpenAI GPT~, ~CTRL~ (PR opened @elk-cloner), ~TransfoXL~ (PR opened @spatil6)

Below is a list of items to follow to make sure the integration of such an architecture is complete:

• Implement XXXForSequenceClassification in modeling_xxx.py or TFXXXForSequenceClassification in `modeling_tf_xxx.py
• Test that architecture in tests/test_modeling_xxx.py or tests/test_modeling_tf_xxx.py
• Add that architecture to __init__.py and docs/source/model_doc/xxx.rst.

Taking a look at the code changes in #7501 would be a good start.

A very good first issue to get acquainted with the library and its architectures!

[pasDamola]

Hi @LysandreJik is this issue still open? I'll like to pick it up

[LysandreJik]

I believe @fmcurti is working on the OpenAI GPT implementation, but both CTRL and TransfoXL are still open! Would love a PR!

[pasDamola]

Hi Lysandre, thanks for assigning this issue to me. I've been trying to
setup transformers on my local machine (Windows 10). I've been having
several issues with the setup.
Is there any guide I could follow?

Thanks much

On Fri, 9 Oct 2020 at 12:38, Lysandre Debut [e-mail]
wrote:

Assigned #7623 https://github.com/huggingface/transformers/issues/7623
to @pasDamola https://github.com/pasDamola.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/huggingface/transformers/issues/7623#event-3859742526,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGGRMAH2H4Q5JYISSW23W3LSJ3Y4HANCNFSM4SGSU6FQ
.

[LysandreJik]

Sure, have you taken a look at the CONTRIBUTING.md document? What issues have you been having?

[pasDamola]

Yes I have.
When I run pip install -e ".[dev]", I always encounter this error. I'm also running it in anaconda environment

`

[LysandreJik]

I believe the repo cannot be installed from conda as of now, can you use a pip virtual environment?

[pasDamola]

Alright, I'll try that now

[pasDamola]

Still having the same error I had while in conda. I'm trying to install tensorflow locally and retry this again

[pasDamola]

Hi @LysandreJik , I'm still having the same errors running on a pip virtual environment

[LysandreJik]

Do you manage to install TensorFlow in your pip environment?

[pasDamola]

Hi @LysandreJik not yet. I get a similar error. I'm trying to look for solutions on the internet

[amritrau]

Hi @LysandreJik – has anyone picked up the CTRL or TransfoXL architectures yet? I'd love to take a crack at one of them if available. Thank you!

[LysandreJik]

No, feel free to take a crack at it! Let me know and I'll put you in the issue description.

[elk-cloner]

is there anybody working on these ? @LysandreJik

[LysandreJik]

I believe CTRL and TransfoXL are still available. Feel free to open a PR!

[spatil6]

Hi @LysandreJik ,

As this Feature request is closed

Do we need TF implementation of causal models GPT-1, Transfoxl and CTRL?
I'm ready to contributed for that as well.

[LysandreJik]

That would be very welcome @spatil6!

[spatil6]

Ok thanks @LysandreJik.

I'm waiting for this PR #8714 to get merge.
Once done, I'll raise PR for these models as well.