Transformers: Add Fast Transformers - Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

[bratao]

🌟 New model addition

Model description

The Fast Transformers repo introduces a fast transformer model based on work to improve attention published in two papers:

• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (https://arxiv.org/abs/2006.16236)
• Fast Transformers with Clustered Attention (https://arxiv.org/abs/2007.04825)

Open source status

• [X] the model implementation is available: (give details)
  https://github.com/idiap/fast-transformers

• [x] the model weights are available: (give details)

• [X] who are the authors: (mention them, if possible by @gh-username)
  @angeloskath

[angeloskath]

Hi guys, let us know how we can help and also kindly add @apoorv2904 to the author list.

Although the model weights are nothing particularly useful we do provide them for our colab so let us know if they are needed and how to provide them.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bratao]

Don´t let it die. In my tests this is the best performing model so far!

[bratao]

@patrickvonplaten @sgugger

I could try to include on huggingface/transformers if there is an interest from the core team. But I would have to depend on https://github.com/idiap/fast-transformers as they created optimized cuda/cpu c++ versions of the proposed attention. A MR with this dependency would be accepted by Huggingface?

[JaeDukSeo]

would love if this comes in!

[patrickvonplaten]

Hey @bratao,

Yes, we would definitely be interested in this model and would also be fine with an optional dependency of https://github.com/idiap/fast-transformers Also pinging @joeddav @TevenLeScao here (in case you guys are interested in helping with the integration).

I would also be happy to help you with the model integration otherwise @bratao :-)

[bratao]

Great, I´m on it @patrickvonplaten

I will work on this on my free time, As soon as I have something, I put it here the fork.

If anyone else want to help or speed it up, just talk to me using the email in my profile!