Transformers: cannot import name AddedToken

[nrjvarshney]

🐛 Bug

Information

Model I am using (Bert, XLNet ...): Albert

Language I am using the model on (English, Chinese ...): English

 27 from typing import List, Optional, Sequence, Tuple, Union
 28

---> 29 from tokenizers import AddedToken, Encoding
30 from tokenizers.decoders import Decoder
31 from tokenizers.implementations import BaseTokenizer

ImportError: cannot import name 'AddedToken'

The problem arises when using:

• [ ] the official example scripts: (give details below)
• [ ] my own modified scripts: (give details below)

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name):
• [ ] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1.
2.
3.

Expected behavior

Environment info

• transformers version:
• Platform:
• Python version:
• PyTorch version (GPU?):
• Tensorflow version (GPU?):
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[nrjvarshney]

Created another environment from scratch and it got resolved.

[wangxupeng]

you need install transformers this way:

git clone https://github.com/huggingface/transformers
cd transformers
pip install .

[JosefJoubert]

you need install transformers this way:

git clone https://github.com/huggingface/transformers
cd transformers
pip install .

This worked for me thanks!

[ahazeemi]

you need install transformers this way:

git clone https://github.com/huggingface/transformers
cd transformers
pip install .

I was having a similar issue on colab:

can't pickle AddedToken objects

This solution also worked for me. Thanks!

[vinejain]

you need install transformers this way:

git clone https://github.com/huggingface/transformers
cd transformers
pip install .

Thanks much, this worked for me!