Transformers: ALBERT tokenizer is not callable

[guoxuxu]

🐛 Bug

Information

when running the follwoing given example:
from transformers import AlbertTokenizer, AlbertModel
import torch
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertModel.from_pretrained('albert-base-v2')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

it alerts "* TypeError: 'AlbertTokenizer' object is not callable"

examples link: https://huggingface.co/transformers/model_doc/albert.html#alberttokenizer

Model I am using (Bert, XLNet ...):
ALBERT

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:

• [X ] the official example scripts: (give details below)
  from transformers import AlbertTokenizer, AlbertModel
  import torch
  tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
  model = AlbertModel.from_pretrained('albert-base-v2')
  inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
  link: https://huggingface.co/transformers/model_doc/albert.html#alberttokenizer'

* TypeError: 'AlbertTokenizer' object is not callable

Expected behavior

Should given a workable example of using ALBERT ?

Environment info

• transformers version: 2.3.0
• Platform: Linux
• Python version: 3.7.4
• PyTorch version (GPU?): 1.4.0
• Tensorflow version (GPU?):
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[sgugger]

Hi there, you should upgrade your transformers library to v3. The version you have does not have callable tokenizers.

Alternatively, the docs for v2.3.0 are here. You can use the dropdown menu on the left (just under the hugging face) to change the version of the documentation you are using.

[guoxuxu]

Thanks very much

[guoxuxu]

Alternatively, I found using tokenizer.ecode is ok.
But, How to view the old doc details for this version?
https://huggingface.co/transformers/v2.3.0/model_doc/albert.html#
This link seems to be archived documentation. No examples provided here.

[sgugger]

This is the documentation of the version 2.3.0. For better documentation, you should really consider upgrading your library :-)

[guoxuxu]

Understand. Thanks very much.