Transformers: "This tokenizer does not make use of special tokens." warning

[weiguowilliam]

❓ Questions & Help

I just updated to the latest version transformers. Now when I use tokenizer to encode word, it always show the warning "This tokenizer does not make use of special tokens."

Is there any way to hide that warning? Thank you.

[iedmrc]

Same here. Is there any way to suppress this warning? I use run_lm_finetuning.py to finetune distilgpt2 and it outputs thousands "This tokenizer does not make use of special tokens.". It's so annoying :(

[rvanasa]

Here's how to suppress the warnings until this is fixed:

import logging
logging.getLogger('transformers.tokenization_utils').setLevel(logging.ERROR)

[weiguowilliam]

Here's how to suppress the warnings until this is fixed:

import logging
logging.getLogger('transformers.tokenization_utils').disabled = True

Thank you!

[iedmrc]

Is this fixed? If not, I think it should be open until it's been fixed.

[thomwolf]

This has been fixed on the master and in the latest release (2.2.1)

[yeliu918]

Hi, I use the latest release but I still have this problem.

[weiguowilliam]

@iedmrc
I close it because the 'log' method works. I don't know whether it's a bug or not.

[LysandreJik]

Hi @yeliu918, could you please show us what you obtain when running this script in your environment?

from transformers import GPT2Tokenizer, __version__
print(__version__)

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(tokenizer.encode("What does this output?"))

[TJKlein]

I am getting warning despite trying everything mentioned above....

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer, __version__
import logging
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(__version__) 
logging.getLogger('transformers.tokenization_utils').disabled = True
tokens_tensor = torch.tensor([tokenizer.encode("some example sentence")])
greedy_output = model.generate(tokens_tensor, max_length=60, num_beams=16)

Version 2.8.0

Setting pad_token_id to 50256 (first eos_token_id) to generate sequence