Sentence-transformers: 'BertTokenizer' object has no attribute 'prepare_for_model'

[RuskinManku]

sentence_features = self.sentence_encoder.get_sentence_features(text, longest_seq)
  File "/usr/local/lib/python3.6/dist-packages/sentence_transformers/SentenceTransformer.py", line 179, in get_sentence_features
    return self._first_module().get_sentence_features(*features)
  File "/usr/local/lib/python3.6/dist-packages/sentence_transformers/models/BERT.py", line 64, in get_sentence_features
    return self.tokenizer.prepare_for_model(tokens, max_length=pad_seq_length, pad_to_max_length=True, return_tensors='pt')
AttributeError: 'BertTokenizer' object has no attribute 'prepare_for_model'

I was using a specific piece of code for many weeks and it was working fine. Just today I am experiencing this error.

[Aniket-Pradhan]

It's because of the new version of transformers. For a quick solution, you could use the previous version of the transformers library. For the actual solution, we must wait for the authors to update the tokenizer API here.

[RuskinManku]

@Aniket-Pradhan yes that's what I guessed, mainly raised the issue for the authors. For people facing similar issue, they can pip install transformers==2.11.0 after doing pip install sentence-transformers to downgrade to a version that works for now.

[nreimers]

Thanks for pointing this out. Will fix it soon

[BrianThomasMcGrath]

I also noticed this, put transformers==2.11.0 in my requirements and it worked

[nreimers]

The method 'prepare_for_model' was added again to Transformers.

A new release based on version 3 will follow soon.