Transformers: sentencepiece 0.1.94 causing segmentation fault

[mejran]

Environment info

• transformers version: 3.4.0 and 3.3.1
• Platform: Linux/Sagemaker
• Python version: 3.7
• PyTorch version (GPU?): 1.4
• Tensorflow version (GPU?):
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

Information

To reproduce

Steps to reproduce the behavior:

1. pip install transformers[torch]
2. from transformers.trainer import TrainingArguments, Trainer
import torch
3. torch.tensor([1,2,3])

transformers 3.3.1 seg faults at step 3, transformers 3.4 seg faults at step 2.

Expected behavior

No segmentation fault

[mejran]

This can be worked around, for anyone hitting this issue, by setting sentencepiece==0.1.91 explicitly.

[LysandreJik]

Maybe we could set sentencepiece==0.1.91 in the setup.py to prevent this from happening, as we already had the issue with the 0.1.92.

Do you want to open a PR for that?

[thomwolf]

This should also be fixed by #8073