Transformers: Segmentation fault (core dumped) after importing transformers

[huangxiaoshuo]

🐛 Bug

Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

• [ ] the official example scripts: (give details below)
• [x] my own modified scripts: (give details below)

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name)
• [x] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1.import torch
2.import transformers
3.torch.rand(4,5)

import torch
import transformers
torch.rand(4,5)

Expected behavior

I got an error 'segmentation fault (core dumped)' while trying to generate a tensor after importing transformers, but if I removed 'import transformers', the tensor could be generated.

Environment info

• transformers version: 2.11.0
• Platform: Ubuntu 18.04.4 LTS
• Python version: 3.7.6
• PyTorch version (GPU?): 1.4.0+cu100
• Tensorflow version (GPU?): 2.2.0
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

[LysandreJik]

Hi, I believe this error may be due to the sentencepiece version 0.1.92 which causes a segmentation fault..

[huangxiaoshuo]

Hi, I believe this error may be due to the sentencepiece version 0.1.92 which causes a segmentation fault..

Thank you very much, this error disappeared when sentencpiece was downgraded to 0.1.91.
BTW, is there some debugging method to find this root cause when we encounter segmentation fault?

[LysandreJik]

Not that I know of :man_shrugging:

[wang-h]

It works. GREAT!!!!

[MaveriQ]

The error still persists after 6 months. Maybe the version 0.1.91 should be hardcoded in package requirements

[LysandreJik]

It was for the last 3.x versions. For version 4.x onwards, sentencepiece is not a requirement anymore.