Simpletransformers: Error: "The current process just got forked. Disabling parallelism to avoid deadlocks"

[tutmoses]

Describe the bug
Running the basic example T5 model produces this initial warning, before going on to complete the task successfully:

The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)

I'm not sure where to set the TOKENIZERS_PARALLELISM argument.

To Reproduce
Run the basic example T5 model.

Desktop (please complete the following information):

• Mac OS Catalina 10.15.5

[ThilinaRajapakse]

T5 seems to run into issues with multiprocessing. You can disable it by specifying use_multiprocessing=False in the model args.

[craigpfeifer]

UPDATE: Added OS / package versions, logging attempt

This also happens with text classification. I tried setting use_multiprocessing=False in the model args and it didn't change anything.

OS : Catalina 10.15.5 & Centos 7.5

package versions:

simpletransformers==0.40.2
tokenizers==0.8.0
torch==1.5.1
sentencepiece==0.1.91

I also tried changing the logging threshold as specified here:
https://github.com/huggingface/transformers/issues/991

To no avail

[ThilinaRajapakse]

I am not sure where it is coming from then. Are you on Mac OS as well, @craigpfeifer ?

Can you guys post a full stack trace to figure out where this is happening?

[maurostaiti]

this also happens for me with text classification on Ubuntu 18.04

INFO:simpletransformers.classification.classification_model: Converting to features started. Cache is not used. The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)

[tutmoses]

T5 seems to run into issues with multiprocessing. You can disable it by specifying use_multiprocessing=False in the model args.

This fix worked for me.

[craigpfeifer]

I tried adding use_multiprocessing=False to the model params:

model = ClassificationModel('bert', 'bert-base-cased',
                            use_cuda=False,
                            weight=class_balance,
                            use_multiprocessing=False,
                            args={"overwrite_output_dir":True, max_seq_length":128,"evaluate_during_training":True})

and I got the runtime error:

Traceback (most recent call last):
  File "SimpleTransformerSentenceClassifier.py", line 140, in <module>
    model = ClassificationModel('bert', 'bert-base-cased/',
  File "/home/cpfeifer/adept-sentence/miniconda3/envs/adept-sentence/lib/python3.8/site-packages/simpletransformers/classification/classification_model.py", line 173, in __init__
    self.model = model_class.from_pretrained(
  File "/home/cpfeifer/adept-sentence/miniconda3/envs/adept-sentence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 655, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'use_multiprocessing'

When i add it to the args map, the code runs with the warnings.

[ThilinaRajapakse]

Yeah, it should be added to the args map.

This may be an issue introduced in 0.8.0 of tokenizers. Can you guys try using tokenizers=0.7.0?

[craigpfeifer]

Verified the warning does not appear with tokenizers=0.7.0. You are right it is a change in 0.8.0.

[drfinkus]

Same behavior, tokenizers=0.8.0rc4 on Ubuntu 20.04. Setting use_multiprocessing=False in the args map has no effect.

Edit: also, setting TOKENIZERS_PARALLELISM=False in the args map has no effect either.

[ThilinaRajapakse]

Please use tokenizers=0.7 for now.

[AlecS12]

tokenizers=0.8.0rc4 work fine. You need to set the environment variable to the string "false"

either by TOKENIZERS_PARALLELISM=false in your shell

or by: import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"

in the python script

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.