Gevent: RuntimeWarning: Monkey-patching not on the main thread; threading.main_thread().join() will hang from a greenlet

[maicss]

First, my English is awful. I'm sorry for that.
Second, below is problem what happened.

I'm learning gevent, so I try to run some examples from docs, but when I debugger some code, pycharm gives some error, I don't how to fixed it, although I search google and stackoverflow.

from gevent import sleep
from gevent.pool import Pool
from gevent.lock import BoundedSemaphore

sem = BoundedSemaphore(2)


def worker1(n):
    sem.acquire()
    print('Worker %i acquired semaphore' % n)
    sleep(0)
    sem.release()
    print('Worker %i released semaphore' % n)


def worker2(n):
    with sem:
        print('Worker %i acquired semaphore' % n)
        sleep(0)
    print('Worker %i released semaphore' % n)


pool = Pool()
pool.map(worker1, range(0, 2))
pool.map(worker2, range(3, 6))

And the Error is:

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/gevent/monkey.py:395: RuntimeWarning: Monkey-patching not on the main thread; threading.main_thread().join() will hang from a greenlet
  _warnings)
Traceback (most recent call last):
  File "gevent/_semaphore.py", line 101, in gevent._semaphore.Semaphore._notify_links (gevent/gevent._semaphore.c:1940)
greenlet.error: cannot switch to a different thread
(<built-in method switch of greenlet.greenlet object at 0x103d880e0>,
 <gevent.thread.LockType object at 0x103f020b0>) failed with error

Someone tells me this problem caused by the IDE, not gevent. But nothing else, so I also raise an issue to the pycharm.

Even this is not gevent's fault, I also feel sorry for my awful English....

Thanks.

[jamadden]

Your English is very understandable! You have nothing to be sorry about.

As I understand it, this _is_ a problem with the IDE. At least, it is a problem with _combining_ that IDE's debugger with gevent. I believe that IDE uses sockets and threads to run the debugger in a target process and communicate with it from the IDE. The use of native threads and switching back and forth between them is a problem.

This has been seen before with other things that use threads, such as with ipython. In some cases there are workarounds that can be used for a specific tool to make it work. I don't know of a workaround for pycharm's debugger, but you can try looking for one (maybe you can make it not use threads?).

I know that plain pdb works just fine with gevent because it doesn't do anything fancy with threads; it is what I use to debug when I'm working with gevent. So you can put import pdb; pdb.set_trace() at the top of your file and run your program on the command line to debug that way.

[maicss]

Thank you very much!
Thanks a lot...
As you say, when I use Pycharm and iPython, scripts give some error, when I run scripts on original REPL, they are run prefect.

And I report this BUG to jetbrains forum.

And I will try to use pdb in my srcipts' debug.

Thanks again!!!

[advance512]

I am facing this same issue, with PyCharm Community Edition 2016.3 and Python 3.5.2.
gevent==1.1.2
greenlet==0.4.11

Interestingly, a very similar repository, that shares much of the code and the exact same initialization flow for monkey patching, works perfectly.

This happens even if the first two lines of the main module are:

from gevent import monkey
monkey.patch_all()

This causes:

../.venv/lib/python3.5/site-packages/gevent/monkey.py:399: RuntimeWarning: Monkey-patching not on the main thread; threading.main_thread().join() will hang from a greenlet
  _warnings)

and:

Traceback (most recent call last):
  File "gevent/_semaphore.py", line 101, in gevent._semaphore.Semaphore._notify_links (gevent/gevent._semaphore.c:2030)
greenlet.error: cannot switch to a different thread
(<built-in method switch of greenlet.greenlet object at 0x7f7ea5552e88>,
 <gevent.thread.LockType object at 0x7f7ea55735f8>) failed with error

If I try to print the main_thread and current_thread in the context of patch_thread(), I see:

<_DummyThread(DummyThread-6, started daemon 140181916427680)>
<_MainThread(MainThread, started 140181979653888)>

Not sure what makes it fail in this project, and succeed in another.

[advance512]

Whoa! Found the issue. That was surprisingly quick.

In PyCharm, go to Settings --> Build, Execution, Deployment -> Python Debugger.
In that dialog, you'll see a "Gevent compatible" checkbox. Not sure how it got unticked in the new project.

Tick that option and enjoy!

[631068264]

Thanks for the setting and maybe it only happen in py3

[VanLiuZhi]

Thanks to answer

[ztxjack]

Thanks a lot , I search the solution for a long time , and yours is the right one for me!

[ronyf]

neither monkey patching nor setting gevent compatibility helped me...

I still get:

  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\mongo_client.py", line 1156, in _reset_on_error
    return func(*args, **kwargs)
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\server.py", line 106, in send_message_with_response
    reply = sock_info.receive_message(request_id)
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\pool.py", line 612, in receive_message
    self._raise_connection_failure(error)
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\pool.py", line 745, in _raise_connection_failure
    raise error
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\pool.py", line 610, in receive_message
    self.max_message_size)
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\network.py", line 173, in receive_message
    _receive_data_on_socket(sock, 16))
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\pymongo\network.py", line 232, in _receive_data_on_socket
    chunk_length = sock.recv_into(mv[bytes_read:])
  File "C:\Users\guardi\Documents\GuardicoreDev\guardicore\python36\lib\site-packages\gevent\_socket3.py", line 433, in recv_into
    self._wait(self._read_event)
  File "src\gevent\_hub_primitives.py", line 265, in gevent.__hub_primitives.wait_on_socket
  File "src\gevent\_hub_primitives.py", line 266, in gevent.__hub_primitives.wait_on_socket
  File "src\gevent\_hub_primitives.py", line 252, in gevent.__hub_primitives._primitive_wait
  File "src\gevent\_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src\gevent\_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src\gevent\_hub_primitives.py", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src\gevent\_waiter.py", line 151, in gevent.__waiter.Waiter.get
  File "src\gevent\_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
greenlet.error: cannot switch to a different thread

any ideas?

[jaddison]

@ronyf I see the same, using django, gevent, docker, trying to debug in PyCharm (yes, I have the gevent checkbox ticked):

Connected to pydev debugger (build 183.5912.18)
django_1         | Traceback (most recent call last):
django_1         |   File "/opt/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 382, in _on_run
django_1         |     r = self.sock.recv(1024)
django_1         |   File "/usr/local/lib/python3.6/site-packages/gevent/_socket3.py", line 382, in recv
django_1         |     self._wait(self._read_event)
django_1         |   File "src/gevent/_hub_primitives.py", line 265, in gevent.__hub_primitives.wait_on_socket
django_1         |   File "src/gevent/_hub_primitives.py", line 266, in gevent.__hub_primitives.wait_on_socket
django_1         |   File "src/gevent/_hub_primitives.py", line 252, in gevent.__hub_primitives._primitive_wait
django_1         |   File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
django_1         |   File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
django_1         |   File "src/gevent/_hub_primitives.py", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
django_1         |   File "src/gevent/_waiter.py", line 151, in gevent.__waiter.Waiter.get
django_1         |   File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
django_1         |   File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
django_1         |   File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
django_1         |   File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
django_1         | greenlet.error: cannot switch to a different thread

I actually monkeypatch in Django's manage.py:

#!/usr/bin/env python
try:
    from gevent import monkey
    monkey.patch_all()
except ImportError:
    pass

try:
    import psycogreen.gevent
    psycogreen.gevent.patch_psycopg()
except ImportError:
    pass

import os  # noqa: E402
import sys  # noqa: E402

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "chatterblock.settings")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)

Which works fine without debugging, and it was working fine in Python3.5 in Vagrant/Virtualbox - but not in docker (python:3.6-stretch docker image).