Gevent: Question: Correct order for monkey patching and imports

[Michael-F-Ellis]

• gevent version: 1.1.0
• Python version: 3.5.3
• Operating System: OS X 10.11.6

Description:

Crashes if monkey.patch_socketIO is run before importing urlllib.request as is done in the tutorial I'm using. I'm looking for clarification on what the intended order is supposed to be and why.

Thanks!

What I've run:

# -*- coding: utf-8 -*-
"""
gevent.monkey.patch_socket bug.
Modified example from http://sdiehl.github.io/gevent-tutorial/#data-structures
Crashes if urllib.request is imported AFTER calling monkey.patch_socket.
"""
import gevent
import gevent.monkey
## import urllib.request #GOOD
import json
gevent.monkey.patch_socket()
import urllib.request #BAD

def example_3a():
    print("Example 3a")
    baseapi = 'https://script.google.com/macros/s/AKfycbyd5AcbAnWi2Yn0xhFRbyzS4qMq1VucMVgVvhul5XqS9HkAyJY/exec'
    url = baseapi + '?tz=Europe/Madrid'

    def fetch(pid):
        response = urllib.request.urlopen(url)
        result = response.read().decode('utf-8')
        json_result = json.loads(result)
        fulldate =  json_result['fulldate']
        print("Process {}: {}".format(pid, fulldate))
        return fulldate

    def synchronous():
        for i in range(1, 5):
            fetch(i)

    def asynchronous():
        threads = []
        for i in range(1, 5):
            threads.append(gevent.spawn(fetch, i))
        gevent.joinall(threads)

    print('Synchronous')
    synchronous()

    print('Asynchronous')
    asynchronous()

    print('-'*40)

if __name__ == '__main__':
    example_3a()          

Traceback

Example 3a
Synchronous
Traceback (most recent call last):
  File "patch_socket_bug.py", line 45, in <module>
    example_3a()
  File "patch_socket_bug.py", line 37, in example_3a
    synchronous()
  File "patch_socket_bug.py", line 28, in synchronous
    fetch(i)
  File "patch_socket_bug.py", line 19, in fetch
    response = urllib.request.urlopen(url)
  File "/Users/mellis/anaconda3/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/mellis/anaconda3/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/Users/mellis/anaconda3/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/Users/mellis/anaconda3/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Users/mellis/anaconda3/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Users/mellis/anaconda3/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Users/mellis/anaconda3/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/Users/mellis/anaconda3/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/Users/mellis/anaconda3/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/Users/mellis/anaconda3/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/Users/mellis/anaconda3/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/Users/mellis/anaconda3/lib/python3.5/http/client.py", line 1261, in connect
    server_hostname=server_hostname)
  File "/Users/mellis/anaconda3/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/Users/mellis/anaconda3/lib/python3.5/ssl.py", line 753, in __init__
    server_hostname)
TypeError: _wrap_socket() argument 1 must be _socket.socket, not SSLSocket

[jamadden]

The intended order is that monkey-patching should be the first thing that happens, before any other imports.

Most of the time you will want to use gevent.monkey.patch_all() and not a single module-patching function; there can be subtle dependencies between them, which is what I believe is happening here. That is, if I turn the call to patch_socket into a call to patch_all, the above script works as given.

[Michael-F-Ellis]

Thanks for the quick response. Using patch_all cures it for me, too, and appears to make the asynchronous case faster. So that's great.

I have to confess that patch_all makes me rather nervous when I think about using it to introduce gevent into existing code for my clients. The code I'm talking about is a set of long-running industrial control processes that make heavy use of Python multiprocessing and zmq. It works quite well with ~5 processes running on an Arm7 but my client's plans going forward could require maybe 4x that number.

Cooperative multitasking with gevent seems like a good fit for the problem. It's a concept I'm quite comfortable with having written a (very much simpler) system in C in the early 90's. I think I would feel better about patch_all() if I knew precisely what it patches and what gotchas to look out for.

Thanks again for the help and for creating and maintaining gevent.

Mike

[jamadden]

and appears to make the asynchronous case faster

It makes it actually asynchronous. Having the patch (any patch) after the import of urllib.request was too late to do any good, so the greenlets wound up running sequentially as the sockets made blocking calls.

I think I would feel better about patch_all() if I knew precisely what it patches

It's is pretty well documented, and of course the source is available---when there is wiggle rooms in the docs, it's likely because it does different things on different versions or we want to keep that possibility open.

what gotchas to look out for.

There is definitely at least one gotcha with multiprocessing.Queue called out in the documentation for patch_thread. Also see the documentation about child processes in gevent.os, among other places.

[Michael-F-Ellis]

Those are useful links. Thanks.

So after reading (skimming, to be truthful) I thought "What the heck. Drop the big one, see what happens."

I cloned an existing project with 13000 lines of code in a dozen modules. It spawns 6 child processes that frequently call time.sleep(), uses zmq, pymodbus3, interfaces to Google Sheets, runs a web service, interleaves logging from all processes to a single file and monitors process status to respawn children that crash. In short, a non-trivial app.

I put the incantation at the top of the main module and launched it.

from gevent import monkey
monkey.patch_all()

To my surprise, it not only didn't fly up its own nose, but appears to run indistinguishably from the unmodified version right down to reporting and restarting a crashed process after an exception within an exception I've not yet fixed.

Now obviously I've not introduced any greenlet spawning into any of the processes, but the global monkey patch in and of itself appears to being doing no harm. _Primum non nocere_, as the MD's say.

I'm very impressed!