Gunicorn: Hitting a RecursionError with gevent

Created on 23 Nov 2018  路  13Comments  路  Source: benoitc/gunicorn

I'm hitting the following RecursionError when a part of an application (pywb delivering static files) tries to respond (the more complex functionalities work. :-)). i think that exception should be handled / propagated since i can't deduce where something went wrong in the app from this output.

Traceback (most recent call last):
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
    resp.write_file(respiter)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 392, in write_file
    if not self.sendfile(respiter):
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 382, in sendfile
    sent += os.sendfile(sockno, fileno, offset + sent, count)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  [Previous line repeated 923 more times]
RecursionError: maximum recursion depth exceeded
(

Most helpful comment

Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the hasattr dance, I would just drop 3.4 and assume socket.sendfile exists 馃槃

But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture os.sendfile before patching and use that capture in the patch.

All 13 comments

What versions of gunicorn and gevent are you using?

oh, sorry for missing that information. thanks for reminding me. it's gunicorn 19.9.09 and gevent 1.3.7.

@funkyfuture looking at the traceback, I think you are using the master version of Gunicorn :)

The following line

return os.sendfile(fdout, fdin, offset, nbytes)

should read

return o_sendfile(fdout, fdin, offset, nbytes)

in Gunicorn 19.9. We've changed it to call os.sendfile() directly in https://github.com/benoitc/gunicorn/commit/e974f30517261b2bc95cfb2017a8688f367c8bf3#diff-24baff6269faf10cb820888842174a8bR37.

@jamadden do we still need to monkeypatch os.sendfile() manually? It looks like we could rely on Gevent: http://www.gevent.org/api/gevent._socket3.html#gevent._socket3.socket.sendfile

you're right, i'm relying on #1897. i just referred to what pip told me.

do we still need to monkeypatch os.sendfile() manually? It looks like we could rely on Gevent:

gevent does provide socket.sendfile on all versions of Python 3. The version gevent provides is cooperative with the gevent loop, which os.sendfile is not. The purpose of your patch at this point is to execute os.sendfile against a non-blocking socket, using a loop, making it basically cooperative. That may or may not be more efficient than what gevent is doing; gevent's version may improve in the future as we incorporate more libuv support, or compile it with cython.

Unless there are fears or reports of substantial performance degradations, I would suggest that you can drop your monkey-patch and just use socket.sendfile.

@benoitc @javabrett bump here. If we switched to Python 3.5 as a minimum, we could use socket.sendfile.

Check #1559?

I am also seeing this after updating from Gunicorn 19.9.0 to 20.0.0 (with gevent 1.4.0, Django 2.2.6, Python 3.7.5, Ubuntu 19.10):

[2019-11-11 09:16:23 +0000] [11635] [ERROR] Error handling request
Traceback (most recent call last):
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
    resp.write_file(respiter)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 397, in write_file
    if not self.sendfile(respiter):
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 387, in sendfile
    sent += os.sendfile(sockno, fileno, offset + sent, count)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  [Previous line repeated 936 more times]
RecursionError: maximum recursion depth exceeded

Is there any known solution?

Now that v20 has been released, this bug is causing more problems (e.g., #2164).

gevent provides socket.sendfile on all versions of Python 3. One option would be to change gunicorn.http.wsgi.Resoponse:sendfile to use socket.sendfile instead of os.sendfile if hasattr(sock, 'sendfile'). Another option would be to override Response.sendfile in the gevent worker. Of course the obvious option would be to go back to what gunicorn 19 did and capture the non-patched os.sendfile as a variable and call it instead of calling os.sendfile from the patch that replaces os.sendfile.

@jamadden what would you suggest?

Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the hasattr dance, I would just drop 3.4 and assume socket.sendfile exists 馃槃

But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture os.sendfile before patching and use that capture in the patch.

That sounds good. Let's capture the old sendfile. Let's also bump the Python version for the next release.

Can we make a 20.x release branch, then, and start working on 21?

Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the hasattr dance, I would just drop 3.4 and assume socket.sendfile exists 馃槃

But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture os.sendfile before patching and use that capture in the patch.

https://docs.gunicorn.org/en/stable/settings.html#sendfile

Was this page helpful?
0 / 5 - 0 ratings