I'm hitting the following RecursionError
when a part of an application (pywb delivering static files) tries to respond (the more complex functionalities work. :-)). i think that exception should be handled / propagated since i can't deduce where something went wrong in the app from this output.
Traceback (most recent call last):
File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
resp.write_file(respiter)
File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 392, in write_file
if not self.sendfile(respiter):
File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 382, in sendfile
sent += os.sendfile(sockno, fileno, offset + sent, count)
File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
return os.sendfile(fdout, fdin, offset, nbytes)
File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
return os.sendfile(fdout, fdin, offset, nbytes)
File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
return os.sendfile(fdout, fdin, offset, nbytes)
[Previous line repeated 923 more times]
RecursionError: maximum recursion depth exceeded
What versions of gunicorn and gevent are you using?
oh, sorry for missing that information. thanks for reminding me. it's gunicorn 19.9.09
and gevent 1.3.7
.
@funkyfuture looking at the traceback, I think you are using the master version of Gunicorn :)
The following line
return os.sendfile(fdout, fdin, offset, nbytes)
should read
return o_sendfile(fdout, fdin, offset, nbytes)
in Gunicorn 19.9. We've changed it to call os.sendfile()
directly in https://github.com/benoitc/gunicorn/commit/e974f30517261b2bc95cfb2017a8688f367c8bf3#diff-24baff6269faf10cb820888842174a8bR37.
@jamadden do we still need to monkeypatch os.sendfile()
manually? It looks like we could rely on Gevent: http://www.gevent.org/api/gevent._socket3.html#gevent._socket3.socket.sendfile
you're right, i'm relying on #1897. i just referred to what pip
told me.
do we still need to monkeypatch os.sendfile() manually? It looks like we could rely on Gevent:
gevent does provide socket.sendfile
on all versions of Python 3. The version gevent provides is cooperative with the gevent loop, which os.sendfile
is not. The purpose of your patch at this point is to execute os.sendfile
against a non-blocking socket, using a loop, making it basically cooperative. That may or may not be more efficient than what gevent is doing; gevent's version may improve in the future as we incorporate more libuv support, or compile it with cython.
Unless there are fears or reports of substantial performance degradations, I would suggest that you can drop your monkey-patch and just use socket.sendfile
.
@benoitc @javabrett bump here. If we switched to Python 3.5 as a minimum, we could use socket.sendfile
.
Check #1559?
I am also seeing this after updating from Gunicorn 19.9.0 to 20.0.0 (with gevent 1.4.0, Django 2.2.6, Python 3.7.5, Ubuntu 19.10):
[2019-11-11 09:16:23 +0000] [11635] [ERROR] Error handling request
Traceback (most recent call last):
File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
resp.write_file(respiter)
File "/data-hub-api/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 397, in write_file
if not self.sendfile(respiter):
File "/data-hub-api/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 387, in sendfile
sent += os.sendfile(sockno, fileno, offset + sent, count)
File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
return os.sendfile(fdout, fdin, offset, nbytes)
File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
return os.sendfile(fdout, fdin, offset, nbytes)
File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
return os.sendfile(fdout, fdin, offset, nbytes)
[Previous line repeated 936 more times]
RecursionError: maximum recursion depth exceeded
Is there any known solution?
Now that v20 has been released, this bug is causing more problems (e.g., #2164).
gevent provides socket.sendfile
on all versions of Python 3. One option would be to change gunicorn.http.wsgi.Resoponse:sendfile
to use socket.sendfile
instead of os.sendfile
if hasattr(sock, 'sendfile')
. Another option would be to override Response.sendfile
in the gevent worker. Of course the obvious option would be to go back to what gunicorn 19 did and capture the non-patched os.sendfile
as a variable and call it instead of calling os.sendfile
from the patch that replaces os.sendfile
.
@jamadden what would you suggest?
Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the hasattr
dance, I would just drop 3.4 and assume socket.sendfile
exists 馃槃
But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture os.sendfile
before patching and use that capture in the patch.
That sounds good. Let's capture the old sendfile. Let's also bump the Python version for the next release.
Can we make a 20.x release branch, then, and start working on 21?
Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the
hasattr
dance, I would just drop 3.4 and assumesocket.sendfile
exists 馃槃But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture
os.sendfile
before patching and use that capture in the patch.
Most helpful comment
Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the
hasattr
dance, I would just drop 3.4 and assumesocket.sendfile
exists 馃槃But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture
os.sendfile
before patching and use that capture in the patch.