Gunicorn: Gevent workers are taking long time to response

[saurabhsri2015]

I have installed gunicorn & using async worker but it is taking too much time to resolve the request & it is showing critical worker timeout after that page is showing. Can please tell me about issue

[benoitc]

is there any way to reproduce your issue? what is your setup and paramd. any detail that could help us to reproduce your issue would help. Thanks :)

[saurabhsri2015]

Gunicorn using gevent workers is responding to basic python application but we have a full running application( using MySQL DB & other resources ) when we try to use gevent with that it stops responding to requests and ends up with worker time out ( No more logs in debug mode ) .

The application works fine with sync & tornado workers .It's a basic WSGI application using Django framework .

We even tried to request a plain text file within that application which doesn't have any dependency ( DB or variable expansions ) , even that ends up with similar worker timeout errors . I am trying to suspect what could be the issue for this .

Is there any further configuration which we are missing while using gevent ?

[jamadden]

Are you using a gevent-compatible MySQL driver? e.g., PyMySQL. If you're using MySQL-Python (MySQLdb) or mysqlclient, those are not compatible with gevent and a query in one greenlet will stop all other greenlets from progressing.

[saurabhsri2015]

We are using MySQL-Python for our application. Is there anyway to use async worker with MySQL-Python ?? We are not bounded with gevent, we can use any async worker which is recommended with MySQL-Python.

[jamadden]

I suggest using PyMySQL with gevent workers. PyMySQL is a drop-in replacement for MySQL-Python that's gevent-compatible (just install it, and do PyMySQL. install_as_MySQLdb() at startup).

Plus, PyMySQL is actually maintained. I can't recommend MySQL-Python to anyone at this point because it's not maintained.

[saurabhsri2015]

I have added 2 line in manage.py but still getting the same issue.
import pymysql
pymysql.install_as_MySQLdb()

I have also tried with eventlet workers & when i am executing wget localhost:8000 so it is throwing error

Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

Is it configuration issue with application or gunicorn ??

[benoitc]

Don't edit the arbiter. You can simply use an hook for that. Also you may want to use one of these drivers:

https://github.com/mthurlin/gevent-MySQL
https://github.com/esnme/ultramysql

[jamadden]

I don't know gevent-MySQL, but I can't recommend ultramysql. It's not a drop in replacement and takes a fair amount of code to make work reliably. YMMV of course. I did make it work with relstorage, and you can copy the code, but it's not pretty.

[benoitc]

@saurabhsri2015 bump. Did you solve your issue?

[saurabhsri2015]

@benoitc I am still facing the issue with async worker but now I am using sync worker. Can you suggest which worker class is good Sync or Async for production

[benoitc]

@saurabhsri2015 both are used in production since a long time. Your issue is due to a misuse of gevent right now. You should make sure that the driver you're using is using the event loop of gevent so it doesn't block. The drivers linked above do it. Or use PyMsQL as suggested by @jamadden :)

[saurabhsri2015]

@benoitc Thanks

[WqyJh]

I just got into the same problem. When I use curl for a specific url, it's blocked until timeout, with correct HTTP response after timeout and the error log showed [CRITICAL] WORKER TIMEOUT.

bind = "127.0.0.1:8000"
workers = 1
accesslog = 'log/access.log'
errorlog = 'log/error.log'
loglevel = 'debug'
proc_name = 'app'
daemon = False
reload = True
worker_class = 'gevent'
worker_connections = 10000
preload_app = False

This is my gunicorn.conf.py, in which I'd tried to change values of workers, preload_app.

asn1crypto==0.24.0
cached-property==1.5.1
cffi==1.11.5
cryptography==2.3.1
Django==2.0.4
django-cors-headers==2.2.0
django-filter==2.0.0
django-rest-framework==0.1.0
djangorestframework==3.8.2
djangorestframework-jsonapi==2.6.0
drf-writable-nested==0.5.1
enum-compat==0.0.2
gunicorn==19.9.0
idna==2.7
inflection==0.3.1
itsdangerous==0.24
lxml==4.2.5
Markdown==2.6.11
mock==2.0.0
numpy==1.15.3
pandas==0.23.4
pbr==4.0.2
pycparser==2.19
PyMySQL==0.9.2
python-dateutil==2.7.4
python-docx==0.8.7
pytz==2018.4
six==1.11.0
xlrd==1.1.0

This is my requirements.txt. Note that I use PyMySQL which works very well with gevent. To prove that, I wrote an simple django project with PyMySQL, and run it with gunicorn + gevent, which worked.

Finally I recalled that I created a process pool with multiprocessing in my project, which seemed to be the biggest difference with my sample project. After I commented that code, everything was ok.

As a conclusion, multiprocessing is not compatible with gevent. You might face with the similar problem if you use multiprocessing when you try to run gunicorn with gevent workers.

Hope to help you!

1407

[jamadden]

As a conclusion, multiprocessing is not compatible with gevent. You might face with the similar problem if you use multiprocessing when you try to run gunicorn with gevent workers.

Only limited uses of multiprocessing (specifically, direct use of multiprocessing.Process) is supported in a gevent monkey-patched environment. Other uses will hang:

Caution Use of multiprocessing.Queue when thread has been monkey-patched will lead to a hang due to Queue’s internal use of a blocking pipe and threads. For the same reason, concurrent.futures.ProcessPoolExecutor, which internally uses a Queue, will hang.