Gunicorn: Gunicorn spawns infinite workers

[kiranruth]

[2020-09-24 11:42:09 +0000] [8] [DEBUG] 8 workers
[2020-09-24 11:42:09 +0000] [1987] [INFO] Booting worker with pid: 1987
[2020-09-24 11:42:09 +0000] [1988] [INFO] Booting worker with pid: 1988
[2020-09-24 11:42:09 +0000] [1989] [INFO] Booting worker with pid: 1989
[2020-09-24 11:42:09 +0000] [8] [DEBUG] 5 workers
[2020-09-24 11:42:09 +0000] [1990] [INFO] Booting worker with pid: 1990
[2020-09-24 11:42:09 +0000] [1991] [INFO] Booting worker with pid: 1991
[2020-09-24 11:42:09 +0000] [1992] [INFO] Booting worker with pid: 1992
[2020-09-24 11:42:09 +0000] [8] [DEBUG] 6 workers
[2020-09-24 11:42:09 +0000] [1993] [INFO] Booting worker with pid: 1993
[2020-09-24 11:42:09 +0000] [1994] [INFO] Booting worker with pid: 1994
[2020-09-24 11:42:09 +0000] [8] [DEBUG] 8 workers
[2020-09-24 11:42:10 +0000] [1995] [INFO] Booting worker with pid: 1995
[2020-09-24 11:42:10 +0000] [1996] [INFO] Booting worker with pid: 1996
[2020-09-24 11:42:10 +0000] [8] [DEBUG] 7 workers
[2020-09-24 11:42:10 +0000] [1997] [INFO] Booting worker with pid: 1997
[2020-09-24 11:42:10 +0000] [8] [DEBUG] 8 workers
[2020-09-24 11:42:10 +0000] [1998] [INFO] Booting worker with pid: 1998
[2020-09-24 11:42:10 +0000] [1999] [INFO] Booting worker with pid: 1999
[2020-09-24 11:42:10 +0000] [2000] [INFO] Booting worker with pid: 2000
[2020-09-24 11:42:10 +0000] [2001] [INFO] Booting worker with pid: 2001
[2020-09-24 11:42:10 +0000] [8] [DEBUG] 7 workers
[2020-09-24 11:42:10 +0000] [2002] [INFO] Booting worker with pid: 2002
[2020-09-24 11:42:10 +0000] [8] [DEBUG] 8 workers
[2020-09-24 11:42:10 +0000] [2003] [INFO] Booting worker with pid: 2003
[2020-09-24 11:42:11 +0000] [8] [DEBUG] 7 workers
[2020-09-24 11:42:11 +0000] [2004] [INFO] Booting worker with pid: 2004
[2020-09-24 11:42:11 +0000] [8] [DEBUG] 8 workers
[2020-09-24 11:42:11 +0000] [2005] [INFO] Booting worker with pid: 2005
[2020-09-24 11:42:11 +0000] [2006] [INFO] Booting worker with pid: 2006
[2020-09-24 11:42:11 +0000] [2007] [INFO] Booting worker with pid: 2007
[2020-09-24 11:42:11 +0000] [2008] [INFO] Booting worker with pid: 2008
[2020-09-24 11:42:11 +0000] [2009] [INFO] Booting worker with pid: 2009
[2020-09-24 11:42:11 +0000] [2010] [INFO] Booting worker with pid: 2010
[2020-09-24 11:42:11 +0000] [2011] [INFO] Booting worker with pid: 2011
[2020-09-24 11:42:11 +0000] [8] [DEBUG] 7 workers
[2020-09-24 11:42:11 +0000] [2013] [INFO] Booting worker with pid: 2013
[2020-09-24 11:42:11 +0000] [2014] [INFO] Booting worker with pid: 2014
[2020-09-24 11:42:11 +0000] [8] [DEBUG] 8 workers

I've tried to increase ram but no luck. How do I go about solving this issue:
I'm using gunicorn==20.0.0

[benoitc]

how can it be reproduced?

[four43]

Are your workers crashing and just getting restarted? Just because they're booting doesn't mean they're still running when Gunicorn tries to make more.

[nilushancosta]

I have been facing the same problem reported in this issue since around the day this issue was created. The same application used to work without a problem before that.

I am running a Flask application using gunicorn (Gunicorn==20.0.4) on Kubernetes using a minikube cluster. The command that is used to start the gunicorn service is gunicorn --bind 0.0.0.0:5005 --worker-class=gevent --worker-connections=10 --workers=1 <path>. This is now causing gunicorn to keep starting workers as described by the issue author above. I noticed the following traps being recorded on the kernel message buffer when this was happening.

[ 2085.254598] audit: type=1701 audit(1601381252.646:1841): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=16952 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2085.324916] traps: gunicorn[16953] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2085.324982] audit: type=1701 audit(1601381252.716:1842): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=16953 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2085.397425] traps: gunicorn[16954] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2085.397523] audit: type=1701 audit(1601381252.788:1843): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=16954 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2090.126193] do_general_protection: 131 callbacks suppressed
[ 2090.126195] traps: gunicorn[17028] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2090.126274] audit: type=1701 audit(1601381257.517:1909): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=17028 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2090.203483] traps: gunicorn[17029] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2090.203620] audit: type=1701 audit(1601381257.594:1910): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=17029 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2090.272145] traps: gunicorn[17030] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2090.272265] audit: type=1701 audit(1601381257.663:1911): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=17030 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2090.364631] traps: gunicorn[17031] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2090.364710] audit: type=1701 audit(1601381257.756:1912): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=17031 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2090.433255] traps: gunicorn[17032] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2090.433367] audit: type=1701 audit(1601381257.824:1913): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=17032 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1
[ 2095.176972] do_general_protection: 145 callbacks suppressed
[ 2095.176973] traps: gunicorn[17132] general protection ip:7f2796d98170 sp:7ffebd15df90 error:0 in libpython3.7m.so.1.0[7f2796c3b000+1c6000]
[ 2095.177003] audit: type=1701 audit(1601381262.567:1986): auid=4294967295 uid=500 gid=500 ses=4294967295 subj=kernel pid=17132 comm="gunicorn" exe="/usr/local/bin/python3.7" sig=11 res=1

These messages keep repeating as long as the container is running.

But when I replace the above gunicorn command with PYTHONPATH=<Path to project> python <Path to file> and run the application directly with Python (also on Kubernetes), it works without an issue.

Unfortunately I am unable to share the project that I am using, as it is private. I will check if there is another way to reproduce this issue.

In the meantime, is there something else that I can provide to narrow down the problem?

[jamadden]

@nilushancosta and others seeing worker crashes using the gevent worker: Please check that you are using compatible versions of gevent and greenlet. The recent release of greenlet 0.4.17 is incompatible with every version of gevent before 20.9 and installing greenlet 0.4.17 with an older gevent is likely to lead to crashing workers. See https://github.com/gevent/gevent/issues/1674

[nilushancosta]

@nilushancosta and others seeing worker crashes using the gevent worker: Please check that you are using compatible versions of gevent and greenlet. The recent release of greenlet 0.4.17 is incompatible with every version of gevent before 20.9 and installing greenlet 0.4.17 with an older gevent is likely to lead to crashing workers. See gevent/gevent#1674

Thank you very much @jamadden for the information. Using greenlet==0.4.16 solved the problem!

[four43]

Is there a way to hint at that from requirements_test.txt? Python package management is a bit of a cluster, but that sounds like something that would be tough to just "know" without being involved in the project.

[geekboy0801]

Thanks, @jamadden @nilushancosta

gunicorn==20.0.4
gevent==20.6.2
greenlet==0.4.16

This solved the problem,