Uwsgi: No request handled by `uwsgi` with queue filling up

[ntarocco]

Hello,
we are recently experiencing a very weird issue, and we are running out of ideas to fix it, I hope you can help.
We use nginx, available on HTTPS/443, which sends requests to uwsgi. Unfortunately, all requests are "hanging" in uwsgi somewhere, with no log on uwsgi or the Python app.
Basically, when we restart uwsgi, it spawns the workers and the application is launched and ready (showed in uwsgi.log). Requests are sent to nginx (we only have load balancers HEAD ping requests for now, no traffic) but after a while, they all fail:
nginx error.log:

date [error] 20802#0: *20753 connect() failed (111: Connection refused) while connecting to upstream, client: [IP], server: _, request: "HEAD /ping HTTP/1.0", upstream: "uwsgi://[IP]:[PORT]"
date [error] 20805#0: *22759 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: [IP], server: _, request: "HEAD /ping HTTP/1.0", upstream: "uwsgi://[IP]:[PORT]"

The Python app is working fine because I can launch it manually and it handles requests.

I don't know if it can help, but when I check the stats server, I see that slowly "queue": 0 increases until it reaches 101, the max ("max_queue":100) and then I see in the logs:
uwsgi.log: *** uWSGI listen queue of socket "ip:port" (fd: 6) full !!! (101/100) ***

If we keep trying to restart uwsgi service, it reproduces the same behavior. Only 1 time out of 10 (apparently randomly) it handles request correctly.

How can I debug what is happening? No useful logs from nginx, no logs from uwsgi, no logs (nothing handled) on the app side.

Our configuration was working fine until a few weeks ago, and then it broke, probably due to some configuration changes and we have hard time to identify which one.
Do you know if there is any known issue with CentOS, for example?

Our setup:
nginx version: 1.12.2
uwsgi version: 2.0.15
CentOS version: CentOS Linux release 7.4.1708 (Core)

uwsgi config:

plugin = python

stats = [IP]:[PORT]
socket = [IP]:[PORT]
safe-pidfile = /PATH/uwsgi.pid
chmod-socket = 666
vacuum = true
buffer-size = 65535
socket-timeout = 60
socket-write-timeout = 60
so-write-timeout = 60
so-send-timeout = 60
socket-send-timeout = 60

master = true

processes = 5
threads = 4
thunder-lock = true

module = mymodule.wsgi
virtualenv = /opt/mymodule

supervisor.d conf:

[program:uwsgi]
command=/usr/sbin/uwsgi --ini /PATH/uwsgi.ini
directory=/PATH
user=USER
startsecs=5
startretries=3
stopwaitsecs=10
stopsignal=QUIT
killasgroup=true
stopasgroup=true
redirect_stderr=true

nginx conf:

location / {
    uwsgi_pass app_server;
    include uwsgi_params;

    uwsgi_buffering off;
    uwsgi_request_buffering off;

    uwsgi_param Host $host;
    uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
    uwsgi_param X-Forwarded-Proto $scheme;

    client_max_body_size 100m;
  }

[ntarocco]

For whom encounters this issue, after 2 days of investigations, we found the problem, and it is very similar to what documented on StackOverflow: https://stackoverflow.com/questions/34023882/django-uwsgi-nginx-requests-hang

Basically, one endpoint of our backend was down (error logging system, we knew that), but given that the Python application alone was running fine even if that endpoint was down, we did not think that it was the cause the issue.

@unbit
Wouldn't it be possible to improve the error logging of uwsgi to show something in such a case?
It looks like a few people waste quite a lot of time to identify the issue...
We were actually planning to abandon uwsgi for something else...