Uwsgi: Unexplainable timeout error when using uwsgi behind nginx

[bow]

Hi everyone,

I'm using uwsgi (version 2.0.13.1) behind nginx (version 1.10.3) to run a Flask application on Python 3 on Ubuntu 14.04.03. The connection to nginx is done via https, but nginx connects to uwsgi via an IPC socket (config below).

I've been getting some timeout errors from nginx that I can't explain. Here are some examples:

2017/08/17 17:38:02 [error] 22887#22887: *613 upstream timed out (110: Connection timed out) while reading response header from upstream, client: <CLIENT_IP>, server: _, request: "POST <URL> HTTP/1.1
2017/08/31 10:39:22 [error] 5904#5904: *464 upstream timed out (110: Connection timed out) while reading response header from upstream, client: <CLIENT_IP>, server: _, request: "POST <URL> HTTP/1.1

Here's my uwsgi config file:

[uwsgi]
module = wsgi:create_app()

master = true
processes = 5

socket = /tmp/app.sock
chown-socket = app_user:www-data
chmod-socket = 660
socket-timeout = 60
harakiri = 60
vacuum = true

enable-threads = true
die-on-term = true
single-interpreter = true
harakiri-verbose = true

logger = rsyslog:localhost:514,app-uwsgi,14
stats = 127.0.0.1:9090

And here's my uwsgi_params settings in nginx:

        include                         uwsgi_params;
        uwsgi_param                     X-Real-IP $remote_addr;
        uwsgi_param                     X-Forwarded-For $proxy_add_x_forwarded_for;
        uwsgi_param                     X-Forwarded-Proto $http_x_forwarded_proto;
        uwsgi_pass                      unix:/tmp/app.sock;
        uwsgi_read_timeout              60;
        uwsgi_send_timeout              60;

The error occurs randomly. With the exact same payload, the first request could result in an error, but sending it again it could work fine. So I'm not exactly sure if increasing the timeout would help.

What I also don't understand are also these parts of the log messages:

22887#22887: *613 upstream timed out

What does 22887#22887 mean? Why does it say *613 upstream ..? These numbers are always different per error message.

And ultimately, is there a way to reproduce this reliably and then fix it?

[funkybob]

Does the problem still happen when using the current release?

[dsully]

I'm seeing the same thing with uWSGI 2.0.15, with the same setup. nginx in front of uWSGI vending a Python 3 Flask app.

[funkybob]

For the requests that don't get a timeout, how long are they typically taking? Are they _close_ to the timeout?

Also, are there any logs on the uwsgi side?

As to the meaning of those numbers in the nginx logs, you'd probably need to ask the nginx folks :)

[dsully]

Not close at all - timeout is 60 seconds. Most requests are serviced in under 1 second.

I see uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] in the uWSGI logs.

[bow]

@funkybob , I haven't switched to a new version. I did eventually increase my socket-timeout and harakiri config values both to 120. I can't say whether this fixed the problem per-se, since I already noticed the problem occuring fewer times even before I did the update. I can say that I'm not seeing the errors anymore (though I am still not 100% sure given that I don't exactly know what the cause was).

This particular error is only ever triggered when one of my POST endpoints is hit (from another, upstream service). This endpoint is hit relatively few times compared to others. I'm not sure if I have response time data from that service unfortunately. I can say that the payload from that upstream service was received completely as I see this application's database updated ~ which makes the reason for erroring out more puzzling to me.

On the uwsgi side, I'm seeing a lot of this:

SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request [POST endpoint]

around the time the error occured.

I also see @dsully's uwsgi error message, but on a different endpoint.

[funkybob]

The "uwsgi_response_writev_headers_and_body_do" message comes when uWSGI is trying to send the headers and body of the response in one go.

It is not, it seems, a timeout - that's caught later in the same function.

So, yes, your app has handled the request, and returned a response for uWSGI to pass on, but something went wrong.

Would be interesting to know exactly how long uWSGI is taking processing that request. I wrote a patch for someone recently that would log request details as it was parsed coming in...

[funkybob]

@bow that looks like a regular "the browser went away" message... most common cause is the user hit refresh, but can also be from XHR cancelling the request. There are ways to tell nginx to signal uwsgi about this, but to have your app notice and handle it well is a little tricky.

[kylemacfarlane]

This seems very similar to the timeouts I get in #1602.

[DiegoGallegos4]

Any advancements on this?

[kmadac]

I have the same issue. So far I didn't solve it properly, but I found that when I set processes to 1, problem is gone.

My stack:

python 2.7.13
debian 9
nginx 1.10.3
uwsgi 2.0.14-debian

[DiegoGallegos4]

Well, in my case I discover uwgsi is very cpu hungry so I had to increase the vcpus of my instance.

[kmadac]

Well, I haven't set number of vcpus, but number of uwsgi workers to 1 in app uwsgi .ini file. You have 5 in your configuration. I know it is not wanted configuration if you have more cpus, but could you please try it to check whether it fixes 'broken pipe' issue.

[dbelieves]

I am facing the same issue. Has anyone found a solution?

[KoleS46]

Hi guys I think we had the same problem that uwsgi closed the connection but continued processing and setting/extending the http-timeout fixed it for us (http://uwsgi-docs.readthedocs.io/en/latest/Options.html#http-timeout).

[ace-han]

I am facing the same issue after I upgraded my app from py2 to py3, with the same nginx and ini files.

OS: CentOS release 6.6 (Final)
Python: 2.7.13 to 3.6.4
uWSGI: 2.0.14 and above

Tue Feb 27 16:56:18 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/some-api/ (ip xxx) !!!
Tue Feb 27 16:56:20 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/some-api/ (ip xxx) !!!
Tue Feb 27 16:56:26 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/some-api/ (ip xxx) !!!
Tue Feb 27 16:56:26 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/some-api/ (ip xxx) !!!
...

# ini file
[uwsgi]
master = true
chdir = /home/yyy/xx
module = xx.wsgi:application
processes = 20
socket = 0.0.0.0:8001
stats = 0.0.0.0:8002
# clear environment on exit
vacuum = true
max-requests = 500
#chmod-socket = 666
daemonize=/home/logs/uwsgi.log
pidfile=/home/uwsgi.pid
# Otherwise there'll be lots of requests logs.
disable-logging=true
# enlarge request query size
buffer-size = 655350
socket-timeout = 300
http-timeout = 300
harakiri = 300

I've tried socket-timeout, http-timeout and harakiri, but no luck...

# nginx.conf
server {
        listen       80;
        server_name  x.y.z.com;
        #access_log  logs/host.access.log  main;

        # for larger client request header
        large_client_header_buffers 4 1096k;

        # for modifier, plz refer to 
        # https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms
        location ^~ /static/ {
                alias /home/ucmt/dv/static/;
        }
        location ~ /(.*)$ {
                include uwsgi_params;
                uwsgi_read_timeout 600s;
                uwsgi_pass ll_uwsgi_backend;
        }
}
....

Please kindly help me out, thx

[mvdpanne]

Like the other people above me, I too was pulling my hair out as to why my requests were timing out.

My symptoms are as follows. At first, everything would work as expected. But as soon as I hit my endpoints a couple of times a second, it just locks up, and Nginx serves me 502 Bad Gateway errors. Only after restarting the uwsgi workers, I can accept requests again. Like the others, I see the following errors:

SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected)
uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296]

The suggestion by @kmadac does wonders, however! Setting processes = 1 in my uWSGI configuration file resolved the problem of my endpoints locking up/timing out. Interestingly enough, the errors keep occurring. So I'm not sure to what extend the error messages are part of the problem. It is furthermore good to note that for me, the problem occurred regardless of whether a GET or POST request was being executed.

I understand that this certainly won't be an appropriate fix for everyone, but it might be a good workaround for others until the underlying issue is found and fixed.

My setup:

• Ubuntu 16.04
• Python 3.6.4
• uWSGI 2.0.15
• Nginx 1.10.3

[mvdpanne]

Unfortunately, after testing the 'workaround' some more, the problem started appearing again. So it does not go away; it happens less frequently. To my perception it occurs randomly.

[redwud]

The following worked for me as work around, as I am temporarily downloading a huge file, might be the case for others.

• uwsgi ini:
  http-timeout = 3600000
• nginx conf:
  uwsgi_max_temp_file_size 20480m;

[hoondongkim]

I met the same problem.
But, Some python code works well. and Only One python code occur that problem.
that code use gensim , keras, tensorlfow library for deep learning serving.
and , word2vec file size is almost 500mb. and deep learning model size is almost 600mb.
that huge file is loaded only one time at first app initial time.

but, same code works well in case of sigle statnd alone Flask mode.
same code occur that error at the case of Nginx + uwsgi + flask.

[adibalcan]

For me the issue was from Redis, just on pages when Redis was called.

[canivel]

same for me, keras uwsgi flask = 504

[tannart]

I'm seeing the same issue with an nginx -> uwsgi -> django configuration, all hooked up in a docker-compose network.

Here's the trio of messages I'm seeing in the logs:

1. SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /my/transaction/hook/ (ip remote-ip) !!!
2. uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during POST [/my/transaction/hook/] (remote-ip)
3. IOError: write error

I then also see my nginx container returning a 499 (Client closed the connection) response code.

And my config:

• Ubuntu 16.04.2
• Python 2.7.9
• uWSGI 2.0.7
• Nginx 1.9.15
• docker 17.05.0
• docker-compose 1.9.0
• Postgres 9.4.15

It's probably also worth noting that I saw an uptick in these exceptions after moving my database from the EC2 it shared with my application on to an external RDS.

[funkybob]

@MacFarlaneBro does anything improve if you update to a current release of uWSGI?

[tannart]

@funkybob I'll try that out and report back

[faxioman]

Exactly the same problem using uwsgi with unix socket:

python 3.5.5 / python 3.6.3
uWSGI 2.0.17 (also tested with uWSGI 2.0.13.1)
nginx 1.12.2
django 2.0.6

Our projects are deployed on a k8s cluster. I have the same problem in all the deployed python 3 projects.
No problem verified in others web projects using:

python 2.7.14
uWSGI 2.0.17
nginx 1.12.2
django 1.x

[tannart]

@funkybob So I upgraded to uWSGI 2.0.17 a couple of days ago and I'm still seeing the issue.

[alexvy86]

I'm experiencing something similar with uwsgi 2.0.17 behind nginx 1.10.3 (communicating with a socket file, uwsgi protocol), running in a Debian 9 Docker container, with Python 3.6.5. In my case the first requests sometimes work. After some random amount of time (or a condition that I haven't been able to identify; has even happened that no requests ever work), all further requests hang. Nginx times out after the configured uwsgi_read_timeout and logs this:

2018/06/20 00:38:05 [error] 7#7: *218 upstream timed out (110: Connection timed out) while reading response header from upstream, client: <REDACTED_IP>, server: , request: "POST /tokenize HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock", host: "<REDACTED_URL>:5000"

But uwsgi is not logging anything at all for those requests (the initialization messages do come up).

[kylemacfarlane]

@adibalcan might be on to something when they mention Redis. The two possibly related bugs I opened (#1602 and #1716) both use Redis. In fact the OSError in #1716 is coming from Redis.

@MacFarlaneBro says that the problem became worse when switching to remote RDS but I find the opposite. When I develop locally with Redis/Postgres I constantly get timeouts even as a single user but it's pretty rare in production with Elasticache/RDS until the server comes under heavier load.

[adibalcan]

@kylemacfarlane I solved the problem, my Redis connexion lasts just 5 minutes, after my app tries to connect at Redis but takes a very long time(maybe infinite) and the server responds with timeout.

[kylemacfarlane]

@adibalcan Are you sure you were getting the same timeout error as everyone else? That sounds completely different to what others are reporting.

My problem (and I assume most others here) is that writing to the web client connected to uWSGI "times out" immediately. I'm not getting any timeouts when connecting to Redis, Postgres, etc.

The reason I bring up Redis is because in #1716 if I suppress the OSError from Redis then it turns out that the web client is dead too which is similar to this bug. To me it seems like a common thread in all the issues I'm having is that uWSGI loses track of open sockets and it may be an incompatibility with redis-py or psycopg2 which causes this. This would also explain why both I and @MacFarlaneBro see a difference (even if opposite) with local vs remote cache/db servers.

[faxioman]

Mmm... same problem after switching (testing purpose only) from uwsgi to phusion passenger.
It definitely must be a lower level problem... continuing the investigation.

[alexvy86]

Small update on my case, if I set uwsgi's harakiri option to 30 seconds, when I run into this issue, leaving the application alone for a minute or so, brings it back to normal. So it sounds like in my case it could be threads getting "locked" somehow (even though my requests are all finishing, so I see no reason why that would be the case).

[faxioman]

Same behaviour here using harakiri option.

[duizendnegen]

Sounds like https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html - worker's mercy, configuring the following in uswgi.ini resolves the situation for me:

harakiri = 240 
http-timeout = 240 
socket-timeout = 240 
worker-reload-mercy = 240 
reload-mercy = 240 
mule-reload-mercy = 240

[alexvy86]

Another small update from my end: the harakiri option that seemed to fix my issue apparently did not, I ran into it again even with that option set to true.

I was able to determine that in my case, starting a background thread was the culprit. At some point, all my web requests hung when nginx tried to pass them to uwsgi, and got terminated after the timeout I configured in nginx, without uwsgi ever logging anything about them.

My latest attempt at fixing that, with apparently good results so far, is to start uwsgi with the --lazy-apps option, so each worker process (of which I'm only starting 1 anyway) loads the whole application on its own, instead of having the master process load everything and then fork into the worker process. In the end we might even go with a different setup where the work done by that background thread is done by a separate process altogether, but for now my conclusion is that when using background threads, it's worth checking if lazy-apps would be beneficial.

[anmolghosh]

Facing similar issue. I am serving django 1.11 application using uwsgi 2.0.17 with nginx 1.10.3 behind AWS ALB. Normally all requests are being served normally, but occasionally few ends with IOError from uwsgi and corresponding 499 status error code in nginx. Couldn't find any pattern and am getting errors even with very lightweight APIs like health check.

Running the application in lazy-apps mode or setting the uwsgi and nginx timeout values didn't help.
Did anyone find a working solution for it?

[mvdpanne]

I have posted in this issue before, and I never managed to solve this problem.

Instead, I switched to Gunicorn, and had no problems while using the same setup (see https://github.com/unbit/uwsgi/issues/1623#issuecomment-370090634). I know, it's not exactly considered polite to 'advertise' a different package, but it might help out some people.

[salilsethi]

I was having the same problem... A quick workaround was not setting the master process.

So in the config file, I removed the line master = true and it works fine for me.

Hope this help.

[Poply]

@salilsethi

Thanks. Removing the master setting is what fixed my issue.

Setup:
Flask application behind nginx using Flask-mail to send mail through my mxroute email server on a google cloud VM.

For a while I thought it was something on Google's side blocking my emails. However I could send my emails fine when I used flask's built-in server running on the Google VM.

Python 3.5.3
Flask 0.12.2
nginx 1.12.2
Flask-mail 0.9.1
CentOS 7.3
uWSGI 2.0.17

[xrmx]

uwsgi may have issues but a lot of the comments here are like "my own application do something silly so that uwsgi workers cannot serve requests and nginx gives up". And that's an issue in your own application not in uwsgi.

[jtlz2]

@xrmx Fair point - noted and comment deleted.

[sacdallago]

Ok, fun but not so fun thing, the exact same error log can occur for a completely different reason: the socket cannot write the temporary response file because of permissions mismatch (or: nginx cannot read the temp file for permissions mismatch).

In this case, what you will see thrown by uwsgi is something like:

SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request ...64.161) !!!
uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 429] during GET /api/runs/...0.0.64.161)

Which is what brought me to this issue in the first place.

Notice one thing: the line of this error is 429, not around * 2XX* as here or here. This should already sorta tell you something is not exactly the same.

Now, to really understand that this was a completely different problem the important bit is checking out nginx's error log, which in my case looks like this:

YYYY/MM/DD HH:MM:SS [crit] 9072#0: *23 open() "/var/lib/nginx/tmp/uwsgi/1/00/0000000001" failed (13: Permission denied) while reading upstream, client: X.X.X.X, server: _, request: "GET /some/thing HTTP/1.1", upstream: "uwsgi://unix:/path/to/cocket.sock:", host: "expl.com", referrer: "https://expl.com/something"

This is very different from https://github.com/unbit/uwsgi/issues/1623#issuecomment-398593515

Here it's clear: the problem is someone can not write (uwsgi) or something can not read (nginx). The easy work around if this is the scenario is to write temp files in an everyone can write, everyone can read dir (if you are OK with that in terms of who has access to your server). /tmp is an excellent candidate. Nginx has a parameter called client_body_temp_path that you can use to set the path BUT, if you are importing the uwsgi nginx params in nginx, that parameter changes to uwsgi_temp_path. You can see that from the path reported in the error, if, like in my case, it's /var/lib/nginx/tmp/uwsgi/ then you'll need to change the uwsgi_temp_path. More: https://github.com/dockerfile/nginx/issues/4

A real nightmare. But now solved.

[yangjerry110]

you uwsgi service is action the work too many time ? so.. you uwsgi is time out for himself, you should see your uwsgi's actiom time and for the action time you should add the time for your uwsgi's action time.. then see the result to adjust your uwsgi's action time for himself can work normal.. if you do this and the problem is exit , you can see the nginx's action time , nginx also can end the work for time's too log.. you also can adjust the action time for the nginx action time the normal nginx's action time is one minute, you can proper add the time for the nginx's setting keepAlive-time the setting prama is the nginx connect you web softwore keep connection time. if you uwsgi action time is too long ,your uwsgi's action time is set normal ,the nginx also connect time out for your web softwore, hope the sug can help you

[titicaca]

same for me, keras uwsgi flask = 504

I got the same problem, have you solved it? @canivel @hoondongkim

[titicaca]

I tried gunicorn instead of uwsgi and everything works now.

same for me, keras uwsgi flask = 504

I got the same problem, have you solved it? @canivel @hoondongkim

[AndreiDrang]

Hi there.
I have the same error when i try to delay(just start task, did not wait result) some celery task from django app view.
POST request + task.delay() = 502 error(if i increase wait time - 504 error)
I try all config variants from this issue, but ...

Debian 9
Nginx 1.10
Uwsgi 2.0.17
Python 3.6
Django 2

[federicopuntoni]

Same issue here:

Debian GNU/Linux 9
Python 3.7.0
Nginx 1.13.12
UWSGI 2.0.17.1
Flask 1.0.2
Docker 18.09.0

[uwsgi]
module = main
callable = app
cache2 = name=search_cache,items=1000

Call an API, store results as json in uwsgi cache, get stored json without problem after few seconds.

But hitting web browser refresh during storing in cache send me:

SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /web?q=asdfasdf (ip 172.17.0.1) !!!

uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /web?q=asdfasdf (172.17.0.1)

OSError: write error

[b0urb0n]

lazy-apps worked for my use case which also showed similar behavior to #1602

When shutting down uwsgi, I get the following errors:

^CSIGINT/SIGQUIT received...killing workers...
*** Error in `/opt/speakeasy_services/venv/bin/uwsgi': free(): corrupted unsorted chunks: 0x0000000006d81d70 ***
!!! uWSGI process 13846 got Segmentation Fault !!!
terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x81489)[0x7f3a7a45c489]
/usr/lib64/titanium_core/core_engine.so(_ZN4core6engine14implementationD1Ev+0xb6)[0x7f3a6c204e26]
/usr/lib64/titanium_core/core_engine.so(_ZN4core6engineD1Ev+0x11)[0x7f3a6c1fecf1]
*** backtrace of 13846 ***
/opt/speakeasy_services/venv/bin/uwsgi(uwsgi_backtrace+0x2e) [0x463efe]
/opt/speakeasy_services/venv/bin/uwsgi(uwsgi_segfault+0x21) [0x464291]
/usr/lib64/libc.so.6(+0x36280) [0x7f3a7a411280]
/usr/lib64/libc.so.6(+0x39acd) [0x7f3a7a414acd]
/usr/lib64/libc.so.6(+0x39bb7) [0x7f3a7a414bb7]
/opt/speakeasy_services/venv/bin/uwsgi() [0x41d90f]
/opt/speakeasy_services/venv/bin/uwsgi(end_me+0x25) [0x461505]
/usr/lib64/libpthread.so.0(+0xf5d0) [0x7f3a7b6215d0]
/usr/lib64/libpthread.so.0(pthread_cond_wait+0xc5) [0x7f3a7b61d965]
/usr/lib64/titanium_core/core_engine.so(_ZN4core13event_manager13sender_threadEv+0x15f) [0x7f3a6c20985f]
/usr/lib64/titanium_core/dep/libstdc++.so.6(+0xc317f) [0x7f3a6868a17f]
/usr/lib64/libpthread.so.0(+0x7dd5) [0x7f3a7b619dd5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f3a7a4d8ead]
*** end of backtrace ***
*** Error in `/opt/speakeasy_services/venv/bin/uwsgi': free(): corrupted unsorted chunks: 0x000000000210a680 ***

[constantinevitt]

I was having the same problem... A quick workaround was not setting the master process.

So in the config file, I removed the line master = true and it works fine for me.

Hope this help.

This solved my problem, thank you.

[mreaston]

We've noticed the same thing for our app running in docker behind nginx. Adding: threaded-logger = true to the config seems to help, now the logging thread crashes after some time instead of the rest of the app workers.

We used strace to determine that uwsgi was stuck in a write with our app logs, so it makes sense that moving it out to another thread helped keep the workers happy.

[ing200086]

Has anyone been able to figure out the root of this issue. A lot of what I've seen is "I changed this random timeout or removed this line, and it works now".... I have tried to make all those random changes, but no such luck. Would love to know what the source of the issue is to fix it...

[ByteOven]

"Would love to know what the source of the issue is to fix it"
I'd guess its timeout on both sides, nginx and uwsgi.

Edit:
I'm getting these as well on big db queries ( small VPS's )

[AceCat]

For those having similar problems, I found that this happened when running a multi-threaded application that would be hit by multiple requests at the same time. I found that passing in the --lazy-apps argument fixed the problem.

[zoltan-fedor]

I have removed master = true, tried adding lazy-apps = true and also tried adding the harakiri option as:

harakiri = 240 
http-timeout = 240 
socket-timeout = 240 
worker-reload-mercy = 240 
reload-mercy = 240 
mule-reload-mercy = 240

None of these made any difference.

The error is still there randomly:

Tue Jun 18 12:49:58 2019 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during POST /ajax/get_similar_cases (172.21.0.1)
OSError: write error

[funkybob]

Does lazy-apps even have an effect without a master?
My understanding was lazy-apps delayed initialising the app from the master to the workers. Workers start slower, but are guaranteed "isolated" instances of the app, instead of sharing what was initialised in the master.

[mreaston]

We've been able to solve it on the project I'm working on. It boiled down to systemd running docker-compose up and at some point closing stdout/stderr of the docker-compose and a write somewhere making its way up to that level and getting stuck (or so I suspect). We rewrote our systemd file to resemble this one and haven't seen a hang at all: https://github.com/docker/compose/issues/4266#issuecomment-474534508

[bukowa]

In my case the cause of a timeout was using http option while passing uwsgi_params to nginx. Solution was disabling uwsgi_params in nginx.

Edit: the second solution was using socket listening on port and passing uwsgi_params normally.

[RNCTX]

Anything (even unused) that is not found or fails due to a permission will cause this, apparently.

In my case I had a database certificate filename listed in an environment variables file. The certificate didn't exist in that location due to a typo.

[gg4u]

Also experiencing similar issue of 504 at random.

my error on uwsgi log shows different line number:

 uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 331] during GET [...]

error on nginx log:

[...] [error] 16250#16250: *275 upstream timed out (110: Connection timed out) while reading response header from upstream, [...]

Maybe I started to get troubles when using certbot, so using a certificate,
but not sure and don't know how to test or check for more insightful logs..

[minyoungjun]

I added this line in my ini file.

lazy-apps = true

It works for me!

[jdugh]

The following worked for me as work around, as I am temporarily downloading a huge file, might be the case for others.

• uwsgi ini:
  http-timeout = 3600000
• nginx conf:
  uwsgi_max_temp_file_size 20480m;

For me, juste uwsgi_max_temp_file_size 20480m; in my nginx site conf solved the problem.
I just adapted the value for my context.
Thank you @redwud

[successgilli]

For me, the problem of upstream timed out (110: Connection timed out) while reading response header from upstream was solved by setting the processes to 1 in the uwsgi.ini file and taking out master = true

Also, if directory permission for writing and reading the socket was an issue, the error message would show permission denied. then you will have to ensure that the user set in your system service unit or Startup file has appropriate privileges with the required group which is usually www-data.

[bekab95]

So what is best config for Behind Nginx when getting too much users online ?

[zoltan-fedor]

In my case (Nginx + UWSGI + Flask) the error was:

uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306]

This happens when the client makes a request and then closes it (either because server took too long to respond or client has been disrupted) but uwsgi is still processing that request.
If the server takes too long to respond, then changing timeouts might help or identifying why it takes too long to respond.

In my case it did not take too long to respond (< 2 seconds), so in my case the issue was the client going away before the response was ready.

To remove these errors from the uwsgi log I have added the following to the NGinx config (for uwsgi routes):

uwsgi_ignore_client_abort on

Alternatively you could just disable the logging of write errors:

ignore-write-errors = true

[davidmontoyago]

I've experienced the same issue when proxying to uwsgi with envoy instead of nginx, causing envoy to fail requests with 503 UC - upgrading to versions below solved it for me:

uwsgi==2.0.18
werkzeug==0.16.1

[shahram4m]

it worked for me by comment

master = true

and put this,
lazy-apps = true

in uwsgi.ini file

[funkybob]

Does lazy-apps even mean anything without master?

My undestanding was it would delay initialising python / the app from the master process into the workers.

[shahram4m]

explain for lazy-apps: load apps in each worker instead of the master

[est]

is it possible to log how much time uwsgi took in uwsgi_response_writev_headers_and_body_do() ?

It's helpful to debug which part was responsible for the timeout.