Sanic: Nginx Reverse Proxy + Sanic

[dsanghan]

I've setup a Nginx reverse proxy that forwards all HTTPS requests to an HTTP sanic backend using a private IP. We receive about a 10 requests/s and generally see about a 100 established ports on nginx, with about 400 in timeout_wait. In about an hour, we're seeing nginx report a bad gateway and all requests start failing.

The error logs start filling with:

2017/10/21 11:05:44 [error] 2447#2447: *26440 connect() failed (110: Connection timed out) while connecting to upstream, client: *.*.*.*, server: example.com, request: "POST /api/negotiate HTTP/1.1", upstream: "http://*.*.*.*:12789/api/negotiate", host: "example.com:12789"
2017/10/21 11:05:44 [error] 2446#2446: *25512 connect() failed (110: Connection timed out) while connecting to upstream, client: *.*.*.*, server: example.com, request: "POST /api/negotiate HTTP/1.1", upstream: "http://*.*.*.*:12789/api/negotiate", host: "example.com:12789"
2017/10/21 11:05:44 [error] 2447#2447: *25191 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: *.*.*.*, server: example.com, request: "POST /api/negotiate HTTP/1.1", upstream: "http://*.*.*.*:12789/api/negotiate", host: "example.com:12789"
2017/10/21 11:05:45 [alert] 2446#2446: *25099 open socket #69 left in connection 49
2017/10/21 11:05:45 [alert] 2446#2446: *24467 open socket #18 left in connection 80
2017/10/21 11:05:45 [alert] 2446#2446: *26148 open socket #21 left in connection 119
2017/10/21 11:05:45 [alert] 2446#2446: *25613 open socket #216 left in connection 122/api/

Exposing sanic via HTTPs works properly. We only start seeing issues as soon as we put sanic behind the nginx reverse proxy.

Here's our nginx conf:

upstream http_backend {
    server *.*.*.*:12789;
    keepalive 512;
}

server {
    listen 12789 ssl;
    server_name example.com;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    keepalive_timeout 10;

    location / {
        proxy_http_version 1.1;
        proxy_pass http://http_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
    }
}

Any suggestions?

[prashantgpt91]

@dsanghan I think you should consider using uwsgi+nginx wherein uwsgi will launch your application and nginx will act as front end reverse proxy. This would give your sanic app a more robust env.
Take a look at this

[prashantgpt91]

@dsanghan I tried doing this, seems like uwsgi does not supports sanic though works for flask.
May be we should look for alternative of uwsgi.
if anyone else has also observed same thing, then let's discuss alternatives.

[ashleysommer]

@dsanghan
Are you using Sanic release 0.6.0 or Sanic #master?
If you are not using master, can you please check it out and test if you are still seeing the same bug with the current master code?
The HTTP connection handling logic of the Sanic server protocol was recently updated, so it might work differently now, and it may have incidentally fixed the bug you are seeing, but also it might not be related at all.

[prashantgpt91]

@ashleysommer sorry to ask but may i know how to install master branch of sanic? whenever I use
pip install sanic it installs 0.6v by default

[dsanghan]

@x0v: https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch

[dsanghan]

@ashleysommer: We're using 0.6.0. I'll update to master and revert.

[prashantgpt91]

@dsanghan I tried with this command pip install git+https://github.com/channelcat/sanic.git@master but still same 0.6v got installed.

[dsanghan]

Master is still tagged as version 0.6 in __init__.py. So you should still see 0.6.

Just in-case, use the --upgrade option when installing. You should see:

Collecting git+https://github.com/channelcat/sanic.git@master
  Cloning https://github.com/channelcat/sanic.git (to master) to /tmp/pip-cq_kn6pi-build
Requirement already up-to-date: httptools>=0.0.9 in /usr/local/lib/python3.5/dist-packages (from sanic===0.6.0)
Requirement already up-to-date: uvloop>=0.5.3 in /usr/local/lib/python3.5/dist-packages (from sanic===0.6.0)
Requirement already up-to-date: ujson>=1.35 in /usr/local/lib/python3.5/dist-packages (from sanic===0.6.0)
Collecting aiofiles>=0.3.0 (from sanic===0.6.0)
  Downloading aiofiles-0.3.2-py3-none-any.whl
Requirement already up-to-date: websockets>=3.2 in /usr/local/lib/python3.5/dist-packages (from sanic===0.6.0)
Installing collected packages: aiofiles, sanic
  Found existing installation: aiofiles 0.3.1
    Uninstalling aiofiles-0.3.1:
      Successfully uninstalled aiofiles-0.3.1
  Found existing installation: sanic 0.6.0
    Uninstalling sanic-0.6.0:
      Successfully uninstalled sanic-0.6.0
  Running setup.py install for sanic ... done
Successfully installed aiofiles-0.3.2 sanic-0.6.0

[prashantgpt91]

okay thanks a lot.
one more question, you said

1. Exposing sanic via HTTPs works properly
2. Put sanic behind the nginx reverse proxy

May i know, how you did both these things? i.e.. How you exposed sanic via HTTPS and how you put sanic behind Nginx reverse proxy.

Pardon me to ask too much, I have just started learning out these things.

Any ref link or brief explanation is enough.

[dsanghan]

@x0v: I feel like these are questions for another issue. Please open one / look at the docs since they are pretty helpful.

[dsanghan]

@ashleysommer: Updating to latest master fixed the issue. Thanks!

[chatreek]

I'm using sanic on version 0.7.0 but still facing same problem.

I runing each component on docker by separate container.

Any suggestion ?