Flask-socketio: IOError: unable to complete websocket handshake while running in uwsgi+Nginx

[Rakin061]

I have been planning to deploy a flask-SocketIO application into production using uwsgi server behind Nginx

Versions including relevant dependencies of my virtual environment:-

OS:-  Ubuntu 16.04.3 LTS
uWsgi:-   2.0.15 
nginx:-   1.12.2
python:- 2.7.12
Falsk:-   0.12.2
Flask-SocketIO:-  2.9.2
gevent:- 1.2.2

I am intended to run the uwsgi server through systemd and load the uwsgi configurations from a .ini file.

Here's my wsgi.py file:

from server import app
if __name__ == "__main__":
    app.run()

Here's the configuration file myproject.ini for uwsgi

[uwsgi]
module = wsgi:app

http = :5000
gevent = 1000
http-websockets = true

master = true
processes = 5

socket = myproject.sock
chmod-socket = 660
vacuum = true

die-on-term = true

Here's the myproject.service file in /etc/systemd/system

[Unit]
Description=uWSGI instance to serve myproject
After=network.target

[Service]
User=user
Group=www-data
WorkingDirectory=/home/user/myproject
Environment="PATH=/home/user/myproject/virtual_frame/bin"
ExecStart=/home/user/myproject/virtual_frame/bin/uwsgi --ini myproject.ini

[Install]
WantedBy=multi-user.target

And, finally my nginx configuration file:

 server {
    listen 80;
    server_name 10.11.201.82:5000;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/home/user/myproject/myproject.sock;
        uwsgi_ignore_client_abort on;
    }

    location /socket.io {
        include proxy_params;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass http://10.11.201.82:5000/socket.io;
    }

}

Errors:-

Actually I'm getting different errors from several times.

1) IOError: unable to complete websocket handshake

Here's my project status while running:_

user@user-HVM-domU:~/myproject$ sudo journalctl -u myproject -n 20
-- Logs begin at শনি 2018-02-03 09:52:05 +06, end at রবি 2018-02-04 15:27:48 +06. --
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/flask/app.py", line 1997, i
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     return self.wsgi_app(environ, start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/flask_socketio/__init__.py"
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/engineio/middleware.py", li
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     return self.engineio_app.handle_request(environ, start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/socketio/server.py", line 3
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     return self.eio.handle_request(environ, start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/engineio/server.py", line 2
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     environ, start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/engineio/socket.py", line 8
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/engineio/socket.py", line 1
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     return ws(environ, start_response)
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:   File "/home/user/myproject/virtual_frame/local/lib/python2.7/site-packages/engineio/async_gevent_uwsgi
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]:     uwsgi.websocket_handshake()
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]: IOError: unable to complete websocket handshake
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]: [pid: 23604|app: 0|req: 21/79] 10.11.201.82 () {50 vars in 1028 bytes} [Sun Feb  4 15:27:39 2018] GET /s
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]: [pid: 23604|app: 0|req: 22/80] 10.11.201.82 () {38 vars in 784 bytes} [Sun Feb  4 15:27:39 2018] GET /so
ফেব্রু 04 15:27:39 user-HVM-domU uwsgi[23592]: [pid: 23604|app: 0|req: 23/81] 10.11.201.82 () {42 vars in 845 bytes} [Sun Feb  4 15:27:39 2018]

2) IOError: write error: uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.

ফেব্রু 04 15:16:53 user-HVM-domU uwsgi[23421]: [pid: 23433|app: 0|req: 52/94] 10.11.201.82 () {42 vars in 845 bytes} [Sun Feb  4 15:16:38 2018] POST /s
ফেব্রু 04 15:16:53 user-HVM-domU uwsgi[23421]: [pid: 23433|app: 0|req: 52/95] 10.11.201.82 () {38 vars in 783 bytes} [Sun Feb  4 15:16:30 2018] GET /so
ফেব্রু 04 15:16:55 user-HVM-domU uwsgi[23421]: [pid: 23433|app: 0|req: 54/96] 10.11.201.82 () {42 vars in 843 bytes} [Sun Feb  4 15:16:55 2018] POST /s
ফেব্রু 04 15:16:55 user-HVM-domU uwsgi[23421]: [pid: 23433|app: 0|req: 54/97] 10.11.201.82 () {38 vars in 783 bytes} [Sun Feb  4 15:16:53 2018] GET /so
ফেব্রু 04 15:17:08 user-HVM-domU uwsgi[23421]: Sun Feb  4 15:17:08 2018 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 
ফেব্রু 04 15:17:08 user-HVM-domU uwsgi[23421]: IOError: write error
ফেব্রু 04 15:17:08 user-HVM-domU uwsgi[23421]: [pid: 23433|app: 0|req: 55/98] 10.11.201.82 () {44 vars in 867 bytes} [Sun Feb  4 15:16:08 2018] GET /so

3) Nginx error log

Last 20 line of /var/log/nginx/error.log

2018/02/04 15:24:42 [error] 23563#23563: *109 upstream prematurely closed connection while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "GET /socket.io/?EIO=3&transport=websocket&sid=b95aa3f7a7374706a7d09d857c1e69fe HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=websocket&sid=b95aa3f7a7374706a7d09d857c1e69fe", host: "10.11.201.82"
2018/02/04 15:25:40 [error] 23560#23560: *83 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "GET /socket.io/?EIO=3&transport=polling&t=1517736280486-2&sid=998d555f29cc49f3949bf080b5c2820c HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=polling&t=1517736280486-2&sid=998d555f29cc49f3949bf080b5c2820c", host: "10.11.201.82", referrer: "http://10.11.201.82/"
2018/02/04 15:25:49 [error] 23563#23563: *98 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "GET /socket.io/?EIO=3&transport=polling&t=1517736289238-36&sid=6535da975fee43bdb49be5f061e04522 HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=polling&t=1517736289238-36&sid=6535da975fee43bdb49be5f061e04522", host: "10.11.201.82", referrer: "http://10.11.201.82/"
2018/02/04 15:25:50 [error] 23563#23563: *138 upstream prematurely closed connection while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "GET /socket.io/?EIO=3&transport=polling&t=1517736290606-40&sid=c13d0711b34b49cb8d5d973642369c33 HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=polling&t=1517736290606-40&sid=c13d0711b34b49cb8d5d973642369c33", host: "10.11.201.82", referrer: "http://10.11.201.82/"
2018/02/04 15:25:54 [error] 23563#23563: *143 upstream prematurely closed connection while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "GET /socket.io/?EIO=3&transport=polling&t=1517736294273-52&sid=aec315362ea644fe93035f7d8c5519a6 HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=polling&t=1517736294273-52&sid=aec315362ea644fe93035f7d8c5519a6", host: "10.11.201.82", referrer: "http://10.11.201.82/"
2018/02/04 15:25:56 [error] 23560#23560: *83 upstream prematurely closed connection while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "GET /socket.io/?EIO=3&transport=polling&t=1517736346555-72&sid=933777c0ced64dc4a459b1ebe33a0d21 HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=polling&t=1517736346555-72&sid=933777c0ced64dc4a459b1ebe33a0d21", host: "10.11.201.82", referrer: "http://10.11.201.82/"
2018/02/04 15:25:56 [error] 23560#23560: *83 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.11.201.82, server: 10.11.201.82, request: "POST /socket.io/?EIO=3&transport=polling&t=1517736356497-73&sid=933777c0ced64dc4a459b1ebe33a0d21 HTTP/1.1", upstream: "http://10.11.201.82:5000/socket.io/?EIO=3&transport=polling&t=1517736356497-73&sid=933777c0ced64dc4a459b1ebe33a0d21", host: "10.11.201.82", referrer: "http://10.11.201.82/

[Rakin061]

Screenshot of my overall project status

[miguelgrinberg]

Have you seen the documentation regarding deployment with uwsgi? I suggest you start from the configuration from there. In particular, you have to use one worker.

To use multiple workers with SocketIO the set up is more complicated. The workers must all be standalone uwsgi instances with one worker, and nginx should be configured to load balance across all these workers. Enabling multiple workers in uwsgi is not supported because this protocol requires sticky sessions, which nginx provides.

[Rakin061]

Yes, I configured each and every parameter except worker for uwsgi following the official documentation expecting that multiple worker will be a more performant option for production deployment.

So, using processes =5 is the root cause of all the errors I stated ?? Or is there any other configuration You could suggest me to remove all the errors ?

And, since I couldn't use multiple worker in the uwsgi part, I could have tried simulating multiple workers in the Nginx configuration.

Thanks.

[miguelgrinberg]

Not sure if it is the only problem, but definitely something you need to fix. Switch to one worker, then test using uwsgi directly, and if that works, then add nginx.

[Rakin061]

Yes, using single worker solved the problem. No problem in message sending and receiving and everything is working pretty fast.

But the errors like unable to complete websocket handshake and IOError: Write Error is still in action.

[miguelgrinberg]

Do all your clients fail to connect over websocket, or just some of them? You'll have to debug this to identify which clients fail to connect, and why.

[Rakin061]

Ok. I will check back accordingly and let you know

[Rakin061]

Hello @miguelgrinberg

After analyzing everything as you said and checking some server-client connections :-

Do all your clients fail to connect over websocket
-- No. All the clients could get connected with the server and get proper responses from the server as well. So, I think websocket connection should be okay.

Then, I tried to remove nginx out of the picture and tried to run the project directly with the uwsgi server. But still I can reproduce those errors IOError:unable to complete websocket handshake and IOError: Write Error. So, it's obvious that this is not a nginx configuration issue.

And, when I run with the default wsgi server python server.py then no error occurred. So, it should be the problem associated with the uwsgi server.

Here's the uwsgi status while running uwsgi directly without nginx

(virtual_frame) user@user-HVM-domU:~/myproject$ sudo uwsgi --http :5002 --gevent 1000 --http-websockets --master --wsgi-file server.py --callable app 
*** Starting uWSGI 2.0.15 (64bit) on [Mon Feb  5 11:32:36 2018] ***
compiled with version: 5.4.0 20160609 on 28 January 2018 14:44:36
os: Linux-4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018
nodename: user-HVM-domU
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /home/user/myproject
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your processes number limit is 31753
your memory page size is 4096 bytes
detected max file descriptor number: 1024
- async cores set to 1000 - fd table size: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :5002 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:42622 (port auto-assigned) fd 3
Python version: 2.7.12 (default, Dec  4 2017, 14:50:18)  [GCC 5.4.0 20160609]
Python main interpreter initialized at 0x1333e30
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 20716928 bytes (20231 KB) for 1000 cores
*** Operational MODE: async ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1333e30 pid: 28257 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 28257)
spawned uWSGI worker 1 (pid: 28266, cores: 1000)
spawned uWSGI http 1 (pid: 28267)
*** running gevent loop engine [addr:0x4830f0] ***

And here's my project status and error log

[pid: 28266|app: 0|req: 1/1] 10.11.201.82 () {38 vars in 622 bytes} [Mon Feb  5 11:33:33 2018] GET / => generated 21514 bytes in 19 msecs (HTTP/1.1 200) 2 headers in 82 bytes (3 switches on core 999)
[pid: 28266|app: 0|req: 2/2] 10.11.201.82 () {38 vars in 596 bytes} [Mon Feb  5 11:33:33 2018] GET /static/utf8.js => generated 5401 bytes in 5 msecs via sendfile() (HTTP/1.1 200) 8 headers in 300 bytes (1 switches on core 999)
[pid: 28266|app: 0|req: 3/3] 10.11.201.82 () {38 vars in 606 bytes} [Mon Feb  5 11:33:34 2018] GET /static/era_logo.jpg => generated 7538 bytes in 1 msecs via sendfile() (HTTP/1.1 200) 8 headers in 287 bytes (1 switches on core 999)
[pid: 28266|app: 0|req: 4/4] 10.11.201.82 () {38 vars in 596 bytes} [Mon Feb  5 11:33:34 2018] GET /static/ERA.png => generated 16328 bytes in 0 msecs via sendfile() (HTTP/1.1 200) 8 headers in 289 bytes (1 switches on core 999)
[pid: 28266|app: 0|req: 5/5] 10.11.201.82 () {38 vars in 708 bytes} [Mon Feb  5 11:33:34 2018] GET /socket.io/?EIO=3&transport=polling&t=1517808814084-0 => generated 119 bytes in 1 msecs (HTTP/1.1 200) 4 headers in 180 bytes (3 switches on core 999)
[pid: 28266|app: 0|req: 6/6] 10.11.201.82 () {42 vars in 842 bytes} [Mon Feb  5 11:33:34 2018] POST /socket.io/?EIO=3&transport=polling&t=1517808814114-1&sid=ffb2449454b34644959fa113f606c6c3 => generated 2 bytes in 1 msecs (HTTP/1.1 200) 3 headers in 117 bytes (3 switches on core 999)
you need to build uWSGI with SSL support to use the websocket handshake api function !!!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask_socketio/__init__.py", line 42, in __call__
    start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/middleware.py", line 47, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/socketio/server.py", line 360, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/server.py", line 274, in handle_request
    environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/socket.py", line 89, in handle_get_request
    start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/socket.py", line 130, in _upgrade_websocket
    return ws(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/async_gevent_uwsgi.py", line 35, in __call__
    uwsgi.websocket_handshake()
IOError: unable to complete websocket handshake
[pid: 28266|app: 0|req: 7/7] 10.11.201.82 () {50 vars in 1028 bytes} [Mon Feb  5 11:33:34 2018] GET /socket.io/?EIO=3&transport=websocket&sid=ffb2449454b34644959fa113f606c6c3 => generated 0 bytes in 1 msecs (HTTP/1.1 500) 0 headers in 0 bytes (1 switches on core 999)
[pid: 28266|app: 0|req: 8/8] 10.11.201.82 () {38 vars in 782 bytes} [Mon Feb  5 11:33:34 2018] GET /socket.io/?EIO=3&transport=polling&t=1517808814120-2&sid=ffb2449454b34644959fa113f606c6c3 => generated 68 bytes in 0 msecs (HTTP/1.1 200) 3 headers in 131 bytes (3 switches on core 999)
[pid: 28266|app: 0|req: 9/9] 10.11.201.82 () {36 vars in 609 bytes} [Mon Feb  5 11:33:34 2018] GET /favicon.ico => generated 233 bytes in 1 msecs (HTTP/1.1 404) 2 headers in 72 bytes (3 switches on core 999)
[pid: 28266|app: 0|req: 10/10] 10.11.201.82 () {42 vars in 843 bytes} [Mon Feb  5 11:33:34 2018] POST /socket.io/?EIO=3&transport=polling&t=1517808814152-3&sid=ffb2449454b34644959fa113f606c6c3 => generated 2 bytes in 0 msecs (HTTP/1.1 200) 3 headers in 117 bytes (3 switches on core 999)
('query:--', 'hi')
session_id:-- 1
[pid: 28266|app: 0|req: 12/11] 10.11.201.82 () {42 vars in 843 bytes} [Mon Feb  5 11:33:37 2018] POST /socket.io/?EIO=3&transport=polling&t=1517808817314-5&sid=ffb2449454b34644959fa113f606c6c3 => generated 2 bytes in 1316 msecs (HTTP/1.1 200) 3 headers in 117 bytes (3 switches on core 998)
[pid: 28266|app: 0|req: 12/12] 10.11.201.82 () {38 vars in 782 bytes} [Mon Feb  5 11:33:34 2018] GET /socket.io/?EIO=3&transport=polling&t=1517808814153-4&sid=ffb2449454b34644959fa113f606c6c3 => generated 76 bytes in 4481 msecs (HTTP/1.1 200) 3 headers in 131 bytes (3 switches on core 999)
('query:--', 'how are you')
session_id:-- 1

So, could you please give me some ideas about the above two errors. I don't even know why they are occurring ? what is the root cause of them ? Without properly understanding the errors I think I couldn't be able to solve the errors myself.

I have searched for IOError:unable to complete websocket handshake and IOError: Write Error but what I'm getting is out of context of Flask-SocketIO. So, currently I am totally helpless unless I could grasp proper reasoning of those errors.

[miguelgrinberg]

Socket.IO can work over HTTP as well as WebSocket. The fact that the application works does not mean websocket is working. The websocket error that you are getting likely indicates that the client cannot use websocket. But I wanted to know if the connection fails every time or only occasionally, or maybe for specific clients.

The error indicates that the connection handshake between uWSGI and the client could not complete. This error is not a SocketIO error, it comes from uWSGI, so you need to find out what some or all of your clients fail to establish a websocket connection with uWSGI.

My guess is that there is a problem with your uWSGI server, specifically in its websocket part. You may want to find a simpler websocket example for uWSGI and see if you can make it work, my guess is that you are still going to see this error.

[Rakin061]

@miguelgrinberg .. I got the point you described

Yes, from the error log I shared it's clearly seen that

generated 0 bytes in 1 msecs (HTTP/1.1 500) 0 headers in 0 bytes (1 switches on core 999)

HTTP has been used.

But I wanted to know if the connection fails every time or only occasionally, or maybe for specific clients.
-- Yes, the connection fails occasionally for specific client. For, some clients I got no errors.

So, how can I switch to work SocketIO from HTTP to WebSocket ?? Is it to be handled in coding section or in the uwsgi configuration ??

Here's the header of one the requests from client. But I couldn't figure out from there whether HTTP or WebSocket has been used.

[miguelgrinberg]

Your client is using HTTP. For WebSocket there are no requests constantly being issued, it's a long term connection.

[Rakin061]

@miguelgrinberg Yes, I mentioned it in my comment as well.

But, how can I make SocketIO switch from HTTP to WebSocket for all available clients ?? Is it to be handled in coding section or in the uwsgi configuration ??

[miguelgrinberg]

It happens automatically. The handshake error prevents the websocket connection from taking over HTTP. As I said above, it could be a uwsgi problem, look for a simpler websocket example and make sure it works before you continue troubleshooting, because my guess is that the problem is at a more basic level than SocketIO.

[Rakin061]

@miguelgrinberg

Yes. It's confirmed that client is connected over HTTP.

So, now I have forgot about my project. I just tested the Flask-SocketIO example which You posted in Your blog tutorial and run app.py with uwsgi. And the similar error IOError:unable to complete websocket handshake appears here as well.

As same error is appearing regardless of different projects it's clear that problem underlies with my uwsgi configuration or in my ubuntu server . After digging out more what I got is:

1) I might build uwsgi with SSL support to use the websocket hankshake api function. May be ssl-certbot will do that for me
2) There could be an issue with my Nginx version (1.12.2). I noticed in your official documentation--

 > However, only releases of nginx 1.4 and newer support proxying of the WebSocket protocol.

So, upgrading my nginx version could help.

Anyway, many thanks for your constant support.

[miguelgrinberg]

Your nginx is fine (1.12 is newer than 1.4). But you may want to try to build uwsgi with ssl and websocket support if you haven't done that already.

[Rakin061]

Dear @miguelgrinberg,

As you already know I have been using uWSGI+ gevent and ssl support for uwsgi seems different and cumbersome for gevent rather than eventlet.

So, I have decided to perform the SSL verification part with Nginx for HTTPS support. Here's my Nginx configuration:-

server {
    listen 80;
    server_name 10.11.201.82:5003;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:///home/user/uwsgi_Test/myproject1.sock;
        uwsgi_ignore_client_abort on;
    }

    location /socket.io {
        include proxy_params;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass http://unix:///home/user/uwsgi_Test/myproject1.sock;
    }

}
server {
        listen 443 ssl;
        ssl_certificate nginx.crt ;
        ssl_certificate_key nginx.key;
        keepalive_timeout  1800;
        server_name 10.11.201.82;

        location / {

                include uwsgi_params;
                uwsgi_pass unix:///home/user/uwsgi_Test/myproject1.sock;
                uwsgi_ignore_client_abort on;
        }

        location /socket.io {
                include proxy_params;
                proxy_http_version 1.1;
                proxy_buffering off;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";
                proxy_pass http://unix:///home/user/uwsgi_Test/myproject1.sock;
    }
}

It works pretty fine allowing clients to use https with port 443. But in the console I am getting the following error:

Mixed Content: The page at 'https://10.11.201.82/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://10.11.201.82/socket.io/?EIO=3&transport=polling&t=1518670604216-0'. This request has been blocked; the content must be served over HTTPS.

It seems that socke.io has been blocked while transferring contents using websockts.

Could you please give me any light to resolve this issue ?

PS:- I have tried with localhost as I found Firefox allow mixed content for localhost ... but still same error is occurring. So, that might be the socket.io configuration problem.

[miguelgrinberg]

The JavaScript connection URL for SocketIO should be https:// not http://.

[Rakin061]

I have been used relative protocol caller for the scripts... Hence I will analyze once again. Thanks!!

[Rakin061]

@miguelgrinberg

Many thanks. No errors now in browser console and network tab.
But now Nginx giving me an error:-

2018/02/15 12:46:15 [error] 10223#10223: *6 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "GET /socket.io/?EIO=3&transport=polling&t=1518677145654-8 HTTP/1.1", upstream: "http://unix:///home/user/uwsgi_Test/myproject1.sock:/socket.io/?EIO=3&transport=polling&t=1518677145654-8", host: "localhost", referrer: "https://localhost/"

I am sensing the error arises because of Unix Socket. So, is there any hindrance using Unix Socket instead of http-socket in Flask-SocketIO ??

[miguelgrinberg]

Does the error go away if you switch to a regular socket? This is saying that the server closed the connection with nginx prematurely, it has nothing to do with SocketIO or WebSocket.

[Rakin061]

@miguelgrinberg
Now, no error in the uwsgi as well as Nginx. But getting following 2 errors in the browser network tab:-

socket.io.min.js:1 GET https://10.11.201.82/socket.io/?EIO=3&transport=polling&t=1518686771064-3 400 (Bad Request)

socket.io.min.js:1 GET https://10.11.201.82/socket.io/?EIO=3&transport=polling&t=1518686791640-7 502 (Bad Gateway)

Clicking on the socket.io link I am seeing

400 Bad Request
Request Header Or Cookie Too Large

My Nginx config of the ssl part once again

server {
        listen 443 ssl;
        ssl_certificate nginx.crt ;
        ssl_certificate_key nginx.key;
        keepalive_timeout  1800;
        server_name 10.11.201.82;

        location / {

                include uwsgi_params;
                uwsgi_pass 10.11.201.82:5003;
                uwsgi_ignore_client_abort on;
        }

        location /socket.io {
                include proxy_params;
                proxy_http_version 1.1;
                proxy_buffering off;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";
                proxy_pass http://10.11.201.82/socket.io;
 }
}

[miguelgrinberg]

There are a couple of things that are odd in your nginx config:

• The server_name should be a name, or a _ if you don't care about giving the server a name. It should not be an IP address.
• The proxy_pass directive in the /socket.io location block does not have a port number.

Hopefully fixing these will address your problem.

[Rakin061]

Hello @miguelgrinberg,

All the errors I addressed regarding this project has been solved. I can now successfully send requests and getting back responses through websokcet. I have used self signed certificate for https support. Now, everything is working just fine.

Onemore thing , as far I understand that socket.io is actually a wrapper over original flask applications. So, I could expect similar behaviors regardless of which server I will be using.

I have already informed that I have been using uwsgi with gevent as asynchronous.

So, when I have been passing certfile and keyfile directly to socketio.run() then it's working fine. Here's the server.py file's socketio.run section

if __name__ == '__main__':
  socketio.run(app, host='0.0.0.0', port=5003, debug=True, certfile='foobar.crt', keyfile='foobar.key')

So, running the server.py directly from virtual environment with python server.py is invoking the default flask development server and working perfectly.

But, when I am running the same server.py file with uwsgi server with gevent with the following command

uwsgi --master --https 0.0.0.0:5003,foobar.crt,foobar.key --http-websockets --gevent 1000 --wsgi-file server.py --callable app

then a certification error occurred.

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

That seems surprising to me as I am passing the same certificated files (foobar.crt, foobar.key), same port, but when I have switched to uwsgi server I am getting the error.

[miguelgrinberg]

Did you also post this question in Stack Overflow? https://stackoverflow.com/questions/48861768/ssl-sslerror-ssl-certificate-verify-failed-certificate-verify-failed-ssl-c

I just answered there, the problem is not with uwsgi, but with the python requests library.

[Rakin061]

Yes, but as I have been using dialogflow's request mechanism, I have no control over their request mechanism. So, I can't do requests.get(url, verify=False) as you described.

In another post, I described more details about it https://stackoverflow.com/questions/48878420/how-to-disable-ssl-verification-for-text-request-of-api-ai-dialogflow

[Rakin061]

And, as you described the problem is not in the chat_message() section because working with HTTPS it only requires the crt and key file.
As I mentioned, when I have been passing the certificate files as arguments in socketio.run

if __name__ == '__main__':
  socketio.run(app, host='0.0.0.0', port=5003, debug=True, certfile='foobar.crt', keyfile='foobar.key')

it's working fine.

But when I am serving it with uWsgi with the certificate files like that

uwsgi --master --https 0.0.0.0:5003,foobar.crt,foobar.key --http-websockets --gevent 1000 --wsgi-file server.py --callable app

then control has been shifted to uwsgi and as a result the chat_message() function resides in the server.py file can't find those certificate files and resulting the error.

So, is there any way to pass the certificate files concurrently to socketio.run() ?

[miguelgrinberg]

I'm very confused. The way I understand the flow of requests in your system is as follows:

• you have a server hosted on uWSGI with a self-signed certificate
• You connect to this server from a browser, and that works well (aside from the browser not being able to validate the certificate)
• The URL that you request installs an application in the browser, which connects to your server over Socket.IO
• The application in the browser then sends a Socket,IO event to the server, and the handler for this event issues an HTTP request to another server, which is also using a self-signed certificate.
• Validation on this HTTP request fails.

Why do you keep insisting that this has something to do with your own uWSGI server, given that the request that fails to validate is the HTTP request that is sent in the Socket.IO event handler? Is this request recursively connecting to itself?

[Rakin061]

I am also confused. I am trying to describe as far I understand.

The application in the browser then sends a Socket,IO event to the server, and the handler for this event issues an HTTP request to another server, which is also using a self-signed certificate.

I think that another server (dialogflow) is using the self-signed certificate which I passed in the socketio.run() method as arguments. Running the application with python server.py and using https in the browser also indicates that.

But, when uwsgi has been used, then socketio.run() has become void (As far my understanding reading the Flask-SocketIO official doc. Pardon me if I'm not correct) and the dialogflow server can't find the self signed certificate files as parameters.

Validation on this HTTP request fails.

Yes, validation failed only when client start chatting in the browser. I'm just getting user queries from client side and just sending the queries to the server side to get a response from the dialogflow server. That's it. So, when https is used in the browser most probably dialogflow server couldn't find those certificate files.

[miguelgrinberg]

You are not really explaining how your application works. I have no idea what this other digiflow server is or who runs it. I don't see how it could "steal" your own self-signed certificate and use it for itself. I think you need to provide a better explanation of how your application is designed, I really can't help you with the information you provided.

[Rakin061]

Okay. I don't know whether the dialogflow is stealing my self-signed certificate or not.
But, the thing is When I'm providing the self-signed certificate as arguments in the socketio.run() it's working, and when I removed the certificates it stopped working magically.

I know you're so busy, but still if you really wanna see that magic, I could connect you to my server. Thanks.

[miguelgrinberg]

I'm really not interested in your server. As far as I can see that works well. The dialogflow server has the problem with an invalid certificate. Who runs that server? Let them know that they need to fix their certificate and that would end your problems.

[Rakin061]

Dialogflow is a NLP Machine Learning Platform owned by Google.

And what I'm using is Dialogflow's Python Client.

[miguelgrinberg]

As I said above, the problem is not in your code, but in the third-party server you are contacting. You need to let them know that they are using an invalid certificate, there is nothing else you can do. See https://discuss.api.ai/t/api-api-ai-certificate-error/1457, there is a bunch of people complaining about the same thing.

Your only solution is to modify their Python client to send the request with verification turned off.

[Rakin061]

Ok. But my question is why it's working when passing certificate file as a argument in socketio.run()

Isn't that surprising and illogical ? ?

I already created an issue in their github. :(

Your only solution is to modify their Python client to send the request with verification turned off.

Yes, exactly with this thoughts I created the issue and post a question on stack overflow .

[miguelgrinberg]

But my question is why it's working when passing certificate file as a argument in socketio.run()

Something else must be different when you run the server in these two modes. Either a different version of Python, any other package, etc.

[Rakin061]

Hello @miguelgrinberg

The problem has been solved. I tried to make it work with _HTTP connection class_ but it doesn't help.

Then, I analyzed Dialogflow's root request mechanism and found they're using _httplib.HTTPSConnection()_ as the connection class and HTTPConnection.putrequest() function for actually preparing the request.
From the official doc of _HTTP PROTOCLO CLIENT_ I got:-

class httplib.HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, source_address[, context]]]]]]])
A subclass of HTTPConnection that uses SSL for communication with secure servers. Default port is 443. If context is specified, it must be a ssl.SSLContext instance describing the various SSL options.
This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior ssl._create_unverified_context() can be passed to the context parameter.

I just extended the request.py of their python client to use the ssl certificate files in the server side which I'm providing with the uWSGI as parameters. Now, the python client will get complete HTTPS support.

And, A great many thanks to You for your constant support and immediate responses all along.

So, now the original error including all the sub-errors regarding this issue has been solved. I feel certain that it's time I could close the issue.
Thanks again !!!

[felocru]

Hi, I was following the whole conversation.
There was a small message in one of the logs. Specifically, the comment of February 4.
The message is: "you need to build uWSGI with SSL support to use the websocket handshake api function !!"
This message is from uwsgi. My question is, is SSL necessarily needed for websocket to work?

I am presenting exactly the same inconvenience.
Thank you

[miguelgrinberg]

@felocru Oh, I missed that message above. In this context "SSL support" means the SSL libraries, which uWSGI must be using to handle some parts of the websocket handshake. It does not mean that WebSocket only works on secure HTTP, those are two completely different things. But yes, try rebuilding uWSGI after installing SSL libs.

[acnebs]

I am getting this same error (you need to build uWSGI with SSL support to use the websocket handshake api function !!) on Ubuntu with Nginx + uWSGI + Gevent. However, I don't see any reason why uWSGI wouldn't have SSL support.

I'm installing from the binary via pip install uwsgi on python3.7, and I've installed the SSL headers just in case apt-get install libssl-dev.

Is there a way to check if uwsgi has SSL support? I couldn't find anything in the docs.

I'm also using uwsgi from within a virtualenv, which from somewhere I've read by the problem, but I tried install uwsgi globally as well and that didn't seem to help.

[Rakin061]

Hello,

No, it will not be enough to install uWSGI and SSL separately for running the application perfectly with uWSGI !
Rather, you have to build the uWSGI while installing with SSL support !!

This guide will lead you to the right direction all along !!

Best of luck !!