Flask-socketio: Issues with socket connection in production using Gunicorn/Eventlet behind Nginx with SSL

[dhhagan]

After successfully implementing flask-socketio locally, I have run into issues in production which is most likely associated with nginx config and ssl (and me not knowing what I'm doing). I have followed the example here, but am receiving two issues in the console:

[1] POST https://[my-url]/socket.io/?EIO=3&transport=polling&t=LsS2rV6&sid=ae355db134324f8da26321a9d6086f25 400 (BAD REQUEST)

[2] WebSocket connection to 'wss://tatacenter-airquality.mit.edu/socket.io/?EIO=3&transport=websocket&sid=ae355db134324f8da26321a9d6086f25' failed: WebSocket is closed before the connection is established.

[3] GET https://[my-url]/socket.io/?EIO=3&transport=polling&t=LsS2rVD&sid=ae355db134324f8da26321a9d6086f25 400 (BAD REQUEST)

Relevant config files are below:

site.conf

...
exec gunicorn -w 1 --worker-class eventlet --timeout 360 --bind 127.0.0.1:8000 wsgi:application

nginx/sites-available/[site]

server {
    server_name [url];
    listen *:80;
    listen [::]*:80;

    return 301 https://$host$remote_uri;
}

upstream socket_nodes {
    ip_hash;

    server 127.0.0.1:8000;
}

server {
    server_name [url];

    ssl_certificate [ssl-cert.chained.crt];
    ssl_certificate_key [ssl-cert.key];

    listen *:443 ssl;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log debug;

    location / {
        proxy_pass http://socket_nodes;
        proxy_redirect off;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /socket.io {
        proxy_pass http://socket_nodes/socket.io;
        proxy_http_version 1.1;
        proxy_redirect off;
        proxy_buffering off;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }

    location ^~ /static/ {
        root /var/www/link-to-site;
    }

    location =/favicon.ico {
        root /var/www/link-to-favicon.ico;
    }
}

Requirements.txt

alembic (0.9.2)
awscli (1.11.36)
blinker (1.4)
boto3 (1.4.3)
botocore (1.4.93)
cffi (1.2.1)
click (6.7)
colorama (0.3.7)
coverage (4.4.1)
cryptography (1.0.1)
docopt (0.4.0)
docutils (0.13.1)
enum34 (1.0.4)
eventlet (0.17.4)
Flask (0.12.2)
Flask-Caching (1.3.2)
Flask-Login (0.4.0)
Flask-Markdown (0.3)
Flask-Migrate (2.0.4)
Flask-Moment (0.5.0)
Flask-PageDown (0.2.1)
Flask-Script (2.0.5)
Flask-SocketIO (2.9.1)
Flask-SQLAlchemy (2.2)
Flask-WTF (0.14.2)
funcsigs (1.0.2)
futures (3.0.3)
gunicorn (18.0)
idna (2.0)
ipaddress (1.0.14)
itsdangerous (0.24)
Jinja2 (2.9.6)
jmespath (0.9.0)
loggly-python-handler (1.0.0)
Mako (1.0.6)
mandrill (1.0.57)
Markdown (2.6.2)
MarkupSafe (1.0)
MySQL-python (1.2.5)
ndg-httpsclient (0.4.0)
nltk (3.2.2)
numpy (1.11.3)
onetimepass (1.0.1)
pandas (0.19.2)
pip (9.0.1)
pyasn1 (0.1.8)
pycountry (17.1.2)

pycparser (2.14)
pyOpenSSL (0.15.1)
PyQRCode (1.1)
python-dateutil (2.6.0)
python-editor (1.0.3)
python-engineio (1.7.0)
python-http-client (1.2.3)
python-socketio (1.8.0)
pytz (2016.10)
PyYAML (3.12)
requests (2.7.0)
requests-futures (0.9.5)
rsa (3.4.2)
s3transfer (0.1.10)
sendgrid (2.2.1)
setuptools (18.0.1)
six (1.10.0)
smtpapi (0.3.1)
SQLAlchemy (1.1.11)
textblob (0.11.1)
tzlocal (1.3)
unpyclib (0.8.1)
Werkzeug (0.12.2)
wheel (0.24.0)
WTForms (2.1)

Any thoughts or recomendations on debugging are greatly appreciated!

[miguelgrinberg]

The 400 error should log more information if you pass engineio_logger=True to your SocketIO constructor. Maybe that gives us a clue.

[dhhagan]

Ahh okay. I see the following in the log file after doing that:

Now, rather than seeing a continuous feed of errors in the console, they only appear once. After about a minute, I receive a 504 gateway Time-out.

[miguelgrinberg]

The 504 status code means that nginx timed out waiting for the application to respond. Does the nginx log tell you which route is timing out? This could be one of your HTTP routes, I don't think this would be for a WebSocket route.

[dhhagan]

Yea, it seems like it is the websocket...which doesn't make sense.

I think my config is okay..but obviously not..

[miguelgrinberg]

Do the suggestions on this issue help in your case?

[dhhagan]

It fixed the 504 issue, but not the others. I assume it's something with eventlet? Not sure though. Here are the errors the console spits out. I'm using sockets to grab data from the server without having to re-load for each new (in the future) data point. Sometimes it gives me a few points while still spitting out all these errors, but often it results in nothing.

[miguelgrinberg]

You need to look in the server logs carefully to find other indications of problems. My guess is that the 400s are because the session ids were disconnected.

[dhhagan]

Yea, I've been searching through, and the only thing I can find is that I continuously get an "Invalid Session" error, but don't know why. I think I have debugging messages logged everywhere, but it doesn't seem to be too informative.

Every now and again, it sort of works..but mostly just results in tons of these errors.

Have you seen this error before or know where I can find more information on it?

[dhhagan]

After about 30 min of letting it sit (and seeing invalid session errors), it began working... I guess that means the config is okay, but somehow am seeing all these invalid sessions which I don't know the cause of..hmm.

[dhhagan]

I think I finally found the issue!

I had an ill-configured ghost blog in a separate block. As soon as it was commented out..websockets worked! Not sure how to configure it correctly, but that's okay.

Thanks!

[tonysepia]

@dhhagan

I had an ill-configured ghost blog in a separate block. As soon as it was commented out..websockets worked!

Could you please explain exactly what was that 'ghost blog'? What did you change? I am getting the same problem