Flask-socketio: Error during WebSocket handshake: Unexpected response code: 400 when long polling

[whenchyiv]

This is a weird one that has shown up recently. Flask-Socketio has been working great (🙌), but recently when the client is polling before upgrading to a websocket connection the server often (though not always) returns an error 400. After upgrading to a socket connection everything works fine, though if a client fails to upgrade it sometimes sees this problem consistently.

I'm seeing this in production on Heroku. I cannot replicate it locally, and haven't been able to find an obvious cause by downgrading/upgrading dependencies, making code changes, etc.

The relevant dependencies are:

eventlet==0.23.0
Flask==1.0.2
gunicorn==19.8.1
python-engineio==2.1.1
python-socketio==1.9.0
redis==2.10.6

JS Client v2.1.1

Procfile:

web: newrelic-admin run-program gunicorn --worker-class eventlet -w 1 app:app

Issue occurs in both Chrome and Firefox.

Any ideas? I've tried a number of configurations and can't isolate the issue, so I'm wondering if it's possibly a known issue with Socketio that has a workaround? The recommended solution I found on the Socketio repo (forcing {transports: ['websocket']} via the client config) didn't resolve the issue.

Thanks!

[miguelgrinberg]

Did you check your logs?

[whenchyiv]

I have, though I'm not seeing the server throwing any errors, nor is the Flask-Socketio error handler logging anything. I didn't enable logging generally for Socketio, though. Completely forgot that is disabled by default. 🙈

Will enable that and report back - sorry I didn't think of that before opening the issue, yikes!

[Hendrikto]

How did you deploy the app? I had a similar problem which was caused by using uWSGI + NGINX. Sometimes it would work, sometimes it would not. I also did not get any error except for something like "unexpected response code 400" on the client. Nothing on the server. After some googling, uWSGI does not seem to properly support socketio.

I switched my deployment over to using the eventlet server. Now everything works all the time. Maybe you can try that if you still have the problem.

[whenchyiv]

@Hendrikto that's really interesting - thanks for chiming in. I'm already using Gunicorn's eventlet workers, so I don't think that's it, but I wonder if Heroku's router is giving me some trouble.

The application works, and eventually the client connects successfully, but it introduces an unfortunate delay sometimes.

[whenchyiv]

Following up on this with some new information:

• The 400 errors are resolved by forcing Socketio to use websockets only with {transports: ['websocket']}. This issue only occurs when polling.
• It doesn't happen consistently. The 400 errors had completely resolved themselves for the past month or so with no code changes, and have now returned. I suspect a config issue along the lines of what @Hendrikto mentioned, rather than an issue with either Flask-Socketio or our app. The app is running on the Heroku-16 stack, if that helps.

Any ideas? I'm at a bit of a loss. Happy to test out any recommended changes and report back. I don't think this is a Flask-Socketio issue, though, so we can close this if you don't have any thoughts.

Thanks so much, and thank you all of your work on Flask-Socketio - it's really fantastic!

[miguelgrinberg]

@whenchyiv what command do you use to start gunicorn? You have to constrain your server to a single worker for things to work, this is a limitation in gunicorn that does not support sticky sessions.

[whenchyiv]

@miguelgrinberg web: newrelic-admin run-program gunicorn --worker-class eventlet -w 1 main:app

We also have session affinity enabled for Heroku, since we're using more than one dyno.

[miguelgrinberg]

Well, I can tell you with 100% certainty that if the sticky sessions don't work well you'll get 400s. Have you verified that Heroku implements this properly? Dynos are supposed to come and go, not sure how they can reliably support session affinity, maybe that's where the problem is.

[whenchyiv]

I believe they do, but I can confirm with support. This support article goes over their implementation. It does sound like that may be the issue, if session affinity failing results in a 400 error. Websockets (rather than polling) work flawlessly, though.

[miguelgrinberg]

Did you see this?

If 4 dynos are each holding 10 connections and a 5th dyno is added, you should therefore expect roughly 2 connections (on average) from each existing dynos to be shifted to the 5th dyno to make sure all resources are used equally.

Each time you scale up or down it is quite possible you'll get a bunch of errors due to them reassigning some clients.

[whenchyiv]

I've noticed that, though this is being seen while the number of dynos is static during the initial client connection. We don't have any autoscaling enabled, so that shouldn't be causing the errors. I wouldn't be surprised if there was something with Heroku's implementation that might be the errors, though.

[nayebare]

Help needed.
This is my error in Chrome: Error during WebSocket handshake: Unexpected response code: 400
And in Firefox :The connection to ws://127.0.0.1:5000/socket.io/?EIO=3&transport=websocket was interrupted while the page was loading.

This is my nginx server:

server {
location / {
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}

    location /static/ {
    autoindex on;
    root /home/nayebare/Desktop/inconnectpoint/flaskapp/clientsfrontend/build
}
   location /socket.io {
    include proxy_params;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_pass http://localhost:5000/socket.io;
}

}

Here is my flask app:
from flask import Flask

pip2 install flask-mysqldb version being used

from flask_mysqldb import MySQL
from flask_cors import CORS
from flask_socketio import SocketIO, emit, join_room
import logging
from producer import *

app = Flask(__name__, static_folder='../clientsfrontend/build/static')

handling gunicorn and flask logs

gunicorn_error_logger = logging.getLogger('gunicorn.error')
app.logger.handlers.extend(gunicorn_error_logger.handlers)
app.logger.setLevel(logging.DEBUG)
file_handler = logging.FileHandler('errorlog.txt')
app.logger.addHandler(file_handler)
app.logger.debug('this will show in the log')

app.config.from_pyfile('config.py')
mysql = MySQL(app)
socketio = SocketIO(app)
CORS(app, resources={r"/": {"origins": ""}})

from views import *

@socketio.on('connect')
def on_connect():
print('user connected')

@socketio.on('stream_data', namespace='/socketio.io')
def test_message(message):
emit('stream_data', {'data': message['data']}, broadcast=True)

run main flask application

if __name__ == '__main__':
socketio.run(app, debug=True, host='0.0.0.0', port='5000')

And here how i am running the app using uwsgi:
uwsgi --http :5000 --gevent 1000 --socket --master --wsgi-file app.py --callable app
And this the output:
[pid: 14500|app: 0|req: 1/1] 127.0.0.1 () {46 vars in 831 bytes} [Fri Mar 20 15:53:18 2020] GET /socket.io/?EIO=3&transport=websocket => generated 11 bytes in 1 msecs (HTTP/1.1 400) 1 headers in 54 bytes (3 switches on core 999)

When running the app using nginx+gunicorn using this command
gunicorn app:app --worker-class eventlet -w 1 --bind 0.0.0.0:5000 --log-file logs/gunicorn.log --log-level DEBUG --reload
There is no communication from the client

Here is the client code:
import io from 'socket.io-client'
const socket = io('http://127.0.0.1:5000',{'transports': ['websocket']})

Here is my pip list
certifi 2019.11.28
chardet 3.0.4
click 7.1.1
dnspython 1.16.0
eventlet 0.25.1
Flask 1.1.1
Flask-Cors 3.0.8
Flask-JWT 0.3.2
Flask-MySQLdb 0.2.0
Flask-SocketIO 4.2.1
gevent 1.4.0
greenlet 0.4.15
gunicorn 20.0.4
idna 2.9
itsdangerous 1.1.0
Jinja2 2.11.1
kafka-python 2.0.1
MarkupSafe 1.1.1
monotonic 1.5
mysqlclient 1.4.6
pip 20.0.2
pkg-resources 0.0.0
PyJWT 1.4.2
python-engineio 3.12.1
python-socketio 4.4.0
requests 2.23.0
setuptools 46.0.0
six 1.14.0
urllib3 1.25.8
uWSGI 2.0.18
Werkzeug 1.0.0
wheel 0.34.2

Any help is appreciated:

[miguelgrinberg]

Could you be having CORS problems? If you need CORS for your Flask routes, you also need it for Socket.IO. Try this:

socketio = SocketIO(app, cors_allowed_origins=['origin1', 'origin2', ...])

[brobey8]

@miguelgrinberg I'm a new user of this library and my issue was solved with your answer @miguelgrinberg thank you!