Flask-socketio: flask-socketio blocking REST calls

[mtShaikh]

Hi Miguel, great library! I have a flask server which facilitates both socketio and rest clients. My problem is that whenever a client connects to my server the server starts to block all rest calls until the client disconnects. This behaviour only occurs on my ec2 instance which runs ubuntu 18.04. I'm using eventlet and gunicorn to run the server behind nginx. I've searched through all the issues here and on the web and unfortunately couldn't find a solution.

The server logs

[2020-09-10 09:42:09 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Sending packet OPEN data {'sid': '9a5ec071ee39465a9e990ae2ffd50eb3', 'upgrades': [], 'pingTimeout': 60000, 'pingInterval': 25000}
[2020-09-10 09:42:09 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Sending packet MESSAGE data 0
[2020-09-10 09:42:09 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Received request to upgrade to websocket
[2020-09-10 09:42:09 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Upgrade to websocket successful
[2020-09-10 09:42:10 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Received packet MESSAGE data 0/caller?profileId=2,
[2020-09-10 09:42:10 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Sending packet MESSAGE data 0/caller
[2020-09-10 09:42:23 +0000] [28639] [INFO] 9a5ec071ee39465a9e990ae2ffd50eb3: Received packet MESSAGE data 1/caller,
[2020-09-10 09:42:23 +0000] [28639] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 134, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 190, in handle_request
    util.reraise(*sys.exc_info())
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
    raise value
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 182, in handle_request
    resp.close()
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 402, in close
    self.send_headers()
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 322, in send_headers
    util.write(self.sock, util.to_bytestring(header_str, "latin-1"))
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/gunicorn/util.py", line 286, in write
    sock.sendall(data)
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/eventlet/greenio/base.py", line 402, in sendall
    tail = self.send(data, flags)
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/eventlet/greenio/base.py", line 396, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/ubuntu/.local/share/virtualenvs/api-development-zGzOrktH/lib/python3.7/site-packages/eventlet/greenio/base.py", line 383, in _send_loop
    return send_method(data, *args)
OSError: [Errno 9] Bad file descriptor
[2020-09-10 09:42:23 +0000] [28639] [INFO] [2020-Sep-10 09:42] 127.0.0.1 GET https /api/test? 200 OK

gunicorn conf

gunicorn --access-logfile - --bind 127.0.0.1:8081 --timeout 1200 -k eventlet -w 1 run:app

run.py module

import eventlet

eventlet.patcher.monkey_patch()
import logging.config
import os

from dotenv import load_dotenv


from app import create_app, socketio

dotenv_path = os.path.join(os.path.dirname(__file__), ".env")
load_dotenv(dotenv_path)
gunicorn_logger = logging.getLogger("gunicorn.error")
app = create_app(os.getenv("FLASK_CONFIG"), gunicorn_logger)

app.app_context().push()


if __name__ == "__main__":

    socketio.run(app, log_output=True)

requirements.txt

alembic==1.4.2
amqp==2.6.1
aniso8601==8.0.0
apispec==3.3.2
apscheduler==3.6.3
attrs==20.2.0
bcrypt==3.2.0
billiard==3.6.3.0
blinker==1.4
boto3==1.14.58
botocore==1.17.58
cachetools==4.1.1
celery==4.4.7
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
click==7.1.2
dnspython==1.16.0
docutils==0.15.2
eventlet==0.27.0
flask-apispec==0.10.0
flask-apscheduler==1.11.0
flask-bcrypt==0.7.1
flask-cors==3.0.9
flask-jwt-extended==3.24.1
flask-mail==0.9.1
flask-marshmallow==0.13.0
flask-migrate==2.5.3
flask-redis==0.4.0
flask-rest-paginate==0.1.5
flask-restful==0.3.8
flask-rollbar==1.0.1
flask-socketio==4.3.1
flask-sqlalchemy==2.4.4
flask==1.1.2
google-auth==1.21.1
greenlet==0.4.16
gunicorn==20.0.4
idna==2.10
importlib-metadata==1.7.0; python_version < '3.8'
itsdangerous==1.1.0
jinja2==2.11.2
jmespath==0.10.0
jsonschema==3.2.0
kombu==4.6.11
mako==1.1.3
markupsafe==1.1.1
marshmallow-sqlalchemy==0.23.1
marshmallow==3.7.1
monotonic==1.5
psycopg2-binary==2.8.6
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycparser==2.20
pyjwt==1.7.1
pyrsistent==0.17.2
python-dateutil==2.8.1
python-dotenv==0.14.0
python-editor==1.0.4
python-engineio==3.13.2
python-socketio==4.6.0
pytz==2020.1
pyyaml==5.3.1
redis==3.5.3
requests==2.24.0
rollbar==0.15.0
rsa==4.6; python_version >= '3.5'
s3transfer==0.3.3
six==1.15.0
sqlalchemy-utils==0.36.8
sqlalchemy==1.3.19
twilio==6.45.1
tzlocal==2.1
urllib3==1.25.10
vine==1.3.0
webargs==6.1.1
werkzeug==1.0.1
zipp==3.1.0

nginx.conf (don't know if it's an nginx issue or not because a curl request from within the ec2 instance also blocks)

server {
        listen 443;
        server_name _;

        include snippets/ssl.conf;
        include snippets/certs/domainname;

        location / {
                include proxy_params;
                proxy_pass http://127.0.0.1:8081/;
        }

        location /socket.io {
                include proxy_params;
                proxy_pass http://127.0.0.1:8081/socket.io;
                proxy_http_version 1.1;
                proxy_buffering off;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'Upgrade';
                proxy_set_header Host $host;
        }

        ssl_certificate /path_to_certificate; # managed by Certbot
        ssl_certificate_key /path_to_key; # managed by Certbot
}

the route that blocks

@ns.route("test")  # this is just a decorator for api.add_resource()
class TestController(BaseController):
    def get(self):
        return jsonify(dict(status=200))

The client im using to test
index.js

const socketUrl = 'https://apidev.domain.com/caller';
// const socketUrl = 'http://localhost:8000/caller';
let connectButton;
let disconnectButton;
let socket;
let statusInput;
let tokenInput;

const connect = () => {
  let error = null;

  socket = io(socketUrl, {
    query: {
        "profileId": 2
    },
    reconnection: true,
    transports: ['websocket']
});

  socket.on('connect', () => {
    console.log('Connected');
    statusInput.value = 'Connected';
    connectButton.disabled = true;
    disconnectButton.disabled = false;
  });


  socket.on('disconnect', (reason) => {
    console.log(`Disconnected: ${error || reason}`);
    statusInput.value = `Disconnected: ${error || reason}`;
    connectButton.disabled = false;
    disconnectButton.disabled = true;
    error = null;
  });
  socket.on('error', (error) => {
    console.log("error",error)
  })

  // socket.open();
};

const disconnect = () => {
  socket.disconnect();
}

document.addEventListener('DOMContentLoaded', () => {
  connectButton = document.getElementById('connect');
  disconnectButton = document.getElementById('disconnect');
  statusInput = document.getElementById('status');
  tokenInput = document.getElementById('token');
});

index.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>Single User Websocket</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.2.0/socket.io.js"></script>
  <script src="index.js"></script>
</head>
<body>
  <h1>Single User Websocket Demo</h1>
  <p>
    <label for="status">Status: </label>
    <input type="text" id="status"
      name="status" value="Disconnected"
      readonly="readonly" style="width: 300px;"
    />
  </p>
  <p>
    <label for="token">My Token: </label>
    <input type="text" id="token" name="token" value="secret token" />
  </p>
  <p>
    <button id="connect" onclick="connect()">
      Connect
    </button>
    <button id="disconnect" onclick="disconnect()" disabled>
      Disconnect
    </button>
  </p>
</body>
</html>

I have also tried using gevent and gevent-websockets too but it still causes the same issue on the instance
I've been at this for the past two days now. Will appreciate some direction
Thanks!!

[miguelgrinberg]

The stack trace that you have in your log has references to the sync.py file in gunicorn which is the sync worker. The gunicorn command that you show me indicates that you are running the eventlet worker, though, so there is a disconnect there that you need to figure out. The correct worker that you should be running is eventlet, as in your command.

[mtShaikh]

Thanks for the help mate!
I am using supervisor to run gunicorn and in my case it wasnt picking changes to run gunicorn with the updated command
Restarted supervisor itself and it worked
cheers!