Gunicorn: Gunicorn + Flask-socketio too high CPU usage

[greggy]

I've created a docker container with a very simple flask-socketio application. It works correctly, but after a couple of days CPU usage rises to 100%. I was trying to profile it with vmprof, but didn't understand how to run my application with it.

Actually, my main question is why it starts "eat" CPU after some time? And how can I use vmprof for the whole application under gunicorn?

Dockerfile:

FROM python:3

EXPOSE 3000
RUN apt-get update && \
    apt-get install -y build-essential python3-dev \
    htop net-tools
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY WS_SERVER ./WS_SERVER
CMD ./WS_SERVER/run_uwsgi.sh

requirements.txt:

flask_socketio
gunicorn
eventlet

run_uwsgi.sh:

#!/bin/bash

export PYTHONPATH="${PYTHONPATH}:/app/WS_SERVER"
gunicorn --worker-class eventlet -w 5 --threads 5 --pid /app/wsserver.pid -b :3000 --log-file /app/LOGS/ws_uwsgi.log server:app

server.py:

import logging

from flask import Flask, request
from flask_socketio import SocketIO, emit

import eventlet
eventlet.monkey_patch()

FORMAT = '%(name)s - %(levelname)s - %(message)s'
logging.basicConfig(level=logging.DEBUG, filename='./LOGS/ws_app.log',
                    filemode='w', format=FORMAT)

app = Flask(__name__)
app.config['SECRET_KEY'] = '...'
socketio = SocketIO(app, cors_allowed_origins='*', async_mode='eventlet')

EVENTS = [
    'event1',
    'event2'
]

def handle_common(message):
    logging.debug('My Event: %s' % message)
    event_name = message.get('eventName', '')
    emit(event_name, message, broadcast=True)

for event_name in EVENTS:
    socketio.on_event(event_name, handle_common, namespace='/')

@socketio.on('connect', namespace='/')
def test_connect():
    logging.debug('Connect!')
    emit('connect', {'data': 'Connected'})

@socketio.on('disconnect', namespace='/')
def test_disconnect():
    logging.debug('Client disconnected %s' % request.sid)

@socketio.on_error_default
def error_handler(e):
    logging.debug('Error: %s' % e)

if __name__ == '__main__':
    socketio.run(app, debug=True)

Thank you!

[jamadden]

And how can I use vmprof for the whole application under gunicorn?

I've had great success using py-spy. You can connect it to the gunicorn master process and use —subprocesses to also automatically record all worker processes at the same time, or you can connect it to a single worker process to watch just that process.

[greggy]

@jamadden Thank you! It seems is what I needed.

[jamadden]

Cool! Does that mean you figured it out? If so, could you please close this issue?

[greggy]

@jamadden I don't know exactly who is the victim of cpu usage. Regarding the attached screeshot seems it is eventlet. Am I right?

[jamadden]

It looks like you're spending most of your time waiting for IO events. Which isn't unexpected.

[greggy]

It looks like you're spending most of your time waiting for IO events.

I'm not sure. This screenshot was made at the weekend, nobody uses it. I'll try to do it today in a working day when it will be under high load.

[greggy]

As you see, nothing was changed. It's very strange.

[jjpaulo2]

@greggy did you find any solution?

[greggy]

Yes, the issue was in code. It takes to much data from db and some threads eat cpu.

[amks1]

Yes, the issue was in code. It takes to much data from db and some threads eat cpu.

Hi @greggy I'm facing a similar issue with my Gunicron + Flask-SocketIO app in production. After several hours, the CPU goes to 100% and stays there. It's probably an issue in the code, but I've not been able to figure out how to debug this. Do you remember what steps you followed?

[greggy]

Hello, @amks1 It was an issue in response from db. It got a too big list of entries from table. We added pagination and the issue gone.