Flask-socketio: Help: Issues with gunicorn behind nginx in docker environment

[DanielCardonaRojas]

I have successfully incorporated flask-socketio into a project and runs fine with gunicorn and eventlet, it also works fine with docker-compose (celery rabbit and redis services). But when nginx is put in front of gunicorn, workers stop serving my regular api endpoints (get a 404, worker closes connection), even though socketio keeps working. I can imagine this is probably not a flask-socketio issue directly but I'm hoping to get some insight to figure this out.

Logs when querying a normal endpoint:

api_1       | 2017-10-12 12:28:18 [7] [DEBUG] GET /api/users
api_1       | 2017-10-12 12:28:18 [7] [DEBUG] Closing connection.

These are the dependency versions:

• gunicorn==18.0
• Flask-SocketIO==2.9.2
• python-socketio==1.8.1
• python-engineio==1.7.0
• eventlet==0.21.0

Docker nginx image is version 1.12 > 1.4 and the nginx configuration file is mostly the same as the one you suggest.

upstream http_nodes {
    server api:5000; #Docker service name
}

upstream socketio_nodes {
    ip_hash;

    server api:5000; #Docker service name
}

server {
    listen 5000;
    server_name _;

    location / {
        proxy_pass http://http_nodes;
    }

    location /socket.io {
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass http://socketio_nodes/socket.io;
    }
}

This is docker-compose file im testing with:

version: '3'
services:
    nginx:
        image: nginx:alpine
        ports: 
            - "5000:5000"
        volumes:
            - ./mobi-api.conf:/etc/nginx/conf.d/default.conf
        depends_on:
            - api
        restart: always
    postgres:
        image: mdillon/postgis
        volumes:
        - db_data:/var/lib/postgresql
        ports:
            - "5432"
        env_file: .env_postgres
    api:
        build: .
        #entrypoint: ./wait-for-it.sh postgres:5432 --
        volumes:
            - .:/api
        env_file: .env
        ports:
            - "5000"
        links:
            - postgres:postgres
        command: gunicorn wsgi:app --bind 0.0.0.0:5000 --worker-class eventlet --workers 1 --log-level DEBUG
        depends_on:
            - redis
            - postgres
            - rabbit
        restart: always

    celery:
        build: .
        volumes:
            - .:/api
        env_file: .env
        links: 
            - rabbit
            - postgres:postgres
        depends_on:
            - rabbit
        command: celery worker -A runcelery.celery --loglevel=info --host rabbit

    beat:
        build: .
        volumes:
            - .:/api
        env_file: .env
        links: 
            - celery
        depends_on:
            - celery
        command: celery beat -A runcelery.celery --loglevel=info

    rabbit:
        image: rabbitmq:3.6.12-alpine
        environment:
            - RABBITMQ_DEFAULT_USER=guest
            - RABBITMQ_DEFAULT_PASS=guest
        ports: 
            - "5672"
    redis:
        image: redis:3.0-alpine
        ports: 
            - "6379"



volumes:
    db_data:
    static_files:

Any ideas on why this happens and how I could fix it?
Any other overall suggestions are welcome.

[miguelgrinberg]

Who's returning 404? Is it nginx without talking to your app, or the app itself?

[DanielCardonaRojas]

The request does reach gunicorn since it logs as shown above and socketio is served correctly. But then a "closing connection" is logged by gunicorn and the 404 logged by nginx.

[miguelgrinberg]

Okay, then can you show me the gunicorn logs that show the 404s?

[DanielCardonaRojas]

Sorry for the delay. Here are the logs for both containers, nginx and the applications itself.

api_1       | [2017-10-13 11:59:47 +0000] [7] [DEBUG] GET /api/users
nginx_1     | 172.19.0.1 - - [13/Oct/2017:11:59:47 +0000] "GET /api/users HTTP/1.1" 404 233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "-"
api_1       | [2017-10-13 11:59:47 +0000] [7] [DEBUG] Closing connection.

And my browser shows the regular 404 from flask:

Not found

The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

[miguelgrinberg]

The logs that you are capturing are the debug stream from gunicorn, not the access log. The access log should have a format that is fairly close to that of nginx, and I would expect to see a 404 there too, if it is gunicorn that is generating the error.

Assuming the error comes from gunicorn, would it be possible that the modules that have your HTML routes are not getting imported?

[DanielCardonaRojas]

I looked at gunicorn access log and its identical to the nginx log. Removing the nginx container and exposing the port directly from application container works fine. I'll inspect the url_map anyways and post the routes when I get home.

[DanielCardonaRojas]

Inspecting the url_map from flasks shell it does show all registered routes as expected.

[miguelgrinberg]

Well, you must be missing something. Look at the gunicorn access log for a request going to /api/users sent directly, versus the failed ones that nginx issues. There's got to be something that's different, since the server is the same.

[DanielCardonaRojas]

The only difference I noticed was gunicorn serving http 1.0 and nginx http 1.1. And for some reason y see two instances of gunicorn running when executing ps aux in the container.

[miguelgrinberg]

What I meant was to look at the access log for gunicorn serving to nginx (ending in a 404) vs gunicorn serving to the client (ending in a 200), both going to the same URL. There's got to be something different between those two besides the 404 vs 200 status code.

[DanielCardonaRojas]

Running gunicorn with:

gunicorn wsgi:app --bind 0.0.0.0:5000 --worker-class eventlet -w 1 --access-logfile gunicorn.log --log-level DEBUG --forwarded-allow-ips "*" 

These are the access logs for gunicorn:

root@029f3cecf826:/api# tail -f gunicorn.log
172.19.0.7 - - [14/Oct/2017:23:29:18 +0000] "GET /api/users HTTP/1.0" 404 233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"

These are docker container logs:

api_1       | [2017-10-14 23:41:32 +0000] [7] [DEBUG] GET /api/users
nginx_1     | 172.19.0.1 - - [14/Oct/2017:23:41:32 +0000] "GET /api/users HTTP/1.1" 404 233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "-"
api_1       | [2017-10-14 23:41:32 +0000] [7] [DEBUG] Closing connection.

This is the project repo:

https://github.com/WalloomProjects/Mobi-API/tree/feature/notification-system

I also tried to look at the access and error log for nginx under /var/log/nginx but nothing gets written there.

[miguelgrinberg]

Not sure I understand how your code works at all. File application/routes/api_1_1/user.py appears to have an undefined symbol (Namespace), or am I mistaken?

[DanielCardonaRojas]

Your right, my mistake, this import is missing:

from flask_restplus_patched import Resource as Res, Namespace, abort

No sure why tests where passing. Any how I fixed it but still is the same as other endpoints respond with the same 404. I will push the fix anyways.

I am using flask-restplus to serve an api with some patches to use webargs and marshmallow.

[miguelgrinberg]

No sure why tests where passing

It's actually impossible that your /api/users route worked with that missing import, right? Isn't that the module that serves the /api/users endpoint you are testing with?

[DanielCardonaRojas]

Hi, Miguel

I finally got it working, all this time while working locally I was making requests to localhost, I recently noticed I would get 404s when accessing through my macbooks ip itself. I started messing around and noticed the issue was related to flasks SERVER_NAME configuration. Setting it to None made things work. Thank's a lot for all your help.

[ColdFire87]

I had a similar issue: gunicorn serving a Flask app. All worked fine when accessing the service directly, all 404s when proxying through nginx. After countless hours of trying to figure it out it seems that setting SERVER_NAME to None for the Flask app did the trick.

Thanks Daniel!

[xsren]

Setting SERVER_NAME to None for the Flask app did the trick.
Thanks!

[erikdao]

@DanielCardonaRojas Thank you very much. After days of confusing, you saved me :D