Flask-socketio: Configuring SSL with eventlet

[chipsenkbeil]

Hi,

Is there documentation anywhere about configuring SSL when using eventlet?

I see there was a comment here about already forwarding additional arguments to the server.

There is a snippet for Flask receiving an ssl_context: http://flask.pocoo.org/snippets/111/

Trying to pass around an ssl_context gives me an unexpected keyword argument 'ssl_context'. The same goes for keyfile and certfile, but I'm assuming that was specific to gevent.

if __name__ == '__main__':
    certfile = os.path.join(ROOT_DIR, 'private/certificate.pem')
    keyfile = os.path.join(ROOT_DIR, 'private/privatekey.pem')

    # Ignore the None value as the keyword error happens before I get anywhere close to setting up SSL
    socketio.run(
        app,
        host='0.0.0.0',
        port=5000,
        ssl_context=None
    )

I'm trying to understand based on the documentation, but I've never done this with Flask, let alone with SocketIO integration. From eventlet's documentation, it looks like you create a GreenSSLSocket? I'm assuming you then pass that to eventlet.wsgi.server(sock, site, ...) as the socket. If that's the case, it doesn't look like I can do that since the socket is hard-coded here: https://github.com/muatik/Flask-SocketIO/blob/9cd254c5fdc45a2aa8f03ec372eddb2f74e8a7bb/flask_socketio/__init__.py#L378

If that's not the case and there is some way to do this (without adding something like nginx in front), please let me know or point me in the right direction!

[miguelgrinberg]

My intention was to pass-through any WSGI server options. In looking at the eventlet documentation, it appears SSL is not enabled by passing extra arguments, looks like the socket needs to be wrapped with a call to wrap_ssl (source: http://eventlet.net/doc/modules/wsgi.html#ssl). So this is a valid issue, I need to add support for doing this wrapping.

I think I'll have you pass the certfile and keyfile arguments in the socketio.run() call, and then I apply those appropriately on my side.

[chipsenkbeil]

Yeah, I came to that conclusion myself after posting this issue. Sounds good! For now, I switched to gevent since I can pass through certfile and keyfile.

[miguelgrinberg]

Fixed in #196. Closing.

[nadaz]

Hi, first of all, thank very much for creating this terrific piece of software :)

I've been using for a very short time, so sorry if this was already reported/explained.

1. Trying to enable SSL (i'll use nginx somewhere in the future probably, but not right now), passing
   certfile and keyfile to app constructor; at first seems everything fine, but after some time (or better, from an unknow browser of a friend), it raises these error

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
func(args, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 719, in process_request
proto.__init__(sock, address, self)
File "/usr/lib/python2.7/SocketServer.py", line 655, in __init__
self.handle()
File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
self.handle_one_request()
File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 330, in handle_one_request
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 200, in recv
read = self.read(buflen)
File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 139, in read
super(GreenSSLSocket, self).read, args, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 113, in _call_trampolining
return func(a, *kw)
File "/usr/lib/python2.7/ssl.py", line 608, in read
v = self._sslobj.read(len or 1024)
SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:1750)

I think that the client is asking plain http element, can you confirm this is the case?

1. And second thing, even if the connection function, all the emit/send logic (from both sides) seem to not work on firefox, which signal it as "mixed content", (can see console logs every three second, as server works -example code below-)

@app.route('/login', methods=['GET','POST'])
def do_login():
    global timers
    if not timers:
        timers = socketio.start_background_task(timer)

 [ ... ]

def timer():
    while True:
        socketio.emit('msg',{'user':'hour','msg':str(time.time())},namespace='/game')
        time.sleep(3)

This code works nice without ssl.

So the question is: beside putting certfile and keyfile in constructor, there's anything else to care about (in emit method for example) ?

[miguelgrinberg]

@nadaz in addition to adding the certificate on the server, you have to change the http:// to https:// in the client connection call. That should be the only changes regarding Socket.IO. You may have other changes to make regarding other subsystems in your application, such as static files, for example.

[Renmusxd]

Using eventlet with ssl and the certfile/keyfile parameters mentioned above in the following way
socketio.run(app, debug=True, use_reloader=False, certfile=CERT_FILE, keyfile=KEY_FILE, port=5500) causes an odd error:

Traceback (most recent call last):
File "[...]/server.py", line 231, in
socketio.run(app, debug=True, use_reloader=False, certfile=CERT_FILE, keyfile=KEY_FILE, port=5500)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/flask_socketio/__init__.py", line 493, in run
run_server()
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/flask_socketio/__init__.py", line 485, in run_server
*ssl_params)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/eventlet/convenience.py", line 126, in wrap_ssl
return wrap_ssl_impl(sock, *a, *kw)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/eventlet/green/ssl.py", line 379, in wrap_socket
return GreenSSLSocket(sock, a, *kw)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/eventlet/green/ssl.py", line 68, in __init__
ca_certs, do_handshake_on_connect and six.PY2, args, *kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 737, in __init__
self._context.verify_mode = cert_reqs
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 479, in verify_mode
super(SSLContext, SSLContext).verify_mode.__set__(self, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 479, in verify_mode
super(SSLContext, SSLContext).verify_mode.__set__(self, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 479, in verify_mode
super(SSLContext, SSLContext).verify_mode.__set__(self, value)
[Previous line repeated 325 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Seems to be related, but can open a new issue if needed.

[miguelgrinberg]

@Renmusxd believe it or not, this looks like a bug in Python 3.6's SSL class. Do you have an older Python at hand to see if I'm correct or not?

[Renmusxd]

How exciting! That's what the stacktrace seemed to suggest but I thought it was little ambitious to start by thinking the problem was there.
Looks like the bug is isolated to python 3.6, in 3.5 the following error occurs:

Traceback (most recent call last):
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
func(args, *kwargs)
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/wsgi.py", line 734, in process_request
proto.__init__(sock, address, self)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socketserver.py", line 681, in __init__
self.handle()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/server.py", line 422, in handle
self.handle_one_request()
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/wsgi.py", line 339, in handle_one_request
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/green/ssl.py", line 204, in recv_into
return self._base_recv(nbytes, flags, into=True, buffer_=buffer)
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/green/ssl.py", line 225, in _base_recv
read = self.read(nbytes, buffer_)
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/green/ssl.py", line 139, in read
super(GreenSSLSocket, self).read, args, *kwargs)
File "[...].virtualenvs/bot35/lib/python3.5/site-packages/eventlet/green/ssl.py", line 109, in _call_trampolining
return func(a, *kw)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 791, in read
return self._sslobj.read(len, buffer)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 575, in read
v = self._sslobj.read(len, buffer)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1977)

[miguelgrinberg]

@Renmusxd Did you monkey patch the standard library?

[Renmusxd]

@miguelgrinberg

If you mean including

import eventlet
eventlet.monkey_patch()

at the top of the file, then yes

[miguelgrinberg]

Okay, that is not the problem then.

There is clearly something off in the interaction between eventlet and the Python ssl support. Not sure if you feel like experimenting a bit more. If you are using the current release of eventlet, which had a major release less than a month ago. You can check the 0.19.0 release, or maybe even the 0.17.4 version, which in my experience was super stable and is likely the version that was current back when I added the support for ssl wrapping the eventlet socket.

[Renmusxd]

@miguelgrinberg Recursion crash disappears in eventlet 0.17.4, still around in 0.19.0, all in python 3.6
Edit: in python 3.5 crash remains in original project independent of eventlet version

[Renmusxd]

Here's my smallest working example of the crash in python 3.6, this doesn't crash in python 3.5, the error I posted for 3.5 seems to be specific to a different project (using more or less the same setup though):

import eventlet
eventlet.monkey_patch()

import os
import socket
from flask import Flask, render_template, request, Response, send_file
from flask import make_response
from flask_socketio import SocketIO
from OpenSSL import SSL, crypto

'''
requirements.txt
cffi==1.9.1
click==6.6
cryptography==1.7.1
enum-compat==0.0.2
eventlet==0.20.1
Flask==0.12
Flask-SocketIO==2.8.2
greenlet==0.4.11
idna==2.2
itsdangerous==0.24
Jinja2==2.8.1
MarkupSafe==0.23
pyasn1==0.1.9
pycparser==2.17
pyOpenSSL==16.2.0
python-engineio==1.1.0
python-socketio==1.6.2
six==1.10.0
Werkzeug==0.11.15
'''

app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
async_mode = None
socketio = SocketIO(app, async_mode=async_mode)
def create_self_signed_cert(certfile, keyfile, certargs, cert_dir="."):
    C_F = os.path.join(cert_dir, certfile)
    K_F = os.path.join(cert_dir, keyfile)
    if not os.path.exists(C_F) or not os.path.exists(K_F):
        k = crypto.PKey()
        k.generate_key(crypto.TYPE_RSA, 1024)
        cert = crypto.X509()
        cert.get_subject().C = certargs["Country"]
        cert.get_subject().ST = certargs["State"]
        cert.get_subject().L = certargs["City"]
        cert.get_subject().O = certargs["Organization"]
        cert.get_subject().OU = certargs["Org. Unit"]
        cert.get_subject().CN = 'Example'
        cert.set_serial_number(1000)
        cert.gmtime_adj_notBefore(0)
        cert.gmtime_adj_notAfter(315360000)
        cert.set_issuer(cert.get_subject())
        cert.set_pubkey(k)
        cert.sign(k, 'sha1')
        open(C_F, "wb").write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
        open(K_F, "wb").write(crypto.dump_privatekey(crypto.FILETYPE_PEM, k))

CERT_FILE = "cert.pem"
KEY_FILE = "key.pem"
create_self_signed_cert(CERT_FILE, KEY_FILE,
                            certargs=
                            {"Country": "US",
                             "State": "NY",
                             "City": "Ithaca",
                             "Organization": "Python-Bugs",
                             "Org. Unit": "Proof of Concept"})
socketio.run(app, debug=True, use_reloader=False, certfile=CERT_FILE, keyfile=KEY_FILE, port=5500)

[Renmusxd]

@miguelgrinberg Should I take this to their github?

[miguelgrinberg]

I assume by "their github" you mean eventlet's? I think they are more likely to be the owners of this, so yes, that is probably a good idea.

[Renmusxd]

Thanks for your help

[vandenbroucke]

I've been spending quite a few hours on the exact same issue: In short: emited events from background_thread took a while before they were visible at the clients. Everyone recommends monkey patching => However SSL issue. After using eventlet.monkey_patch(socket=False), my emits from the background thread were immediate and my issues were resolved.