Uwsgi: PyPy + Django + gevent loop engine

[dmk23]

I am trying to deploy the configuration in the headline and having no luck.

As a starting point, I am using the working configuration on CPython and just replacing virtualenv with the one created under PyPy (with uWSGI executable installed via pip on that virtualenv)

This configuration fails in multiple ways, specifically:

1. First of all gevent loop engine does not load and if strict = True logs [strict-mode] unknown config directive: gevent. I was expecting gevent support to be built under pip
2. With strict turned off, the Django app fails to load *** no app loaded. going in full dynamic mode *** . Here I was expecting for PyPy virtualenv to honor the same options as CPython
3. I did look up the options for PyPy module and tried pypy-wsgi = django.core.handlers.wsgi:WSGIHandler() resulting in the following error:

From cffi callback <function uwsgi_pypy_loader at 0x00007f2f93ad6f98>:
Traceback (most recent call last):
  File "c callback", line 304, in uwsgi_pypy_loader
AttributeError: 'module' object has no attribute 'WSGIHandler()'
*** no app loaded. going in full dynamic mode ***

At this point I am not quite sure how to proceed. I find it counter-intuitive that it is not possible to simply swap the virtualenv and preserve the same options and behaviors.

In any case would appreciate help figuring out the proper configuration. At the moment the docs are silent on compatibility of PyPy with Django, Gevent or any other plugins / apps.

[unbit]

Gevent under pypy is still an experimental effort (you need a special almost-working branch of gevent for it)

The PyPy plugin has no explicit support for virtualenv, but PEP405 should work

The colon-based form is not supported for pypy-wsgi, just use pypy-wsgi-file with wsgi.py

[dmk23]

I get a sense that any use of PyPy plugin ought to be treated as experimental... Gevent issue aside, is anyone known to run Django apps with PyPy at a significant scale in production?

[WST]

I only tested PyPy with a Django website on my local machine; had no problem though.

[dmk23]

Thanks. In my case I'd say Gevent would be more important than PyPy. So I am switching back to CPython and continuing some rigorous testing, with a few new issues

strict = True
gevent = 100
gevent-monkey-patch = True
#gevent-early-monkey-patch = True
gevent-wait-for-hub = True
processes = 2
threads = 4

Here are the problems with the above config:

1. Under "strict" uWSGI 2.0.9 does not recognize gevent early monkey patch option, so I got it commented out. Are the docs for this option obsolete or it is due in the next release?
2. If I try to reload my page a few times, under Django DEBUG= True, I get the following error. I was under impression that monkey.patch_all() should make MySQLdb just work:

DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 139750555608720 and this is thread id 139750339108048.

1. If I turn off monkey patch in uwsgi.ini file and instead put monkey.patch_all() into Django settings.py I get the same error.
2. I added some "print" statements into settings.py and saw that they only get triggered when a new worker gets a request, not when master loads. Isn't it supposed to load once by master?
3. As another experiment I've added the code below before monkey patch code in settings.py. This in turn caused the second page load to simply hang.

if 'threading' in sys.modules: del sys.modules['threading']

Would appreciate any help sorting out what is going wrong here... Have you tested monkey-patched Django/ Gevent with MySQL driver ? Did anything like this come up?

[unbit]

(please do no hijack issues)

Btw,

1 - early patching is a 2.1 thing
2 - threading + gevent does not make sense and it is not supported (monkey patching even maps threads to greenlet so only evil things will happen ;)
3 - like 2
4 - django loading is lazy by design, edit wsgi.py for such kind of usages (it is loaded on startup)
5 - like 2

[dmk23]

Sorry for the hijack - but I thought these are related issues from the same evolving configuration. If necessary, we can always create a separate issue...

Thanks for the tips, I was able to solve the DatabaseWrapper problem and shared pre-loading by creating a custom wsgi module, putting monkey patch there first and then adding shared initialization.

Unfortunately, I cannot avoid monkey-patching threads since redis-py client blocks/fails if sockets are patched and threads are not. Here are the details: https://github.com/kennethreitz/grequests/issues/55#issuecomment-75628252

The above configuration has been working fine with gevent = 100, but I then I tried to set gevent = 1000 and started seeing segfaults after a couple requests OR even on touch-reload of a server that has not served a single request (but was up for a few minutes). Are there any best practices or expected limits on the number of gevent cores? The docs say nothing about tuning that setting...

[parasyte]

gevent v1.1a2 currently has very good support for PyPy (still an alpha release at present).

Are there plans to port the uWSGI gevent plugin to work with PyPy?

[dmk23]

+1. we want PyPy JIT'd applications with the best event loop

[unbit]

Work on it already started last week. It should be ready soon

[voluntas]

+1

[tebanep]

Really cool! This can give python, nodejs' performance!

[parasyte]

Out of curiosity, is this PyPy+gevent work public somewhere? It does not appear in any of the branches.

[ephillipe]

+1

[trim-fly]

is there any updates? just want to bring pypy+gevent straight to production :)

[ddorian]

People of the thread, what are you using currently to host pypy+gevent ?

[parasyte]

@ddorian I'm using bottle + upstart. Bottle for the WSGI interface, and Upstart to control process creation/restarts/etc.

[ddorian]

@parasyte
Does that support multiple processes/workers and load-balancing between them (like normal uwsgi/gurnicorn etc) ?

[parasyte]

@ddorian No, we have nginx acting as a front end load balancer for the cluster.

[sf-jin-ku]

@unbit Are there any updates on this issue? It looks like gevent 1.1.1 works very well with pypy 5.1

[rockallite]

Gevent 1.2.1 on PyPy 5.7.1 compiled and run with no problem. But there's no PyPy-based gevent loop engine support in uWSGI 2.0.x (compiling uWSGI gevent plugin with PyPy failed with errors), which is a pity...

[arthuriantech]

@unbit Current uWSGI version is 2.0.18. What about any updates for PyPy with gevent engine?