Unit: python virtualenv

[elijahsgh]

I read from the beta that you could now use virtual environments, but I keep encountering
Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings'

I'm assuming this is because the virtualenv isn't quite right since I can use my unitd without a virtualenv and everything seems fine.

Did I follow the directions incorrectly?

working_directory should be the parent where home and path are relative. path is the path to the virtualenv (ie: python3 -m venv env then "path" would be venv). If I omit working_directory and set home I always get the encodings error even if I set it to a venv directory. No combination of venv works with home or path.

[elijahsgh]

After trying the same thing in Unit 1.0 it seems to work fine now.
Closing.

[VBart]

This is strange, because I can't remember any changes in 1.0 that can be related. Please keep us informed, if something will not work again.

[elijahsgh]

Thanks for the response. I'll reopen a new issue if it happens again. I should have some time this weekend to try some python2 and python3 projects I have under uWSGI with unit. :)

[kalieye]

i'm experiencing the exact same problem on 1.0, i get this error:

Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings'

weirdly enough, even installing the requirements globally i get the same error

[VBart]

@kalieye Thanks for the report. Could you share steps to reproduce?

[elijahsgh]

I'm now having some trouble again, too.

```git clone https://github.com/nginx/unit
cd unit
git checkout 1.0
./configure
./configure python --config=/usr/bin/python-config
./configure python --config=/usr/bin/python3-config
make all
build/unitd --control 127.0.0.1:8000 --log ./unit.log --pid unit.pid --state unit-state --modules build

unit.log says:

2018/04/17 05:27:21 [notice] 15907#15907 module: python 2.7.14 "build/python.unit.so"
2018/04/17 05:27:21 [notice] 15907#15907 module: python 3.6.5 "build/python3.unit.so"

I have a `build/python.unit.so` and a `build/python3.6.unit.so`. At this point none of my projects will start. They all fail whether I use `python` or `python3` as the type.

2018/04/17 05:27:54 [info] 15985#15985 "demo" application started
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

If I `rm build/python3.unit.so` the python2 projects start successfully. Very strange!


This workstation is Fedora 27.

example "applications" section:

"applications": {
"demo": {
"type": "python",
"home": "/tmp/nginx-playground/unit/env-py2",
"path": "/tmp/nginx-playground/unit",
"module": "demo_app",
"processes": 1
}
}
```

Substitute 'python3' for python3 app and home directory. Directories env-py2 and env-py3 were generated with python2 -m virtualenv env-py2 and python3 -m venv env-py3.

[VBart]

Unit by default uses the most recent version of interpreter (among available modules) that matches the type field. If you have two modules (Python 2.x.y and Python 3.a.b) you have to specify python 2 in type in order to use the smaller version.

Does it fix the problem?

[elijahsgh]

The python2 applications now start with this:

  "applications": {
    "demo": {
      "type": "python2",
      "home": "/tmp/nginx-playground/unit/env-py2",
      "path": "/tmp/nginx-playground/unit",
      "module": "demo_app",
      "processes": 1
    }

However, the python3 apps still do not start. I have tried python, python 3, python 3.6, python 3.6.5 and I still receive the Py_initialize error.

[VBart]

Could you share the minimal application (or an example one) that reproduces the issue?

[elijahsgh]

demo-py2.json

{
  "listeners": {
    "*:8300": {
      "application": "demo"
    }
  },

  "applications": {
    "demo": {
      "type": "python2",
      "home": "/tmp/nginx-playground/unit/env-py2",
      "path": "/tmp/nginx-playground/unit",
      "module": "demo_app",
      "processes": 1
    }
  }
}

demo-py3.json

{
  "listeners": {
    "*:8300": {
      "application": "demo"
    }
  },

  "applications": {
    "demo": {
      "type": "python",
      "home": "/tmp/nginx-playground/unit/env-py3",
      "path": "/tmp/nginx-playground/unit",
      "module": "demo_app",
      "processes": 1
    }
  }
}

demo_app.py

#!/usr/bin/env python3
from wsgiref.simple_server import make_server

def application(environ, start_response):
    status = '200 OK'
    headers = [('Content-type', 'text/plain; charset=utf-8')]
    start_response(status, headers)

    return 'Hello\n\n'

[VBart]

Thank you. I've reproduced the problem.

[VBart]

Ok. I've found the problem. Looks like we don't support PEP 405 (which is used for Python 3.3 or above).

Quick and dirty fix for Python 3.3+:

diff -r 44f8dcca7f58 src/nxt_python_wsgi.c
--- a/src/nxt_python_wsgi.c     Sun Apr 15 19:44:38 2018 +0300
+++ b/src/nxt_python_wsgi.c     Tue Apr 17 15:21:45 2018 +0300
@@ -206,19 +206,20 @@ nxt_python_init(nxt_task_t *task, nxt_co

         len = nxt_strlen(c->home);

-        nxt_py_home = nxt_malloc(sizeof(*nxt_py_home) * (len + 1));
+        nxt_py_home = nxt_zalloc(sizeof(*nxt_py_home) * (len + sizeof("/bin/python")));
         if (nxt_slow_path(nxt_py_home == NULL)) {
             nxt_alert(task, "Failed to allocate buffer for home path");
             return NXT_ERROR;
         }

 #if PY_MAJOR_VERSION == 3
-        mbstowcs(nxt_py_home, c->home, len + 1);
+        mbstowcs(nxt_py_home, c->home, len);
+        mbstowcs(nxt_py_home + len, "/bin/python", sizeof("/bin/python"));
+        Py_SetProgramName(nxt_py_home);
 #else
         nxt_memcpy(nxt_py_home, c->home, len + 1);
+        Py_SetPythonHome(nxt_py_home);
 #endif
-
-        Py_SetPythonHome(nxt_py_home);
     }

     Py_InitializeEx(0);

[ahankinson]

Does this patch hardcode the path to the python interpreter? My Python3, for example, is not at /bin/python but at /usr/bin/python3.6.

[VBart]

@ahankinson This path doesn't matter for Unit, because it's a path to command-line interpreter. Unit doesn't use command-line interpreter, it uses its own module specified in the configuration API.

The same is true for any Python application server with embed interpreter for performance.

[kalieye]

@VBart the patch solved the dependency problem, but i receive a new error now which is more difficult to solve:

2018/04/24 14:40:31 [info] 2913#2913 "myapp" application started 2018/04/24 14:40:31 [alert] 2913#2913 Python failed to import module "mymodulename" 2018/04/24 14:40:31 [notice] 2818#2818 process 2913 exited with code 1 2018/04/24 14:40:31 [warn] 2821#2821 failed to start application "myapp" 2018/04/24 14:40:31 [alert] 2821#2821 failed to apply new conf

PS. the app starts normally and works as intended in the virtualenv, it's a flask application

[VBart]

@kalieye Could you show unit configuration and directory listing with the application?

[kalieye]

sure @VBart, here's the listing and the json config

list.txt
fltest.json.txt

and if needed here's the complete tarball (with everything already installed and the virtualenv ready): https://we.tl/Q283PyWVur

i tried creating a very basic flask application, but the result is exactly the same, i built unit from sources with the patch you posted and with python3.6 on ubuntu 17

hope this helps, please let me know if you need me to do further testing

[elijahsgh]

I believe your "home" and "path" are backwards, @kalieye

After flipping them I was able to get your demo app running. I also changed the ports a bit, but I think that was unrelated. :)

[kalieye]

@tamarintech tried to flip them to no avail, now i'm thinking that it may be something in my installation! what version of python are you using? (thanks)

[VBart]

@kalieye First of all, check your paths. The virtual environment is located at /www/fltest/env/ (so this is the home directory), while the WSGI application module is at /www/fltest/ (that's should be set in path).

Also the content of app.py doesn't look like a WSGI module with application() callable. Try this instead:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_nginxunit():
    return 'Hello, NGINX Unit'

application = app

[VBart]

I've committed the fix for ModuleNotFoundError: No module named 'encodings' problem. Thanks all for testing.

[kalieye]

thanks a lot @VBart, everything works perfectly now!