Uwsgi: In python3, uwsgi fails to respond a stream from BytesIO object.

[eyal-stratoscale]

in python3, uwsgi fails to respond a BytesIO stream.

If I use the following falcon application in test.py:

import falcon
import io

class Resource(object):
    def on_get(self, req, resp):
        resp.stream = io.BytesIO(b'moshe')

api = falcon.API()
api.add_route('/', Resource())
application = api

When I run (python3 uwsgi): uwsgi --http :8000 --module test and curl it curl http://localhost:8000, the client gets an error curl: (52) Empty reply from server and uwsgi log shows:

*** Starting uWSGI 2.0.11.2 (64bit) on [Thu Dec 10 13:45:07 2015] ***
compiled with version: 5.1.1 20150618 (Red Hat 5.1.1-4) on 10 December 2015 12:41:16
os: Linux-4.2.6-300.fc23.x86_64 #1 SMP Tue Nov 10 19:32:21 UTC 2015
nodename: localhost.localdomain
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/eyal/work/hammock
detected binary path: /home/eyal/work/hammock/.tox/py35/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 30621
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :8000 fd 7
spawned uWSGI http 1 (pid: 4212)
uwsgi socket 0 bound to TCP address 127.0.0.1:42015 (port auto-assigned) fd 6
Python version: 3.5.0 (default, Dec 10 2015, 11:41:08)  [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x285da00
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72768 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x285da00 pid: 4211 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 4211, cores: 1)
io.UnsupportedOperation: fileno

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eyal/work/hammock/.tox/py35/lib/python3.5/site-packages/falcon/api.py", line 232, in __call__
    body = self._get_body(resp, env.get('wsgi.file_wrapper'))
  File "/home/eyal/work/hammock/.tox/py35/lib/python3.5/site-packages/falcon/api.py", line 653, in _get_body
    return wsgi_file_wrapper(stream, self._STREAM_BLOCK_SIZE)
SystemError: <built-in function uwsgi_sendfile> returned a result with an error set
[pid: 4211|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 303 bytes} [Thu Dec 10 13:45:11 2015] GET / => generated 0 bytes in 2 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

when I run the same app with gunicorn (in python 3 enivornment): gunicorn test:application, The curl returns 'moshe'.

[xrmx]

Can you dig a bit more and track this exception "io.UnsupportedOperation: fileno" in falcon please?

[eyal-stratoscale]

Thanks for the quick response, I dug already it is not in falcon, it is in uwsgi.
In the last two lines of the bug I show that the same code works under gunicorn.

[xrmx]

Looks relevant https://github.com/benoitc/gunicorn/issues/595
Also this:
http://permalink.gmane.org/gmane.comp.python.wsgi.uwsgi.general/7598

[pavel-stratoscale]

Are there any plans on fixing this?
This is a blocker for us, if it won't be fixed in the next few weeks we will need to move to an alternative platform.

[rdeioris]

wsgi.file_wrapper in uWSGI is a true optimization using sendfile() or offloading. A BytesIO is not based on a file descriptor so it cannot be optimized. The best (or the only) solution is disabling wsgi.file_wrapper:

mport falcon
import io

class Resource(object):
    def on_get(self, req, resp):
        del(req.env['wsgi.file_wrapper'])
        resp.stream = io.BytesIO(b'moshe')

api = falcon.API()
api.add_route('/', Resource())
application = api

In both git master and uwsgi-2.0 branch, a new option --wsgi-disable-file-wrapper forces this from the configuration

[methane]

https://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling

Note that even if the object is not suitable for the platform API, the wsgi.file_wrapper must still return an iterable that wraps read() and close() , so that applications using file wrappers are portable across platforms.

[eyal-stratoscale]

Thanks @rdeioris , the first option of deleting the request environment variable worked. However, there is not such option, in any version, --wsgi-disable-file-wrapper. Am I missing anything?

[rdeioris]

@methane this is a long standing issue in uWSGI, but improving it would require a complex object. Definitely something to look at for 2.1.

[rdeioris]

@eyal-stratoscale you need versions from github. The patch will be in 2.0.12 (scheduled for 30th december)

[eyal-stratoscale]

Thanks

[gergelypolonkai]

Any news on this? Just bumped into it with Flask’s send_file(). I fallback to creating a Response() object and sending that, but using send_file() would be much more elegant for me…

[arianmaykon]

Just had the same problem as @gergelypolonkai reported.

[spitzbubchen]

Also having this issue with Flask send_file()

[j0hnsmith]

Just got this with Python 3.6.1 and flask send_file(), running uwsgi 2.0.15 with --wsgi-disable-file-wrapper makes the problem go away.

[Sweenu]

Same problem here with Python 3.6 and Flask, I'm gonna try to disable the wsgi file wrapper right away

[merwok]

The option is a useful stopgap measure, but it would be preferable if uwsgi would follow the WSGI standard and not error. This is not a very user-friendly scenario given that wsgi.file_wrapper is mostly invisible (devs use send_file in Flask or FileResponse with Django for example), the response returns 200 OK but the body is empty.

For the implementation, uwsgi could be inspired by gunicorn’s detection of fileno methods that raise io.UnsupportedOperation.

[jpic]

Disabling file wrapper works for me too <3

[builder7777]

The issue was open for 4 years now, are there updates? is it on the road map or just abandoned all together?
It's still happening on python 3.5.0 and 3.6.8

[Ddasch]

I am having the same issue. Worse still is that the uwsgi parameter --wsgi-disable-file-wrapper now no longer fixes the issue. I am running uwsgi version 2.0.18 and python 3.7.

[gfairchild]

Is this issue why Django's official documentation on outputting PDFs doesn't work for me? I'm using uwsgi.

[jpic]

This is working for me:

        f = io.BytesIO(self.object.binary)
        content_type = self.object.mimetype or 'application/octet-stream'
        response = http.FileResponse(f, content_type=content_type)
        if self.object.encoding:
            response['Content-Encoding'] = self.object.encoding
        response['Content-Length'] = len(self.object.binary)
        response['Cache-Control'] = 'public, max-age=31536000'
        return response

I have these options amongst others:

  --vacuum \
  --enable-threads \
  --post-buffering=8192 \
  --ignore-sigpipe \
  --ignore-write-errors \
  --disable-write-exception \
  --mime-file /etc/mime.types \
  --thunder-lock \
  --offload-threads '%k' \
  --file-serve-mode x-accel-redirect \

Hope this helps

[danielrussellLA]

bump on this issue. In python 3.6 i need to add wsgi-disable-file-wrapper = true because my app was failing when using flask's send_file for a BytesIO stream in one route. Would be great if this issue was resolved so that I don't need to leave other parts of my application to suffer from the lack of optimization

[jonbulica99]

Bumpity bump. It would be really great if #2069 got merged. The issue itself is almost 5 years old and even the PR is almost 6 months old.

[rdeioris]

@jonbulica99 i have reviewed the patch. It requires a minor fix then i can merge it.

[heeplr]

Is a98261f7d0c2 already included in the 2.0.18 release? I'm still having trouble and wonder if i'm doing it wrong or if it's just not included in the version i'm running.

[kylemanna]

Is a98261f already included in the 2.0.18 release? I'm still having trouble and wonder if i'm doing it wrong or if it's just not included in the version i'm running.

Looks like it's in 2.0.19

$ git tag --contains a98261f7d0c2fb2117756ad9c0a204edc8445dc3
2.0.19
2.0.19.1