Uwsgi: HTTP1.1 chunked transfer not supported

[shimzim]

Hi,

If I send a stream to a uwsgi server, with no content-length (which is the recommended way for a chunked transfer with HTTP1.1), the server getting the response gets an empty data stream from the read() function, as can be seen here (req->post_cl is null in this case):
https://github.com/unbit/uwsgi/blob/master/core/reader.c#L301

Is there a plan to support chunked streaming?

Thanks,
Shimshon.

[sharbov]

+1

[unbit]

uWSGI has an api for chunked input, but the webserver needs to pass it to the backend:

http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html

If you are using the http router you can add this option:

--http-chunked-input

(automatically detect chunked input requests and put the session in raw mode)

[shimzim]

Which webserver? I tried running uwsgi as-is on my computer (with falcon on top of it). I also tried now with the --http-chunked-input but nothing changed, also tried http-raw-input. BTW - couldn't find the http-chunked-input flag in the docs, can you please point to a reference?
I have the following app which works with falcon over gunicorn but not with falcon over uwsgi. Am I missing any configuration parameter?

def on_put(self, req, resp):
    print "123"
    while 1:
        data = req.stream.read(4096)
        if not data:
            print "XXXXXXX"
            break
        print "LEN = %s" % len(data)

When I run it with uwsgi, I see "123" and then "XXXX".

[unbit]

You cannot use the WSGI interface to read chunked input (it does not support it, albeit some application server tried to manage it is some way). You need to use this api:

http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html

The alternative is to transform the chunked input in a non chunked one (like nginx does), but the uWSGI api assumes that the chunked input usage is to implement some kind of framed input streaming, so the classic read() method is suboptimal

[xrmx]

Can we close this?

[shimzim]

Given you're not planning on fixing it - yes. :(

On Wed, Jan 4, 2017 at 1:17 PM, Riccardo Magliocchetti <
[email protected]> wrote:

Can we close this?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/unbit/uwsgi/issues/1428#issuecomment-270348959, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHHH8cy7Ho-rna3R4Esgqhs5FqFO_8Xcks5rO3-_gaJpZM4LQHS-
.

--
Shimshon Zimmerman

Web http://www.stratoscale.com/ | Blog http://www.stratoscale.com/blog/
| Twitter https://twitter.com/Stratoscale | Google+
https://plus.google.com/u/1/b/108421603458396133912/108421603458396133912/posts
| Linkedin https://www.linkedin.com/company/stratoscale

[unbit]

@shay-stratoscale why you think it is a bug ?

[sharbov]

It's a bug because chunked transfer is currently not supported.

A solution based on transforming the chunked input into a non chunked one (like nginx does) is not practical because it buffers entire requests in order to calculate the content length, which could be of very large sizes.

I believe some kind of solution is in order otherwise users which requires this functionality will prefer alternatives like gunicorn over uwsgi.

[unbit]

@shay-stratoscale chunked transfer IS supported by uWSGI, it is not supported by the WSGI standard. The PEP3333 does not cover it, and instead you always need a Content-Length. You are expecting your WSGI layer to automatically manage it, but it is a violation of the standard. In uWSGI we prefer to not enforce ad-hoc behaviour where a standard is expected, by adding specific features in the uwsgi module in the best possible way.

The http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html page should show why a specific api is "better" for chunked input. Think about a XMPP/BOSH layer where input could arrives with days of late, expliciting how you want to deal with this looks way saner to me.

[c4milo]

I just read parts of PEP3333 and chunked encoded responses seem to be considered:

• https://www.python.org/dev/peps/pep-3333/#handling-the-content-length-header
• https://www.python.org/dev/peps/pep-3333/#buffering-and-streaming

So given that @unbit also mentioned that uWSGi supports sending chunked encoded responses, I would agree with closing this issue.

A different matter is chunked encoded requests (push) which seems to be what http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html is all about and was indeed left out from PEP3333.

[mitsuhiko]

@unbit this keeps coming up more and more. One of the proposals that Werkzeug implemented is that a mode can be supported by the server where data is uncompressed/unchunked on the way in and a wsgi.input_terminated bool is set. If it's set to True then client code is supposed to ignore the content length header but instead read until the EOF marker on the input stream.

Werkzeug does this for instance: https://github.com/pallets/werkzeug/blob/1f0d4e3309ecd1b10de767e7e83868ef89b13cc0/werkzeug/wsgi.py#L207-L222

Since this issue keeps being pushed my way on an almost weekly basis now I want to know if you would be interested in supporting this?

The intended behavior would be a flag where if turned on, the server decodes and de-chunks. This means the individual chunks are no longer visible to the consumer. The server (uwsgi) then sets wsgi.input_terminated to True. The server then just needs to ensure that its input stream does not block if one were to read past the end of it but yield empty strings.

[unbit]

@mitsuhiko let's see if i got everything:

• i should add an option like '--wsgi-manage-chunked'
• when this option is set and uWSGI recognize a chunked body the env['wsgi.input_terminated'] item is set to True
• when the user calls read([n]) uWSGI will returns the decoded chunk

[mitsuhiko]

@unbit Correct. And in case the read is expected to read N bytes and only the current chunk does not contain that many bytes it would wait until the next (or final) chunk comes in and yield accordingly.

[unbit]

@mitsuhiko ok, seams reasonable. I will implement it in the master branch, if all goes well (as we are near to a release of the stable branch) @xrmx should be able to cherry-pick it in time

[unbit]

I have pushed the first implementation into master:

https://github.com/unbit/uwsgi/commit/11ca00432176f4246881870ea6c7c4ea5a26481f

just add --wsgi-manage-chunked-input to enable it

I was able to implement read([amount]), tomorrow i will try to implement readline().

In addition to this, how the iterable version of wsgi.input is supposed to work in this extension ?

@xrmx wait for some feedback before cherry-picking

[bertjwregeer]

WebOb also supports something like it, except it is currently hidden behind a webob.is_body_readable environ flag. If wsgi.input_terminated is what we officially want to start using moving forward, I'd be okay with supporting it, although the wsgi.input_terminated is not officially part of any PEP.

[mitsuhiko]

@bertjwregeer the pep process in WSGI land sadly never goes anywhere but if one wants I'm more than happy to draft a mini pep for WSGI that specifies it.

[amol-]

This is a nearly weekly question in most frameworks, if there is a general agreement to make a "standard de facto" I think it would be good to make it happen independently from a PEP acceptance.

[bertjwregeer]

@mitsuhiko if there is some document we collectively can point to that documents the de facto standard, that would be absolutely fantastic. With uWSGI supporting it, I will support it in WebOb, and then others can fall in line as well.

Having that document will help though with somewhere to point users that are wondering what it is, and somewhere to point other implementors as "hey, here's how the WSGI request/response and or web frameworks are handling it".

Thank you!

[mitsuhiko]

@bertjwregeer this is the gist where I proposed it first: https://gist.github.com/mitsuhiko/5721107

[bertjwregeer]

I've released WebOb 1.7.4 which supports the wsgi.input_terminated flag in the environ.

[davidism]

@unbit As a follow up, uWSGI should set the wsgi.input_terminated flag any time it terminates input, not just for chunked encoding. For example, if it already limits the stream to Content-Length when present, there's no need for the application to do this too.

[ronlut]

@unbit Thanks for implementing it.
Any estimation of when will it be released as part of the uwsgi package? I don't see it in the last release (2.0.17).

Update: tried it with the master codebase and the option is present, but I'm still getting an empty body while consuming the environ['wsgi.input'] stream, although the wsgi.input_terminated flag is set to True

Update2: After 2 days of testing, here are my conclusions:
The only non-working set up is nginx using uwsgi_pass + uwsgi + flask.
Other set-ups work.

• If I use nginx with: proxy_request_buffering off;, proxy_http_version 1.1; and proxy_pass HTTP_ENDPOINT_OF_UWSGI;

  and uwsgi with: wsgi-manage-chunked-input = true everything works.

• If I use nginx (without uwsgi) with: proxy_request_buffering off;, proxy_http_version 1.1; and proxy_pass HTTP_ENDPOINT_OF_FLASK_DEV_SERVER; it works.

• If I use uwsgi http socket (no nginx) with: wsgi-manage-chunked-input = true it works.

• If I use only flask (no nginx and uwsgi) it also works.

BUT, If I use nginx with:
include uwsgi_params;, uwsgi_request_buffering off; and uwsgi_pass unix:PATH_TO_myproject.sock;

and uwsgi with: socket = myproject.sock, chmod-socket = 660, wsgi-manage-chunked-input = true; it doesn't work and the stream is empty.

If you have any suggestions for why this is happening, I would be more than happy to hear.
The only thing I can detect as different is that uwsgi_pass doesn't have a proxy_http_version equivilant.
In addition the nginx docs say:

When HTTP/1.1 chunked transfer encoding is used to send the original request body, the request body will be buffered regardless of the directive value.

for uwsgi_request_buffering, while saying something a bit different for proxy_request_buffering:

When HTTP/1.1 chunked transfer encoding is used to send the original request body, the request body will be buffered regardless of the directive value unless HTTP/1.1 is enabled for proxying.

[minghuig]

I am also curious about whether or not we know when a new release with the --wsgi-manage-chunked-input flag will come out? Thanks!

[shanytc]

2018 and chunked data is still missing ... facepalm

[unbit]

@shanytc thanks for your kindness toward people working for you for free, this issue is about wsgi.input_terminated support, that is currently only in master while waiting for more feedbacks.

Chunked support is in uWSGI by 2012: https://uwsgi-docs.readthedocs.io/en/latest/Chunked.html

[yoobles]

Is it possible to release https://github.com/unbit/uwsgi/commit/11ca00432176f4246881870ea6c7c4ea5a26481f?

For those that come here looking for a solution, you should look at the following:
https://uwsgi-docs.readthedocs.io/en/latest/Changelog-1.9.13.html?highlight=uwsgi.chunked_read#chunked-input-api
https://github.com/sileht/gnocchi/blob/61a975e5780c396e6a642e4333911560327f0e6e/gnocchi/rest/influxdb.py#L143-L152