Tornado: parse_multipart_form_data for stream_request_body

[Lookyan]

Hi!

We have a good feature for streaming HTTP body (decorator stream_request_body).
Using stream_request_body we get chunks of HTTP body in data_received method. But this data is raw. And It will be cool to have a support for higher level tool to work with streaming body (in case of multipart_form_data).

If it is not solved yet I can do this, but I'm not sure about architecture for solution.
I want to implement smth like StreamParser, which will get boundary and infinite generator which will return file-like objects. This parser we can instantiate in prepare method and then simply pass chunks in data_received method to it. Parser will write to these file-like objects and efficiently use RAM. But this solution has drawbacks. At least, we don't have universal solution in case if we want to handle data with some logic. Also we can't return any meta info from these file-like objects. But there can be important information such as ObjectId in case of using for example mongodb gridfs.

Another solution is to implement a subclass of RequestHandler with stream_request_body decorator and overridden prepare and data_received methods. We can add some extra abstract methods such as open_file, close_file, new_data and pass to them info about current state and chunks which we can write to file and do whatever we want without thinking about boundaries, headers.

What do you think about it?

[ajdavis]

This seems like a useful idea; I believe you can implement it with Tornado's public API, so you could publish it as a separate package that provides a RequestHandler subclass for those users who need it. But perhaps it's close enough to Tornado's core feature set that it should be part of Tornado itself.

Ben, curious what you think?

[bdarnell]

I think this belongs in Tornado itself, although maybe it should first be introduced as a separate package so we can be sure we have the API right. Whatever we come up with should be extensible to handle other kinds of parsing too, so it can't just be a subclass of RequestHandler (although we could provide one for convenience if that's the only parser you need).

My thought on the API is to define a StreamingFormDataParser class, whose constructor takes a StreamingFormDataParserDelegate (and maybe a request object or a headers dict). The delegate interface defines methods like start_file, file_data_received, finish_file. The parser has a data_received and finish method, which parses a chunk of data and calls methods on the delegate. You could use this on its own, but the most common use case would be to create a parser in prepare using self as the delegate, and forward data_received and finish calls to the parser. Or there could be pre-made delegates that could store the files directly in s3/gridfs/etc.

[siddhantgoel]

We ran into the same problem recently, so I ended up writing a small module for this purpose. I started with the API interface suggested in Ben's comment, but then ended up changing it slightly as I made progress. Although if I think about it, it's really not all that different.

Here is the source, and it's pip-installable under the name streaming-form-data. It uses Cython (and the generated .c file is included), so it'll end up building a C extension on installation.

So far, it seems to work fine. To be extra sure about correctness, I also imported the test cases for tornado.httputil.parse_mutlipart_form_data, in addition to writing some tests of my own.

Anyway, hope this is helpful, and I'm curious about thoughts!

[sizeoftank]

Really need this feature too,
Is any appropriate way to process file upload stream now?

[Lookyan]

@sizeoftank I'll publish my library soon.

[Lookyan]

@sizeoftank https://github.com/Lookyan/tornado-stream-request