Falcon: How do I handle multipart form based file uploads?

[dukedougal]

There seems to be no convenient way to handle multipart form based file uploads. Where should I look to work out how to do this?

Is there documentation to explain how to handle this?

thanks

[kgriffs]

You should be able to do this using cgi.FieldStorage. For example (not tested):

https://gist.github.com/kgriffs/fe371bfbb2d63211889b

[kgriffs]

(TODO: consider supporting this out of the box, or adding to a separate "talons" project)

[oscarcp]

@kgriffs I think supporting multipart forms is a must, but I don't know your design scope for Falcon.

It will certainly make forms way easier, because right now we have to consume and process the stream by hand, which could be done directly by Falcon, and leave the logic part to the user, I don't mean a fully fledged wrapper like django, but something like req.files that can deal with the storage would be nice (if that makes sense)

[kgriffs]

The primary design scope for Falcon has been around REST APIs, where data would just be POSTed or PUT directly, not using HTML forms. That being said, if something is a common ask (the two prime examples being form and cookie support), and it doesn't require pulling in dependencies and/or bloating the framework, I think it makes sense to go ahead and implement it directly in Falcon rather than relying on middleware or hooks.

[mcm]

@kgriffs would you want to use cgi.FieldStorage for this, or would you want to implement it natively? This is probably something we could implement pretty easily after the various "how to access self.stream" issues, and the form parsing, are sorted out.

It would likely make sense to allow to specify a max file size to keep in memory, and then pass off to a temporary file instead.

[dukedougal]

@kgriffs some REST API's are a little loose and output JSON with REST like interfaces but accept HTML forms.

You may want to contact Marcel Hellkamp and ask if you can integrate this into Falcon https://pypi.python.org/pypi/multipart/0.1

[mcm]

@kgriffs I think this would be a perfect thing for the collection of middlewares we talked about this week. I had a few thoughts on how to implement the file stuff in a middleware. Let me know if you want to start a repo for that stuff and I can work on something for this.

[ourway]

Well, I personally ditched Pyramid in favor of Falcon, But for a simple file upload, I had to write 20 lines of code and two hours of research. I think it's very bad. I am using Falcon in my daily tasks, but for some simple tasks, I have to switch to bottle. Please consider implementing this feature as a native option.

[dukedougal]

@ourway it is super easy to use werkzeug with Falcon and this provides the functionality you need.

[yohanboniface]

@ourway why not publishing your work as a falcon plugin? A framework is not just the core, it's also the ecosystem, which is missing right now for falcon, and I think what we, falcon users can do, is to build this ecosystem. :)
I'm not saying the multipart parsing must only be in a plugin, but that may be a first step :)

Offtopic: @kgriffs it could be nice to have a page to start referencing the known plugins. Maybe a wiki page for now?

[dukedougal]

Here's a little bit of code showing how I use werkzeug for form handling in Falcon. Sorry it's not a standalone fully functioning app but just clipped out of a larger app:

You'll need to install werkzeug

pip install werkzeug

Then at the top of your code, import it:

from werkzeug.wrappers import Request

Then in your Falcon app use it to handle forms like this:

def ProfileVARPUT(self):
    # updates a users profile fields
    werkzeug_request = Request(self.req.env)

    full_name = werkzeug_request.form.get('full_name', None)
    if full_name and (helpers.utf8len(full_name) > 250):
        description = 'Full name cannot be longer than 250 characters.'
        raise HTTPError(status.HTTP_400, title=status.HTTP_400, description=description)

    row = { 'website'        : werkzeug_request.form.get('website', None),
            'ssh_key'        : werkzeug_request.form.get('ssh_key', None),
            'github'         : werkzeug_request.form.get('github', None),
            'location'       : werkzeug_request.form.get('location', None),
            'twitter'        : werkzeug_request.form.get('twitter', None),
            'image'          : werkzeug_request.form.get('image', None),
            'full_name'      : werkzeug_request.form.get('full_name', None),
    }
    db_blah.Session.query(Users).filter_by(user_id=self.user_id).update(row)
    db_blah.Session.commit()

    self.users_service.trigger_git_newuser(user_id=self.user_id)
    self.resp.status = status.HTTP_201 # created

[philiptzou]

A req.files mentioned in #493 could be a good idea. Also, with my further suggest this can be introduced as a (standard) plugin into Falcon.

[yohanboniface]

I needed multipart support for a project of mine, so I moved out and packaged @kgriffs suggestion on a very simple plugin: https://github.com/yohanboniface/falcon-multipart

It works for my current use case and tests, but there is not guaranty it will work for yours ;)
Usually, multipart parsing is a black box I just need to call from outside, so I'm moving forward step by step. But hey, better than nothing ;)

From my point of view, aim is to focus discussion around this code until we have a fully functional (tested) parser, and then discuss about merging it in Falcon core or keep it as plugin.

BTW, I also needed to add multipart encoding support to pytest-falcon.

[kgriffs]

@yohanboniface cool, thanks for sharing! Would you mind posting to the mailing list as well to help loop in more members of the community?

[vectorfrog]

@yohanboniface Thanks! Your plugin is working great for me!

[rr-]

@yohanboniface can I combine JSON data with binary data in one POST request as seen here using your solution? If so, would you be so kind and provide some CURL+mini Python example?

[kgriffs]

BTW, falcon-multipart has been added to a wiki page along with a few other goodies. Everyone's welcome to include their projects on this page: https://github.com/falconry/falcon/wiki/Add-on-Catalog

I'd like to have a design session around form parsing during the upcoming PyCon sprints. It may make sense to pull this functionality into the core framework, or at least make it possible to attach new attributes to the Request and Response classes in lieu of implementing custom child classes.

[Camilo-HG]

Hi

I'm trying to implement POST request for uploading files.

I have used falcon-multipart in order to multipart/form-data, this allow me to retrieve my file in a cgi.FieldStorage() in which file is in binary format, but now, I need to write this file in a directory with the original extension.

This is the code I'm using.

app.py:

import falcon

from .files import Resource

from falcon_multipart.middleware import MultipartMiddleware

api = application = falcon.API(middleware=[MultipartMiddleware()])

files = Resource()
api.add_route('/files', files)

files.py:

import io
import os
import shutil

import falcon
import json


class Resource(object):

   _storage_path = './uploaded_files'

   def on_post(self, req, resp):
        """
        POST METHOD
        """
        # Retrieve file extension
        ext = req.get_param('extension')

        # Retrieve input_file
        input_file = req.get_param('file')

        # Read file as binary
        raw = input_file.file.read()

        # Retrieve filename
        filename = input_file.filename

        # Define file_path
        file_path = os.path.join(self._storage_path, filename)

        # Write to a temporary file to prevent incomplete files from
        # being used.
        temp_file_path = file_path + '~'

        # Finally write the data to a temporary file
        with open(temp_file_path, 'wb') as output_file:
            shutil.copyfileobj(raw, output_file)

        # Now that we know the file has been fully saved to disk
        # move it into place.
        os.rename(temp_file_path, file_path)

        resp.status = falcon.HTTP_201

[branelmoro]

I have added changes in request.py in falcon framework to parse application/x-www-form-urlencoded and multipart/from-data.
I have raised pull request - https://github.com/falconry/falcon/pull/1236 it is not yet merged in master.
Check this - https://github.com/branelmoro/falcon
I hope this will make file upload and post parameters easy.

[vytas7]

Hi all involved in the discussion!
I'm going to close this now since the work is primarily tracked as https://github.com/falconry/falcon/issues/953 (within the Falcon 3.0 milestone).