Django-import-export: Import file flow breaks when horizontally scaled app

[gone]

I'm getting this error on file upload of my prod server, only recently when I bumped up the number of dynos:

  File "/app/.heroku/python/lib/python2.7/site-packages/import_export/admin.py", line 122, in process_import
    import_file = open(import_file_name, input_format.get_read_mode())

IOError: [Errno 2] No such file or directory: u'/tmp/tmpPWSbPm'

The only thing that makes sense to me is the upload and process is happening in separate web requests? Am I missing a config value?

[bmihelac]

I just found this related SO entry:

http://stackoverflow.com/questions/17289459/heroku-tmp-folder-deletion

It looks like temporary file should be stored using File storage API for such cases.

[gone]

I'm a little worried about using the storage api for a temporary file - I point to an s3 bucket for my default file store in my applications, and I apply world readable perms to the bucket. Some of the files I'm importing contain sensitive data. That means I'd need to setup a file store just for import-export and introduce a few new setting value to control that store's settings. if that's stlil the way that makes the most sense to you I could give it a try.

I just took a shot at a difference approach in #133 by serializing the imported data to json and including that with the confirm response. I don't think this swells the response too much because we're already including the imported data - so we go from n to 2n.

I could see going a step farther and turning the confirm form into something that contains a dynamically generated formset via a resource -> form transformation and treating the whole second step as a form submit, which would allow last min changes to the data before import without leaving the page. Interested in feedback!

[marksweb]

I've had this running from an EC2 server (single instance so not an issue with load balancing between multiple servers).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bubbassauro]

Leaving this here since I had the same issue and it wasn't clear to me from the documentation.

What may happen if you're running on an EC2 server behind a load balancer is:

1. You upload the file
2. You see a preview. By default the temporary file is stored in the server's file system
3. You click upload and there's a chance you'll hit the other server behind the LB, which doesn't have the file.

So you can use the CacheStorage instead in your admin.py, for example:

from import_export.tmp_storages import CacheStorage

class CustomBookAdmin(ImportMixin, admin.ModelAdmin)
    tmp_storage_class = CacheStorage
    resource_class = BookResource

In our case, we're using the ImportExportMixin in multiple classes, so I added this:

from import_export.tmp_storages import CacheStorage

class ImportExport(ImportExportMixin):
    """Override temporary storage to use cache instead of file system because
    filesystem has issues when the application is running behind a load balancer"""
    tmp_storage_class = CacheStorage

And replaced on the admin classes that were using it:

class CustomBookAdmin(ImportExport, admin.ModelAdmin):
    # other admin options...