Django-storages: Google App Engine Issue with Tempfile Library

[pasiorovuo]

This is not a real bug, but an issue caused by Google App Engine...

I'm trying to run a Mezzanine CMS app on Google App Engine (with Google Cloud SQL and Google Cloud Storage backends). The app fails on start with

ERROR    2017-03-12 20:04:24,811 wsgi.py:263] 
Traceback (most recent call last):
  File "/Users/por/Applications/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Users/por/Applications/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Users/por/Applications/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject
    __import__(cumulative_path)
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/marttilantia/wsgi.py", line 18, in <module>
    application = get_wsgi_application()
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/lib/django/core/wsgi.py", line 13, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/lib/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/lib/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/lib/django/apps/config.py", line 90, in create
    module = import_module(entry)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/storageutils/__init__.py", line 1, in <module>
    import storages.backends.gs
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/lib/storages/backends/gs.py", line 5, in <module>
    from storages.backends.s3boto import S3BotoStorage, S3BotoStorageFile
  File "/Users/por/Sites/marttilantia-mezzanine/marttilantia/lib/storages/backends/s3boto.py", line 6, in <module>
    from tempfile import SpooledTemporaryFile
ImportError: cannot import name SpooledTemporaryFile

This is caused by a feature in Google App Engine described briefly in here. Python Standard Library tempfile implementation is severely limited, and does not contain SpooledTemporaryFile.

It would be nice to have a way around this and use django-storages with Google App Engine.

[scjody]

@pasiorovuo Interesting idea. Note that a bunch of backends use SpooledTemporaryFile so it might be worth coming up with a solution that works for all backends, although I realize a GAE user is most likely to use Google Cloud Storage.

If you have time to work on a PR, I suggest branching off #236 so your changes can be merged after those. (I don't think it's a good idea to combine the two PRs because your change will likely impact more than one backend.)

[EssaAlshammri]

I have had this issue before. and my project is in appengine standard environment
and I used this and everything worked like a charm but with slight changes.
it hasn't be touched for a while though.

if you don't wanna use it let's wait for this PR to be merged :)

[madisona]

I'm sure it is less efficient, but I've just added this monkey patch in appengine_config.py and it allows the library to work just fine. It would be nice to have a more efficient permanent solution

import tempfile
tempfile.SpooledTemporaryFile = tempfile.TemporaryFile

[pasiorovuo]

Quite a simple and nice solution. I circumvented the issue by writing a custom tempfile.py module that returns StringIO objects.

[ibrokemycomputer]

@madisona Your fix works great for getting the app to run again, however it interferes with form file uploads.

TemporaryFile() got an unexpected keyword argument 'max_size'

The difference between the spooled and non-spooled temporary files is the extra 'max_size' parameter that causes a 500 error on form submission. If I find a solution I will post it here!

[stusayre]

@madisona this worked for me too thank you

[cyrilguerard]

@ibrokemycomputer You can make it work by using original version in Python librairies 2.7.

Copy https://svn.python.org/projects/python/tags/r27/Lib/tempfile.py into a file tempfile2.py

import tempfile
import tempfile2
tempfile.SpooledTemporaryFile = tempfile2.SpooledTemporaryFile

[jschneier]

@ibrokemycomputer @cyrilguerard something like this?

import tempfile

class SpooledTemporaryFile(tempfile.TemporaryFile):
    def __init__(self, *args, **kwargs):
        kwargs.pop('max_size', None)
        tempfile.TemporaryFile.__init__(*args, **kwargs)

tempfile.SpooledTemporaryFile = SpooledTemporaryFile

[drmauij]

No idea if it can be of any help. After applied the monkey patch suggested from @madisona I kept having the following error:
TransportError: ('Connection broken: IncompleteRead(173 bytes read)', IncompleteRead(173 bytes read))
which I solved applyng another monkey patch :-P

import requests import requests_toolbelt.adapters.appengine requests_toolbelt.adapters.appengine.monkeypatch()

Now it works fine.

[jesteria]

If this helps anyone, I couldn't create a SpooledTemporaryFile shim by subclassing TemporaryFile – the latter is actually a function, (which returns the opened file object).

So instead what worked for me was just:

import tempfile

def SpooledTemporaryFile(*args, **kwargs):
    kwargs.pop('max_size', None)
    return tempfile.TemporaryFile(*args, **kwargs)

tempfile.SpooledTemporaryFile = SpooledTemporaryFile

(And, I also needed to install requests_toolbelt and run that monkeypatch.)

Thanks, all.

---

@ibrokemycomputer @cyrilguerard something like this?

import tempfile

class SpooledTemporaryFile(tempfile.TemporaryFile):
    def __init__(self, *args, **kwargs):
        kwargs.pop('max_size', None)
        tempfile.TemporaryFile.__init__(*args, **kwargs)

tempfile.SpooledTemporaryFile = SpooledTemporaryFile