Django-storages: S3Boto3Storage raises ValueError: I/O operation on closed file.

[tsifrer]

When running python manage.py collectstatic we get the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "items()venvs/app_root/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
    collected = self.collect()
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 139, in collect
    for original_path, processed_path, processed in processor:
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 246, in post_process
    for name, hashed_name, processed, _ in self._post_process(paths, adjustable_paths, hashed_files):
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 312, in _post_process
    hashed_name = self.hashed_name(name, content_file)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 109, in hashed_name
    file_hash = self.file_hash(clean_name, content)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 86, in file_hash
    for chunk in content.chunks():
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/files/base.py", line 76, in chunks
    self.seek(0)
ValueError: I/O operation on closed file.

This only happens when using django-storages 1.6.4 or above.
Versions 1.6.3 and lower work fine.

We're using Django 1.11.4, python 3.5.2, boto3 1.4.6.

[jschneier]

@tsifrer thanks for the report. Just to clarify -- this happens on both 1.6.4 and 1.6.5?

[tsifrer]

@jschneier Correct, happens on both, 1.6.4 and 1.6.5.

[Alberick]

I'm also having this issue, any update on it ?

[xrmx]

It looks like the save of S3Boto3Storage now closes the file. I think it's a side effect of https://github.com/jschneier/django-storages/commit/c73680e7d7f906b841f77dd1aa4f5c3cfa8fb3a2 but haven't tried with the commit reverted.

[abhinavnair]

I am also facing this issue but not with collectstatic. But rather with uploading a file. I just moved from boto 2.40.0 to boto3 1.4.7 and django-storages from 1.1.8 to 1.6.5. After reading the comments above, I changed django-storages to 1.6.3 but the issue still occurs. Any help?

[abhinavnair]

I just tried going back versions even further and at 1.5.0, this issue doesn't occur. It occurs on every version later than 1.5.0.
Moreover this issue only arises when I am trying to upload an image file after cropping it. For all other cases, there is no problem at all.

Here is a snippet for uploading a cropped image file:-

def decodeImage(dataurl):
    from django.core.files.temp import NamedTemporaryFile
    from django.core.files import File
    imgstr = re.search(r'base64,(.*)', dataurl).group(1)
    img_temp = NamedTemporaryFile(delete=True)
    img_temp.write(imgstr.decode('base64'))
    img_temp.flush()
    return File(img_temp)

def post(self, request, *args, **kwargs):
    dataurl = request.POST['cropped']
    new_pic = decodeImage(dataurl)
    from django.utils.timezone import now
    self.myobject.profile_picture.save(str(now()), new_pic)

[Flimm]

I experience this issue with version 1.6.5, but not with 1.6.3.

[ghost]

Same as @abhinavnair for me. Uploading a file results in a crash, but collectstatic is perfectly fine.

The bug seems to be filetype independent, The same issue occurs on image and text files.

The 1.5 version works well, but the bug shows up on every version greater than it.

[WhyNotHugo]

Has anyone manged to work around this?

[xrmx]

@WhyNotHugo yes, you don't need to assume that the saved file handler is open https://github.com/jschneier/django-storages/issues/382#issuecomment-327761800

[RaidoS]

@abhinavnair
thx, downgrade to 1.5.0 helps

[WhyNotHugo]

@xrmx It's not an assumption I'm making; it's an assumption ManifestFilesMixin makes (and that's part of Django itself). I've pushed #437 to work around this (I'm working on fixing a few tests right now).

Note that, if you're using ManifestFilesMixin, you also need to set AWS_QUERYSTRING_AUTH = False; otherwise there's an infinite loop rewriting file path URLs.

[coderberry]

I had to lock the Pipfile to this:

"boto3" = "==1.5.0"
django-storages = "==1.6.3"

[akalinmehmet]

I had this issue too with django-storages == "1.6.5" but after downgrading to "1.5.0" with suggestion by @abhinavnair problem doesn't occured.

[Flimm]

I get this issue with django-storages version 1.6.6 too.

[charlesthk]

Using a custom StorageClass fix the issue :

from storages.backends.s3boto3 import S3Boto3Storage, SpooledTemporaryFile
import os


class CustomS3Boto3Storage(S3Boto3Storage):

    def _save_content(self, obj, content, parameters):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose instance
        super(CustomS3Boto3Storage, self)._save_content(obj, content_autoclose, parameters)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

[BojanKogoj]

I tried suggestion by @coderberry and it didn't work.

I tried @WhyNotHugo pull request (did pip install git+git://github.com/WhyNotHugo/django-storages.git@fix-valueerror-boto3) but it didn't help either.

One thing I noticed was that when I tried to upload an image the first time it crashed (but uploaded), but second attempt worked. Every image worked in second attempt but not in first.

I managed to get it working with example by @charlesthk

[Safrone]

the PR from @WhyNotHugo worked with Boto3==1.5.36 and it seems @charlesthk's fix works with the latest version of everything django-storages[boto3]==1.6.6 and boto3==1.7.3

[SlipJ]

@charlesthk can you make a pull request with this to the master?

[nikssardana]

I tried the solution by @charlesthk but it gives me an error: cant' start new thread.
Can anyone help me with this?

Edit: The error occurs once in a while but not every time. I have been searching for the solution but found nothing.

[martinvol]

I confirm that collectstatic works downgrading django-storages to 1.6.3. (I'm using the Storage class used in #151)

[martinvol]

And I also confirm that @charlesthk solution works with django-storages 1.6.6 (currently the lastest)

Thanks @charlesthk! 🎉

[sww314]

481 fixes similar issue in Google Cloud Storage. All the storages need to act like the Django file storage, which calls content.seek(0) before doing anything else.

[Flimm]

I still experience this issue in django-storages 1.7.1 .

[rnegron]

Also having this issue. The custom class in @charlesthk's comment solves it.

[probabble]

What needs to happen to get this fix enabled by default?

[selected-pixel-jameson]

@charlesthk Do you know if your custom save content method will work with the ManifestFilesMixin? I'm haven't had a problem running this until I added the mixin and now I'm seeing this exact error.

[tisdall]

@selected-pixel-jameson - I also wasn't having an issue until I started using ManifestFilesMixin and charlesthk's solution fixed it.

[singhravi1]

Guys, Any updates? When the updated version is going to be released?
@charlesthk Did you submit a pull request with your solution?

[chescales]

+1 on this, I also used charlesthk solution temporarily and it worked with:

django-storages==1.7.1
boto3==1.9.168
botocore==1.12.168
Django==1.11.21

[nikolas]

Pull request with @charlesthk's solution is here: https://github.com/jschneier/django-storages/pull/717

[rh0dium]

Would it be possible to get a new release out?

[Flimm]

django-storages 1.7.2 fixes this issue for me. Thank you for your work, @jschneier !

[magraeber]

I tried the new release 1.7.2. It did not solve my issue. But @charlesthk solution fixed it.

[jschneier]

Please give detailed reproduction steps.

On Tuesday, September 10, 2019, magraeber [e-mail] wrote:

I tried the new release 1.7.2. It did not solve my issue. But @charlesthk
https://github.com/charlesthk solution fixed it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jschneier/django-storages/issues/382?email_source=notifications&email_token=AAREDWHWXNSTHEOHARPIOV3QI7X77A5CNFSM4DW6MAYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6MG6FI#issuecomment-530083605,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAREDWB3X773VN5VEAZEUFLQI7X77ANCNFSM4DW6MAYA
.

[jschneier]

Also note that the bug was fixed in 1.10 so if you are on a Django before that (upgrade!) you will still see this issue.

[magraeber]

@jschneier, thanks for your fast reply.
I was using Django 2.2.1 und just now updated to 2.2.5. This update didn't change anything.
It's is very hard to give detailed information to reproduce the step: It only happens with certain files, other files of the same type work perfectly. And our Django project is quite large.

For me the fix with custom save functions is totally fine. I just wanted to report the issue. Maybe there are other users, that also still experience problems.

If you are interested, I could try to copy our Django project and strip it down to only the relevant parts and share it via Github. Or are there any information (code snippets, package versions) I can give you right now?

[jschneier]

Could it be related to file size? Whether or not the file is being compressed (gzipped)? If there was a nice way to reproduce that would be best obviously but I don't want to ask for too much.

[magraeber]

That's a good hint. It looks like file size does influence the error. Larger files work. Here is how I test it locally:

import requests

# Works perfectly with a large file > 10 MB
files = {'file': ('large.file', open('large.file', 'rb'))}
response = requests.post('http://localhost:8000/api/v1/file/',  files=files)

# Get server-side error "ValueError: I/O operation on closed file." with a smaller file < 3 MB
files = {'file': ('small.file', open('small.file', 'rb'))}
response = requests.post('http://localhost:8000/api/v1/file/',  files=files)

As you can see, I'm not compressing the files before sending. Of course I'm not really sure, what the requests library is doing internally. Or do you mean some server-side compressing in boto3 or django-storages?

[jschneier]

django-storages gzips certain kinds of files if you tell it to, that’s what I meant by compression. Can you give an abbreviated overview of your settings as well. Also double check the output of storages.__version__ just for sanity!

On Sep 11, 2019, at 00:54, magraeber [e-mail] wrote:

That's a good hint. It looks like file size does influence the error. Larger files work. Here is how I test it locally:

import requests

Works perfectly with a large file > 10 MB

files = {'file': ('large.file', open('large.file', 'rb'))}
response = requests.post('http://localhost:8000/api/v1/file/', files=files)

Get server-side error "ValueError: I/O operation on closed file." with a smaller file < 3 MB

files = {'file': ('small.file', open('small.file', 'rb'))}
response = requests.post('http://localhost:8000/api/v1/file/', files=files)
As you can see, I'm not compressing the files before sending. Of course I'm not really sure, what the requests library is doing internally. Or do you mean some server-side compressing in boto3 or django-storages?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/jschneier/django-storages/issues/382?email_source=notifications&email_token=AAREDWDDRRG5JT2LUWSYPITQJCP3NA5CNFSM4DW6MAYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6NTS4I#issuecomment-530266481, or mute the thread https://github.com/notifications/unsubscribe-auth/AAREDWF5ONPNFWGE5TKRJLDQJCP3NANCNFSM4DW6MAYA.

[magraeber]

I checked the version again. It is 1.7.2.

This seems to be the relevant part in settings.py:

AWS_STORAGE_BUCKET_NAME = env('DJANGO_AWS_BUCKET_NAME')
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
        'CacheControl': 'max-age=86400',
    }
AWS_S3_SIGNATURE_VERSION = 's3v4'
AWS_S3_REGION_NAME = 'eu-central-1'

MEDIAFILES_LOCATION = 'media/public'
MEDIA_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'apps.core.custom_storage.MediaStorage'

STATICFILES_LOCATION = 'static/public'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'apps.core.custom_storage.StaticStorage'

And here is the code in custom_storage.py (before I added the custom save function from charlesthk):

class MediaStorage(S3Boto3Storage):
    location = settings.MEDIAFILES_LOCATION
    querystring_expire = 1800  # 30 min until the generated link expires
    default_acl = 'private'
    custom_domain = False

[KanishkTanwar]

@magraeber +1
Is this issue fixed i just used the above class and it did the trick.
Does any one know any another work around.

[scream4ik]

I dont think that the issue can be closed. I still get ValueError: I/O operation on closed file.
And the custom class in @charlesthk's comment solves it.

Django==2.2.9
django-storages==1.8
boto3==1.10.42
botocore==1.13.42

[voiddragon]

Sadly the custom class has stopped working due to S3Boto3Storage not having a _save_content function anymore

[pasevin]

@voiddragon

Note: as others have found out there's a bug in this rewrite, refer to https://github.com/jschneier/django-storages/issues/382#issuecomment-592876060 for a fixed version.

Here's rewritten custom class:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose
        # instance
        super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

[voiddragon]

@pasevin

It doesn't seem to work though

When I upload a file it still shows up as no file on the django admin for example.

[pasevin]

@voiddragon that's odd, it works for me. I don't see the reason why it shouldn't work.

[mannpy]

@pasevin Thank you! It doesn't work because the method doesn't return a result from the parent class.
@voiddragon
Here is the fixed version:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

[pasevin]

Thanks, @mannpy it just saved me, I totally haven't noticed that bug...

[selected-pixel-jameson]

Has anyone verified that @mannpy 's solution works with manifestfilesmixin?

[oriolcp]

this issue should be reopened. I still get ValueError: I/O operation on closed file.
And the custom class https://github.com/jschneier/django-storages/issues/382#issuecomment-592876060 solves it.

Django==2.2.12
django-storages==1.9.1
boto3==1.13.19
botocore==1.16.19

[shuckc]

Thank you to the kind souls who update this page. I visit after every major Django update to grab the latest workaround. You don't have unit tests, CI, or release notes, but like migrating birds we know where to come. In the spirit of all the duct tape that holds the internet together, Bravo.

[jschneier]

Thanks for the poetry.

Is there a reproducible slimmed down Django app? That is the easiest way to fix this...several years later.

[erdseb]

Thanks for the poetry.

Is there a reproducible slimmed down Django app? That is the easiest way to fix this...several years later.

Do you have a writable S3 test account to test with?

I first saw ValueError: I/O operation on closed file. when I switched from plain S3Boto3Storage to that plus ManifestFilesMixin on to today:

from django.contrib.staticfiles.storage import ManifestFilesMixin
from storages.backends.s3boto3 import S3Boto3Storage


class S3StaticFilesStorage(ManifestFilesMixin, S3Boto3Storage):
    pass

I believe it's about 180 static files total and django-storages 1.7.1 in that setup.

[loicgasser]

I can confirm that I have this issue as well with.

Django==3.0.7
django-storages==1.9.1
boto==2.49
boto3==1.14.1

Last traceback line

  File "/home/loicgasser/Workspace/service-zuudoo/.venv/lib/python3.7/site-packages/storages/backends/s3boto3.py", line 546, in _save
    content.seek(0, os.SEEK_SET)
ValueError: I/O operation on closed file.

Storage class:

class MediaStorage(S3Boto3Storage):
    location = 'media'

[ironworld]

@pasevin Thank you! It doesn't work because the method doesn't return a result from the parent class.
@voiddragon
Here is the fixed version:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

Thanks! This fix works for django-storages version 1.9.1.

[nikolas]

Pull request opened for this fix here: https://github.com/jschneier/django-storages/pull/905

[kshpytsya]

For me, a patch like this (which does not involve copying) fixes the issues.

--- site-packages/storages/backends/s3boto3.py.orig     2020-07-31 11:46:19.309504453 +0300
+++ site-packages/storages/backends/s3boto3.py  2020-07-31 11:58:25.400758620 +0300
@@ -38,10 +38,16 @@


 boto3_version_info = tuple([int(i) for i in boto3_version.split('.')])


+# https://github.com/boto/s3transfer/issues/80#issuecomment-562356142
+class NonCloseableBufferedReader(io.BufferedReader):
+    def close(self):
+        self.flush()
+
+
 @deconstructible
 class S3Boto3StorageFile(File):

     """
     The default file object used by the S3Boto3Storage backend.
@@ -542,11 +548,12 @@
         obj = self.bucket.Object(encoded_name)
         if self.preload_metadata:
             self._entries[encoded_name] = obj

         content.seek(0, os.SEEK_SET)
-        obj.upload_fileobj(content, ExtraArgs=params)
+        with NonCloseableBufferedReader(content) as reader:
+            obj.upload_fileobj(reader, ExtraArgs=params)
         return cleaned_name

     def delete(self, name):
         name = self._normalize_name(self._clean_name(name))
         self.bucket.Object(self._encode_name(name)).delete()

[loicgasser]

This looks like a better solution, can we possibly have a patch for this and new version? I know it's because of s3transfer now, but this could be nice to have in django-storages.

[robertpro]

@pasevin Thank you! It doesn't work because the method doesn't return a result from the parent class.
@voiddragon
Here is the fixed version:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

Thanks! This fix works for django-storages version 1.9.1.

Can I see a full version of your code? where does ABC and SpooledTemporaryFile comes from?

[pasevin]

@robertpro
ABC (Pycharm added this to silence "Class must implement all abstract methods" error): https://docs.python.org/3/library/abc.html
SpooledTemporaryFile: https://docs.python.org/3/library/tempfile.html#tempfile.SpooledTemporaryFile

[robertpro]

@pasevin Thanks!

These are the imports:

import os
from abc import ABC
from tempfile import SpooledTemporaryFile

from storages.backends.s3boto3 import S3Boto3Storage

[Yeppman]

November 2020 , I downgraded django stotages from 1.9.0 to 1.5.0, it solved the issue for me

[jschneier]

For me, a patch like this (which does not involve copying) fixes the issues.

--- site-packages/storages/backends/s3boto3.py.orig     2020-07-31 11:46:19.309504453 +0300
+++ site-packages/storages/backends/s3boto3.py  2020-07-31 11:58:25.400758620 +0300
@@ -38,10 +38,16 @@


 boto3_version_info = tuple([int(i) for i in boto3_version.split('.')])


+# https://github.com/boto/s3transfer/issues/80#issuecomment-562356142
+class NonCloseableBufferedReader(io.BufferedReader):
+    def close(self):
+        self.flush()
+
+
 @deconstructible
 class S3Boto3StorageFile(File):

     """
     The default file object used by the S3Boto3Storage backend.
@@ -542,11 +548,12 @@
         obj = self.bucket.Object(encoded_name)
         if self.preload_metadata:
             self._entries[encoded_name] = obj

         content.seek(0, os.SEEK_SET)
-        obj.upload_fileobj(content, ExtraArgs=params)
+        with NonCloseableBufferedReader(content) as reader:
+            obj.upload_fileobj(reader, ExtraArgs=params)
         return cleaned_name

     def delete(self, name):
         name = self._normalize_name(self._clean_name(name))
         self.bucket.Object(self._encode_name(name)).delete()

I much prefer this non-copying solution given that the reason that upload_fileobj was added was to add support for very large files.

[jschneier]

Can someone verify that this branch resolves their issue?

https://github.com/jschneier/django-storages/pull/955

[magraeber]

Yes, it solves my issue.

[israellias]

@jschneier I need to wait to see this change on pypi?
It is secure to put master of this repo in my requirements.txt?

[jschneier]

@jschneier I need to wait to see this change on pypi?
It is secure to put master of this repo in my requirements.txt?

I'm going to push in a breaking change at some point so I wouldn't recommend. I'll just deploy as-is actually.

[jschneier]

@israellias just released.

[jschneier]

Unfortunately the fix seems to have broken things more generally (see #967) so I've put together an alternative one. My solution is to separate out the collectstatic handling since that seems to be quite different & not as common. Please take a look at #968.

This way it'll just be a quick update to a setting and any additional fixes are logically separated.

[djackson-exo-inc]

@jschneier Can you please specify what tag/release this is fixed in?