Django-storages: OSError: File does not exist

[luccascorrea]

I am using django-storages version 1.7.2 and boto3 version 1.5.22.

A crash has started happening lately when saving an image field, I am not sure why. The crash doesn't happen every time. Here's the full stack trace:

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/storages/backends/s3boto3.py" in _open
 461.             f = S3Boto3StorageFile(name, mode, self)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/storages/backends/s3boto3.py" in __init__
 69.             self.obj.load()

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/boto3/resources/factory.py" in do_action
 505.                 response = action(self, *args, **kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/boto3/resources/action.py" in __call__
 83.         response = getattr(parent.meta.client, operation_name)(**params)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/botocore/client.py" in _api_call
 317.             return self._make_api_call(operation_name, kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/botocore/client.py" in _make_api_call
 615.             raise error_class(parsed_response, operation_name)

During handling of the above exception (An error occurred (404) when calling the HeadObject operation: Not Found), another exception occurred:

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
 35.             response = get_response(request)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
 128.                 response = self.process_exception_by_middleware(e, request)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
 126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
 44.         response = view_func(request, *args, **kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/wagtail/admin/urls/__init__.py" in wrapper
 102.             return view_func(request, *args, **kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/wagtail/admin/decorators.py" in decorated_view
 34.             return view_func(request, *args, **kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/wagtail/admin/utils.py" in wrapped_view_func
 220.                 return view_func(request, *args, **kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/wagtail/images/views/chooser.py" in chooser_upload
 161.             form.save()

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/wagtail/admin/forms/collections.py" in save
 71.         return super().save(commit=commit)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/forms/models.py" in save
 456.             self.instance.save()

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/base.py" in save
 729.                        force_update=force_update, update_fields=update_fields)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/base.py" in save_base
 759.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/base.py" in _save_table
 842.             result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/base.py" in _do_insert
 880.                                using=using, raw=raw)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/manager.py" in manager_method
 82.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/query.py" in _insert
 1128.         return query.get_compiler(using=using).execute_sql(return_id)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in execute_sql
 1284.             for sql, params in self.as_sql():

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in as_sql
 1237.                 for obj in self.query.objs

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in <listcomp>
 1237.                 for obj in self.query.objs

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in <listcomp>
 1236.                 [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in pre_save_val
 1186.         return field.pre_save(obj, add=True)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/fields/files.py" in pre_save
 287.             file.save(file.name, file.file, save=False)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/fields/files.py" in save
 88.         setattr(self.instance, self.field.name, self.name)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/fields/files.py" in __set__
 345.             self.field.update_dimension_fields(instance, force=True)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/fields/files.py" in update_dimension_fields
 447.             width = file.width

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/core/files/images.py" in width
 19.         return self._get_image_dimensions()[0]

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/core/files/images.py" in _get_image_dimensions
 28.             self.open()

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/db/models/fields/files.py" in open
 76.             self.file = self.storage.open(self.name, mode)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/django/core/files/storage.py" in open
 33.         return self._open(name, mode)

File "/opt/ivala/ivalavenv/lib/python3.6/site-packages/storages/backends/s3boto3.py" in _open
 464.                 raise IOError('File does not exist: %s' % name)

It happens when an image file field is being saved. From the stack trace, it seems that what happened is:
1) The file was uploaded.
2) Django was going to calculate the dimensions of the image and for that, it needed to open the file.
3) As it was trying to fetch the file, S3 returned a 404 response.

The main issue is that I checked and step 1 did in fact work. The file was correctly uploaded and by looking at the timestamps (of both the upload and the crash), it seems to have completed before step 2 took place.

Please let me know if any other information is necessary, I have the full stack trace with variable values.

Any help will be greatly appreciated. Thanks.

[luccascorrea]

I just came across this while reading AWS' documentation:

Additionally, Amazon S3 might return the "NoSuchKey" error if the object was recently uploaded. When you upload an object, Amazon S3 replicates your data across multiple servers, so the newly uploaded object might not be found until after the object is completely replicated across the multiple servers. Amazon S3 provides eventual consistency for read-after-write if a GET or HEAD object call is made before the object upload. To avoid the "NoSuchKey" error for a newly uploaded object, wait a few seconds after the PUT request, and then you can make the first attempt to access the object.

With that in mind, would it be possible to add a second wait after receiving a 404 response from AWS and then try again or something along those lines?

[oilycoyote]

Did you find a solution?