Django-storages: ContentFile with str content throws TypeError

[radimsuckr]

Hello, I've run into problems when I tried saving Django ContentFile. When I save the file with, for example, 'hello', it throws following exception: TypeError: Unicode-objects must be encoded before hashing.

However everything works fine with bytes. Below is traceback from Django shell. I've tried it on Django 1.11 and 2.2 with latest versions of both boto3 and django-storages.

Traceback:

>>> S3Boto3Storage().save('test.txt', ContentFile('hello'))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/storages/backends/s3boto3.py", line 506, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/storages/backends/s3boto3.py", line 521, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 621, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj
    return future.result()
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/client.py", line 642, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/hooks.py", line 360, in emit_until_response
    return self._emitter.emit_until_response(aliased_event_name, **kwargs)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/hooks.py", line 243, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/handlers.py", line 212, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/handlers.py", line 190, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/home/grelek/projects/djstorages-test/venv/lib/python3.7/site-packages/botocore/handlers.py", line 204, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

[sww314]

Try this:
S3Boto3Storage().save('test.txt', ContentFile(b'hello'))

[radimsuckr]

Yeah, bytes work just fine, but the documentation states that str should work too. :( Or do I miss something?

https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#model

[sww314]

str works in Python 2. In Python 3, you need to encode it or use Bytes.

[sww314]

See #707

[LincolnPuzey]

@sww314 This issue should be re-opened.

It is perfectly valid Django to initialize a ContentFile with either a string or bytes in Python 3.

This can be seen in the Django docs: https://docs.djangoproject.com/en/dev/ref/files/file/#django.core.files.base.ContentFile
And in the source for ContentFile itself: https://github.com/django/django/blob/master/django/core/files/base.py#L126
Where __init__ checks if the content arg is a str.

So this is a bug in django-storages that should be fixed.

[LincolnPuzey]

Note that the same error occurs when saving a Django File object initialized with a file object opened in text mode.

S3Boto3Storage().save("testfile.txt", File(StringIO("foo")))

[LincolnPuzey]

@jschneier Can this issue be re-opened? As per my above comment I believe this is a valid bug in django-storages.

[radimsuckr]

Hi @LincolnPuzey, I've implemented a workaround for this issue in module druids/django-chamber. You might want to take a look at it.

Nevertheless I agree with you that this behavior in django-storages is wrong and should be fixed.