Boto3: Type error when submitting to S3 bucket

[dbenjamin22]

I receive the following Type error when submitting to S3, this previously worked fine in python 2.7, just not in 3.

Traceback (most recent call last):
File "/home/foo/config/plugins/python/helpers/pydev/pydevd.py", line 1596, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/foo/config/plugins/python/helpers/pydev/pydevd.py", line 974, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/foo/config/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/foo/bar/foobar.py", line 97, in
s3.Bucket('foo').put_object(Key=fooName, Body=fooFile, ContentType='text/html')
File "/home/foo/.local/lib/python3.5/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, args, kwargs)
File "/home/foo/.local/lib/python3.5/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(params)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/client.py", line 538, in _make_api_call
request_signer=self._request_signer, context=request_context)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/hooks.py", line 242, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(kwargs)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/handlers.py", line 201, in conditionally_calculate_md5
calculate_md5(params, *kwargs)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/handlers.py", line 179, in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
File "/home/foo/.local/lib/python3.5/site-packages/botocore/handlers.py", line 193, in _calculate_md5_from_file
md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

[stealthycoin]

Body must be a bytes or a file type as per the documentation. Py2 just happens to not know the difference before a string and bytes, which means it kind of "works" as long as your bytes all happen to fall in utf-8.

[marianobrc]

I'm having a very similar issue with python 3.5 and Django 1.11
I'm creating my file like this:

from django.core.files import File
unified_parts_file_obj = File(open(unified_parts_file_path,'r'))

File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/client.py", line 593, in _make_api_call
request_signer=self._request_signer, context=request_context)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/hooks.py", line 242, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(*kwargs)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/handlers.py", line 201, in conditionally_calculate_md5
calculate_md5(params, *kwargs)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/handlers.py", line 179, in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/handlers.py", line 192, in _calculate_md5_from_file
for chunk in iter(lambda: fileobj.read(1024 * 1024), b''):
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/botocore/handlers.py", line 192, in
for chunk in iter(lambda: fileobj.read(1024 * 1024), b''):
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/s3transfer/utils.py", line 430, in read
data = self._fileobj.read(amount_to_read)
File "/home/mariano/venv-chekin/lib/python3.5/site-packages/s3transfer/upload.py", line 85, in read
return self._fileobj.read(amount)
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 5669: invalid start byte

Should I set content-type somewhere in the file object?

Thanks

[aaronkw]

I just encountered this error and fixed it by opening as a binary file, for example:

from django.core.files import File
unified_parts_file_obj = File(open(unified_parts_file_path,'rb'))

might be useful for others

[direwolf424]

@aaronkw Worked for me by opening it as binary file. Thanks .