Boto: Py3 error on S3 build_post_form_args

[yuvadm]

>>> s3_conn.build_post_form_args(BUCKET_NAME, filename)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/yuval/.virtualenvs/proj/lib/python3.4/site-packages/boto/s3/connection.py", line 333, in build_post_form_args
    policy_b64 = base64.b64encode(policy)
  File "/Users/yuval/.virtualenvs/proj/lib/python3.4/base64.py", line 62, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
TypeError: 'str' does not support the buffer interface

[yuvadm]

The policy built is a py3 str, while b64encode operates on bytes only. Assuming the policy is always an ASCII string, the fix might be as simple as calling:

policy_b64 = base64.b64encode(policy.encode('ascii'))

[ztane]

Ping, this is still there today.

S3 docs say that "The policy required for making authenticated requests using HTTP POST is a UTF-8 and Base64 encoded document written in JavaScript Object Notation (JSON) that specifies conditions that the request must meet." This is rather clear as it is.

On the other hand, the boto code currently is an abomination if one considers that the json module is in the python standard library. Currently no UTF-8 extra characters, nor " nor anything such is allowed in the key name by boto; instead they just cause errors. And consider this piece:

    conditions.append('{"bucket": "%s"}' % bucket_name)
    if key.endswith("${filename}"):
        conditions.append('["starts-with", "$key", "%s"]' % key[:-len("${filename}")])
    else:
        conditions.append('{"key": "%s"}' % key)

Now, even in Python 2 if the key happens to contain some fancy characters, it will suddenly be a no-no, though unfortunately, an unicode object with ascii-only works as such.

Thus for Python 3, the immediate proper fix is to .encode('UTF-8'), but the code should be refactored to use json and not this haphazard pasting of strings.

[ztane]

Example fix that I am using is available at https://github.com/ztane/boto/commit/0819a88b088092172da30b07db2abfd4c5a12826 - though I am not able to do the test yet

[ztane]

Also should fix https://github.com/boto/boto/issues/2615 too

[lsowen]

Any progress/movement on getting this issue fixed/closed?

[marquesds]

I'm still having this problem. How can I fix this?

[ztane]

me too, I guess I have to monkeypatch build_post_policy.

[gtmanfred]

I was able to get around this by monkeypatching b64encode

import base64
import six

if six.PY3:
    old_b64encode = base64.b64encode

    def b64encode(s, altchars=None):
        if isinstance(s, six.text_type):
            return old_b64encode(s.encode('utf-8')).decode('utf-8')
        return old_b64encode(s)

    setattr(base64, 'b64encode', b64encode)

@mfschwartz or @dstufft it would be really awesome to get this fixed?

This is holding me up on a migration to python3.

Thanks!
Daniel

[jsadn]

i don't believe boto3 has an equivalent of this helper so this should be a particularly prioritized fix since only alternatives are monkeypatching and forking.