Django-storages: S3Boto3StorageFile always returns bytes iterator (not string)

Created on 9 Oct 2017 · 8Comments · Source: jschneier/django-storages

When I run this code:

import csv
storage = S3Boto3Storage()
with storage.open('test.csv', 'r') as f:
    reader = csv.reader(f)
    print (list(reader))

I get the following message:

Error: iterator should return strings, not bytes (did you open the file in text mode?)

The problem is that the __file_ property of S3Boto3StorageFile is always instantiated in 'w+b' mode in order to write the contents from the s3boto3 Object. Therefore, when I read from that file, the iterator always returns bytes, and not string.

I've worked around this problem by using the codecs library:

import codecs
import csv
storage = S3Boto3Storage()
with storage.open('test.csv') as f:
    reader = csv.reader(codecs.iterdecode(f, 'utf-8'))
    print (list(reader))

But I still think this issue should be resolved, since the S3Boto3Storage _open() function has a mode parameter.

s3boto

Source

icpetcu

👍9

Most helpful comment

Hi everybody,

I've fixed the issue and made a PR (see above). Using S3Boto3Storage is now transparent, compatible with the Django provided FileStorage and doesn't require additional code. Please push package originator to include a PR into the next release

nnseva on 25 Jan 2019

👍4

All 8 comments

+1, I was bit by this as well. With regular file storage, storage.open('test.csv', 'r').read() returns a unicode str, while storage.open('test.csv', 'rb').read() returns bytestring. With S3Boto3Storage both return bytestring. S3Boto3Storage should follow the same function signature as file storage.