Smart_open: s3 seek

[perrygeo]

If I try to use smart open to seek/read parts of an s3 file, I get NotImplementedError: seek other than offset=0 not implemented yet.

Arbitrary seeking, especially when the seek was specified relative to the beginning of the file (seek(..., whence=0), should be possible through the Range HTTP header

>>> import boto
>>> s3 = boto.connect_s3()
>>> bucket = s3.lookup('bucket')
>>> key = bucket.lookup('key')
>>> parts = key.get_contents_as_string(headers={'Range' : 'bytes=12-24'})

seek could establish a pointer to the starting byte and subsequent reads would define the end.

Are there any technical limitation or design restrictions that would prevent this?

[piskvorky]

No, I think it's doable, and useful. Only no one did it yet.

Note though that smart_open doesn't use get_contents_as_string, which would read the entire file into RAM. It uses the streamed interface of boto, so that it can process large files too.

[perrygeo]

get_contents_as_string _should_ respect the HTTP Range header but it doesn't always behave that way. In particular, I found that the first call read the entire contents while subsequent calls (with the exact same args and kwargs) pulled in only the requested bytes. I believe this to be a bug in boto. Unfortunately I couldn't find another way to implement this in boto2.

However, switching to boto3 I was able to put together a working s3 reader using the object abstractions. I wrapped it in a file handle interface that does arbitrary seeks and reads: https://gist.github.com/perrygeo/9239b9ab64731cacbb35#file-s3reader-py . It's very effective and allowed me to read TIF tags off 2000 x 1.1 GB files stored on S3 in just a few minutes.

I haven't yet considered how something like this could integrate with smart_open but I figured it might be useful.

[piskvorky]

Ah, great. get_contents_as_string didn't work well when I tried, but your analysis is clearly more exhaustive :)

As long as it supports streamed reads and writes (large files), it's a welcome addition! I think we'd have to implement buffering internally (for the for line in x: line-streaming interface, looking for newlines), but that shouldn't be too difficult. The only tricky part is interaction between stream buffering and read: user does seek, then read vs. user does seek, then iterates a few lines, then does read...

What does switching to boto3 entail? I vaguely remember issues with that library (unfinished), but that was some time ago.

[tmylk]

Blocked by #43

[mpenkov]

@menshikh-iv I think this is done. We can seek S3 files now.

[menshikh-iv]

Resolved in https://github.com/RaRe-Technologies/smart_open/commit/fbc82cc04bf92e8c588c49c43ff5aff8234ea87d