Boto: S3 parallel download thread-safety issues

[dacut]

I'm not sure if this is by design or not, but S3 connection objects appear to have a thread-safety issue when used for parallel range downloads. (Interestingly, they don't have an issue with parallel multipart uploads.) Not sure if this is related to the recent SSLv3 deprecation, but I don't think so.

I'm still digging into the issue (feel free to ping me at AMZN -- cuthbert@), but here's a framework for the code to repro (AmazonLinux 2015.03 with updates applied today, running on a c3.8xl in us-gov-west-1):

s3 = boto.s3.connect_to_region('us-west-1')
bucket = s3.get_bucket('cuthbert-usw1')
key = bucket.get_key(key_name)

def thread_worker():
    # ... code to figure out which range of bytes to fetch next
    range_data = key.get_contents_as_string(headers={"Range": "bytes=%d-%d" % (start_byte, end_byte)})

# thread_count ranges from 1-128.
threads = [Thread(target=thread_worker) for i in xrange(thread_count)]
for thread in threads:
    thread.start()

Examples of stack traces seen:

   File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1650, in get_contents_to_file
     response_headers=response_headers)
   File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1482, in get_file
     query_args=None)
   File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1535, in _get_file_internal
     for bytes in self:
   File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 386, in next
     data = self.resp.read(self.BufferSize)
   File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 413, in read
     return http_client.HTTPResponse.read(self, amt)
   File "/usr/lib64/python2.7/httplib.py", line 573, in read
     s = self.fp.read(amt)
   File "/usr/lib64/python2.7/socket.py", line 380, in read
     data = self._sock.recv(left)
   File "/usr/lib64/python2.7/ssl.py", line 714, in recv
     return self.read(buflen)
   File "/usr/lib64/python2.7/ssl.py", line 608, in read
     v = self._sslobj.read(len or 1024)
SSLError: [SSL: BLOCK_CIPHER_PAD_IS_WRONG] block cipher pad is wrong (_ssl.c:1750)

# Same as above until...
   File "/usr/lib64/python2.7/ssl.py", line 608, in read
     v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1750)

Plus occasional reproductions of this forum's issues:
https://forums.aws.amazon.com/message.jspa?messageID=460058

Moving the s3, bucket, and key assignment into the thread loop (i.e. making them thread-local) fixes this issue.

I'm not sure if this is a bug in Python's HTTP library, its integration with OpenSSL, or Boto's usage, but I'm inclined to think it's the first. Nonetheless, I'm wondering if there's some bit of defensive coding we (I) can put into Boto to avoid this.

Feel free to assign or triage to me (however that works here) -- my main goal in writing this up is to provide Google a hit for "Boto" + these particular error messages so future spelunkers can resolve it faster than I did. :-)

[cgordoncarroll]

THANK YOU FOR DOCUMENTING THIS. I've been having this problem for days and was mashing my head against a wall crawling over the S3 documentation trying to figure out these rather cryptic error messages.

[ixaxaar]

+1 having the same issue accompanied by the head mashing.

[diwu1989]

so basically the way to work around it is to have a S3 connection instance per thread, vs sharing the same one

[manugarri]

pinging since this affects dask when reading from s3.