Boto3: How can I set s3 connection to keep alive?

Created on 30 Sep 2016 · 5Comments · Source: boto/boto3

I wanna uploading lots of files to s3 service every time.
I made 10 workers to upload files using thread.
But a message, "Resetting dropped connection", is shown repeatedly.
I guess s3 connection is newly opened per one time.
How can I set connection to keep alive?

import threading
import boto3

s3 = boto3.resource('s3')

class Worker(threading.Thread):
    ...
    def run(self):
        s3.Object(bucket, key).put(Body=body)

question

Source

odg0318

Most helpful comment

The session itself isn't thread safe, you'll need to create a new one for each thread:

import threading
from boto3.session import Session


class Worker(threading.Thread):
    ...
    def run(self):
        s = Session()
        s3 = s.resource('s3')

JordonPhillips on 6 Oct 2016

😄5

All 5 comments

I suggest that you move the boto3.resource('s3') into the thread. I believe the general advice is to have one resource per thread as they're not thread-safe.

themanifold on 30 Sep 2016

😕1

I moved boto3.resource('s3') into each thread, but the same message is shown.

odg0318 on 4 Oct 2016

The session itself isn't thread safe, you'll need to create a new one for each thread:

import threading
from boto3.session import Session


class Worker(threading.Thread):
    ...
    def run(self):
        s = Session()
        s3 = s.resource('s3')

JordonPhillips on 6 Oct 2016

😄5

I created new session in each thread.
But Resetting dropped connection message still is shown.
I guess S3 doesn't support Keep-Alive or I quickly send request to server repeatedly.

Thank you for your help.

odg0318 on 7 Oct 2016

What's the difference between