Botocore: Make clients pickleable

[AlexDiede]

My project creates a kinesis client and spawns multiple sub-processes that I pass the client to. This worked when using boto but does not with boto3 because clients are not pickleable. Could this feature be added?

Here's the minimal code to reproduce the issue:

import boto3
import pickle
c = boto3.client('kinesis')
pickle.dump(c, open('c','w'))

The error I get is: _pickle.PicklingError: Can't pickle : attribute lookup Kinesis on botocore.client failed

In my actual project I do something similar to:

from multiprocessing import Process, Queue
import boto3

c = boto3.client('kinesis')
q = Queue()
q.put(c)
p = Process(target=other_function, args=(q,))
p.start()
p.join()

I get the same pickle error as the previous example at p.start()

I'm running Ubuntu 14.04, the interpreter is Python 3.4.0 (gcc 4.8.2), and botocore version is 1.1.10.

[kyleknap]

We would have to do more research to see how doable this is. Marking this as a feature request.

[amatthies]

+1 for pickleable clients

[rayluo]

In general, pickling a complex object is not easy, especially when the object (indirectly) contains low level resources such as file handlers.

A much safer way is to pickle the primitive parameters you use to create current client instance, so that you can recreate a new client with same behavior.