Botocore: Support asyncio

[jamesls]

This is a tracking issue for the feature request of supporting asyncio in botocore, originally asked about here: https://github.com/boto/botocore/issues/452

There's no definitive timeline on this feature, but feel free to +1 (thumbs up 👍) this issue if this is something you'd like to see. Also, if you have any additional information about asyncio you'd like to share (even just about your specific use case) feel free to chime in.

[jab]

Thanks for logging this! Currently I'm only calling list_distributions(..) and create_distribution(..) on a boto3.client('cloudfront') instance, which I know is a small fraction of the boto3 API, but it'd be nice if those calls cooperatively yielded in between making the request and receiving the response so that other coroutines in my app could run in the meantime. (I could always just roll my own client using aiohttp too since I'm using so little of the API, but figured it was worth asking about anyway.)

In case it helps, aiodns provides an example of supporting asyncio on Python 2.6+ (see https://github.com/saghul/aiodns#python-versions).

If boto3 was already written with the assumption of blocking rather than asynchronous I/O throughout, I'm not sure how disruptive a change this would be.

[Jc2k]

I'm not that familiar with the internals of botocore, but I think the right approach is to create a custom version of endpoint.py and client.py that used aiohttp under the hood and yielded as you want.

I would probably start by subclassing Endpoint and making an aiohttp version of _send_request and _get_request. I'd also subclass ClientCreator and override _create_api_method (that is where the calls to the endpoint come from). I haven't yet figured out how i'd get from a service object to a client created with my ClientCreator subclass.

Does that sound right, @jamesls? I might have a go...

[rdbhost]

I've started a port of botocore to asyncio, at:

https://github.com/rdbhost/botocore

So far, the S3 integration tests pass, as do about 2/3 of the unit tests. I hope to have all tests converted and passing by a week from now.

The port depends on yieldfrom.requests and yieldfrom.urllib3 , asyncio ports of the requests and urllib3 libraries.

[jab]

@rdbhost Awesome! Thanks for working on this. Definitely interested to follow your progress.
@jamesls / botocore maintainers, had a chance to check this out / evaluate for merge potential?

[koliyo]

+1

[AlexNigl]

+1

[rdbhost]

This issue has gotten a couple of +1s in the last few days, so I thought I
would point out that the work is done, at:

https://github.com/rdbhost/yieldfromBotocore

It implements an asyncio version of botocore, not boto3. I may eventually
convert boto3, but my own needs have been satisfied by botocore, so other
things now have priority.

David

On Thu, Jun 18, 2015 at 6:59 AM, AlexNigl [email protected] wrote:

+1

—
Reply to this email directly or view it on GitHub
https://github.com/boto/botocore/issues/458#issuecomment-113148024.

[amatthies]

+1

[jmehnle]

@rdbhost, thanks for tackling that! It seems, however, that https://github.com/rdbhost/yieldfromBotocore is 271 commits behind upstream. Are you actively maintaining it?

[rdbhost]

My intention has been to do a merge when upstream reached v1.0; that seems to have happened without my noticing.

I will be merging commits from upstream, this weekend.

[r39132]

+1 @jamesls are you planning on pulling rdbhost's changes into botocore itself? We are very interested in this feature.

[rdbhost]

It is not a very promising candidate for merging back into botocore.

The changes are numerous, and the changes needed to make botocore functional within asyncio make it non-functional outside asyncio. I expect it to be a seperate product indefinitely, with a parallel API, meaning an API as similar as possible within the asyncio constraints.

[jamesls]

One thing that also complicates things is that botocore supports as far back as python 2.6.5, so we'll need to figure out how we can support asyncio and still maintain py2 support. I see that there are asyncio backports to python2, so perhaps something could be done there.

[jmehnle]

Trollius (asyncio port to Python 2.x) is complete and stable.

[harai]

+1

[jettify]

I have working port of botocore for asyncio: https://github.com/jettify/aiobotocore using aiohttp for async http requests. I am trying to reuse as much botocore code as possible, so I patched only several classes and just import rest of the code as result library has few hundreds lines of code. And this approach helps to keep up with upstream, but obvious downside, I rely on internal interfaces which is subject of change for new libs. API almost the same as botocore just yield from or awaite (python 3.5) should be added before calls.

For now I am using aiobotocore with s3 and ported almost all s3 test, except I need to work more on pagination since it is not easy to implement iterator protocol with yield from.

[mpaolini]

+1 integrating aiobotocore might be doable

[balihoo-gens]

+1

[mikeplavsky]

+1

[cgst]

+1

[thomascellerier]

+1

[lsbardel]

OK guys, I've written a production ready, and hopefully useful to the community, asyncio extension to botocore. The library is currently included in https://github.com/quantmind/pulsar-cloud in the asyncbotocore module but it is self-contained and could be striped out if needed.

The API is the same as botocore but with the addition of http_session keyword which is an optional asyncio compatible HTTP client (it must expose the _loop attribute) and API similar to requests.
As far as I know there are two of such clients:

• pulsar http client
• aiohttp http client (@asvetlov does aihttp client/session have the _loop attribute?)

If used with pulsar, the library can also use greenlet in an implicit asynchronous fashion. Check https://github.com/quantmind/pulsar-cloud for more info

By the way, thanks to @jettify for the initial effort from which I leveraged from

Feedbacks welcome!

[asvetlov]

@lsbardel yes, aiohttp.ClientSession has _loop but I don't encourage using private attributes.
BTW we use https://github.com/jettify/aiobotocore as aiohttp-based library.

[lsbardel]

@asvetlov cool, I understand the private attribute thing, but the _loop attribute should be (almost) a standard for asyncio objects ;-) maybe we should ask guido.

I would be happy to use https://github.com/jettify/aiobotocore has low level async botocore library but currently it does not work with pulsar http client. At the moment it clearly requires aiohttp while cloud.asyncbototore does not require an asyncio library as such.

So maybe we should converge to a library that allows for different http clients?

[lsbardel]

By the way all that is python 3.4 or above

[asvetlov]

@lsbardel At asyncio active development times Guido was against public loop attribute, he motivated it as "user always may pass explicit loop if needed". I don't think the decision has changed now.

[nadad]

+1

[jmorris0x0]

+1

[frnkvieira]

+1

[Nath-P]

+1

[vinayan3]

+1

[smorin]

+1

[smorin]

How much work do you think it would take to add this functionality to boto3?

[jamesls]

Boto3's interesting because of the resources layer which can make multiple API calls, e.g:

s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
    for key in bucket.objects.all():
        print(key.key)

What would the ideal async API look like for resources?

[smorin]

Ideally the implementation would allow for callbacks/extension points for people to integrate the non-blocking solution of their choice whether, gevent or asyncio etc.

It can be tricky to do expose that, is boto3 using requests or something like that under the hood?

For example, prompt_toolkit didn't originally have async support but added the ability to be used in a event loop.

• http://python-prompt-toolkit.readthedocs.io/en/stable/pages/building_prompts.html#prompt-in-an-asyncio-application

Also here is an example:

• https://github.com/kennethreitz/grequests

Here is another example discussing: Async code design

• http://stevedower.id.au/blog/async-api-for-python/

[rdbhost]

Steve Morin:

If what you are wanting is a way to make requests to AWS from an asyncio
app, using an API very similar to botocore, look at:

https://github.com/rdbhost/yieldfromBotocore

It is based on an asyncio port of requests:

https://github.com/rdbhost/yieldfromRequests

The yieldfromBotocore library is not maintained anymore, as the boto3 API
library was evolving too fast to keep up with, and I have other priorities.
It does work on my production site, though, and possibly others.

Simply adding asyncio support to botocore itself is probably not viable, as
the architectural differences between the asyncio and traditional blocking
approaches is too profound to address both in one code base.

On Mon, May 23, 2016 at 8:39 PM, Steve Morin [email protected]
wrote:

Ideally the implementation would allow for callbacks/extension points for
people to integrate the non-blocking solution of their choice whether,
gevent or asyncio etc.

It can be tricky to do expose that, is boto3 using requests or something
like that under the hood?

For example, prompt_toolkit didn't originally have async support but added
the ability to be used in a event loop.

-
http://python-prompt-toolkit.readthedocs.io/en/stable/pages/building_prompts.html#prompt-in-an-asyncio-application

Also here is an example:

• https://github.com/kennethreitz/grequests

Here is another example discussing: Async code design

• http://stevedower.id.au/blog/async-api-for-python/

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/boto/botocore/issues/458#issuecomment-221151568

[jettify]

@jamesls using python3.5 features api is pretty straitforward:

s3 = boto3.resource('s3')
async for bucket in s3.buckets.async_all():
    async for key in bucket.objects.async_all():
        print(key.key)

[jettify]

@smorin prompt toolkit is using threads for some blocking calls, in same fashion you can do with boto, just create big enough ThreadPoolExecutor and use it for each boto call that performs IO. This solution will work absolutely fine (with some exceptions of course).

[smorin]

I am looking to run this on AWS Lambda so trying to avoid threads for that
reason and the performance penalty.

On Tue, May 24, 2016 at 2:03 PM, Nikolay Novik [email protected]
wrote:

@smorin https://github.com/smorin prompt toolkit is using threads for
some blocking calls, in same fashion you can do with boto, just create big
enough ThreadPoolExecutor and use it for each boto call that performs IO.
This solution will work absolutely fine (with some exceptions of course).

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/boto/botocore/issues/458#issuecomment-221391484

Steve Morin | Hacker, Entrepreneur, Startup Advisor
twitter.com/SteveMorin | stevemorin.com
_Live the dream start a startup. Make the world ... a better place._

[foobarna]

+1

[bmamouri]

I'm downloading files from S3 in my lambda function and my lambda cost is extremely high because boto3 file download operation is blocking. I am really looking forward to implement this without asyncio that works in Python 2.7

[JohnDzialo]

+1

[floostmodern]

+1

[graingert]

twisted now supports asyncio. Twisted supports both Python 2 and Python 3.

If you ported botocore to twisted you could expose a blocking wrapper in boto3, and users could use botocore asynchronously via asyncio and twisted in Python 3 and just twisted in Python 2.

[jettify]

@graingert I do not think it is best approach.

Better approach is to move request preparation and response parsing logic out of IO library, as result anyone can add support for new framework regardless async or sync (https://sans-io.readthedocs.io/). botocore is doing good job there, I managed to hack asyncio support in aiobotocore very quickly and with little amount of code.

[graingert]

@jettify yeah that's an even better solution.

[lsbardel]

Updated pulsar-cloud to be compatible with botocore 1.4.61

• Explicit native asyncio support
• Implicit asyncio+greenlet support
• Works with pulsar http client and possibly aiohttp (need testing)

Feedback welcome

[atif1996]

+1

[pfreixes]

There is open a new thread at aiobotocore side [1] with a new proposal to align botocore and aiobotocore, we will like to get more opinions and especially the botocore maintainers such as @jamesls

[1] https://github.com/aio-libs/aiobotocore/issues/213

[ravidbro]

+1

[mnolet]

+1000!

[mikepashin]

+1 000 000

[aehlke]

@jamesls can you please edit your issue description to encourage adding a 👍 emoji reaction to your issue instead of "+1 this issue"? Now that github added has since added that feature, it's annoying for people to respond with "+1" replies, pinging all subscribers. The issue text seems to encourage this. Thanks!

[ashwiniadiga]

+1

[lsbardel]

@aehlke this ticket has been open since 2015!
Three years!! :joy:

People can put whatever they like to draw some attention.
Emoji, plus one or plus whatever.
If you don't like to pinged you can aways unsubscribe and swing by from time to time!

https://github.com/aio-libs/aiobotocore is the answer at the moment.
Lets rally behind that project!

[kadut]

I'd be happy to be wrong, pls check into it enough to confirm either way. I was going from:
https://github.com/boto/botocore/issues/458 which is still open, although there is a project mentioned there that supports some AWS services but but not SSM. Also despite the issue I thought I was safe as long as thread per call, but perhaps not?

[mojimi]

I'm assuming the NodeJS version is naturally async because of the nature of javascript, but we really need this on Python.

As mentioned, you guys could simply build on top of aiobotocore, which shows that it's not that difficult, the hardest part is testing.

Being able to download several S3 Items asynchronously or invoke several lambdas and wait on the returned response asynchronously is more than necessary! In some cases it would execute codes 20x faster.

[belegnar]

+1

[atif1996]

+1

[arairyoto]

+1

[Zeus1984]

+1

[sharkguto]

+1

[Catastropha]

+1

[patrickwerz]

+1

why would someone be against a async / non blocking version of boto?

[Naddiseo]

@danketrotzdem we're downvoting people who only add "+1" as a comment, which spams emails to everyone subscribed to this thread without providing any content, when instead they should be using the github reactions on the very first post. We're subscribed to know when this issue is resolved, not to see a constant stream of "+1" emails.

Perhaps it would be time to lock this thread?

Sorry everyone for spamming you.

[sharkguto]

please asyncio support !! we are in 2020 Oo python2 is dead

[agronick]

@jettify there would be no point in using asyncio if all your IO is blocking on a TheadPool. Adding an event loop doesn't make non-io code faster. In fact it makes it slower since you're putting every method on the event loop for no reason. I've tested it. The only way you get the crazy performance benefits from asyncio is to have everything one thread, thus reducing the CPU context switching and eliminating the GIL locking that comes from multiple threads.

[graingert]

@agronick aiobotocore doesn't use a TheadPool

[rob-blackbourn]

I'm looking at deploying a Python 3 asyncio micro-service stack into AWS, but I find I cannot use any of the native Amazon services (which would save me a bunch of money) because there is no asyncio support. None of the other cloud suppliers have asyncio support, so I'll still choose AWS, but without the cool service based products.

It would seem there are two approaches to adapting this library.

The most complicated (but probably the most desirable) would be the state machine approach (like the h11 and h2 libraries have adopted). This would move the sync/async decision to downstream libraries, and keep the core library a clean implementation of the protocol.

A more simple approach would be to remove the explicit sync implementation and to provide sync and async abstract classes. These would then be implemented in downstream libraries. This would require careful design to prevent syntactically invalid code being presented to Python 2 interpreters.

[hlongmore]

I just gave the issue a +1 because trying to use other packages that use botocore along with aiobotocore is painful, due to that project being so tightly coupled with botocore. It is possible with only two packages depending on botocore, or with six, to end up with all but one package that have a compatible set of botocore versions, but aiobotocore's pinning to one specific version means neither pipenv nor poetry can find a match. My solution thus far has been to fork aiobotocore, change the version for botocore, and hope it works, but that is bound to break eventually.

[kkozmic]

One thing that also complicates things is that botocore supports as far back as python 2.6.5, so we'll need to figure out how we can support asyncio and still maintain py2 support. I see that there are asyncio backports to python2, so perhaps something could be done there.

This I assume is no longer a consideration since Python 2 is now obsolete. Is there any official plan/work under way to make the async support happen?

What are the obstacles preventing that from happening now?

[graingert]

You can support anyio and python 2 with one codebase using https://pypi.org/project/unasync/

[martindurant]

Adding my voice here as the maintainer of s3fs, which has recently adopted async functions, and the mismatch between aiobotocore and botocore versioning is hurting us ( https://github.com/dask/s3fs/issues/357 ).

+1 +1 +1

[BastianZim]

Hi, I was just wondering if there is an official statement on whether this is going to be implemented or not, since this issue has been stale for some time now.
Thanks!

[BastianZim]

Since @jamesls is not with the boto team anymore: Is there anyone else at AWS that would be able to comment on this or should we consider this stale and try to find a different alternative?
I'm not sure who the main maintainer is but issue #764 has an answer by @kyleknap so maybe you're able to comment on this?
Thanks!

[aaronclong]

Is there anything the community can do to accelerate or help out with this work?

[BastianZim]

Also pinging @swetashre since you’re listed on most issues here. Is there any comment from AWS?

[rafidka]

I needed the same functionality and ended up creating a simple package, boto3async, to help me with this.

The trick I used is best illustrated with the following example:

async def list_bucktes_async():
    s3_client = boto3.client('s3')
    return await asyncio.to_thread(s3_client.list_buckets)

async def main():
    response = await list_bucktes_async()
    print(json.dumps(response, default=str, indent=2))

asyncio.run(main())

then, to generalize the solution so I don't need to do this for every operation I need, I defined the following method:

def asyncify_client(client):
    """
    Adds async methods to each of the sync methods of a boto3 client.

    Keyword arguments
    client -- The client to add sync methods to. Notice that the client
        will be updated in place, and will also be returned as a return
        value.

    Returns:
    The same client.
    """

    def create_async_func(sync_func):
        async def async_func(*args, **kwargs):
            return await asyncio.to_thread(sync_func, *args, **kwargs)
        return async_func

    for operation in client._service_model.operation_names:
        operation_camelcase = _camel_to_snake(operation)
        sync_func = getattr(client, operation_camelcase)
        async_func = create_async_func(sync_func)
        setattr(client, f'{operation_camelcase}_async', async_func)

    return client

Finally, I defined a method inside boto3async that calls boto3's client() method, followed by calling the above method, to return to the user a boto3 client with _async version for each operation. So, all the user has to do is call boto3async.client instead of boto3.client, passing in the same exact arguments, and then start using the _async methods.

There are two caveats, unfortunately:

• This library depends on Python's to_thread function. Unfortunately, this function is only supported in Python 3.9 and beyond, hence this library only runs on Python 3.9 and beyond. It is possible to copy the implementation of to_thread from Python 3.9's source code to make the library work on Python 3.7 and later.
• The trick works with boto3 clients only; resources are higher level and require manual work (unless one can think of a similar track that works with resources.)

Hope some find this useful.