Tornado: PEP-0492 async/await with AsyncHTTPClient

[arthurdarcet]

I'm trying to get the example from the docs working: http://tornado.readthedocs.org/en/latest/guide/coroutines.html#python-3-5-async-and-await
with the following code

import asyncio
from tornado.httpclient import AsyncHTTPClient

async def main():
    http_client = AsyncHTTPClient()
    response = await http_client.fetch('http://httpbin.org')
    return response.body

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

but all I get is a RuntimeError: Task got bad yield: <tornado.concurrent.Future …>

Am I doing something wrong or is the tornado support for PEP-0492 not complete yet?

Thanks for your help

(I am on master for tornado and Python 3.5.0rc1)

[bdarnell]

The PEP 492 stuff is mostly finished, although there are two rough edges that I'm not sure how to handle. They're both related to the fact that Tornado is fairly liberal about what you can yield or await, while asyncio is more strict.

First, it's the top-level coroutine runner that decides what is awaitable for the entire stack. If the top level is asyncio's run_until_complete(), the entire stack must play by the asyncio rules. If it's a tornado @gen.coroutine, Tornado's rules apply (if one were not a superset of the other, then it would be very difficult to mix them). Although actually in this case tornado.concurrent.Future _should_ be Awaitable so I'm not sure why it's not working.

Second, even in a pure-Tornado world, we had a few features that are not exposed for await. A Tornado coroutine can yield a list of futures, but await won't accept a list. There doesn't seem to be much we can do here except require an explicit wrapper object around the list.

[arthurdarcet]

Looks like the asyncio.Task class wants asyncio.Future objects, I can make the example work by rewriting the Future.__await__ method:

import asyncio
import tornado.concurrent
import tornado.httpclient
import tornado.platform.asyncio

def aw(self):
    fut = asyncio.Future()
    tornado.concurrent.chain_future(self, fut)
    return (yield from fut)
tornado.concurrent.Future.__await__ = aw

async def main():
    http_client = tornado.httpclient.AsyncHTTPClient()
    fut = http_client.fetch('http://httpbin.org/get')
    response = await fut
    print(response.body)

tornado.platform.asyncio.AsyncIOMainLoop().install()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Should I submit a pull request for this ?
The tests fail miserably with this, so I guess it's not the right solution at all

[zguangyu]

I just tested the following code with tornado 4.3 and got the right result. I use to_asyncio_future to convert tornado Future to asyncio.Future.

import asyncio
import tornado.concurrent
from tornado.httpclient import AsyncHTTPClient
import tornado.platform.asyncio

async def main():
    http_client = AsyncHTTPClient()
    response = await tornado.platform.asyncio.to_asyncio_future(http_client.fetch('http://httpbin.org'))
    return response.body

tornado.platform.asyncio.AsyncIOMainLoop().install()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

[arthurdarcet]

@zguangyu I didn't know to_asyncio_future, but yes, that's what I ended up doing in https://github.com/tornadoweb/tornado/issues/1493#issuecomment-133424820

Is this to_asyncio_future supposed to be required? (It's either missing from the examples in the doc or missing from the implementation of __await__)

[zguangyu]

@arthurdarcet You need to use to_asyncio_future because you use the asyncio event loop. It's asyncio event loop not await that require a asyncio.Future object.

There is no need to do any conversion when using await in tornado IOLoop. The following code works well in my RequestHandler. The gen.Task absolutely returns a tornado Future object. I don't know how to run a native coroutine function directly in tornado IOLoop.

async def post(self):
    global _user_cache
    user = await gen.Task(User.auth, self.get_argument("user"), self.get_argument("pw"))
    if user is not None:
        self.set_secure_cookie("user", self.get_argument("user"))
        _user_cache[self.get_argument("user")] = user
        self.redirect(self.get_argument("next", "/"))
    else:
        self.render("login.html", error="Either user or password is incorrect!")

[arthurdarcet]

OK. I don't have to use the asyncio IOLoop so here is what I have working right now:

import tornado.gen
import tornado.httpclient
import tornado.ioloop


async def main():
    http_client = tornado.httpclient.AsyncHTTPClient()
    fut = http_client.fetch('http://httpbin.org/get')
    response = await fut
    print(response.body)

io_loop = tornado.ioloop.IOLoop.current()
io_loop.run_sync(lambda: tornado.gen.convert_yielded(main()))

The convert_yielded is necessary otherwise nothing happen (other than a RuntimeWarning saying that the coroutine was never awaited)
Is this the right way to run_sync a coroutine ?

[zguangyu]

@arthurdarcet Wow I didn't know the convert_yielded function. It solves my problem. I think this is a "good" way to run_sync a coroutine.
I'd like to run the coroutine with the following code, just like the call_soon function in asyncio. In this way I can run multiple tasks in a IOLoop.

import tornado.gen
import tornado.httpclient
import tornado.ioloop

async def main():
    http_client = tornado.httpclient.AsyncHTTPClient()
    fut = http_client.fetch('http://httpbin.org/get')
    response = await fut
    print(response.body)

tornado.gen.convert_yielded(main())
io_loop = tornado.ioloop.IOLoop.current()
io_loop.start()

[arthurdarcet]

Any exception with in the main function will not be displayed I think.
You should probably use io_loop.add_future(tornado.gen.convert_yielded(main()), lambda fut: fut.result())

[zguangyu]

But I can get exception displayed with my code when I try to fetch a nonexistent url. I wonder what's the advantage of spawn_callback instead of calling it directly.

[bdarnell]

There are several issues here, all related to the concept of the _coroutine runner_. All coroutines need a runner, whether they use yield, yield from, or async def. The runner determines what can be used in yield or await expressions. Asyncio's runner is fairly strict, allowing only asyncio.Future objects. Tornado's runner is more liberal, allowing Tornado, asyncio, and concurrent Futures and Twisted Deferred objects (and it's extensible via convert_yielded, although much of this flexibility is in accessible in async def coroutines).

In Tornado-style coroutines with yield, each function gets its own runner; in coroutines with yield from or await, the entire stack shares a single runner. The shared runner is much more efficient, but it means that the link between a coroutine and its runner is no longer clear. Unless the world standardizes on a single coroutine runner, interoperability gets trickier (you need explicit bridging functions like to_asyncio_future to create additional runners when calling across framework boundaries).

The IOLoop is only indirectly involved here. In general, you can mix one framework's event loop with another's coroutine runner. However, in idiomatic use of asyncio, you never see the runner - it is created by the event loop when it sees a raw coroutine. (In Tornado, the runner is created by the @gen.coroutine decorator. We don't yet have a clean way to start an async def coroutine with the tornado coroutine runner, except by yielding it from a top-level function that uses @gen.coroutine (convert_yielded works for this usage, but it's supposed to be internal-only. I think run_sync() (and most other functions that accept Futures) should probably call convert_yielded for you).

So the question is, what do we do with the Tornado coroutine runner with async def coroutines? Do we keep the two separate worlds, and document/clean up the steps required to cross framework boundaries? (This seems like an unfortunate step backwards after we just introduced convert_yielded to make integration more seamless with yield-based coroutines, although I'm not sure await gives us any alternatives). Or do we embrace asyncio, using their runner and Futures for async def coroutines? I'm not sure if we'd be giving up any functionality with the asyncio runner - the Tornado runner's convert_yielded flexibility doesn't work with async def anyway. I'm more concerned about the complexity of making this switch in a backwards-compatible way.

Your original idea of returning an asyncio.Future from tornado.concurrent.Future.__await__ seems like the right way to unify the two worlds. The reason the tests were failing is that asyncio Futures depend on the asyncio event loop policy, which we do not currently synchronize with Tornado's instance() and current().

[arthurdarcet]

@bdarnell : thank you for your detailed response. I was actually updating my first post when you commented:

When I use the AsyncIOMainLoop, everything is working correctly until I try to use asyncio.gather which works only with pure asyncio coroutines, if any "child" of the coroutine await a tornado Future, it fails.
My (wild) guess would be that asyncio.gather returns a future instead of an awaitable and tornado isn't wrapping the result as it should

import asyncio
import tornado.gen
import tornado.ioloop
import tornado.platform.asyncio

async def do_tornado():
    await tornado.gen.coroutine(lambda: None)()
    print('tornado')

async def do_asyncio():
    await asyncio.sleep(.1)
    print('asyncio')

async def main():
    await do_asyncio()
    await do_tornado()
    await asyncio.gather(do_asyncio())
    await asyncio.gather(do_tornado())

tornado.platform.asyncio.AsyncIOMainLoop().install()
loop = tornado.ioloop.IOLoop.current()
loop.run_sync(lambda: tornado.gen.convert_yielded(main()))

The first three awaits works correctly, but the await asyncio.gather(do_tornado()) fails with a RuntimeError: Task got bad yield: <tornado.concurrent.Future …>

Funny thing is await asyncio.gather(do_asyncio()) works prefectly well, and both do_tornado() and do_asyncio() are coroutines.
So any coroutine awaited through asyncio.gather cannot await a tornado Future ?

Concerning the convert_yield, I think having run_sync call convert_yield itself would break backward compatibility, since currently this works: tornado.ioloop.IOLoop.current().run_sync(lambda: 1)

[arthurdarcet]

@bdarnell : how about this:
https://github.com/tornadoweb/tornado/compare/master...arthurdarcet:master

All the tests seems to be passing on travis https://travis-ci.org/arthurdarcet/tornado/builds/77324193
Not sure how (in)efficient the IOLoop.current() call is

[bdarnell]

Hmm, we really don't want to limit the use of async/await to the asyncio IOLoop (unless maybe we made asyncio the default whenever it's available). It's annoying that asyncio.Future depends on a running event loop.

[arthurdarcet]

I think it would make sense defaulting to the asyncio event loop when available. But I agree that having the PEP492 syntax available on every loop would be much better.

What do you think of this: https://github.com/tornadoweb/tornado/compare/master...arthurdarcet:mixed-futures

[bdarnell]

Does that work? If it does then we should just make tornado.concurrent.Future a subclass of asyncio.Future (when it is available) so to_asyncio_future wouldn't be necessary. But I'm concerned that it wouldn't actually work. Asyncio uses a lot of non-public methods on its Future class, and I'm doubtful that it would work to mash the two classes together like this.

We could almost use asyncio.Future as-is in place of the Tornado Future. The main feature we need that asyncio doesn't have is set_exc_info() (important for python 2 compatibility). This would be easy to add in a subclass of asyncio.Future. The fact that asyncio.Future depends on the current event loop at construction time is a thornier problem, though.

[arthurdarcet]

It seems to be working: https://travis-ci.org/arthurdarcet/tornado

Having the tornado Future class extend asyncio.Future would probably simplify some stuffs too.

Another solution would be to add a Tornado to Asyncio IOLoop adapter that would emulate an asyncio loop over a tornado one and use this class to add a _loop attribute to the asyncio.Future returned by Future.__await__

[bdarnell]

While there's some merit to the idea of "hollowing out" tornado and replacing our abstractions with asyncio's (when available), that's something that would have to wait at least until Tornado 5.0 (and even then, it's difficult to fully embrace asyncio while we still support Python 2, so it might have to wait until we can go py3-only).

For Tornado 4.3, then, the status quo is unchanged: you couldn't yield from a tornado coroutine in an asyncio coroutine before, and you can't await it now. We'll just need to document this fact and its implications, since the dependency between an async def function and its coroutine runner is not obvious (while PEP 492 was in review, I thought as you did, that it would make the choice of coroutine runner moot).

TODO for 4.3:

• [x] Document coroutine runner issues; encourage use of Tornado coroutine runner for hybrid apps.
• [x] Ensure that anything that handles a Future can also handle an async def (and anything convert_yielded accepts). Specifically, IOLoop.run_sync.
• [x] Document the conversion functions like tornado.platform.asyncio.to_asyncio_future more prominently.
• [x] Document (and possibly rename) tornado.gen.multi_future, since you'll have to refer to it by name in async def coroutines.
• [x] When asyncio is available, register it for convert_yielded whether t.p.asyncio is imported or not (although this may have issues since asyncio futures will generally require the asyncio event loop policy to be set as well)
• [x] Add some sort of cycle detection in convert_yielded (#1499). (doesn't look feasible)

[bdarnell]

Closing this since I've done all of the above.

[gwillem]

For anyone Googling how to use Tornado's AsyncHTTPClient with asyncio, to get @arthurdarcet's example above working, you would need to install the AsyncIOMainLoop and convert the Tornado Future like this:

import asyncio
from tornado.httpclient import AsyncHTTPClient
from tornado.platform.asyncio import to_asyncio_future, AsyncIOMainLoop


async def main():
    http_client = AsyncHTTPClient()
    response = await http_client.fetch('http://httpbin.org')
    return response.body

AsyncIOMainLoop().install()
loop = asyncio.get_event_loop()
loop.run_until_complete(to_asyncio_future(main()))

[arthurdarcet]

Additionally, using asyncio.gather will prevent you from yield from / await tornado Futures/coroutines. You will need to use tornado.gen.multi instead:

import asyncio
import tornado.gen
import tornado.ioloop
import tornado.platform.asyncio

async def do_tornado():
    await tornado.gen.coroutine(lambda: None)()
    print('tornado')

async def main():
    await tornado.gen.multi([do_tornado()])
    # this will fail:
    await asyncio.gather(do_tornado())

tornado.platform.asyncio.AsyncIOMainLoop().install()

coro = main()
coro = tornado.platform.asyncio.to_asyncio_future(coro)

loop = tornado.ioloop.IOLoop.current()
loop.run_sync(lambda: coro)
# or:
#loop = asyncio.get_event_loop()
#loop.run_until_complete(coro)

@bdarnell : is this an expected behavior? Could a warning be added to this section: http://www.tornadoweb.org/en/stable/guide/coroutines.html#python-3-5-async-and-await ?

[bdarnell]

No, it's not expected, although now I can see why. It looks like asyncio.gather is creating an asyncio runner for these coroutines. (the problem is in passing do_tornado() into asyncio.gather, not in awaiting the result of gather). I guess the docs do need a mention of how some functions can create runners on their own and so it may be necessary to use the tornado functions instead of their asyncio counterparts.

[arthurdarcet]

I have opened #1684 to track this