Tornado: Tornado 4.0.1, AsyncHTTPTestCase, IOLoop is already running

[hgenru]

After the upgrade, there was a problem with the use of tornado.testing tools.

I'm run this simple code: https://gist.github.com/hgenru/47ad817a199de2e33333

(env)➜ pip install tornado==3.2

(env)➜ env/bin/nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.082s

OK


(env)➜ pip uninstall tornado && pip install tornado
(env)➜ env/bin/nosetests
E
======================================================================
ERROR: test_response (test.HelloHandlerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/testing.py", line 118, in __call__
    result = self.orig_method()
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/testing.py", line 494, in post_coroutine
    timeout=timeout)
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/ioloop.py", line 418, in run_sync
    return future_cell[0].result()
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/concurrent.py", line 109, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/gen.py", line 160, in wrapper
    result = func(*args, **kwargs)
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/testing.py", line 480, in pre_coroutine
    result = f(self, *args, **kwargs)
  File "/home/hgen/test/47ad817a199de2e33333/test.py", line 15, in test_response
    res = self.fetch('/')
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/testing.py", line 373, in fetch
    return self.wait()
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/testing.py", line 303, in wait
    self.io_loop.start()
  File "/home/hgen/test/47ad817a199de2e33333/env/lib/python3.4/site-packages/tornado/ioloop.py", line 704, in start
    raise RuntimeError("IOLoop is already running")
RuntimeError: IOLoop is already running

----------------------------------------------------------------------
Ran 1 test in 0.075s

FAILED (errors=1)

[bdarnell]

@gen_test and self.fetch() are not compatible with one another, although this is not documented (it needs to be, and I need to see if there's some way to make this interaction better). It happened to work before because you never used yield, but now it's failing due to some changes in 4.0 to improve stack traces on timeouts. Since you don't yield, you can simply remove the @gen_test decorator.

For more complex tests, you'll need to choose between using self.fetch, self.stop, and self.wait for your asynchronous operations, or using yield like you would in a coroutine. For the latter, you'll probably want some helper functions to construct the correct url like self.fetch does.

[hgenru]

@bdarnell maybe add more information into docs? This can cause difficulty in migrating to the new version.

[bdarnell]

To be clear, gen_test and fetch have always been incompatible. It happened to work in 3.2 only if you didn't actually make use of gen_test (by yielding anything).

[helderm]

_@gen_test and self.fetch() are not compatible with one another, although this is not documented (it needs to be, and I need to see if there's some way to make this interaction better)_

Does this statement still holds true (Tornado 4.2)? I'm trying reproducing the same example documented here and I'm receiving the same 'IOLoop is already running' error.

Here is my minimal Gist, am I doing something wrong? What is the correct way of doing this?

[ajdavis]

Instead of this:

    res = yield Task(self.fetch('/'))

Do this:

    res = yield self.http_client.fetch(self.get_url('/'))

The "self.fetch" method on AsyncHTTPTestCase assumes the IOLoop is not
currently running. self.fetch() calls self.io_loop.start(), which throws
the error you see.

If you're using gen_test, then the IOLoop is always running during your
test, so you can't call self.io_loop.start(). Instead of calling
self.fetch, get a Future from self.http_client.fetch, then wait for it to
complete by yielding it.

On Thu, Nov 6, 2014 at 1:23 PM, Helder Martins [email protected]
wrote:

_@gen_test and self.fetch() are not compatible with one another, although
this is not documented (it needs to be, and I need to see if there's some
way to make this interaction better)_

Does this statement still holds true (Tornado 4.2)? I'm trying reproducing
the same example documented here
http://tornado.readthedocs.org/en/latest/testing.html#tornado.testing.gen_test
and I'm receiving the same 'IOLoop is already running' error.

Here is my minimal Gist
https://gist.github.com/helderm/8e44e583dff94a34ad76, am I doing
something wrong? What is the correct way of doing this?

—
Reply to this email directly or view it on GitHub
https://github.com/tornadoweb/tornado/issues/1154#issuecomment-62025837.

[helderm]

I tried this, but the problem is that Tornado enforces the gen_test decorator for any test function with a yield, raising the error below:

TypeError: Generator test methods should be decorated with tornado.testing.gen_test

What worked for me in the end was doing the AsyncTestCase.stop / wait manually, instead of using AsyncHTTPTestCase.fetch

    def test_one(self):
        self.http_client.fetch(self.get_url('/'), self.stop)
        res = self.wait()

I created this working gist for those interested.

[ajdavis]

In your original gist you _did_ decorate your test_one method with
@gen_test. This looks like a reasonable way to write a test, to me:

class HelloHandlerTest(AsyncHTTPTestCase):
def get_app(self):
return Application(
[(r'/', HelloHandler)]
)

@gen_test
def test_one(self):
    res = yield self.http_client.fetch(self.get_url('/'))
    assert res.error is None, 'error: %s' % str(res.error)

Was there any problem with that approach?

On Fri, Nov 7, 2014 at 12:29 PM, Helder Martins [email protected]
wrote:

I tried this, but the problem is that Tornado enforces the gen_test
decorator for any test function with a yield, raising the error below:

TypeError: Generator test methods should be decorated with tornado.testing.gen_test

What worked for me in the end was doing the AsyncTestCase.stop / wait
manually, instead of using AsyncHTTPTestCase.fetch

def test_one(self):
    self.http_client.fetch(self.get_url('/'), self.stop)
    res = self.wait()

—
Reply to this email directly or view it on GitHub
https://github.com/tornadoweb/tornado/issues/1154#issuecomment-62180909.

[bdarnell]

Note that self.http_client.fetch() and self.fetch() are different: the former is a standard http client call like you'd use in application code, and must be used in conjunction with either gen_test and yield or stop and wait. self.fetch is shorthand for the use of self.http_client.fetch with stop and wait.

[helderm]

_Was there any problem with that approach?_
@ajdavis
Yes, that way it returns the "IOLoop is already running" error.
As far as I investigated, that occurs because both gen_test post_coroutine and AsyncTestCase wait (called by AsyncHTTPTestCase fetch ) attempts to start the IOLoop.

[ajdavis]

Right. As Ben said, don't call AsyncHTTPTest's fetch: that is, do not call
self.fetch(). Instead, call self.http_client.fetch(). Here's a complete
gist that works correctly:

https://gist.github.com/ajdavis/69b12c7ee77d6ae010d0

On Tue, Nov 11, 2014 at 12:12 PM, Helder Martins [email protected]
wrote:

_Was there any problem with that approach?_
@ajdavis https://github.com/ajdavis
Yes, that way it returns the "IOLoop is already running" error.
As far as I investigated, that occurs because both gen_test post_coroutine
https://github.com/tornadoweb/tornado/blob/master/tornado/testing.py#L499
and AsyncTestCase wait
https://github.com/tornadoweb/tornado/blob/master/tornado/testing.py#L281
(called by AsyncHTTPTestCase fetch
https://github.com/tornadoweb/tornado/blob/master/tornado/testing.py#L372
) attempts to start the IOLoop.

—
Reply to this email directly or view it on GitHub
https://github.com/tornadoweb/tornado/issues/1154#issuecomment-62580600.

[Groxx]

Is this still an issue? Tornado 4.5.1, and docs say

class MyTest(AsyncHTTPTestCase):
    @gen_test
    def test_something(self):
        response = yield gen.Task(self.fetch('/'))

but I get the same error unless I do this (left everything intact for completeness):

from tornado.web import Application, RequestHandler
from tornado.testing import AsyncHTTPTestCase, gen_test

class Http(RequestHandler):
    def get(self):
        self.write('OK')

class TestHttp(AsyncHTTPTestCase):
    def get_app(self):
        return Application([(r'/', Http)])

    @gen_test
    def test_which_works(self):
        response = yield self.http_client.fetch(self.get_url('/'))

http://www.tornadoweb.org/en/stable/testing.html#tornado.testing.gen_test

[bdarnell]

Nothing has changed: @gen_test and self.fetch are incompatible with each other; you must choose one or the other.

[Groxx]

Yes, but the documentation directly contradicts this, at the location I linked.

[bdarnell]

Ah, thanks for the report! I don't think those docs were ever accurate, even when they were written.