Aiohttp: Failing test in Python 3.8

[Mariatta]

Long story short

I'm seeing tests related to aiohttp that fail in Python 3.8+, but pass in Python 3.7.2.
I encountered this in bedevere, you can check the travis CI log in there. (https://github.com/python/bedevere/pull/154)

Expected behaviour

Tests should be passing.

Actual behaviour

The tests pass for Python 3.7.2, but fail for Python 3.8

Steps to reproduce

I created the following test case: (copy pasted from the docs: https://docs.aiohttp.org/en/stable/testing.html#pytest)

from aiohttp import web

async def hello(request):
    return web.Response(text='Hello, world')

async def test_hello(aiohttp_client):
    app = web.Application()
    app.router.add_get('/', hello)
    client = await aiohttp_client(app)
    resp = await client.get('/')
    assert resp.status == 200
    text = await resp.text()
    assert 'Hello, world' in text

$ pytest -k test_hello
=========================================================================================================== test session starts ===========================================================================================================
platform darwin -- Python 3.8.0a1, pytest-4.2.0, py-1.7.0, pluggy-0.8.1
rootdir: /Users/mariatta/python_github/bedevere, inifile: pytest.ini
plugins: asyncio-0.10.0, aiohttp-0.3.0
collected 126 items / 125 deselected / 1 selected                                                                                                                                                                                         

tests/test___main__.py F                                                                                                                                                                                                            [100%]

================================================================================================================ FAILURES =================================================================================================================
___________________________________________________________________________________________________________ test_hello[pyloop] ____________________________________________________________________________________________________________

aiohttp_client = <function aiohttp_client.<locals>.go at 0x103232040>

    async def test_hello(aiohttp_client):
        app = web.Application()
        app.router.add_get('/', hello)
        client = await aiohttp_client(app)
>       resp = await client.get('/')

tests/test___main__.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.8/site-packages/aiohttp/test_utils.py:294: in request
    resp = await self._session.request(
venv/lib/python3.8/site-packages/aiohttp/client.py:497: in _request
    await resp.start(conn)
venv/lib/python3.8/site-packages/aiohttp/client_reqrep.py:844: in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <aiohttp.client_proto.ResponseHandler object at 0x1034937d0>

    async def read(self) -> _T:
        if not self._buffer and not self._eof:
            assert not self._waiter
            self._waiter = self._loop.create_future()
            try:
>               await self._waiter
E               aiohttp.client_exceptions.ClientOSError: [Errno 54] Connection reset by peer

venv/lib/python3.8/site-packages/aiohttp/streams.py:588: ClientOSError
---------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------
Exception ignored in: <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 58013), raddr=('127.0.0.1', 58014)>
ResourceWarning: unclosed <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 58013), raddr=('127.0.0.1', 58014)>

Your environment

Mac OS Mojave 10.14.3 Python 3.8.0a1.

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are https://github.com/aio-libs/aiohttp/issues/537 (Drop python 3.3 support), https://github.com/aio-libs/aiohttp/issues/3458 (Beta release is failed on Python 3.7 bot for Linux), https://github.com/aio-libs/aiohttp/issues/467 (Failed building wheel for aiohttp on Python 3.5.0rc1), https://github.com/aio-libs/aiohttp/issues/1302 (Make test suite passed on python 3.6 and default), and https://github.com/aio-libs/aiohttp/issues/2148 (Failed tests).

[Mariatta]

🤔 Should I opened this in https://github.com/aio-libs/pytest-aiohttp instead?
Just not sure about the underlying issue and this is outside of my domain.

[webknjaz]

Nope, here is fine. That repo is just a wrapper to auto-add fixtures. The real code of those is here.

[webknjaz]

From prompt look it seems like server isn't listening to port while client sends a request there...

[Mariatta]

Same code works in Python 3.7 though. Would this be a regression from Python side or aiohttp?

[webknjaz]

I believe it might be an asyncio regression. Our nightly job is failing but for another reason: https://travis-ci.com/aio-libs/aiohttp/jobs/177507070.

Let's wait till Andrew takes a look.

[webknjaz]

Update: flake8 issue is gone, but the real tests started failing and they get completely stuck at some point.

Ref: https://travis-ci.com/aio-libs/aiohttp/jobs/177768192#L764

[rockwelln]

I ran into the same error with the following build: Python 3.8.0a2+ (heads/master:f45813d, Mar 12 2019, 21:49:03).

[pardo-bsso]

Same here with 3.8.0a2. When running the examples from aiohttp I see on netstat that the port is indeed listening but refuses connections. However other stuff I have using asyncio works fine (for example, lx200/examples/tcpserver.py )

[rockwelln]

deep diving into the output with PYTHONASYNCIODEBUG=1, the minimal example produce:

DEBUG:asyncio:<Server sockets=[<socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 8080)>]> got a new connection from ('10.0.7.7', 62830): <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.0.23.5', 8080), raddr=('10.0.7.7', 62830)>
ERROR:asyncio:Error on transport creation for incoming connection
handle_traceback: Handle created at (most recent call last):
  File "server.py", line 37, in <module>
    web.run_app(app)
  File "/opt/cc/lib/python3.8/site-packages/aiohttp/web.py", line 401, in run_app
    loop.run_until_complete(_run_app(app,
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/base_events.py", line 576, in run_until_complete
    self.run_forever()
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/base_events.py", line 544, in run_forever
    self._run_once()
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/base_events.py", line 1756, in _run_once
    handle._run()
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/selector_events.py", line 188, in _accept_connection
    self.create_task(accept)
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/base_events.py", line 408, in create_task
    task = tasks.Task(coro, loop=self, name=name)
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8-dev/lib/python3.8/asyncio/selector_events.py", line 197, in _accept_connection2
    protocol = protocol_factory()
  File "/opt/cc/lib/python3.8/site-packages/aiohttp/web_server.py", line 57, in __call__
    return RequestHandler(self, loop=self._loop, **self._kwargs)
  File "/opt/cc/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 136, in __init__
    self._keepalive = False
AttributeError: 'RequestHandler' object has no attribute '_keepalive'

Which seems totally related to #3644.
I fixed it locally and unblock the situation :-D

[asvetlov]

Fixed by #3727

[Mariatta]

Nice. Thanks!