Aiohttp: Long network delay

[rickcoup]

Long story short

Under load around 6 transactions per second, our http requests and asyncio calls are timeout'd at 10 seconds. It makes our API unusable.

Expected behaviour

The http request should be responded within 1 second or hundreds of milliseconds.

Actual behaviour

From cProfiler, it shows

ncalls tottime percall cumtime percall filename:lineno(function)

1375 73906.971 53.751 73906.971 53.751 {method 'poll' of 'select.epoll' objects}

The network connections are increased
from

(app-root) sh-4.2$ ss -s
Total: 10042 (kernel 0)
TCP: 711 (estab 11, closed 698, orphaned 0, synrecv 0, timewait 210/0), ports 0

Transport Total IP IPv6

• 0 - -
  RAW 0 0 0
  UDP 0 0 0
  TCP 13 13 0
  INET 13 13 0
  FRAG 0 0 0

to

(app-root) sh-4.2$ ss -s
Total: 10065 (kernel 0)
TCP: 800 (estab 54, closed 744, orphaned 0, synrecv 0, timewait 259/0), ports 0

Transport Total IP IPv6

• 0 - -
  RAW 0 0 0
  UDP 0 0 0
  TCP 56 56 0
  INET 56 56 0
  FRAG 0 0 0

Steps to reproduce

async def fetch(method, session, url, pData, head, timeout):
''' send reqest by async session with timeout '''
#async with getattr(session, method)(url, data=pData, headers=head, timeout=timeout) as response:
    #return {'data': await response.json(), 'status': response.status}
#Trying to change, to check the performance
if method == 'get':
    async with session.get(url, data=pData, headers=head, timeout=timeout) as response:
        return {'data': await response.json(), 'status': response.status}
elif method == 'post':
    async with session.post(url, data=pData, headers=head, timeout=timeout) as response:
        return {'data': await response.json(), 'status': response.status}
elif method == 'delete':
    async with session.delete(url, data=pData, headers=head, timeout=timeout) as response:
        return {'data': await response.json(), 'status': response.status}
elif method == 'put':
    async with session.delete(url, data=pData, headers=head, timeout=timeout) as response:
        return {'data': await response.json(), 'status': response.status}

Your environment

python 3.6.3, aiohttp==3.5.4 and asyncio==3.4.3 in openshift 3.9

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are https://github.com/aio-libs/aiohttp/issues/962 (Handling very long connection timeouts), https://github.com/aio-libs/aiohttp/issues/1600 (aiohttp for long data processing), https://github.com/aio-libs/aiohttp/issues/2522 (aiohttp and network is unreachable), https://github.com/aio-libs/aiohttp/issues/2726 (Failing tests without network connection), and https://github.com/aio-libs/aiohttp/issues/2729 (400 milliseconds delay using aiohttp).

[MasterGroosha]

I've noticed this happens when using IPv6 instead of IPv4

[asvetlov]

I don't see the mentioned behavior on my boxes.
The snippet from the very first message doesn't contain a comprehensive instruction for reproduction.
Until I have a way to analyze the problem most likely will not be fixed, sorry.

Please provide a stable way of how to get the slowdown.

[GoEdgar]

@asvetlov
socket.getaddrinfo returns a list of supported sockets whose first indexes, if IPv6 is enabled, contain the supported parameters of the IPv6 socket server _(this is true on my local machine. on my phone, the first indexes of the result of calling this method contain information about IPv4, not IPv6)_
In asyncio, socket.getaddrinfo is called before the connection, each result of which participates in attempts to connect a socket to a host socket until it fails.
However, if one of the routers fails to support this protocol (IPv6) on the way to the host, no response is received, and the connection attempt is timed out.

For example, if IPv6 is enabled in the network settings of your local machine, the remote host supports IPv6, but on the way to it one of the nodes does not support IPv6, the connection will be very long, because the first attempt will be to establish an IPv6 connection, which will last up to the timeout value (on Windows it is about 21 seconds)

[asvetlov]

It sounds like a plausible explanation.
If aiohttp works slow on the misconfigured system -- that's fine I guess.
Backporting Happy Eyeballs from Python 3.8 may improve the situation; it requires a lot of work though.

[GoEdgar]

@asvetlov
But in RFC8305 are talking about what is preferable to IPv6, while most on the way to the host support IPv6 is not provided, despite the fact that in the connection settings of the local machine it is enabled.

[asvetlov]

IIRC it is for trying to connect to multiple addresses at once, waiting for the first successful connection and dropping all others

[asvetlov]

You can use TCPConnector(family=socket.AF_INET) to disable IPv6