Uvloop: UDP sendto performance regression 0.8 -> 0.9

[qdbp]

• uvloop version:
  0.8.1 -> 0.9.1
• Python version:
  3.6.3 / Cython version 0.27.3
• Platform:
  Linux 4.13.12-1-ARCH
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:

I upgraded uvloop from 0.8.1 to 0.9.1 and noticed a significant (5x slower) regression in the performance of sendto on the UDP transport. I've attached call profiling information as well as the relevant code that triggered it.

The profiling information below is for completely unchanged code, just the different versions of uvloop installed.

Version 0.8.1:
ncalls=48609413 tottime=161.6 percall=3.325e-06 cumtime=161.6 percall=3.325e-06 dht.pyx:124(send_msg)
https://imgur.com/XAnLthE
Here profiling does not give any details below send_msg, suggesting what's running is pure cython code without profile=True.

Verstion 0.9.1
ncalls=73392211 tottime=649.4 percall=8.848e-06 cumtime=1127 percall=1.535e-05 dht.pyx:126(send_msg)
https://imgur.com/FMZzVjv
You can see the same call dips into pure python (socket.py)

Below is an abstracted form of my codepath.

cdef class Listener:
    ...
    cpdef void connection_made(self, object transport):
        self.transport = transport
    cdef void send_msg(self, bytes msg, tuple addr):
        self.transport.sendto(msg, addr)

def run():
   ...
   the_loop = uvloop.new_event_loop()
   run_listener = the_loop.create_datagram_endpoint(
        lambda: Listener(), local_addr=('0.0.0.0', 12345)
    )
   aio.ensure_future(run_listener, loop=the_loop)
   ... other tasks
   ...

My guess is that the transport became Pythonic vs. C level (that would explain the self-time increasing due to python-level attribute access). Also, from the line numbers into socket.py, for some reason the socket's type (python-level @)property is being accessed on every send.

If I can find the time I'll see if I can poke around, but I'm not that familiar with the uvloop codebase.

I think this is a pretty big issue. For me at least, 0.9.1 is currently unusable.

EDIT:
I give 0.8 credence this is the culprit:
https://github.com/MagicStack/uvloop/commit/2ecaa72e2319e470c788d293f5480b79b270b78f#diff-9aed5173d50c6a444fca0f39d08987e0R145

[1st1]

No need to debug or profile. Just read the release notes: in 0.9.x UDP has a different implementation, because libuv doesn't provide enough APIs for uvloop to maintain full compatibility with asyncio. I'll be making a PR to libuv to add new APIs so that we can have the old high-performance UDP implementation back.

10x slower is surprising though, I didn't expect this to happen.

[1st1]

If anyone want to help me with this: we need to add uv_udp_connect() function to libuv (similar to uv_tcp_connect()). As soon as we have it there, we'll have a fast UDP in uvloop. I myself am super busy with other stuff, and don't have any ETA on when I can make that libuv PR (it could be this weekend or 3 months later).

[1st1]

This will potentially unblock us: https://github.com/libuv/leps/pull/10 As a last resort we can consider shipping patched libuv.

[1st1]

There's a debate about adding uv_udp_connect() in libuv 1.x here: https://github.com/libuv/libuv/pull/1872#issuecomment-464080195

/ Fingers crossed. /

[jlaine]

Let's hope so! I had a quick chat with @saghul at FOSDEM mentioning UDP performance improvements in uvloop were currently tied to uv 2.x plans..

[saghul]

@jlaine I think https://github.com/libuv/libuv/pull/1872#issuecomment-464080195 is indeed it.

[jlaine]

Yay, this PR landed:

https://github.com/libuv/libuv/pull/2217

.. and is released in libuv 1.27

@1st1 could we have an update of the bundled libuv so we can try and make use of this feature?

[1st1]

I think it would be easier to wait until they release it.

That said we can start working on this ;) A good place to start would be to find the old UDP implementation in the history and try to resurrect it.

[jlaine]

The release has actually been tagged :)

[1st1]

That was quick ;) I'm reading this on my phone so i missed the fact they released it already.

We can release uvloop 0.13 as soon as we update our UDP layer.

[jlaine]

Thanks for bumping the libuv version, I'll do a bit of archeology to see what the old implementation looked like.

Can we retain the caching logic I added for address validation though? In the case of an unconnected socket it's still nice not to hit the slow validation code.

[1st1]

I'm actually working on it, and we have a new problem: libuv's uv_udp_t doesn't seem to support AF_UNIX sockets which we now support... We might need to open a PR @ libuv. But so far I don't see the end of the rabbit hole. I'll push my branch soon so that you can also play with it.

[1st1]

The https://github.com/MagicStack/uvloop/pull/238 PR is updating uvloop to start using uv_udp_t handles again. Anyone interested in seeing improved UDP performance please try to test/benchmark that PR.

[jlaine]

This is fixed in uvloop 0.13.0! 🎉