Uvloop: Computing IO-Wait time

[pranavtbhat]

I'm investigating an IO-boundedness issue with a python server, and would like to instrument the uvloop to compute the amount of time spent polling for IO.

The way to do this in libuv would be to register a prepare handle that records the start_time and a check handle that records the the stop_time and writes it to some form of global state. I went through the UVLoop code, and there seems to be no way to register custom check or prepare handles.

Is there another cleaner way to do this? I suppose I could subclass uvloop.Loop and register the handles myself, but that seems to be a fairly ugly approach.

[1st1]

You can try to build uvloop in debug mode. It will have many debug attributes then; if total polling time isn't one of them we can add it.

In general we do need to design a deep tracing API for uvloop. https://github.com/MagicStack/uvloop/pull/171 is meant for that, but I didn't have the time to work on it.

[pranavtbhat]

Total polling time doesn't seem to be a debug attribute. Also, we probably need aggregate data to debug this, so debug-mode may not be feasible for extensive analysis.

The ability to register custom prepare/check handles could allow users like me to self service some tracing/profiling needs. Do you think this would be clean and worth adding to UVLoop?

[1st1]

TBH I don't want to add any non-asyncio API to uvloop. It will increase maintenance overhead and will split the ecosystem a bit.

If you can implement the desired functionality for the debug build, I can consider accepting it.

[pranavtbhat]

I'll follow up with a PR then. Thanks Yury!

[1st1]

I'll follow up with a PR then. Thanks Yury!

NP. Just to clarify: whatever is in the debug build can be removed in the future, i.e. it's not a stable API. But before removing something like that we should figure out a better way.

[pranavtbhat]

@1st1 I tried building the functionality into the debug build, but the debug build itself is a fairly large performance hit.

I considered another option. Inheriting from uvloop.loop.Loop. This requires me to cimport from uvloop. The required cython header (pxd) files are packaged as expected.

However, the libuv .h/.c files are not, and the C compilation fails. Since UVLoop packages it's own libuv, it's probably too hacky to define a separate dependency on libuv.

Do you think we can package the vendor/libuv directory too?