Pyzmq: Loop-less zmq.asyncio

[f0t0n]

Hi guys,

Is it possible to re-work the zmq.asyncio package and remove the need of usage of zmq.asyncio.ZMQEventLoop?

Say if I use zmq in my aiohttp application in combination with uvloop I can't just import and use zmq.asyncio easily. So that this forces me to use third-party libraries like aiozmq instead of the native pyzmq.

Also please take a look at https://github.com/KeepSafe/aiohttp/issues/1092#issuecomment-241133022 describing specific issues with zmq.asyncio.ZMQEventLoop.

I'm not proficient in this particular area therefore I delegate the next discussion to @asvetlov and other guys who know this area well.
If someone is able to work on this, it would be really nice! And as @asvetlov said he can point on required changes.

[asvetlov]

The implementation of loop-less transport is here: https://github.com/aio-libs/aiozmq/blob/master/aiozmq/core.py#L667

While it's based on Transport protocol the code may be transplanted into zmq.asyncio.Socket easily.

The key point is not adding writer but getting notification file descriptor by fd = zmq_sock.getsockopt(zmq.FD) and subscribing for events from it.

The tricky part of implementation is: after every read/write operation the file descriptor should be requested again for upcoming events and if there are events to process .read_ready() should be scheduled again on the next loop iteration.

From my understanding it's because zmq.FD works on edge-triggered level, not level triggered one.
@tailhook had consulted me about the patch, he knows better how zeromq works in this mode.

[minrk]

Having dealt with integrating the edge-triggered FD in an eventloop in zmq.green, I don't want to be responsible for maintaining another implementation that relies on it. I've pushed for a level-triggered separate read/write FD API in libzmq for years, so if that ever happens I will happily drop the zmq_poll implementations and use it.

Fortunately, aiozmq exists and takes on that challenge, so please feel free to use that. I don't view zmq.asyncio as a replacement for it at all, just one simple implementation of AIO-compatible use of zmq.

[ssolari]

+1 I threw out a polling suggestion in https://github.com/KeepSafe/aiohttp/issues/1223. I'm not an expert. Having pyzmq capable of operating with the base asyncio loop would be pretty dreamy.