Uvloop: Probably leaking file objects somewhere

[iceboy233]

Reproduce:

import asyncio
import os
import uvloop

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()

async def main():
    try:
        os.mkfifo('/tmp/fifo')
    except FileExistsError:
        pass
    while True:
        reader = asyncio.StreamReader()
        protocol = asyncio.StreamReaderProtocol(reader)
        fd = os.open('/tmp/fifo', os.O_RDONLY | os.O_NONBLOCK)
        print(fd)
        await loop.connect_read_pipe(lambda: protocol, os.fdopen(fd))
        os.close(os.open('/tmp/fifo', os.O_WRONLY))
        await asyncio.sleep(.1)

loop.run_until_complete(main())

When using the builtin loop, the fds opened every time is the same, and while using uvloop, the fds are increasing.

[1st1]

What if you explicitly close the transport?

trans, photo =  await loop.connect_read_pipe(lambda: protocol, os.fdopen(fd))
os.close(os.open('/tmp/fifo', os.O_WRONLY))
await asyncio.sleep(.1)
trans.close()

[iceboy233]

What if you explicitly close the transport?

Then it doesn't leak. However the default loop doesn't leak even if the transport not explicitly closed.

[1st1]

What OS are you testing this on? For me, on macOS, the behaviour with uvloop is the same as without it...

[iceboy233]

Linux Mint 18.1 Serena
Linux 4.4.0-53-generic 64bit
Python 3.5.2 64bit
uvloop 0.8.0

[iceboy233]

Tested on Linux 3.13 and 4.8, and Python 3.6 - they all leak.

[1st1]

I see that transports aren't GCed. I'm not sure how I feel about this bug: on one hand you have to close transports explicitly. On the other, we want uvloop to behave exactly as asyncio.

Anyways, I'll try to get to the bottom of this.

[1st1]

I found the problem that causes this. The core problem is that the Transport is kept alive while its underlying FD is not closed, which shouldn't be an issue for web servers etc. In any case, I have a fix in mind, will try to implement it soon.

[1st1]

The core problem is that the Transport is kept alive while its underlying FD is not closed

Actually, asyncio behaves in exactly the same way -- the Transport is kept alive by the _read_ready bound method as long as selector is monitoring its socket.

With 3.6.3 I don't see any difference between uvloop and vanilla asyncio. On Linux, FDs get reused at some point of time when a GC happens (probably because _read_ready callback is triggered because the other end of the pipe is closed). On MacOS, _read_ready doesn't receive any indication that the other end of the pipe is closed, so both uvloop and vanilla asyncio leak transports/FDs.

The advice is still the same: always close your transports.

I'm going to close this as "not a bug". Please feel free to reopen if you find another example where transports are closed/kept alive differently from asyncio.