Uvloop: memory leak with 0.12.0

[amir20]

• uvloop version: 0.12.0
• Python version: 3.7.2
• Platform: Mac os and Docker
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: Have not tried yet

Recently I upgraded from 0.11.3 to 0.12.0. I use uvloop to constantly pull data from an API and insert into a database. After running for a couple of minutes, I found that my memory usage jumped from ~100MB to ~450mb. Has anybody else run into this issue? I have a lot going on so reproducing it with a smaller gist would be hard right now. But reverting back to 0.11.3 immediately fixed the issue. So I am guessing something is related to the upgrade.

[1st1]

Wow, this is unfortunate.

Does your application use SSL connections?

[amir20]

Yes it does.

[1st1]

UDP? 0.12.0 has a couple of UDP-related changes. If you use UDP please try to run 0.12.0rc1 too.

The 0.12 release has a brand new SSL implementation, so the leak might be related to that. Ideally we need a script to reproduce it though.

[amir20]

I am not using UDP. It is only HTTPS and called about 51 requests per second. I'll see what I can do to reproduce it. I have an API, mongo storage and a bunch of other things that need to be stripped.

I might just create a tight loop with the two versions and record the memory.

[amir20]

Alright. I was able to reproduce this pretty easily.

Here is my script.

import asyncio
import os

import aiohttp
import uvloop
from async_timeout import timeout

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())


def headers():
    return dict(authorization=f"Bearer {os.getenv('API_TOKEN')}")


async def fetch(url):
    async with aiohttp.ClientSession(headers=headers()) as session:
        async with timeout(8):
            async with session.get(url) as response:
                data = await response.json()
                return response.status, data


for i in range(1000):
    future = fetch("https://api.clashofclans.com/v1/clans/%23UGJPVJR")
    code, response = asyncio.get_event_loop().run_until_complete(future)
    print(i, code)

I ran this with 0.11.x and after 200 iterations my memory was around 25MB. I used Docker so here is a screenshot.

I did pip install -U uvloop and docker-compose build with the latest uvloop and after 200 iterations the memory is around 64MB.

Screenshot attached.

I have not investigated any further as to what could be causing this.

[1st1]

@amir20 Thank you so much.
@fantix please take a look.

[fantix]

Thanks for the reproducing! I'm checking into this.

[1st1]

@fantix were you able to figure it out?

[fantix]

@1st1 I'm still trying to find a clue, but I can reproduce the issue.

[1st1]

Great, thank you! I'll try to find some time to help in the next couple of days.

[fantix]

Thanks for the merge!

@amir20 please feel free to let me know if it has further issues. Thanks!

[amir20]

Nice. Would you be able to get a patch released?

[1st1]

asap; I'm working on another fix right now.

[1st1]

Please try the new uvloop 0.12.1; hopefully it fixes your case.

[amir20]

I am testing right now. Will let it run for an hour and post an update.

[amir20]

I think it works. After 30 minutes, its still at 135MB which is a good sign. Great job @1st1 :)

[1st1]

I think it works. After 30 minutes, its still at 135MB which is a good sign. Great job @1st1 :)

Great news! The fix is to @fantix's credit though :)