Discord.py: Rewrite: Memory leak

[Shea4]

`import discord
from discord.ext import commands

bot = commands.AutoShardedBot(command_prefix="prefix")

bot.run("token")`

My bot has been leaking memory for quite a while now but now it's in around 2.4k servers it's becoming a much bigger issue, I've been through multiple steps to try and find out what the possible leak was and I haven't found anything, so I decided to run the script above on the bots token to find out if that leaked memory and it did so I've came here to ask is there something I need to disable or do to try and stop the leak.

Extra info: Memory climbs to 1.5gb within 5 hours, the bot is in 2.4k servers running 2 shards

[shikhir-arora]

That's probably just normal cache, not a memory leak. Discord sends all entities and those (to list a few: users, servers, channels, etc.) are stored in memory. discord.py, like all Discord API libraries, manages the cache and chunks of data from the Discord gateway, but it's likely pretty normal with 2,400 servers -- especially if there are lots of users in those servers -- for the memory usage to increase. After the initial period, it shouldn't keep climbing (at a very high rate) if everything remains constant though, and by that I mean over several days it shouldn't be going up gigabytes if nothing changes. If it is, then there's probably an issue that should be looked into.

[Shea4]

It continues to go up at a high rate, I think the highest It's reached is 4gb within 4 days If i remember correctly so I don't believe It's due to caching.

[Hornwitser]

See https://stackoverflow.com/questions/1435415/python-memory-leaks for various methods you can try in order to trace down what's leaking memory.

[Shea4]

I have used some of those methods before and there's nothing obvious, my main point is even the simplistic code above leaks.

[numbermaniac]

What version of discord.py rewrite are you on? Try pip show discord.py - I think the latest version starts with 1.0.0a1634.

[Hornwitser]

Unfortunately I don't have a 2.5k guilds bot to test this.

[Shea4]

The version is 1.0.0a1590

[Hornwitser]

Latest version should be a1634, also there should be the actual commit in the version string.

[Shea4]

Yeah sorry here's the full thing: 1.0.0a1590+g860d6a9

[Hornwitser]

Soo.... Are you going to update?

[numbermaniac]

You may want to try re-running the installation command to update to the newest version of discord.py, then try running your bot again.

[Shea4]

Yeah I'll get back to you when I've updated

[Shea4]

Just updated, going to monitor the memory over the next few hours

[Shea4]

It still leaks memory, definitely not as quickly as before, climbed about 300mb in 5ish hours compared to 1gb. Could at this point it just be caching, but I'm thinking it's a bit excessive for just caching. Note the same results happened on the simple code in my first comment.

[Hornwitser]

Did you even read the issue? There's a grand total of 4 lines of code in the test bot that observes the memory leak. It is most certainly not the bot code.

[NotSoSuper]

With my experience of previously running a >200k guild AutoSharded - across multiple processes - discord.py bot, it's the state/cache. @shikhir-arora explains it pretty well, with the exception that Python has a much larger object overhead than other languages.

I've done my own memory leak searching and couldn't find anything that was obvious.
Currently, my 170k guild bot starts with ~24gb of memory during a normal load day; it goes up to 32gb within two days.

If there is a deeper rooted memory leak though, I'd be open to performing any tests.

[giveen]

I've honestly been trying to find a memory leak as well. 37462 people in 33 guilds currently using 94MB, but it slowly grows. I rebooted it recently but it was at 124MB before the reboot.

[Vexs]

Its natural for it to grow to a point- when members are offline, they (depending on a couple things) might not be cached. As they come online, they're added to the cache, and as far as I know, stay there forever.

[giveen]

Its natural for it to grow to a point- when members are offline, they (depending on a couple things) might not be cached. As they come online, they're added to the cache, and as far as I know, stay there forever.

Thats what I learned on the discord that it caches members, but it seems rather odd behavior, and I guess I dont understand the point of doing that. It would seem some kind of release or garbage collection should be performed.

[Vexs]

It's not odd at all- discord doesn't give you before and after states for... pretty much anything. I don't really know if they do at all, frankly. If you want to find out when someone gets a role, you have to have the state they were in prior stored somewhere. The cache is necessary for much of the expected functionality.

You could rewrite large sections of dpy to store this on disk if you really wanted to, or just dump it entirely- or, more sensibly, remove when they go offline.

As a note, there isn't a way to distinguish between offline and invisible members, which if I had to guess, is why dpy doesn't do this to start with.

[Shea4]

Well I feel like since the version update of discord.py my issue has been solved as I'd say the state of my memory is exactly how @shikhir-arora described what discord.py does, Thank you all for the help.