Discord.js: Excessive memory usage for message

[MattIPv4]

Please describe the problem you are having in as much detail as possible:

First off, I'm not really sure this is actually a bug yet, but I'm opening this so I can track the digging and document things so far.

In my Grafana logging for my bot, I noticed that the memory usage for one of my shards had started to increase, out-of-line from all the other shards. Looking further, I then noticed that this seemed to match the calculated memory usage for cached messages in the shard.

Memory RSS is generated from process.memoryUsage() so should be accurate, whereas Cached Messages Memory is an approximation using object-sizeof with an array containing every message in cache. However, this inaccuracy shouldn't matter as this is all about comparison between other shards and other messages.

Alone, these two graphs don't say much, there are probably just more messages in that shard for one reason or another. Alas, there are not. Grafana also has a graph set up for the total number of messages in cache, using similar logic to Cached Messages Memory but without using object-sizeof, and we can see that there are a similar number of messages in the problematic shard to the other shards.

So, at this point, its safe to say that something is different about these messages that's causing them to use a load of extra memory. My next step in debugging what was going on was to get that largest message from each shard and just manually confirm that the messages in shard 7 were indeed much larger. This was done by executing the following script in all shards:

const sizeof = require('object-sizeof');
const messages = client.guilds.cache.array().reduce((prev, guild) => prev.concat(guild.channels.cache.array().reduce((prevC, channel) => { if ('messages' in channel) { return prevC.concat(channel.messages.cache.array()); } return prevC; }, [])), []).sort((a, b) => sizeof(b) - sizeof(a));
return [messages.length, sizeof(messages[0]), sizeof(messages[messages.length-1])];

With the following results:

Shard 0 Result
2404, 3814174, 3096

Shard 1 Result
2103, 96504, 3118

Shard 2 Result
5176, 30996, 3102

Shard 3 Result
3440, 1138488, 3106

Shard 4 Result
1820, 31540, 3128

Shard 5 Result
2514, 28520, 3012

Shard 6 Result
2373, 248242, 3154

Shard 7 Result
4050, 84625818, 3130

This confirmed for me that the messages in shard 7, the problematic shard, were definitely much larger than those in the other, nominal shards. Again, object-sizeof likely isn't being completely accurate with the size estimations, but what matters here is that it is consistent for the comparison between shards and messages.

With this confirmed, the next step was to understand what about the messages in this problematic shard were using so much memory. Again, using object-sizeof, I got the largest messages in the shard and calculated the size for a set of attributes that each message has in Discord.js:

const sizeof = require('object-sizeof');
const message = client.guilds.cache.array().reduce((prev, guild) => prev.concat(guild.channels.cache.array().reduce((prevC, channel) => { if ('messages' in channel) { return prevC.concat(channel.messages.cache.array()); } return prevC; }, [])), []).sort((a, b) => sizeof(b) - sizeof(a))[0];

return JSON.stringify(Object.keys(message).reduce((prev, key) => { prev[key] = sizeof(message[key]) / 1024 ** 2; return prev; }, {}), null, 2);

With the following result:

{
  "channel": 0.0026187896728515625,
  "deleted": 0.000003814697265625,
  "id": 0.000034332275390625,
  "type": 0.0000133514404296875,
  "system": 0.000003814697265625,
  "content": 0,
  "author": 0.0003662109375,
  "pinned": 0.000003814697265625,
  "tts": 0.000003814697265625,
  "nonce": 0,
  "embeds": 0.0007877349853515625,
  "attachments": 0,
  "createdTimestamp": 0.00000762939453125,
  "editedTimestamp": 0.00000762939453125,
  "reactions": 88.01104545593262,
  "mentions": 0.0001068115234375,
  "webhookID": 0,
  "application": 0,
  "activity": 0,
  "_edits": 107.88620185852051,
  "flags": 0.00002288818359375,
  "reference": 0
}

I also ran the same script in another shard (5) where memory usage appeared far more nominal, for comparison:

{
  "channel": 0.0026721954345703125,
  "deleted": 0.000003814697265625,
  "id": 0.000034332275390625,
  "type": 0.0000133514404296875,
  "system": 0.000003814697265625,
  "content": 0,
  "author": 0.0003490447998046875,
  "pinned": 0.000003814697265625,
  "tts": 0.000003814697265625,
  "nonce": 0,
  "embeds": 0.0044078826904296875,
  "attachments": 0,
  "createdTimestamp": 0.00000762939453125,
  "editedTimestamp": 0.00000762939453125,
  "reactions": 0.06316184997558594,
  "mentions": 0.0001068115234375,
  "webhookID": 0,
  "application": 0,
  "activity": 0,
  "_edits": 0.06797599792480469,
  "flags": 0.00002288818359375,
  "reference": 0
}

Include a reproducible code sample here, if possible:

See above.

Further details:

• discord.js version: 12.3.1
• Node.js version: 12.18.0
• Operating system: Windows Server 2016 (deal with it)
• Priority this issue should have – please be realistic and elaborate if possible: N/A

Relevant client options:

• partials: none
• gateway intents:

(new Intents()).add(
    Intents.FLAGS.GUILDS,
    Intents.FLAGS.GUILD_MEMBERS,
    Intents.FLAGS.GUILD_BANS,
    Intents.FLAGS.GUILD_EMOJIS,
    Intents.FLAGS.GUILD_MESSAGES,
    Intents.FLAGS.GUILD_MESSAGE_REACTIONS,
    Intents.FLAGS.DIRECT_MESSAGES,
)

• other:

messageCacheMaxSize: 10,
messageCacheLifetime: 30 * 60,
messageSweepInterval: 5 * 60,

• [ ] I have also tested the issue on latest master, commit hash:

[MattIPv4]

So, as previously noted, its only these two messages that have this excess memory usage. And, although object-sizeof might not be perfectly accurate, the memory usage is seen in process.memoryUsage()

object-sizeof points to the reactions on these messages being the source of the issue, however, I ran this script to start looking at the reactions further, and this seems very odd. Are there undocumented properties in the ReactionManager that I'm missing?

const sizeof = require('object-sizeof');
const messages = client.guilds.cache.array().reduce((prev, guild) => prev.concat(guild.channels.cache.array().reduce((prevC, channel) => { if ('messages' in channel) { return prevC.concat(channel.messages.cache.array()); } return prevC; }, [])), []).sort((a, b) => sizeof(b) - sizeof(a)).slice(0, 3);

return JSON.stringify(messages.map(message => ({ url: message.url, message_size: sizeof(message) / 1024 ** 2, content_size: sizeof(message.content) / 1024 ** 2, embeds_size: sizeof(message.embeds) / 1024 ** 2, reactions_size: sizeof(message.reactions) / 1024 ** 2, attachments_size: sizeof(message.attachments) / 1024 ** 2, reactions_count: message.reactions.cache.size, reactions: message.reactions.cache.array() })), null, 2);

[
  {
    "url": "https://discord.com/channels/583669738203381780/630766891308154910/755655653246238721",
    "message_size": 86.51883506774902,
    "content_size": 0,
    "embeds_size": 0.0007877349853515625,
    "reactions_size": 86.51856803894043,
    "attachments_size": 0,
    "reactions_count": 0,
    "reactions": []
  },
  {
    "url": "https://discord.com/channels/660789921627504671/713583644731768924/755396833953579059",
    "message_size": 51.920021057128906,
    "content_size": 0,
    "embeds_size": 0.0009555816650390625,
    "reactions_size": 51.91975402832031,
    "attachments_size": 0,
    "reactions_count": 0,
    "reactions": []
  },
  {
    "url": "https://discord.com/channels/729127979665326101/732378107083751525/756162481209999481",
    "message_size": 0.023365020751953125,
    "content_size": 0,
    "embeds_size": 0.003021240234375,
    "reactions_size": 0.023097991943359375,
    "attachments_size": 0,
    "reactions_count": 0,
    "reactions": []
  }
]

[MattIPv4]

So I looked through the source for ReactionManager and BaseManager, and I believe this script captures every property of ReactionManager:

const sizeof = require('object-sizeof');
const messages = client.guilds.cache.array().reduce((prev, guild) => prev.concat(guild.channels.cache.array().reduce((prevC, channel) => { if ('messages' in channel) { return prevC.concat(channel.messages.cache.array()); } return prevC; }, [])), []).sort((a, b) => sizeof(b) - sizeof(a)).slice(0, 1);

return JSON.stringify(messages.map(message => ({
  url: message.url, message_size: sizeof(message) / 1024 ** 2,
  content_size: sizeof(message.content) / 1024 ** 2,
  embeds_size: sizeof(message.embeds) / 1024 ** 2,
  attachments_size: sizeof(message.attachments) / 1024 ** 2,
  reactions_size: sizeof(message.reactions) / 1024 ** 2,
  reactions_cache_size: sizeof(message.reactions.cache) / 1024 ** 2,
  reactions_cache_count: message.reactions.cache.size,
  reactions_holds_size: sizeof(message.reactions.holds) / 1024 ** 2,
  reactions_client_size: sizeof(message.reactions.client) / 1024 ** 2,
  reactions_cacheType_size: sizeof(message.reactions.cacheType) / 1024 ** 2,
  reactions_message_size: sizeof(message.reactions.message) / 1024 ** 2,
})), null, 2);

I ran this on the problematic shard:

[
  {
    "url": "https://discord.com/channels/583669738203381780/630766891308154910/755655653246238721",
    "message_size": 87.24896812438965,
    "content_size": 0,
    "embeds_size": 0.0007877349853515625,
    "attachments_size": 0,
    "reactions_size": 87.24870109558105,
    "reactions_cache_size": 0,
    "reactions_cache_count": 0,
    "reactions_holds_size": 0,
    "reactions_client_size": 0.008192062377929688,
    "reactions_cacheType_size": 0,
    "reactions_message_size": 87.24896812438965
  }
]

This seems odd, as it appears that the size (and memory usage) come from nowhere (with message being circular).

Running this on a nominal shard (5):

[
  {
    "url": "https://discord.com/channels/707537675397365761/751867754671243274/756164149880291409",
    "message_size": 0.034259796142578125,
    "content_size": 0.0005474090576171875,
    "embeds_size": 0.0015201568603515625,
    "attachments_size": 0,
    "reactions_size": 0.03344535827636719,
    "reactions_cache_size": 0,
    "reactions_cache_count": 0,
    "reactions_holds_size": 0,
    "reactions_client_size": 0.008192062377929688,
    "reactions_cacheType_size": 0,
    "reactions_message_size": 0.034259796142578125
  }
]

With this in mind, I'm thinking the whole reactions thing might not be right and the memory usage is coming from elsewhere in the message, with it showing in reaction due to the circular reference.

[MattIPv4]

Don't mind me, the whole reactions looking like the issue was due to a circular reference. I've now dumped the size for every property of the message and it appears that Discord.js stores the full edit history for the message, hidden away, that's causing the memory usage.

[advaith1]

not exactly hidden, displayed in Message#edits

[MattIPv4]

I'm blind. I guess this issue is probably going to want to transform into a way to limit edit history considering its currently contributing 100MB+ for a single message currently in one of my shards.

[wasdennnoch]

It makes sense that this issue exists, given that while the message cache itself has a maximum capacity, a single message has an infinite edit history in DJS. However at the same time, realistically this issue should only come to light with API abuse. There have to be thousands of edits of a single message to have them take up that much space.

Actually let's make some calculations. If I read your post correctly a single message takes up roughly 3100 bytes. Your problematic shard draws about 100MB more RAM than the others. That's 100,000,000/3,100 = 32,250 messages (message edits). You said that your message sizes were inflated due to circular dependencies, that means that there were even more edits than that - within one day. 32,250/24/60 = 22.3 edits per minute. That is assuming it's only a single message that's being edited. And assuming that's actually the issue.

With that being said, it's probably a good idea to cap the edit history at something like the 100 most recent edits.

[MattIPv4]

@wasdennnoch Your maths is probably somewhat off, a SINGLE message has an edits size of 107.88620185852051 MB.

[kyranet]

I personally would remove the edits altogether in v13, they're not really needed and users can still get the old behaviour with a structure extension and a guildMemberUpdate event.

I think it makes sense to add a message edit limit in the ClientOptions just like there is one for message cache limits.

[MattIPv4]

const sizeof = require('object-sizeof');
const message = client.guilds.cache.array().reduce((prev, guild) => prev.concat(guild.channels.cache.array().reduce((prevC, channel) => { if ('messages' in channel) { return prevC.concat(channel.messages.cache.array()); } return prevC; }, [])), []).sort((a, b) => sizeof(b) - sizeof(a))[0];

return JSON.stringify({ edits_memory: sizeof(message.edits) / 1024 ** 2, edits_length: message.edits.length }, null, 2);

The problematic shard, with the largest message by memory usage:

{
  "edits_memory": 110.65471267700195,
  "edits_length": 71730
}

On a normal shard, the largest message there:

{
  "edits_memory": 0.06339836120605469,
  "edits_length": 26
}

[wasdennnoch]

50 edits per second, that's even better. Didn't know the rate limit was that high. Looks like some bot is constantly editing a message. I suppose it's worth reporting the message link for API abuse.

[MattIPv4]

👍 Will take a look at adding a client option to set a limit for edit history.

[Skillz4Killz]

@MattIPv4 @kyranet I had a similar issue on mine and I managed to patch it by ignoring message edit events that didn't actually edit anything. MESSAGE_UPDATE is sent even when a URL is in the content. When that URL is embedded Discord treats it as a MESSAGE_UPDATE and you can get multiple MESSAGE_UPDATE per message. Edit that message content and it triggers 5 new MESSAGE_UPDATE for those 5 URLs again. If i had to guess, i would wager this is happening on your end right now.

A simple solution might be to not allow edited messages to trigger message.patch when it isn't actually a change in anything.

[MattIPv4]

I'll look at adding some checks for if the message has actually changed when patch gets called, as well as adding the limit in.

[advaith1]

Well it sends a MESSAGE_UPDATE because it's adding the embed, if you just check if content changes then you won't see the embed

[MattIPv4]

Just as an update, I've had my initial fix for this running in prod on my bot for a while and it seems to be pretty stable (though I've observed some other memory creeps at points).

Here you can see the growth reported above, followed by the deployment of the edit history limiting:

(The first annotation was the deployment of edit history limiting to the problematic shard, with the second annotation being the roll-out of edit history limiting to all other shards)

I still want to put some time into checking that every patch being made to a message is actually changing something before I create a PR, and I need to figure out what breaks when you store no message history at all.

Hoping to have a PR up by the end of this week :)