My bots appear to be suffering from memory leaks. I took a heap dump of my patron bot, which has been running for 30 days with JDA 3.0.0_200. Looking at the heap dump I see that java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue is retaining 53% of my heap.
Clarification: The 30 days is with a relatively small bot. For my bot with over 1400x as many guilds this problem is about 10x-15x faster to take up the heap to the point of impacting audio playback.

The problem appears to be that we have numerous AudioConnections which still holds references to my send handler, which in my case is called GuildPlayer:

I'm not sure why these connections are being retained, but it doesn't look like these connections are getting references from my code without going through JDA code.
This could possibly be related to https://github.com/DV8FromTheWorld/JDA/issues/357
I use JDA-NAS
This is how the inbound references looks like to those AudioConnections

+1 Its been killing my bot for a while now
Don't know if you want the fix on a specific branch, but I found the culprit.
Fixed as of 69efe74
Most helpful comment
This is how the inbound references looks like to those

AudioConnections